Experiences of conducting online exam proctoring in low- resource settings: a Sri Lankan case study

online proctoring academic test

Qualitative Data

Findings relating to problems in internet connections, issues in devices used, factors that led to stress during the exam, and the factors that made cheating easy or hard during the assessment, obtained as responses to open-ended questions, are discussed below.

Issues regarding internet connectivity

Participants faced several types of troubles during the exam due to low bandwidths, instability, and sudden disconnection of the internet. The common issue many of the participants faced was having to reconnect two devices to the Zoom session because of sudden disruption of the internet connection. One respondent mentioned, «I lost my connection when trying to access the internet with two devices and had to record again and again».

We noticed participants leave the seat and adjust their smartphones in some instances, which is evident in the comment:

«The need to use two devices to log in to the meeting several times displayed that my network bandwidth is unstable and on one occasion my mobile phone was disconnected. After I noticed that, I went back and re-connected it and then continued doing the assignment'.

Therefore, we can conclude that the instability of the internet connection is one of the reasons for unavailability of certain views. When the internet connection is unstable, Zoom automatically switches off the videos and the participant is not aware of it. Many of the students reported that often the internet connection was unstable, unless it was a fiber connection that is only available in urban areas. In cases of power outages, the participants tried to connect the devices via a mobile hotspot. Mentioned below is a comment regarding the mobile data signals:

«'It's really hard to manage our bandwidth if we don't have a fiber connection. Else if we got a power cut it's hard to connect via mobile hotspot, during power cuts 4G connection becoming to 2G or 3G (but the bandwidth is really low at that time)»

Another issue that a considerable number of participants had to face was difficulties in uploading the recorded video within the given time limit. One participant commented, «Connection was not stable, and it took me more than 6 hours to upload the assessment videos». Average size of a video file ranged between 150 MB to 220 MB, and this was a problem that could not be seen from the videos. Some people can forge the videos using this issue as an excuse as well. Hence, there should be a mechanism that reports the behaviors as soon as the exam is completed, whether the video is submitted or not.

Issues regarding devices used

Poor battery health in both laptops and smartphones was the major concern participants had regarding the devices. Using Zoom in smartphones for more than 1 hour seemed problematic for many participants. Some responses regarding the issues faced due to bad battery health of the devices used is stated below.

«My phone is having a low RAM and when the camera is turned on the battery drains first. Luckily, I was able to finish the assessment before it died… phone storage space is also very low… I had to delete several necessary applications in order to download zoom to my mobile phone»

«My mobile phone runs out of battery very fast, so it is difficult to stay in zoom for around 1 hour of time without charging.»

«Smartphone storage full and Battery is weak, so I have to keep the charger connected during the assessment, which makes the mobile heated.»

Other than battery health issues, broken in-built webcams, echo in microphones and technical glitches in laptops were among the issues reported regarding the devices. Participants were not in favor of using the smartphone as a proctoring device for a prolonged time due to the above reasons.

Factors that led to stress during the assessment

The time constraints of the assessment, feeling of being recorded and fearing the technical issues discussed above, were the factors that added stress and anxiety during the exam, which is not the case in an exam hall. 57.8% of the participants strongly agreed with the statement that this proctored assessment was more stressful than a traditional invigilated exam.

Several respondents reported that there were many other factors to pay attention to, on top of answering the questions, which is not the case in an exam hall: «Unlike an exam centre, our home environment is not peaceful, it's too noisy, which makes it more difficult to concentrate in an exam. When we are in an examination hall, there is less probability that external factors affect our results, only the knowledge we gain by studying will affect our exam results. But when it is online, other external factors affect our results, like electricity, or internet connection, our home environment. In an exam centre we only need to worry «how to answer the questions», but online we also worry what will happen when other external factors affect the exams, thus we lose our concentration. Trust me I want to be supportive and help the academics to conduct the exam successfully, but as these factors disturb and take away my concentration, I'm not able to be supportive, I'm sorry about that.»

The participants said that external factors such as noisy environments, interruptions by family members, malfunctions of the devices made it difficult for them to concentrate on answering the questions. The participants did not have a peaceful exam environment at home. For example, one commented, «There were a few disturbances such as background noises or neighbors interrupting by knocking on door. Additionally, there wasn't a comfortable place to do the exam at the house since I have no separate room. Anyone could come to the house at the time and interrupt what I was doing.»

The major stress-factor of this assessment was the fear that one would be accused of cheating when facing a technical issue. A few responses that resonate with the above statement are listed below.

«I had no proper place to set up my back camera so had to tape it to a wall - and it fell off multiple times. I had to check it constantly and reposition it multiple times»

«If there is a power cut during an online exam, we do not know how it will affect our exam, and how do we prove it to examiners? And if the power comes again, will it give us back the lost time? If so, how? These factors are stressing me.»

From the above statements, it is evident that the participants were worried about proving their innocence to the invigilators, if they were caught up with some issue during the assessment.

Furthermore, many participants' opinion was that the need to constantly check on the proctoring setup reduced their attention on answering the questions. Below are some examples,

«Continuously I have to check out the devices are working well or not and also if one of the devices or both disconnected then I have to re-connect. That is time consuming. So those are stressful things when it comes to an exam.»

«It increased the stress since many uncontrollable factors contributed to the final output other than our performance. Such as device performance, battery lifetime of the device, network connection, meeting the deadline with uploading speed and everything had to be looked into during the assessment and that reduced the focus on the assessment.»

Before inquiring the opinions on cheating in a proctored exam setup, we asked the respondents to rate if the proctoring setup made cheating easy or hard, on a 5-point Likert Scale (See Figure 5). 82 of 146 respondents said it was neither hard nor easy. Only 19 of 146 respondents believed that cheating was easy in that proctored assessment. According to the findings of [24], the lack of clarity in examination instructions regarding what resources might be used led to confusion over what constituted cheating. Therefore, we explicitly mentioned that the participants could only use the module textbook for the assessment.

The participants mentioned that they did not have enough time to consider cheating, because of the time constraints and the proctoring setup: «I do not think I had the proper motivation to even attempt the assignment successfully given the conditions and the stress. That alone discouraged any attempts to score any higher than I could at my best via cheating in anyway». This aligns with the findings of [24] but one respondent who mentioned that cheating is somewhat easy in this setup stated that «examiners have to go through every second in every recording to find out if any cheating happens», which is conflicting. It is evident that students think that it is a difficult task to manually review videos that are hours long, hence they consider it as an opportunity to cheat.

Figure 5. Student responses to the statement: «The online assessment made cheating…»

This study concludes that it is impractical to develop a fully automated online proctoring system in the current context. The best suited proctoring method is semi-automated proctoring that reports the actions of an exam candidate during an exam session. Upon examining the records, the final decision should be taken by invigilators. Moreover, there should be a fine balance between deterring academic misconduct using proctoring and the inconvenience faced by the students during proctoring. Our findings say that a complex proctoring setup itself is a factor that affects students' performance during an exam, regardless of academic misconduct. Academics should exercise caution when creating proctoring systems, especially in developing nations, where the resources at their disposal make it difficult to completely map online exams to traditional exams. Based on the aforementioned conclusions, it is advised to add the following components to the typical features of an online proctoring system for low-resource situations.

The main cause of most of the issues faced by the students is the secondary proctoring device (mobile phone). Setting up a secondary device provides some useful information but the impediments of having to set up a secondary device exceed the advantages. The two - device setup doubles the network data consumption and doubles the work of the invigilators because they must review two videos per candidate. The main issue reported regarding the devices is the battery drainage of the mobile phones. The candidates say that they had to give extra attention to ensuring the mobile phone is properly connected. Therefore, an ideal solution should eliminate the need for the two-way camera setup.

Difficulties candidates face because of unstable internet connection are the next important problem that should be addressed when designing online proctoring solutions. A solution should minimally depend on the internet connectivity. Trying to identify the performance constraints of a proposed solution to determine whether on-device proctoring can be facilitated in common devices is another step that should be taken. Such facilitation will enable offline mode where the internet will be required at the start and the end for submission purposes only. Once the candidate is authenticated, the system should run offline and should submit a behaviour report as soon as the exam is over instead of sending a bulky video, which will reduce the time for video uploading. Moreover, an ideal solution should include reports about internet connectivity if achieving offline mode is not possible. If disconnected, the timestamps of the disconnected duration should be obtained so that the student can continue answering when the connection becomes stable again without losing writing time. If the behavior report is sent instead of uploading the video, the video must be saved on the device until an examiner request it upon inspection of the behavior report. Another possible feature which could lessen the gap of access to e-learning would be expanding a solution functionality to include mobile phones and tablets, since it is reported that many students in Sri Lanka use mobile devices to engage in online learning activities [4].

Along with the rise of online education, online examination is becoming more popular. However, it is obscure how to use technology to proctor online tests simply and efficiently with limited resources. Our findings strongly suggest that online exam proctoring solutions should be developed after carefully examining the constraints of the environment that it is intended to be operationalized. Based on the outcomes of this study, future research could look at a variety of topics on e-exam proctoring such as offline test proctoring, live - transmission of action detection records or saving behavior report securely until the exam is completed. Integration of the functions into a full online test system would aid in the seamless operation of online education in contexts where internet connectivity and infrastructure are unreliable. To scale the availability of trustworthy online exam processes, a possible future work is to develop e-proctoring solutions that may well operate in various devices such mobile phones, and tablets which would be highly beneficial in facilitating online education in low-income counterparts of the society.

References (translated and transliterated)