Effectiveness of online testing in general English university course from teacher and student perspectives

Размещено на http://www.allbest.ru/

Effectiveness of online testing in general English university course from teacher and student perspectives

Olena O. Kucherova

PhD of Philological Sciences, Associate Professor of the English Language Department National University of “Kyiv-Mohyla Academy”, Kyiv, Ukraine

Iryna O. Ushakova

PhD of Pedagogical Sciences, Senior Lecturer of the English Language Department National University of “Kyiv-Mohyla Academy”, Kyiv, Ukraine

Abstract

With most switching to distance education due to the pandemic caused by COVID-19, the efficacity of assessment is a major concern. The research was aimed to analyze how Moodle learning management system (LMS) can be applied to online language testing, study the effectiveness of online testing and compare the students' and teachers' attitude towards online testing in General English university course. Online tests were administrated as a synchronous component of the distance learning to 857 first-year bachelor's degree students of “Kyiv-Mohyla Academy” (Ukraine) by 20 teachers during the 2020-2021 academic year. A mixed research design was employed, which involved collecting data using an online questionnaire completed by students and teachers anonymously as Microsoft Forms; Excel spreadsheets were used for the analysis afterward. A quantitative descriptive study was conducted to evaluate the students' and teachers' satisfaction with online testing. The expert evaluation method was prioritized to define the expediency of the effectiveness of the online test by the specified criteria and indicators based on the judgments expressed by 7 experienced teachers who are competent in test design. In addition, Pearson's correlation coefficient was calculated to compare the results of an online test and oral exam. The qualitative research method allowed to analyse and interpret data of the experimental learning. Based on the results of our study, we can conclude that different types of Moodle LMS questions can be successfully applied to online language testing as part of course assessment at the university level. The paper argues that online language testing can be effective and relevant to course objectives from both students' and teachers' perspectives with positive washback on education. The results of the study can be employed by university teachers for language course design both for distance and blended learning.

Keywords: online testing; test effectiveness; language skills assessment; formative assessment; Moodle.

Анотація

ЕФЕКТИВНІСТЬ ОНЛАЙН ТЕСТУВАННЯ В УНІВЕРСИТЕТСЬКОМУ КУРСІ АНГЛІЙСЬКОЇ МОВИ З ТОЧКИ ЗОРУ СТУДЕНТІВ ТА ВИКЛАДАЧІВ

кандидат філологічних наук, доцентка кафедри англійської мови Національний університет «Києво-Могилянська академія», м. Київ, Україна

Ушакова Ірина Олександрівна

кандидат педагогічних наук, старша викладачка кафедри англійської мови Національний університет «Києво-Могилянська академія», м. Київ, Україна

В умовах масового переходу до дистанційного навчання, обумовленого світовою пандемією COVID-19, проблема ефективності онлайн оцінювання залишається надзвичайно актуальною. Метою нашого дослідження було проаналізувати шляхи застосування системи управління навчанням Moodle (Moodle LMS) до онлайн тестування з англійської мови, вивчити ефективність онлайн тестування та порівняти ставлення студентів та викладачів до нього під час вивчення/викладання курсу англійської мови. Упродовж 2020-2021 навчального року онлайн тестування проводилось викладачами кафедри англійської мови (20 викладачів) у режимі синхронного викладання для студентів першого року навчання бакалаврських програм (857 студентів) Національного університету «Києво-Могилянська академія» (Україна). Під час дослідження було використано комплекс методів, що передбачали збір даних за допомогою онлайн анкети, заповненої студентами та викладачами анонімно в Microsoft Forms; для аналізу отриманих результатів були використані електронні таблиці Excel. Для оцінки рівня задоволеності студентів та викладачів онлайн тестуванням було проведено кількісне описове дослідження. Метод експертних оцінок був пріоритетним для визначення ефективності онлайн тестів відповідно до наведених критеріїв та показників на основі суджень 7 досвідчених викладачів, компетентних в розробці тестових завдань. Крім того, для порівняння результатів онлайн тесту та усного іспиту було розраховано коефіцієнт кореляції Пірсона. Якісний метод дослідження дозволив проаналізувати та інтерпретувати дані, отримані під час експериментального навчання. На основі результатів нашого дослідження ми можемо зробити висновок, що різні типи завдань Moodle LMS можуть бути успішно використані для онлайн тестування з англійської мови в закладах вищої освіти. У статті стверджується, що онлайн тестування є ефективним, відповідає цілям курсу та має позитивний вплив на навчання загалом за оцінкою студентів і викладачів. Результати дослідження можуть бути використані викладачами університетів для розробки мовних курсів для дистанційного і змішаного навчання.

Ключові слова: онлайн тестування; ефективність тестування; оцінювання мовних навичок; формувальне оцінювання; Moodle.

INTRODUCTION

online testing english university course

The global pandemic caused by COVID-19 has significantly affected the main areas of human activity: medicine, science, business, tourism, culture, and education. Under total lockdown, remote management and online education technologies have become widespread, and people's homes have transformed into temporary offices for parents and classrooms for students. All these external circumstances inspired teachers to adopt new ways of communication with students and organization of learning, teaching, and assessment. University professors and teaching staff started exploring various tools to digitize both study sessions and examinations. Without access to traditional examination methods, they were required to find new effective and relevant ways to assess students online. As a result, teachers had to go beyond traditional forms of teaching and assessment, implementing new digital tools and methods which, in turn, became decisive in the design and evaluation of the effectiveness of online testing.

The problem statement. National University of “Kyiv-Mohyla Academy” has been providing distance education since March 2020, using a university-wide online platform called `DistEdu', which is based on Moodle LMS. One silver lining is that the platform had been implemented and tested before the quarantine, but not all the features and tools were fully employed because it was used as an extra source for teaching. The unexpected lockdown was a trigger for the creation of online courses by each teacher that greatly simplified and generalized the teaching process at the university.

Asynchronous teaching has been extensively encouraged at the university, which means that the teachers and the students engage with the course content at their convenience. Using DistEdu the teachers may upload media, start discussion boards, assign reading or writing as an asynchronous form of teaching and learning. The teachers can also run online tests as a synchronous form of teaching and learning. The teachers guide the students, provide them with feedback and assess their progress. The functionality of the DistEdu platform enabled to replace all study materials and make them available to students quite fast. It was the time to look at everything from different perspectives, and offline classes quickly moved to online Zoom and MS Teams classes, opening the way for teachers to use extra tools and software, such as Quizlet, Padlet, Kahoot, Jamboard, Classtime, or Miro to digitalize teaching and assessments.

It is worth mentioning that the 2020-2021 academic year compared with the previous one was unprecedented just because the teachers who worked with first-year students had never met them in the offline classroom before, and students also had never had face-to-face communication with each other, which was, in any case, a certain psychological barrier and caused some difficulties for an adequate assessment. Furthermore, the question of students' psychological health is becoming ever more vital. For this reason, online testing of the 2020/2021 academic year is featured by a specific educational environment, permanent restrictions, and information overload because of the COVID-19 pandemic.

The analysis of the recent studies and publications. Undoubtedly, the topics of distance education, employing web platforms and e-learning tools for online teaching and assessment are not completely new, they have been studied extensively by the methodologists for several decades, but before the lockdown, they were considered mostly in the context of blended learning or as some extra possibilities of on-site testing, for example, in the computer lab at university. Rapid digital transformation of higher education in connection with COVID- 19 has posed challenges to faculty members, forcing them to search crisis-response methods and search for opportunities for distance education [1]. In this regard, Y. Krylova-Grek and M. P. Shyshkina [2] study the online learning trends of higher education and suggest strategies for how to organize distance learning. Different aspects of technology integration into higher education have been investigated by S. Naidu [3] and the guiding principles for online teachers, emphasising the main problems and challenges that are present in the online classroom were provided by D. A. McFarlane [4]. M. Decuypere et al. [5] and M. Arshad [6] overview the key features of digital education platforms to enhance education.

Due to the pandemic and global move to distance education, the issue of effective online assessment has gone to the fore. On this basis, F. J. Garcia-Penalvo et al. [7] conclude that ways of assessment in Higher Education should be reconsidered. Accordingly, B. O'Sullivan et al. [8] focus on language test preparation strategies and E. Stradiotova et al. [9] compare online testing the language skills with on-site testing. One of the topical issues of online testing in the un-proctored environment is academic dishonesty and student cheating. I. Arnold [10], W. Bloemers et al. [11], D. L. King and C. J. Case [12], J. G. Nguyen et al. [13],

D. Steger et al. [14] draw attention to the impact of cheating while testing on final term grades and suggest ways how to minimize online cheating for online assessments.

In spring 2020 because of experts exploring assessment in different world universities and colleges as well as the impact of technology on education and assessment, the report was published with the aim of helping address some of the concerns about assessment and highlighted some opportunities [15]. According to the report, the assessment should be 1) authentic, designed to prepare students to use technology they will use in their future careers; 2) accessible, designed with accessibility-first principle; 3) appropriately automated, with a balance of automated and human marking for maximum benefits to students; 4) continuous, exploring the opportunities for continuous assessment in order to improve learning experience; 5) secure, adopting authoring detection and biometric authentication for identification and remote proctoring. The abovementioned report also set out the main goals for assessment, arguing that assessment should become more 1) relevant to contemporary needs; 2) adaptable in terms of addressing the students' and providers' needs; 3) trustworthy.

The research goal. Taking into consideration the urgency of online testing as a learning and assessment tool, the given research aims at studying the effectiveness of online testing in General English university course from teacher and student perspectives. To achieve the goal, the following tasks were formulated: 1) to explore the opportunities provided by Moodle LMS to implement online testing in General English course design both for small-scale and large-scale tests; 2) to study the students' and teachers' attitude toward online testing and its washback on education; 3) to compare the results of an online test and oral exam (as a combination of innovative and traditional forms of assessment).

THE THEORETICAL BACKGROUNDS

Being fundamentally important for teaching and learning, high-quality assessment contributes markedly to achieving learning outcomes and enhancing students' satisfaction. Following C. A. Chapelle et al. [16, p. 294], assessment is understood as the process of analysing the ability of learners to use the language based on the information collected. In this regard, the main dichotomies suggested for the investigation of assessment are informal versus formal assessment and formative versus summative assessment [17].

In recent years we have observed the shift towards the assessment for learning approach rather than the assessment of learning [18], [19]. With assessment for learning students take the responsibility for improving their performance in the course. Among recent approaches to assessment, there should be mentioned learning-oriented assessment [20] and dynamic assessment [21]. Built on the idea of formative assessment, they both put learning at the center of the assessment process and “capture all assessment as a vehicle for learning” [15, p. 307] with the primary purpose to promote learning. In this regard, tests as a subset of assessment are viewed as “powerful tools to promote lasting learning in their own right” [22, p. 1864].

Depending on what language tests intend to measure, they can be classified as ability approach (indirect) and performance approach (direct) [15, p. 297]. Ability testing measures grammatical ability, vocabulary knowledge, reading comprehension, and listening comprehension, while performance testing measures writing and speaking. J. D. Brown and T. Hudson employ the term criterion-referenced test to refer to “any test that is primarily designed to describe the performance of examinees in terms of the amount that they know of a specific domain of knowledge or set of objectives” [23, p. 5]. To put it differently, criterion- referenced tests are designed to provide feedback on specific course objectives.

In J. Lewkowicz and C. Leung classroom-based assessment is defined as “any teacher- led classroom activity designed to find out about students' performance on curriculum tasks that would yield information regarding their understanding as well as their need for further support and scaffolding with reference to their situated learning needs” [24, p. 48]. Classroom-based assessment for language courses centers on communicative performance and correlates with the Common European Framework of References for Languages (CEFR). CEFR initially meant as a guide for comparing objectives internationally provides a common foundation for six general levels of language proficiency. It uses descriptions of communicative language activities that learners perform, such as comprehension, production, interaction, and mediation, which draw on linguistic and general competences [25]. Thus, the CEFR, emphasising that evaluation is inextricably connected to teaching and learning, provides principles for the development of language curricula to support the teaching, as well as assessment tools.

As it is stated in CEFR, linguistic competence includes active and passive vocabulary (lexical competence); knowledge of the rules and structures and the ability to use them correctly (grammatical competence); the organization of meaning (semantic competence); hearing and producing sounds (phonological competence); the ability to spell correctly (orthographical competence); the ability to read from a written text, pronouncing correctly (orthoepic competence) [25, p. 129]. That is, grammatical accuracy, vocabulary range, and vocabulary control are essential components of `linguistic competence'. As mentioned above, the fact that some expressions can be used to convey vastly different messages adds to the complexity of language [26, p. 12]. Thus, language tests mainly aim to check grammatical and vocabulary range and accuracy.

In this paper, assessment is understood as an umbrella term that includes formal testing and other types of qualitative assessment, such as informal observation of students' language use, portfolios, and reflection journals. In J. D. Brown's words, a test is “a method of measuring a person's ability, knowledge, or performance in a given domain” [16, p. 3]. As a method or an instrument, it is a set of techniques that require a certain type of performance from the student. And this performance or competence within a particular domain can be measured by a test.

RESEARCH METHODS

The study employed a mixed research design method. To achieve the purpose of our research we used a quantitative research method which involved collecting data by means of an online questionnaire completed by students and teachers anonymously as Microsoft Forms; Excel spreadsheets were used for the analysis afterward. The data analysis was conducted with the use of descriptive statistics, which allowed to summarise and interpret the data obtained. Qualitative data was received through analysing information of individual feedback, which allowed the researchers to compare the responses from students and teachers. In order to determine the effectiveness of online language testing in our study, we also employed the method of expert evaluations and Pearson's correlation coefficient.

Participants

The participants of the study were 857 first-year bachelor's degree students in a wide variety of majors (aged 17-18 years) and 20 teachers (aged 28-56 years) of the National University of “Kyiv-Mohyla Academy” (Ukraine). The research also involved seven experts, who are experienced teachers (with 15-25 years of teaching) from the same university. The study was carried out in the academic year 2020-2021 (autumn and spring semesters) in General English course. The students took part in the experiment as a part of their coursework as they had to pass their examination in a lockdown-forced distance mode.

Instruments

The Likert scale was chosen to measure the students' and teachers' attitudes towards online testing at the end of the second semester and for expert evaluation since this instrument is widely used by researchers in education for that kind of purpose. In addition, Guttman scaling was chosen to find out the most favorable statement in an online questionnaire completed by students and teachers. Pearson's correlation coefficient [27] was chosen to compare the results of the online test with the oral exam to assess its effectiveness.

THE RESULTS AND DISCUSSION

Applying Moodle LMS for online language testing

Testing has always been an important element of the learning and teaching process in General English course. The course lasts for two semesters, during which two main types of tests are used: 1) three Current Tests for each semester (six for the course) and 2) one Final Test at the end of the semester (two for the course). The main purpose of Current Tests is to determine what the students need help with. In contrast, the main purpose of Final Tests is to determine if learning goals have been met. The tests are prepared by the teachers and are concerned with measuring what has been learned as a part of the course, in other words, `achievement assessment' [15, p. 294]. The salient feature of these tests is that they are conducted on the same day by all students of the course.

These tests were quite extensive (100 questions), aimed to check learned lexical and grammar material in the most comprehensive and objective way first for one semester and then for the whole course. That is why the paper-based Final Tests had a complex structure, containing the exercises of various types, which are usually used in preparation for international exams in English and adhere to modern methodological requirements for the indirect type of test items and discrete-point testing. While preparing the Final Tests, teachers of the English Language Department always focus on modern didactic principles and generally accepted criteria to evaluate the effectiveness of any test which are: validity and reliability [28, p. 381]. Moodle LMS can meet all the above criteria. Thus, we were able to adapt and create quizzes of different types to check the studied lexical and grammar material: 1) drag the words from the list below into the correct place in the text, 2) arrange the words into a sentence, or 3) rearrange the jumbled sentences into a logical text (drag and drop into text question type on Moodle LMS); 4) choose the correct answer to complete the sentences; 5) find a mistake in the underlined parts of the sentence; or 6) choose the odd word in each line (multiple choice question type); 7) find the words for the following definitions (matching question type) (Appendix A).

J. D. Brown outlines different assessment types and classifies them into four basic categories: receptive-response, productive-response, personal-response, and individualized- response [29]. The scholar goes on to suggest 12 assessment types: true-false items, matching items, multiple-choice items, fill-in items, short-answer items, performance assessment, conference assessment, portfolio assessment, self/peer assessment, continuous assessment, differentiated assessment, and dynamic assessment. These assessment types fall into two main groups: those that focus on single language points and those that represent how to collect assessment data.

During the research assessment categories and assessment types for tests that were proposed by J. D. Brown [29] were applied to Moodle LMS capacities. Table 1 demonstrates the assessment options that can be employed in online testing. Items for large-scale tests (Final Test) should be relatively quick and easy to administer and score objectively. The selected-response category is suitable for testing the receptive skills of reading, listening, knowledge of grammar, and vocabulary. The constructed-response category is fundamentally different from the selected-response category as students are expected to produce written language ranging from single words (as in fill-in items) to longer answers (as in short-answer items) and a paragraph or essay writing (as in performance assessment). Fill-in items, which require students to write in the missing word or words, are relatively easy to score and quick to administer, hence suitable for large-scale tests (Final Test). Short-answer items, which require students to write a few words, and performance assessment, which requires students to write a paragraph or essay, are more difficult to administer and less objective to score compared to the items mentioned earlier. As a result, they are suitable for small-scale tests (Current Tests).

Table 1 Assessment options for online testing