An inset course on language assessment for university teachers in Ukraine

В статье предлагается обоснование потребности в курсе повышения компетентности в языковом тестировании преподавателей иностранных языков, работающих в университетах Украины. Охарактеризовано понятие “грамотность в языковом тестировании и оценивании ”, на основе чего определен предмет обучения в таком курсе. Описаны содержание и структура курса. Представлен один из модулей курса, прошедший пилотную апробацию в нескольких университетах страны. Выводы, сделанные на основе данных пилотирования, могут быть использованы в дальнейших исследованиях проблемы.

Ключевые слова: компетентность в языковом тестировании и оценивании, курс повышения компетентности в языковом тестировании, содержание и структура курса.

Language testing and assessment (LTA), which has long occupied centre-stage position in FL teaching and learning practices in the West, is rapidly gaining its significance in FL education in this country. Accountability which accompanies the implementation of ECTS has put forth the necessity to use language tests that are known as objective and practical tool of summative assessment.

However, a test can only yield a true picture of students' achievements if its designers follow the major principles or guidelines for test construction and validation, adhere to the testing cycle rules, administer tests properly. Using tests for summative assessment (mid-term or end-of-year tests) urges the grass roots teachers to perform multiple roles of test designers, item writers, test raters for which the majority of teachers are not prepared. This suggests that classroom tests designed with violations of certain testing principles can hardly supply reliable information about learning outcomes and may cause negative impact on language education both in the local and national contexts.

The issue of LTA literacy of Ukrainian university teachers appears to be crucial under the circumstances. So far this literacy has been developed through teachers' participation in trainings, workshops, conferences, self-study work. However important it has been for certain individual representatives of teacher community, this work was mostly sporadic and should be replaced by more systemic and such as involving more teachers from a variety of higher educational settings in the country.

The present paper aims to describe the underlying principles, analyse existing theory and practices and propose the content, structure and teaching methods as components of an in-service teacher training course in LTA designed by the author. Initially, the analysis of the contemporary research in the field of teacher training in LTA is offered.

The issues of teacher assessment literacy have been discussed by Western educationalists since 1990s; a comprehensive account of this evolution is given by A. Green [6]. Since the standards of general assessment literacy were first introduced by R. Stiggins [17] and American Federation of Teachers the role of language assessment literacy (LAL) has been increasing due to new societal needs and approaches to FL teaching and assessment.

Despite ongoing work on enhancing teachers' LTA skills, Western educationalists consistently point to a deficit in teacher LAL. The main reason for this is the systemic shortcomings of modules/ courses on LTA included in pre-service and in-service curricula; they are generally marginalized, are too short and often only optional [18]. Another reason lies in difficulties of defining the range of assessment skills and the scope of training needed to develop these skills properly. The analyses of existing courses undertaken by J. D. Bailey and K. Brown, Y. Jin established that the course had similar constituents which had been only insignificantly changed since mid 1990s [2, 371; 11]. The core topics of LTA training included principles and practice of item writing, validity and reliability issues, item analysis, etc. which help teachers learn to write language tests proper. As A. Green notes, such traditional course content makes courses fairly conservative since they neglect newly developed assessments (such as learner's portfolio and self-assessment) and therefore fail to “meet teachers' needs in rapidly changing educational environments” [6, 14].

Theoretically, LAL received multiple definitions. The most frequently cited definition was made by G. Fulcher according to which LAL is “the knowledge, skills and abilities required to design, develop, maintain or evaluate, large-scale standardized and/or classroom based tests, familiarity with test processes, and awareness of principles and concepts that guide and underpin practice, including ethics and codes of practice. The ability to place knowledge, skills, processes, principles and concepts within wider historical, social, political and philosophical frameworks in order understand why practices have arisen as they have, and to evaluate the role and impact of testing on society, institutions, and individuals” [5, 125]. D. Dunkley, in his turn, calls this definition restrictive as it contains a description of teachers' competence and excludes LAL of other stakeholders [4, 105]. This criticism is in line with L. Taylor's considering LAL of different groups of stakeholders and specifying several assessment skill profiles relevant for these groups (core, intermediary, and peripheral levels). Teachers' assessment needs, in her view, belong to the intermediary level and relate primarily to language pedagogy, knowledge of technical skills, sociocultural values and local practices; the core level concepts of LTA (knowledge of theory, principles and concepts, scoring and decision making) are only complementary to the list of basics needed by teachers [18]. Indeed, teachers need less theory than testing professionals and more practical classroom-relevant assessment skills.

K. O'Loughlin seems to have given the most concrete definition of LAL expressed in terms of setting objectives of LTA courses for student teachers. The course should envisage enabling students “to develop (a) a sound understanding of key concepts in second language assessment; (b) their ability to critically evaluate existing assessment documents, and (3) their capacity to design or adapt assessment instruments for the particular teaching context” [16, 73].

Coming back to practical considerations about the course content, it is obvious that it should reflect the levels of LAL proposed by L. Taylor and, therefore, place “the emphasis in professional development - at least initially - on the role of assessment in the learning process rather than on theoretical and statistical issues in testing [Brindley 2001, 129, cited in 16]. Content of courses, aka teaching, is inseparably connected with and/or is based on the existing LTA course books. The latter are criticized by O. Inbar-Lourie [10] and G. Brindley [cited in 16, 14] for being too arcane for non-specialists and, more importantly, for giving little or no attention to the issues of classroom based assessment. The most popular text books (e.g. Bachman and Palmer, 1996) [1] recommended today for course reading were published over 20 years ago, offer little guidance on development of either classroom based tests or alternative assessments and are hardly relevant to the changing reality of FL teaching and assessment.

All this calls for redesign of LTA courses, their content and text books as well as the training model itself. A. Davies suggested filling the gap within a “knowledge + skills” training model with “scrutiny of testing practice” including evidence of performance on tests and its consequences [Davies 2008, 328, cited in 16, 15). Moreover, applied linguists oppose principles of standarised and non-standardised test designing and propose reconceptualisation of validity and reliability applied in classroom based assessment [Moss and Brokhart 2000, cited in 16, 19) or even reformulation of the competencies needed for conducting assessments in the educational context [10]. These suggestions along with ideas of embracing sociocultural paradigm within LTA courses and developing assessment and/or testing cultures belong to the future. To date, however, “classroom assessment should be judged according to the traditional standards for test validity” [6, 18) and, therefore, LTA courses for teaching practitioners should provide the trainees with solid theoretical grounding and practical skills of constructing, validating and interpreting assessments.

A. Green summarises the principles of developing effective assessments by the acronym PRICE: Planning and Reflection lead to Improvement, when supported by Co-operation and informed by Evidence [7, 21-25]. These suggest that teachers preparing language tests should work methodically and collegially, contribute to discussion of test construct, techniques employed, and further reflect on what and how has been done, provide feedback on items written by each other: in A. Green's words test designers should follow the plan-do-review schedule [7, 21]. Teachers should learn to work in teams during the whole process of test preparation and accept possible critique from colleagues. Moreover, it is essential for them to learn to collect and analyse evidence of the ability of a test to yield true picture of learners' achievements. So far the quality of teacher constructed tests has not been questioned by their users although a study by

D. Coniam [3] suggests the opposite. O. Kvasova provides a detailed examination of major faults within teacher-made MCQ test-tasks written by Ukrainian university teachers [14].

The issues of designing LTA courses for pre-service and in-service contexts were discussed in numerous papers of J. D Brown and K. Bailey K. M., D. Coniam, Ј. Hatipoglu, L. Herrera and D. Macias, Y. Jin, R. Johnson et al., O. Inbar-Lourie, R. C. Kleinsasser, K. O'Loughlin, Tempus ProSET project (www.proset-tempus.netj; the work of Erasmus+ TALE project aimed at designing nine on-line training courses is in progress now (http://taleproject.eu). The study of these theoretical and practical researches as well as own practice of conducting workshops on test writing allows to formulate the goal and objectives of the Course developed specifically for Ukrainian university teachers.

The overall goal of the Course is to promote the enhancement of LTA literacy of university foreign language teachers, to enable them to carry out high quality classroom assessments and, in the long run, to maximize the efficiency of FL teaching and learning. The attainment of the major goal is ensured through achieving the following objectives: systematizing the knowledge of Methods of teaching foreign languages obtained by teachers during pre-service training course and, especially, its part devoted to language assessment; systematizing teachers' own experience of carrying out various forms of assessment; introducing the teachers to the fundamentals of LTA through presentations of LTA theory, participating in workshops and through reflecting on own practices; developing teachers' skills of constructing quality test items/task/ tests of different types (objective and subjective) intended to measure students' various skills (language use, reading, listening, writing, oral interaction and oral production); getting the teachers familiar with the statistics applicable in the classroom to determine the validity of self-made test tasks, encourage their ability to interpret the data and use it in order to improve own test items/tasks; promoting the development of teachers' reflective skills to carry out fair self- and peer- assessment; encouragement of team-work skills as well as skills necessary for proper logistics and mechanics; encouraging teachers' readiness for life-long learning, extensive reading on LTA, participation in workshops of experts, willingness to share own expertise, acceptance of newly developed forms of assessment.

The structure of the Course is made up of six modules that cover the major theoretical and practical aspects of LTA. Each module consists of identical parts:

1 Independent individual (pre-session) work (4 hours). This invites teachers to a) study the recommended literary sources on LTA issues and check its acquisition by answering the offered questions; b) doing a practical assignment.

2 Training session (4 contact hours). This envisages participation in two workshops on the topic that combine theoretical input and doing practical task and are aimed to build relevant skills in LTA. The workshop may be conducted either on the same day or on two subsequent days.

3 Independent team (post-session) work (12 hours). This suggests teaming up teachers (24) in order to do the following: a) construct test tasks to measure a particular language skill using the checklist for item writers; b) try out the constructed tasks and modify them if necessary; c) administer the test tasks in their own student groups, mark and grade them; d) do the statistical analysis appropriate for classroom based assessment; e) interpret the data focusing on the validity of the test tasks, discuss the quality of the tasks, suggest ways to further modify and, possibly, enter into the test task bank; f) collect feedback from testees, discuss the possible impact of the test on the testees' further learning of English; e) answer questionnaires, reflect on their achievements or failures, elaborate plans for further development.

4 Follow-up conference (2 contact hours). The teachers are invited to a) present the outcomes of their team work; b) get feedback from colleagues and trainers and share their impressions of other teams' test tasks; c) give feedback on the efficiency of the module implementation, suggest improvements, make own plans about doing the next module/s.

On successful completion of all tasks that are offered by any of the modules of the Course teachers receive a Certificate of completion. The teachers who successfully completed the tasks of all modules receive a Certificate that will allow them to plan and conduct their own teacher training.

The sustainable outcomes of the Course lie in: a) development of banks of quality test tasks to measure various FL skills which can be used at universities across the country;

b) dissemination of the outcomes of the Course by its participants through university/national conferences, scholarly articles, own workshops and master-classes.

In the long-term perspective the Course could entail: a) cooperation with acclaimed experts on LTA who could be invited to make presentations, conduct workshops and training accessible for grass roots FL teachers from all over Ukraine; b) setting Centres/Resource Centres for enhancing assessment literacy of Ukrainian university teachers in all regions of the country;

c) introduction of Master's programmes in LTA; d) implementation of research into LTA including PhD studies; e) setting up an e-platform for all teachers' independent use and selfstudy.

One of the course modules was piloted in universities of Kyiv, Kharkiv, Cherkasy, Lviv and proved effective. Below are the content and structure of the piloted module.

MODULE 2:

Construction and validation of classroom reading test tasks

Independent individual (pre-session) work (4 hours):

а) study the recommended sources: Kvasova O. Learning to ensure good testing practice (notes on a workshop) // Іноземні мови. - 2011. - №1. - С. 62-66 та // Іноземні мови. - 2011. - № 4 (68). - С. 57-60.

б) do the task: construct two test tasks (MCQ and T/F) to check the understanding of the recommended texts.

Training session (4 contact hrs.):

Workshop 1. Constructing reading test tasks

CEFR descriptors for reading skills. Expeditious and careful reading (skills and subskills).Objective and subjective test formats. How to construct MCQ andT/F/NG. Analysis of MCQ tasks. Faulty items. Standardized rubrics.

Workshop 2. How to write valid reading test tasks

Basic statistics for teacher-made test tasks (item difficulty, item discrimination, distractor analysis). How to interpret the data. Analysis of some MCQ items/tasks and interpretation of the data.

Independentppost-session work (12 год.):

a) study the recommended sources:

Common European Framework of Reference for Languages: learning, teaching, assessment. Council of Europe. - Cambridge: CUP, 2001. - Pp.68-82.

Гнаповська Л. В., Квасова О. Г. Оцінювання вмінь читання: з досвіду пілотування тестових завдань в межах Проекту незалежного тестування з іноземних мов // Іноземні мови. - 2007. - № 2. - С. 3-10.

Квасова О. Г. Тестова перевірка рівня сформованості англомовної компетентності у читанні у майбутніх філологів // Іноземні мови. - 2014. - № 3 (79). - С. 38-44.

Hughes A. Testing for Language Teachers. - Cambridge University Press, 1989. - Pp. 161-162.

b) do the task:

contruct, pilot, modify and administer two own test tasks (MCQ and T/F/NG) in your classroom. Do the statistical analysis and interpret the data.

Follow-up conference (2 contact hrs.):

Present your tasks, the data of statistical analysis and its interpretation. Discuss your tasks and the tasks of your colleagues.

As is seen from the excerpted module, the coursework contains both independent and contact work, individual as well as team work, readings and discussions, and tasks to be carried out. The course presents the balanced and versatile selection of activities which will strengthen the participants' sense of agency and stimulate and motivate them.

Apart from workshop materials, several tools were designed to collect the data of LAL and scrutinize teachers' progress. To measure the initial level of teachers' assessment literacy a test comprised of twenty questions was developed and offered to trainees. It intended to check the trainees' familiarity with the introductory content area of LTA. Another tool - a test constructor questionnaire - was aimed to collect feedback from teacher trainees upon completion of the module; it provided useful information on the further improvements of the course including the content of learning, techniques and organisation and management related issues. Since the constructed tests were piloted by teachers in real-life context, in their academic groups, a special questionnaire was compiled to elicit students' feedback on the test tasks offered to them, and compare their quality with that of test tasks from IELTS or FCE examinations. Finally, the quality of tests developed by teachers and the materials of statistical analysis performed by them was thoroughly examined by the researcher and two other experts.

These tools appeared quite efficient to identify the strengths and weaknesses of the proposed methodology of teacher training in LTA. As a result of participation in the training the teachers were credited by the heightened quality of self-constructed test tasks although it would be erroneous to claim that all of them excelled in preparing the assessment. The trainees also expressed positive evaluation of the module's content and the variety of techniques used by the trainer, although suggested that the course should be delivered over a longer period allowing more time for post-session independent and team activities (constructing, piloting, reflecting, reviewing, modifying test tasks).

The case of piloting the module together with the research data and implications for reviewing and modifying the course were publicized [14] and reported at the 12th Annual Conference of European Association for Language Testing and Assessment (EALTA) (Copengahen, 2015). The perspectives of further research depend upon decisions stipulating the introduction of the Course at universities within in-service teacher training programmes. Meanwhile, this experience is disseminated via activities of Ukrainian Association for Language Testing and Assessment (UALTA).

References

1. Bachman L. F., Palmer A. S. 1996. Language Testing in Practice. - Oxford : Oxford University Press.

2. Brown J. D., Bailey K. M. Language testing courses: What are they in 2007? Language Testing, 2007, 25/3: P. 349-383.

3. Coniam D. 2009. “Investigating the quality of teacher-produced tests for EFL students and the effects of training in test development principles and practices on improving test quality” System 37 (2):226-42.

4. Dunkley D. Language assessment in theory and practice. Accessed 8 August 2016 at: kiyou.lib.agu.ac.jp/pdf/kiyou_14F/14.../14_40_1_103.pdf

5. Fulcher G. Assessment literacy for the language classroom. Language Assessment Quarterly, 2012. Vol. 9 (2), pp.113-132.

6. Green A. Assessment Literacy for language teachers. In Tsagari D. (ed.) Classroom-based Assessment in L2 Contexts - Cambridge Research Publishing, 2016. - P. 8-28.

7. Green A. Exploring language assessment and testing. Language in action. Routledge Taylor & Francis Group. - 269 p.

8. Hatipoglu Q. Summative evaluation of an English language testing and evaluation course for future English teachers in Turkey. - ELTED Vol. 13.Winter 2010. - P. 40-51.

9. Herrera L., Macias D. A call for language assessment literacy in the education and development of teachers of English as a foreign language. Colomb. Appl. Linguist. J., (2015). Vol. 17 (2). - P. 302-312.

10. Inbar-Lourie O. Constructing a language assessment knowledge base: A focus on language assessment courses. - Language Testing, 2008. - Vol. 25 (3). - P. 385-402.

11. Jin Y. The place of language testing and assessment in the professional preparation of foreign language teachers in China. - Language Testing, 2010. - Vol. 27/4. - P. 555-584.

12. Johnson R., Preston B., Floyd O. Teaching the Second-Language Testing Course through Test DevelopmentbyTeachers-in-Training. - Teacher education quarterly, 1999. - P. 71-82.

13. KleinsasserR. C. Transforming a postgraduate level assessment course: A second language teacher educator'snarrative. - Prospect, 2005. - Vol. 20. - P. 77-102.

14. Kvasova O. A. Case of Training University teachers in Developing and Validating Classroom Reading Test Tasks. In Tsagari D. (ed.) Classroom-based Assessment in L2 Contexts - Cambridge Research Publishing, 2016. - P. 54-74.

15. Kvasova O., Kavytska T. The assessment competence of university foreign language teachers: A Ukrainian perspective. - Language Learning in Higher Education. 2014.Volume 4, Issue 1. P. 159-177.

16. O'Loughlin K. Learning about second language assessment: Insights from a postgraduate student on-line subject forum. - University of Sydney Papers in TESOL, 2006. - Vol. 1. - P. 71-85.

17. Stiggins R. J. Evaluating classroom assessment training in teacher education programs. - Educational Measurement : Issues and Practice, 1999. - Vol. 18 (1). - P. 23-27.

18. Taylor L. Communicating the theory, practice and principles of language testing to test stake holders: Some reflections. - Language testing, 2013. - Vol. 30 (3). - P. 403-412.

Размещено на Allbest.ru