Research university bachelor students spoken production assessment criteria: design and approbation

Moreover, the interview method is used in the process of testing the designed criteria. After assessment sessions, I want to receive the feedback from practicing teachers. The feedback includes:

their thoughts on the criteria;

whether there were any issues while applying them;

suggestions for criteria improvement.

The interviews are carried out straight after implementing monologue criteria and dialogue criteria. The interview format is semi-structured: while there is a list of things (see above) that I want to elicit, there is no strict structure of the questions, so the interviews are close to discussions. This method enables me to get the first-hand experience of the teachers at our programme who were applying our criteria while assessing their 3rd year students' speaking performance, for which the criteria are designed.

The context of testing and assessment process

The speaking assessment in focus of this thesis is a part of the Speech practice course, which is a compulsory course at the 3rd year of studies at the Department of foreign languages. It belongs to the professional cycle of disciplines. The monological speech assessment is carried out at the end of the Unit 3, while the dialogical speech evaluation takes place after the students finish Unit 5.

Course pre-requisites: the students should have achieved the B2+ (Upper-Intermediate) level of language proficiency, being ready to use the foreign language as a tool to derive information from foreign sources for various purposes and critically analyze the data; they should have acquired the basics of cultural awareness, efficient communication, research activity, as well as ICT (information and communication technology); finally, the students should be able to use cognitive reflection to assist in their awareness of boundaries of their knowledge, new cognitive goals and mental processes.

The knowledge gained during the course is supposed to be useful for other disciplines at the university. The online component constitutes 20% of the course grade. Generally, the course is aimed at developing “students' communicative competence in the area of English for Academic purpose and is connected to students' research activities” (Bogolepova & Kolesnikova, 2018, p. 2). The main goal is “the formation of foreign language communicative competencies at the C1+ level” (Bogolepova & Kolesnikova, 2018, p. 2).

Learning outcomes for speaking, which is of particular interest for this study, include:

the use of academic vocabulary;

the ability to make use of speaking samples;

the use of background knowledge;

the use of linking devices;

the ability to prove one's point of view employing evidence;

the source reference done in an appropriate way;

the text(s) summary;

the participation in dialogues using functional language;

the presentation of research findings supported with visual aids (Bogolepova & Kolesnikova, 2018).

Moreover, special attention is paid to developing the students' academic skills, such as identifying logical connections, analyzing and synthesizing information and identifying problems and providing possible solutions. The complete list can be found in the course syllabus (Bogolepova & Kolesnikova, 2018).

To sum up, while the main course objective is the formation of the communicative competence at C1+ level, the emphasis is on the use of English for other professional disciplines and research purposes.

Procedures for developing test materials

The process of developing the test materials included the following stages:

thorough analysis of the TLU situation and the students' needs;

statement of the objective of testing;

definition of the test construct;

writing the test draft (including the criteria draft);

test modification;

editing the test (including editing the criteria).

The categories underlying the speaking construct and the subskills tested in the process of assessment can be observed in Figures 7, 8 and 11 respectively.

The initial criteria that were designed to be tested can be found in Appendices 1,2.

Figure 7. Categories underlying the speaking construct (dialogical test)

Figure 8. Categories underlying the speaking construct (monological speech)

Tasks description and analysis

Now I will briefly analyze the test tasks that were used in the two speaking tests where the designed speaking criteria were tested. When approaching the tasks, I will use G. Fulcher's framework for describing tasks (Fulcher, 2003, pp. 144-145).

The first test was aimed at assessing the dialogical speech of the 3rd year students. They were divided into pairs and asked to complete two tasks.

Figure 9. Dialogical task (number 1)

Task 1.

Now I would like you to talk about something together for about 2 minutes. Here are some spheres where cultural awareness is important. First you have some time to look at the task.

Now talk to each other about why cultural awareness is important in these spheres. Don't forget to use language for comparison and contrast and active vocabulary.

Thank you. Now you have a minute to decide in which spheres cultural awareness plays the most important role.

Task 1 (see similar tasks in other test variants in Appendices D - H) incorporates a spidergram with a question. The tester's speech is prompted by the card, while the students get the scheme only. The format of the task is identical to the CAE (Cambridge English: Advanced) Speaking part 3 exam task.

Task orientation: guided. The task requires the students to talk about the mentioned aspects, however, they are by no means restricted to speak only about them.

Interactional relationship: two-way. It is clear that the students co-construct the communication and are expected to play equal, or almost equal, role in achieving common goals. The assessor does not intervene in the conversation, only signals the transition from one part to another.

Goal orientation: convergent.

Interlocutor status and familiarity: same status, as the interlocutor is a fellow student; medium to high familiarity, provided that it is the third year of the students being acquainted with each other.

Topics (for this exact task and the other similar ones, see Appendices D-H): cultural awareness, cultural difference, national character. These topics are a part of the course syllabus, as well as the unit 3. After studying the unit for a set amount of lessons, the students took the test.

Referencing the course syllabus, this task is aimed at testing such expected outcomes as the ability to prove one's point of view using evidence and to participate in a dialogue using functional language. The students need to demonstrate their rich advanced vocabulary and grammar as well as manage the successful interaction and come to a certain agreement with each other, while being respectful and mostly formal. The topics were chosen according to the syllabus, so that students could demonstrate their background and professional knowledge

Task 2.

You are going to discuss something with your partner. Follow the plan given, remember to use active vocabulary and language for comparison and contrast. Be polite and stick to the formal style.

Russians and Americans: are we that different?

Talk about:

The similarities between Russian and American national characters

The differences between them

How the national character traits of the Russians and the Americans may be beneficial in terms of working on a business project

Whether these traits may become an obstacle

Take 1 minute to prepare and 3-4 minutes to discuss the topic with your partner. At the end of your discussion make a short conclusion.

Task 2 (see similar tasks in other test variants in Appendices D - H) asks the students to engage in a conversation based on the given topic and the prompts.

Task orientation: guided.

Interactional relationship: two-way.

Goal orientation: convergent.

Interlocutor status and familiarity: same status, medium to high familiarity.

Topics: cultural awareness, national character, interpersonal conflicts, personal development, stereotypes. These topics are a part of the course syllabus, as well as the unit 3.

The task focused on such aspects of the desired outcomes as the ability to take part in a dialogue using functional language, identifying problems and providing possible solutions. The students need to be aware of the time limit to manage the conversation efficiently. They are expected to use advanced and academic vocabulary, while being polite.

Moving to the second speaking test, which evaluates the monological speech of the students, there is only one task included. The students receive one of the eight texts (see a sample text in Appendix I) to give its summary and express their opinion about the problem raised in the text. They are informed that they have 10 minutes to read the text and get ready and 3-4 minutes to deliver the summary and the monologue. They are also allowed to take notes while preparing.

Task orientation: guided.

Interactional relationship: one-way. The student is asked to make a short summary of the text and to communicate their opinion on the topic, and they are expected to speak the whole time, with to interruptions by the teacher, who only signals the end of the time available for the answer.

Goal orientation: convergent.

Interlocutor status and familiarity: high status, as the interlocutor is the teacher; high familiarity, as it is their Speech practice teacher.

Topic: technological progress and its current and future implication.

This monological task references the expected course outcomes, namely the ability to summarize and present the information from one or multiple sources and to analyze and synthesize the information. The students are also required to use evidence while proving their point of view; their language should be formal.

Having described the test tasks that were used, it is important to highlight that all the three task formats were familiar to the students of all groups, as they had trained in the completion of similar tasks during the course and as a part of revision and preparation for the tests. This is generally aimed at helping the students perform better by eliminating any difficulties caused by the unknown task format.

The test procedure was standard: the students were presented with the task one by one, so that everyone had equal amount of time to prepare. Both the preparation and the oral performance was timed by the teacher, with whom the students were familiar (as it was their Speech practice teacher) and who later evaluated their speeches. The teachers also gave the feedback on the students' performances.

Table 1. Skills and subskills tested in the two speaking tests

Skills	Subskills
Monological speaking	fluency accuracy with words and pronunciation using functional language appropriacy ability to speak at relevant length use of range of words and grammar use of discourse markers academic skills: manage time and resources; identify logical connections; analyze and synthesize information; identify problems and suggest possible solutions
Dialogical speaking	fluency accuracy with words and pronunciation using functional language appropriacy turn-taking skills ability to speak at relevant length responding and initiating use of range of words and grammar use of discourse markers academic skills: manage time and resources; identify logical connections; analyze and synthesize information; identify problems and suggest possible solutions

After carefully testing the rating scales with the practicing teachers and interviewing them, I processed the feedback and reported in in the form of tables.

Table 2. The feedback for the analytic criteria (dialogical speech)

Positive aspects	Negative aspects/ Recommendations/ Things to reconsider
Enjoyed the criterion “Creativity and analytical skills”, as “they really need to be tested”	In the parameter “Grammar and vocabulary”: 1) specify “wide range of grammar”, 2) active and academic vocabulary - how many items?
“No focus on the amount of active vocabulary, so there is no need to write everything down” (and the teacher can focus on the overall performance)	Criterion “Creativity and analytical skills”: 1) “creative ideas” - may cause bias, need to reconsider, 2) “deep” understanding - specify
“Concise, and as a result, easy to navigate”	Criterion “Content”: 1) “some aspects are not fully covered”, “partly corresponds” - specify
Such criteria make it easier to explain the mark to the student	Criterion “Pronunciation”: “phonological features” - specify
	Mistakes VS use of active vocabulary
	Make the wording more precise - for less prepared raters
	“In the dialogues the focus should be probably on ideas, not vocabulary” “Specify what type of hesitations you mean - they are only natural”

The experts had mixed opinions about this first scale. Some appreciated its structure and brevity, pointing out the ease of use and highlighting the transparency of the criteria both for the teacher and the students. Hence, it was easier to justify the marks the students received and reach the same evaluation of the student's performance. Moreover, one expert emphasized the relevance of the “Creativity and analytical skills” parameter, as they acknowledged the need to focus on such skills while testing speaking, not just the form, but the content as well. However, the teachers mentioned the necessity to specify such concepts as “wide range of grammar”, “deep understanding”, “creative ideas”, “partly corresponds”, “phonological features”. Although the experts had some understanding of the given concepts, they advised to clarify them to make the rating scale easier to use and to avoid inconsistencies. One expert focused my attention on the concept of fluency and advised to specify the type of “hesitations” I was referring to in the criteria, as hesitation pauses are only natural not only in the colloquial speech, but also in academic setting (particularly when mentioning cognitively demanding ideas, concepts, links).

Table 3. The feedback for the holistic criteria (monological speech)

Positive aspects	Negative aspects/ Recommendations/ Things to reconsider
Easier to use than the analytic criteria	“Where does the distinction lie between mistakes that impede and don't impede the communication?”
User friendly (underlined phrases and words in bold)	The issue of marking a performance with mixed characteristics (some from one band and some from another)
Precise wording and clear concepts used	Conflicts with students may arise considering the marks

After testing the monological speech rating scale, the experts were unanimous in their opinion on the feasibility of the criteria. The teachers considered the scale to be user friendly and easy to use. They drew my attention to the precise wording and clear concepts employed in the descriptors. Nonetheless, a common issue that arose was the difficulty of evaluating mixed characteristics performances, which could potentially lead to conflicts with students. It was recommended to create a scheme for the resolution of such instances.

To conclude, some experts preferred the holistic criteria to the analytic ones due to their ease of use, and some told me they would rather use analytic rating scales as they are used to them, being trained as the Russian State Exam experts (which relies on analytic scoring to evaluate the speaking performance). The feedback I received from the experts allowed me to revise and improve the rating scales, and this is what I will focus on in the next part.

After getting the feedback of the practicing teachers it was crucial to revise the criteria, implementing the advice and the tips I received.

The revised dialogue criteria (Appendix J), compared to the initial version (Appendix A), contain the following alterations:

The “Content criterion” has been specified in terms of what it means to “partly correspond with the communicative task”;

Hesitations and pauses have been specified, taking into account that hesitation related to difficult conceptual ideas is only natural;

“Wide range of grammar” has been specified;

The footnotes have been provided to specify such concepts as “deep understanding of the topic” (“Deep understanding relates to the student's ability to compare, contrast, analyze ideas rather than just enumerate/retell them. It is also about being able to hypothesize, generalize, criticize”, see Appendix J) and “creative ideas” (“Creativity is a mental characteristic that allows a person to think outside of the box, which results in innovative or different approaches to a particular task. If the student's ideas are clichй, and nothing about them shows the students' own input in relation to the task, the marking should be 1 point for this parameter (“little to no creativity is shown”)”, see Appendix J);

The phonological features of interest for the “Pronunciation” criterion have been enumerated.

In turn, the revised monologue criteria (Appendix K), compared to the initial version (Appendix B), contain no alterations in the wording of the descriptors. The references were added to clarify the terms “mistakes that impede communication” and “mistakes that do not impede communication” and to give suggestions on how to deal with the mixed features performances (see Appendix K).

Mistakes that impede vs don't impede communication are differentiated based on their amount, as the more mistakes are made, the more difficult it is for the listener to grasp meaning.

1. If the examinee's performance incorporates the features of the adjacent bands, it is recommended to pay attention to the content of the talk (if the student's ideas show deep understanding of the topic, it is suggested to opt for the higher score).

2. If the mixed features are from the completely different mark categories, say the summary part can be evaluated as a “4” and the opinion-based talk is closer to an “8”, it is advised to look at the lower mark and add 1 or 2 point to it, depending on the quality of the stronger part of the talk (if, as in our example, the talk - the better part - contains deep topic understanding, it is suggested to add 2 points).

These recommendations are by no means obligatory and are intended for guidance purposes.

You may also take into account the progress made by the student and favour their performance in case 1.

Recommendations for developing EAP speaking tests and criteria

Given the conducted analysis of a number of prominent works focused on spealing testing and rating scales design, as well as the process of designing the criteria for the HSE students and revising them according to the received feedback, it is possible to provide the recommendations for developing the English for Academic purpose (EAP) speaking tests and rating scales. These recommendations may be used by teachers or test developers designing speaking tests for the higher education setting.

speaking education language

Figure 10. Recommendations for developing EAP speaking tests and criteria

The aim of the paper is to design two rating scales - for monological and dialogical speech assessment - for the 3rd year students' assessment at the Speech Practice course. These rating scales are aimed at the final speaking assessment at the end on the two units. The rating scales design based on the learning objective of the course, namely “the formation of foreign language communicative competencies at the C1+ level in the field of English for Academic Purpose” (Bogolepova & Kolesnikova, 2018). It is vital to note that these competencies are aimed at being used in other professional disciplines and for research purpose in particular. So, I designed and tested the dialogical and monological speech criteria, as well as received the feedback from the teachers. Then I improved the criteria according to the feedback.

As part of this work I also:

described the fundamental features on the speaking activity;

examined speaking subskills;

explored rating scales design approaches;

described rating scales design challenges;

analyzed the aims of the “Speech practice” course;

designed two sets of speaking assessment criteria;

tested the two sets in assessing the 3rd year bachelor students speaking skills;

carried out a number of interviews with the Department's teachers of “Speech practice”;

improved the two sets of criteria using the feedback.

As a part of this work, I created several schemes and figures that can be useful for anyone planning a test a criteria design (see Figures 4,5), as well as a table for evaluation of the test materials (see Figure 15). The recommendations for developing EAP speaking tests and rating scales (see Figure 14) can assist in designing and editing the rating criteria by both teachers and test developers.

Table 4. Evaluation of the test materials

Parameters	Dialogical test	Monological test
Based on analysis of students' needs	+	+
Validity
Content validity	+	+
Face validity	+	+
Reliability	+	+
Communicative competence
Linguistic competence	+	+
Sociolinguistic competence	+	+
Discourse competence	+	+
Strategic competence	+	+
Text types (authentic or non-authentic)		+

Conclusion

Since speaking is a complex process unique to humans, there are challenges in assessing it properly. To analyze the speaking skill, it seems feasible to divide it into subskills. As has been discussed in this thesis, speaking subskills form the basis for the design of the spoken production criteria. The choice of the rating scale is influenced both by the test purpose and the context of learning. As the context of the study is taken into account, it is important to state that the spoken production assessment criteria were intended to suit the English for Academic purpose context, as well as the research university context.

The aim of this paper, which is to design a set of speaking criteria for the 3rd year students of the HSE programme “Foreign languages and cross-cultural communication”, was achieved. The final versions of the monological and dialogical speech assessment criteria are expected to have the potential of being used as a part of the Speech practice course, providing grounds for the improvement of the system of assessment.

The limitations of the research lie in its limited scope, being that I focused on the feedback of six teachers who work at HSE at the Department of foreign languages. As a result, the final product is supposed to suit the students who study English for Academic purpose, as well as intend to carry out research in English. We can predict that the criteria are subjective to the context of learning and may not suit other context to the fullest extent. However, we do believe that they can be modified to cater for other contexts, too.

This work can assist in paving the way for the further research on the issue of speaking skills assessment and criteria design, encouraging further exploration of the field and facilitating the educational process by means of improved students' speaking skills assessment criteria.

References

Alderson, J. C. (1991). Bands and scores. In J. C. Alderson & B. North (Eds.), Language testing in the 1990s: The communicative legacy (pp. 71-86). London, England: Macmillan.

Anderson, J. A., Bresciani, M. J., & Zelna C. L. (2004). Assessing student learning and development: A handbook for practitioners. Washington, DC: National Association of Student Personnel Administrators.

Allen, D., & Tanner, K. (2006). Rubrics: Tools for making learning goals and evaluation criteria explicit for both teachers and learners. CBE-- Life Sciences Education, 5 (3), 197-203. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1618692/

Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.

Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice. Oxford: Oxford University Press.

Bachman, L. F., & Savignon, S. J. (1986). The evaluation of communicative language proficiency: a critique of the ACTFL Oral Interview. Modern Language Journal, 70, 380-390.

Bogolepova, S., & Kolesnikova, E. (2018). Syllabus for the Speech practice course. Retrieved from https://www.hse.ru/data/2019/02/06/1150125018/program-1557813735-7Mh8Uf7DhP.pdf

Brown, H., & Abeywickrama, P. (2010). Language assessment: Principles and classroom practices (2nd ed.). Harlow: Pearson Education Limited.

Brown, A., Iwashita, N., & McNamara, T. (2005). An examination of rater orientations and test-taker performance on English-for-Academic-Purposes speaking tasks (TOEFL Monograph Series, MS-29). Princeton, NJ: Educational Testing Service.

Bygate, M. (2010). Speaking. In R. B. Kaplan (Ed.), The Oxford Handbook of Applied Linguistics (2nd ed.). Retrieved from http://proxylibrary.hse.ru:2344/view/10.1093/oxfordhb/9780195384253.001.0001/oxfordhb-9780195384253-e-4?rskey=sLHb6I&result=1

Chapelle, C. (1999). Validity in language assessment. Annual Review of Applied Linguistics, 19, 254-72.

Chuang, Y. (2009). Foreign language speaking assessment: Taiwanese college English teachers' scoring performance in the holistic and analytic rating methods. The Asian EFL Journal, 11 (1), 150-173.

Council of Europe. (2018). Common European framework of reference for languages:

Learning, teaching, assessment. Companion volume with new descriptors. Retrieved from https://rm.coe.int/cefr-companion-volume-with-new-descriptors-2018/1680787989

Davies, A. (1990). Principles of language testing. Oxford: Basil Blackwell.

Davies, A., Brown, A., Elder, C., Hill, K., Lumley, T., & McNamara, T. (1999). Dictionary of language testing. Cambridge: Cambridge University Press.

Douglas, D. (1997). Testing speaking ability in academic contexts: Theoretical considerations. Princeton, NJ: Educational Testing Service.

Douglas, D. (2001). Language for specific purposes assessment criteria: Where do they come from? Language Testing, 18 (2), 171-185.

Dvoretskaya, E. V. (2016). English for academic purposes in higher education curriculum. Voprosy sovremennoj nauki i praktiki. Universitet im. V. I. Vernadskogo, 1(59), 147-152.

Fetzer, A. (2013). The structuring of discourse. In M. Sbisа, K. Turner & W.Bublitz (Eds.), Pragmatics of speech actions (pp. 685 - 711). Berlin: De Gruyter, Inc.

Fulcher, G. (1996). Does thick description lead to smart tests? A data-based approach to rating scale construction. Language Testing, 13 (2), 208-238.

Fulcher, G. (1999). Assessment in English for academic purposes: putting content validity in its place. Applied Linguistics, 20 (2), 221-236.

Fulcher, G. (2003). Testing second language speaking. Abingdon, Oxon: Routledge.

Ginther, A. (2013). Assessment of speaking. In C. A. Chapelle (Ed.), The Encyclopedia of Applied linguistics. Retrieved from https://www.researchgate.net/publication/277707664_Assessment_of_Speaking

Hamp-Lyons, L. (2011). English for Academic purposes. In E. Hinkel (Ed.), Handbook of Research on Second Language Learning and Teaching (pp.89-105). New York, NY: Routledge.

Hamp-Lyons, L., & Hyland, K. (2002). EAP: Issues and directions. Journal of English for Academic Purposes, 1, 1-12.

Harmer, J. (2007). The practice of English language teaching (4th ed.). Harlow: Pearson Education Limited.

Hughes, R. (2011). Teaching and researching speaking (2nd ed.). C. N. Candlin & D.R. Hall (Eds.). New York, NY: Routledge.

Isaacs, T. (2016). Assessing speaking. In D. Tsagari & J. Banerjee (Eds.), Handbook of second language assessment (pp. 131 - 146). Berlin: De Gruyter, Inc.

Jordan, R. (1997). English for Academic purposes: A guide and resource book for teachers. Cambridge: Cambridge University Press.

Khabbazbashi, N. (2015). Topic and background knowledge effects on performance in speaking assessment. Retrieved from https://journals.sagepub.com/doi/full/10.1177/0265532215595666

Kramsch, C. (1993). Context and culture in language teaching. Oxford: Oxford University Press.

Krekeler, C. (2006). Language for special academic purposes (LSAP) testing: The effect of background knowledge revisited. Language Testing, 23(1), 99-130.

Lackman, K. (2010). Teaching speaking sub-skills: Activities for improving speaking. Ken Lackman & Associates. Educational Consultants. Retrieved from http://www.kenlackman.com/files/speakingsubskillshandout13poland_2_.pdf

Lantolf, J. P., & Frawley, W. (1985). Oral proficiency testing: A critical analysis. Modern Language Journal, 69(4), 337-45.

Levelt, W. (1989). Speaking: From intention to articulation. Cambridge, MA: MIT Press.

Luoma, S. (2004). Assessing speaking. Cambridge: Cambridge University Press.

Nation, P., & Newton, J. (2008). Teaching ESL/EFL listening and speaking. New York, NY: Routledge.

O'Sullivan, B. (2012). Assessing speaking. In C. Coombe, P. Davidson, B. O'Sullivan, & S. Stoynoff (Eds.), The Cambridge Guide to Second Language Assessment (pp. 234-246). New York: Cambridge University Press.

Romaine, S. (2000). Language in society: An introduction to sociolinguistics (2nd ed.). Oxford: Oxford University Press.

Taylor, L. (2011). Examining speaking. Research and practice in assessing second language speaking. Cambridge: Cambridge University Press.

Thornbury, S. (2005). How to teach speaking. Harlow: Pearson Education Limited.

UCLES (2011). Assessing speaking performance - level C1. Retrieved from: http://www.cambridgeenglish.org/images/168620-assessing-speaking-performance-at-level-c1.pdf

Ur, P. (2012). A course in English language teaching. Cambridge: Cambridge University Press.

Размещено на Allbest.ru

...