Assessment of academic performance with learning monitoring system for licensing integrated exam preparation in specialty of dentistry

Размещено на http://www.allbest.ru/

УДК 378.046.4:616.314-051

DOI 10.11603/me.2414-5998.2017.1.7503

ASSESSMENT OF ACADEMIC PERFORMANCE WITH LEARNING MONITORING SYSTEM FOR LICENSING INTEGRATED EXAM PREPARATION IN SPECIALTY OF DENTISTRY

O.M. Doroshenko, O. F. Sirenko

P. Shupyk National Medical Academy of Postgraduate Education, Kyiv

The aim of the study - to discover the ways of improving the level of interns' academic performance by development of competency- based testing formats in dental education with learning monitoring system application for licensing integrated exam preparation.

Materials and Methods. In order to improve preparedness to licensing integrated exam, learning monitoring system was used. The results of training tests, mock examination and licensing integrated exam were compared.

Results and Discussion. Learning monitoring system application for licensing integrated exam preparation in specialty of Dentistry allowed to reach higher results of state examination: the average percentage increase and the increase of number of interns, who passed licensing exam, were observed. High level of success pass predictability of learning monitoring system application was proved.

Conclusions. An implication of the on-line distance learning and learning monitoring system gives an opportunity to provide the monitoring of academic performance and to evaluate the competency level during the primary specialization in the specialty of Dentistry.

Key words: dental education; interns; learning monitoring system; integrated licensing exam.

Received 20.01.17

E-mail for correspondence: sirenko_a@ukr.net

trainee dental testing licensing

ОЦІНКА АКАДЕМІЧНОЇ УСПІШНОСТІ ЗА ДОПОМОГОЮ СИСТЕМИ МОНІТОРИНГУ НАВЧАННЯ ДЛЯ ПІДГОТОВКИ ДО ЛІЦЕНЗІЙНОГО ІНТЕГРОВАНОГО ІСПИТУ ЗА СПЕЦІАЛЬНІСТЮ “СТОМАТОЛОГІЯ”

О. М. Дорошенко, О. Ф. Сіренко

Національна медична академія післядипломної освіти імені П. Л. Шупика, м. Київ

Мета дослідження - визначення шляхів покращення рівня академічної успішності лікарів-інтернів шляхом удосконалення форматів оцінювання, основаних на визначенні компетентності, у стоматологічній освіті із застосуванням системи моніторингу навчання для підготовки до ліцензійного інтегрованого іспиту.

Матеріали та методи дослідження. Система моніторингу навчання була застосована з метою покращення готовності лікарів- інтернів до складання ліцензійного інтегрованого іспиту. Було порівняно результати тренувальних тестів, пробних іспитів та ліцензійного іспиту.

Результати й обговорення. Використання системи моніторингу навчання для підготовки до ліцензійного інтегрованого іспиту за спеціальністю “Стоматологія” дозволило досягти кращих результатів державної атестації: відзначалось зростання середнього відсотка складання та зростання числа інтернів, які склали ліцензійний іспит. Доведено високий рівень прогнозування успіху складання іспиту при застосуванні системи моніторингу навчання.

Висновки. Застосування дистанційного он-лайн навчання та системи моніторингу навчання дає можливість здійснювати моніторинг академічної успішності та оцінити рівень компетентності під час первинної спеціалізації за спеціальністю “Стоматологія”.

Ключові слова: стоматологічна освіта; інтерни; система моніторингу навчання; інтегрований ліцензійний іспит.

In recent years, there has been an increasing interest in the development of interactive learning monitoring and assessment methods in postgraduate dental courses. Competency-based testing formats in dental education include: multiple-choice questionnaire; structured oral examination; modified essay question; key features exam; objective structured clinical examination; objective structured practical examination; standardized patients; clinical evaluation exercise; mini-clinical evaluation exercise; entrustable professional activities; directly observed procedural skills; portfolio; multi-source feedback (multisource evaluations) [1].

There is a growing body of literature that recognizes the importance of assessment methods, measuring descriptive knowledge and identifying the ability to apply theoretical knowledge in a specific context to solve practical problem or to reach a clinical decision [1, 4].

Multiple-choice questions are commonly used in dental education for testing theoretical descriptive knowledge. Its validity and reliability is sufficient provided that high-quality questions and sufficient quantity are used [1]. The multiple-choice questionnaire is established as an effective assessment format, considering the low cost in light of the numbers and frequency of tests. The multiple-choice questionnaire requires less time and resources after test pool composition. The availability of distance performing for the significant number of participants simultaneously with easy online access makes such tests widespread [1]. The formed question pools can be applied by several universities [1].

Single best-answer multiple-choice questions consist of a question (the stem), two or more choices from which examinees must choose the correct option (the distractors) and one correct or best response (the key) (Cizek & O'Day, 1994, as cited in Tarrant, Ware & Mohammed, 2009) [5].

However, the number of distractors is argued, results from different studies show that in most circumstances, ideally multiple-choice items should consist of as many options as feasible given the item content and the number of plausible distractors [5]. On the other hand, three option-items have many advantages for both writers and examinees, and additional nonfunctioning distractors are not likely to improve item or test psychometric properties [5].

It was stated that writing high quality distractors is an important part of the test development process and four- or five-option multiple choice questions are the common standard in health science disciplines on certification-level examinations [5]. Properly constructed multiple-choice questions are able to test higher levels of cognitive reasoning and can accurately discriminate between high- and low-achieving examinees [5].

The rapid development of computer technologies caused widespread popularity of distance learning in different fields of medical education. Learning monitoring systems play an important role in the assessment of combination of theoretical knowledge, practical skills and ability to success in professional decision-making in dentistry [1]. Several systematic reviews [2, 3] indicate that internet-based learning, using pre- and post-tests, has demonstrated high effectiveness in obtaining of knowledge (factual or conceptual understanding), practical skills acquisition and making adequate clinical decisions. Principles of the evidence-based medicine have the main role in learning process in Dentistry during primary specialization and continuous professional development.

The aim of the work. The specific objective of this study was to discover the ways of improving the level of interns' academic performance by improvement of competency-based testing formats in dental education with learning monitoring system application for licensing integrated exam preparation.

Materials and Methods. To improve interns' preparedness to licensing integrated test, learning monitoring system was used. Question pools were formed in the on-line testing system (http://lms.inmeds. com.ua/ilias.php). The largest question pool consists of 2000 tasks from test bank, including tests on General Dentistry, Therapeutic Dentistry, Prosthetic Dentistry, Pediatric Dentistry, Orthodontics, and Maxillofacial Surgery. Multiple-choice questions are also divided into several subtests, such as clinical examination; treatment of dental patients requiring special tactics because of physiological states and physical illnesses; emergency aid; the organization of dental health care, including prevention; initial visits; repeated visits (treatment ongoing). Moreover, separate question pools based on previously issued test booklets were compiled in order to help interns familiarize themselves with the structure of future exam. The training tests and the mock examination were performed during preparation period, including the period on clinical bases. The main convenience of the learning monitoring system is the possibility of distance learning and passing tests online. The first cohort of interns (70 participants) started to use on-line training in learning monitoring system on the second year of internship during extramural part of training. The second cohort included first-year interns (47 participants) who trained on-line in ILIAS from the beginning of the course within 5 months fulltime training cycle. Initial and final testing exams were performed for both cohorts of interns.

Results and Discussion. The qualification requirements, stated by the Order of the Ministry of Health of Ukraine from 14.02.2012 № 121, provide licensing integrated examination “STEP 3. Dentistry” as a part of the state attestation of interns in specialty of Dentistry (the order of the Ministry of Health of Ukraine from 20.11.2006 № 763). It consists of 200 multiple-choice tests of general dental knowledge with the accent on clinical cases and reaching a clinical decision. The integrated licensing exam gives the main conception of academic performance, mastering the learning program and the necessary level of competence to be allowed to further part of state attestation. Licensing integrated test in Dentistry is the compulsory element of the final theoretical and practical examination performed by the state examination board.

Provision of training tests was successful as it gave the opportunity to identify students who had some weaknesses in their knowledge (low-achieving examinees). Formation of the group of risk helps to work out measures of strengthening their knowledge on specific topics, which have been revealed by training test. It also helps to provide individualized approach in training curriculum for low-achieving interns. Some additional tests can be accessible for the manually added participants for their better preparation and selfconfidence during state examination.

The scheme of on-line training tests is similar to the paper-based state licensing exam (200 tasks per 200 minutes). The results of the on-line test are available immediately after finishing the test, therefore interns don't have to wait for the result of the training test. Additionally on-line testing helps to be prepared and acquainted with the structure and the timing of the actual licensing examination psychologically, avoiding the impact of stressful situation on the results of the test.

During the training test or mock examination, each participant gets a different set of questions. Questions are randomly selected from one or more question pools. Participants can work through the test only for a specified period of time. Time is clocking away from the moment a user starts a test for the first time. ILIAS displays the test results after the test pass is finished. After finishing the test pass participants can access their test results on the `Info'-tab at any time. The summary of results is appended at the point in time indicated in the setting above: A `Table of Detailed Test Results' shows the titles of the questions and the points scored in a particular test pass. The test with a random set of questions is available furthermore. The user's learning progress status is set to `Completed' after the user has passed the test.

Results of all participants are presented in the table with the detailed evaluation for each intern or question. Obtained data can be analyzed as a prognostication criterion for the forecasting an exam success.

Aggregated test results are available to the tutor of the course and consist of total number of participants who started the test, total finished tests (participants that used up all possible passes), average test processing time, total passed tests, average points of passed tests and average processing time of all passed tests.

The mean percentage of initial training of interns from the first cohort was 58.8 %. There was no significant discrepancy with the second cohort interns' initial results - 56.55 %. After series of training tests, the final mock examination was performed in both cohorts of interns. Interns, who started on-line training from the beginning of the course showed higher mean percentage results - 87.04 % than examinees from the first cohort of interns, who have got the opportunity of distance training on the second year of studying, and passed the final test with the mean percentage of 83.5 %.

According to the results of training tests and mock examination, the group of low-achieving examinees was formed in the first cohort (21 interns) and in the second cohort (13 interns). Additional training tests were performed for interns with low results under tutors' supervision. Mean results of first cohort interns' licensing exam was observed as 78.26 % which didn't differ significantly from the final mock examination results.

Comparing results of the mock on-line examination and the state licensing integrated exam, we came to the conclusion that training tests in learning monitoring system can be applied as a high level of success pass predictability instrument.

Conclusions. The multiple-choice questionnaire suffers from some disadvantages, however it still gives an appropriate apprehension to reveal the competency level provided that high-quality tests are performed and functional distractors are used.

An implication of the on-line distance learning and learning monitoring system gives an opportunity to provide the monitoring of academic performance and to evaluate the competency level during primary specialization in specialty of Dentistry. Improving the test context and providing regular on-line training tests and mock examinations are required for licensing integrated exam preparation, developing clinical reasoning, professional thinking and self-confidence during internship.

The prospect of further research. The improvement of the competency-based testing formats is a leading task for dental educational institutions, providing postgraduate training, including effectiveness assessment of the existing professional tests and development of constantly updating high-quality multiple-choice question pools with functional distractors.

List of literature

1. Assessment formats in dental medicine: An overview / S. Gerhard-Szep, A. Guntsch, P. Pospiech [et al.] // GMS Journal for Medical Education. - 2016. - No. 33 (4). doi: 10.3205/zma001064.

2. E-learning programs in oncology: a nationwide experience from 2005 to 2014 / J. Degerfalt, S. Sjostedt, P. Fransson [et al.] // BMC Research Notes. - 2017. - No. 10. - P. 39. doi: 10.1186/s13104-017-2372-8.

3. Internet-based learning in the health professions: a meta-analysis / D. A. Cook, A. J. Levinson, S. Garside [et al.] // JAMA. - 2008. - No. 300. - P. 1181-1196. doi: 10.1001/jama.300.10.1181.

4. Gerhard-Szep, S., Guntsch, A., Pospiech, P., Sohnel, A., Scheutzel, P., Wassmann, T., & Zahn, T. (2016). Assessment formats in dental medicine: An overview. GMS. Journal for Medical Education, 33 (4). doi: 10.3205/zma001064.

5. Degerfalt, J., Sjostedt, S., Fransson, P., Kjellen, E., & Werner, M.U. (2017). E-learning programs in oncology: a nationwide experience from 2005 to 2014. BMC Research Notes, 10, 39. doi: 10.1186/s13104-017-2372-8.

6. Cook, D.A., Levinson, A.J., Garside, S., Dupras, D.M., Erwin, P.J., & Montori, V.M. (2008). Internet-based learning

7. Assessment of clinical competence / V. Wass, C. van der Vleuten, J. Shatzer, R. Jones // Lancet. - 2001. - No. 357 (9260). - P. 945-949. doi: 10.1016/S0140- 6736(00)04221-5.

8. Tarrant M. An assessment of functioning and nonfunctioning distractors in multiple-choice questions: a descriptive analysis / M. Tarrant, J. Ware, A. M. Mohammed // BMC Medical Education. - 2009. - No. 9. - P. 40. doi: 10.1186/1472-6920-9-40.

9. in the health professions: a meta-analysis. JAMA, 300, 11811196. doi: 10.1001/jama.300.10.1181.

10. Wass, V., van der Vleuten, C., Shatzer, J., & Jones, R. (2001). Assessment of clinical competence. Lancet, 357 (9260), 945-949. doi: 10.1016/S0140-6736(00)04221-5.

11. Tarrant, M., Ware, J., & Mohammed, A.M. (2009). An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis. BMC Medical Education, 9, 40. doi: 10.1186/1472-6920-9-40.

Размещено на Allbest.ru