Tracking the dynamics of lecturer rating indicators to ensure the quality of higher education

Размещено на http://www.allbest.ru/

Tracking the dynamics of lecturer rating indicators to ensure the quality of higher education

Junior Researcher of Digitization of Education Research Lab Borys Grinchenko Kyiv University, Kyiv, Ukraine a.

Анотація

ВІДСЛІДКОВУВАННЯ ДИНАМІКИ РЕЙТИНГОВИХ ПОКАЗНИКІВ ВИКЛАДАЧА ДЛЯ ЗАБЕЗПЕЧЕННЯ ЯКОСТІ ВИЩОЇ ОСВІТИ

Тютюнник Анастасія Володимирівна

молодший науковий співробітник НДЛ цифровізації освіти

Київський університет імені Бориса Грінченка, Київ, Україна

Не можна заперечувати факт важливості ролі викладача у забезпеченні якості вищої освіти. Система внутрішнього забезпечення якості передбачає наявність розроблених і оприлюднених критеріїв, правил і процедур оцінювання науково-педагогічних, наукових працівників, а також щорічне їх оцінювання відповідно до європейських стандартів, Закону України "Про вищу освіту", рекомендацій Національного агентства із забезпечення якості вищої освіти. В Київському університеті імені Бориса Грінченка для формування якісного складу науково-педагогічного складу працівників була розроблена система рейтингів викладачів "Е-портфоліо", яка є однією з компонентів інформаційно освітнього середовища. Для аналізу діяльності науково-педагогічного складу та всіх підрозділів Університету Грінченка в системі "Е-портфоліо" було представлено декілька типів звітів, які можна побудувати за такими критеріями як основні види діяльності, рейтингові бали, структурні підрозділи, посади, вчене звання, ступінь тощо. Протягом декількох років використовувалсь лише вбудована статистики для аналізу діяльності викладачів, підрозділу, але з'явилась потреба у моніторингу більш детальної динаміки усіх показників. Було проаналізовано сучасні інструменти бізнес-аналітики для розробки візуалізації та вирішено використовувати інструмент для бізнес аналітики Power BI, який дасть змогу мати повне уявлення про результати діяльності викладача за усіма потрібними критеріями. В статті представлено аналітичний звіт Power BI в основі якого покладена модель "Схема Зірка". Завдяки створеним «Мірам» та стандартним функціям зведених таблиць стала можлива реалізація візуалізація вхідних даних рейтингу, створення деталізації та фільтрів для більш зручного перегляду та аналізу даних. Звіт після завершення всіх налаштувань візуалізації та фільтрів було опубліковано за допомогою інструменту серверу звітів Power BI у вільному доступі. Використання даного інструменту дає змогу топ-менеджерам університету мати цілісну картину результатів у діяльності працівників та підрозділів, а також визначати пріоритети діяльності, приймати справедливі рішення при продовженні контрактів, що цілком і безумовно сприяє забезпеченню якості вищої освіти.

Ключові слова: якість вищої освіти; рейтингові показники; е-портфоліо; моніторинг рейтингових показників; візуалізація; Power BI

Introduction

The main task of a modern higher education institution is to provide quality educational services and ensure their compliance with national, European and international standards. The role of the lecturer is one of the crucial in ensuring the quality of the educational process. According to the Law of Ukraine "On Higher Education" (Verkhovna Rada of Ukraine. Law №1556-VII, 2014), the system of ensuring the quality of educational activities and the quality of higher education (internal quality assurance system) provides for the availability of developed and published criteria, rules and procedures for the evaluation of pedagogical, scientific and pedagogical, scientific workers, as well as their annual evaluation. The National Agency for Quality Assurance in Higher Education (NAQA) in its Recommendations for the Implementation of the Internal Quality Assurance System (National Agency for Higher Education Quality Assurance, 2019) focuses on the professional development of lecturers and leadership as components of the internal quality assurance system.

Analysis of the recent research and publications

lecturer rating quality education

A number of studies by both foreign and domestic scientists are devoted to the study of higher education quality. The International Network for Quality Assurance Agencies in Higher Education (INQAAHE) in its reports (The International Network for Quality Assurance Agencies in Higher Education, 2022) offers best practices to improve both external quality assurance and internal quality of the internal system. R. Heintze (Heintze, 2017), S. Yaved (Javed, 2017) are engaged in the research of external and internal quality of higher education; measuring the quality of higher education services - K. Fawad Latif, I. Latif, U. Farooq Sahidzada, M. Ullah (Latif, Sahibzada, Ullah, 2019); research of the effectiveness of rating assessment N. Pauflera, E. Sloat (Pauflera, Sloat, 2020), V. Ogneviuk (Ogneviuk, 2016), V. Bykov, O. Spirin (Bykov, Spirin, & Pinchuk, 2020), (Bykov et al., 2020), A. Hurzhii, V. Lapinskyi (Lapinskyi, Hurzhii, Kartashova, 2018), D. Karamyshev (Karamyshev, 2020), focus on the problems and prospects of assessing the quality of Ukrainian education. Tools for measuring the lecturer's activity are presented in the works of N. Morze (Morze, Buinytska, 2017), (Morze, Buinytska, Kocharian, 2015), (Morze, Buinytska, 2019), O. Rayevnyeva (Rayevnyeva, Stepurina, 2017), L.Varchenko-Trotsenko (Morze, Varchenko- Trotsenko, 2016) and others.

However, little attention is paid to the organization and analysis of tracking the dynamics of rating indicators of both lecturers and units.

The purpose of the article

With the help of modern business intelligence tools to develop visualization of rating indicators of lecturers and departments for making successful management decisions that will contribute to the quality of higher education.

Presentation of the main material

In order to understand the holistic picture of the qualitative composition of the academic staff of the University, the implementation of the Law of Ukraine on Higher Education, in which the quality and openness of the results of the higher education institution is one of the priorities, Borys Grinchenko Kyiv University has developed a system of internal ratings of lecturers - the E-portfolio system (https://eportfolio.kubg.edu.ua/) (“E-portfolio”, 2022). The system reflects the activities of scientific and pedagogical staff, which affects the quality assurance indicators of the university's educational activities in accordance with European standards. The E-portfolio accumulates all the professional activities of lecturers according to certain quantitative and qualitative indicators for assessing the main activities of each lecturer and all university departments in order to objectively analyze the quality of staffing and quality assurance of higher education (Buinytska, Tiutiunnyk, 2022). The main objectives of the introduction of rating evaluation are the development of leadership potential of scientific and pedagogical and scientific workers, motivation for effective and efficient activities that promote innovation and improve the quality of educational activities; the formation of high-quality teaching staff of the university (“Regulations on the annual rating assessment of the professional activity of scientific- pedagogical and scientific employees of Borys Grinchenko Kyiv University "Leader of the year"”, 2020).

The rating system provides for the coverage of all aspects of lecturers' activities by indicators; the possibility of their addition, change. Performance indicators were developed based on internal quality standards (according to UNESCO), indicators of international university rankings, corporate standards of the university (scientific standard and standard of ICT (digital) competence). Every year, in accordance with the needs of the university, the indicators for the next year are modified and approved by the Academic Council after a joint discussion by the staff of the university's structural units.

The main requirements for the "E-portfolio" system are: establishing the rating of lecturers and departments; the ability to create, fill, update and improve the database and use data from the components of the created information and educational environment of the university: institutional repository, database of registers of the University, e-learning system, data from open profiles in Google Academy; consideration of the results of the rating of lecturers, departments, research laboratories, faculties, institutes and tracking their dynamics (Morze, Buinytska, 2017, p. 39).

In order to reflect reliable information in the lecturer's portfolio, all information is entered into the system either from the institutional repository https://elibrary.kubg.edu.ua/ or from the registers of the University's activity base https://rg.kubg.edu.ua/. Responsibility for entering information into these systems rests with certain employees, determined by a separate order.

When forming rating lists, it is possible to form ranking by structural units, departments, positions, academic titles, academic degrees. It is also implemented the observance by lecturers of certain, so-called mandatory, conditions, in case of non-fulfillment of which the lecturer cannot be a leader in the university.

The lecturer's portfolio page provides basic information about the lecturer - education, position, department, public activities, etc. Further, according to the types of activities and defined evaluation indicators, three main blocks are formed - research activities, professional development and teaching activities.

For the sake of clarity, openness and transparency of the calculation of points for certain activities, it is enough to review the details of the rating, which reflects the calculation of points for each of the rules defined in the current year.

In order to analyze the activities of lecturers, structural units and the university as a whole, the main types of statistical reports were implemented in the E-portfolio system:

- on the rating scores of lecturers, which are determined in terms of positions, academic degrees, departments, etc;

- by structural units;

- by rating indicators of the main activities;

- for each of the weight indicators by which the ratings are calculated;

- by average indicators.

The first few years of the introduction of the E-portfolio rating system and the use of built-in statistics was enough for us to analyze the activities of the lecturer, structural unit. But there was a need to track the dynamics of indicators.

For a more detailed comparative analysis of the activities of lecturers, structural units of Borys Grinchenko Kyiv University in the period 2017-2021 and the presentation of a visualized detailed report, the Power BI tool was chosen.

Power BI is a new direction of business intelligence development, which is a set of cloud- enabled business intelligence services for data analysis and visualization. The main advantage of this tool is the ability to build interactive dashboards with key performance indicators that are available for viewing from any device connected to the Internet. In addition, Power BI allows you to quickly work with real-time analytics, perform professional visualization, connect different data sources into a single model, get the necessary details and data samples (“What is Power BI?”, 2022). In order to create an interactive analytical report and visualize this data, the pre-processed results (in the form of summary tables) (Fig. 1) of professional activity, separately for each specific rule of scientific and pedagogical and scientific workers of Borys Grinchenko Kyiv University, collected in the E-portfolio system during 2017-2021, were loaded into Power BI. After loading the data into Power BI, it was necessary to process them according to our needs.

Fig. 1. The "Data" working window in Power BI

In the working window "Model" were formed the necessary relationships between the tables in the input data.

The optimal way to create the report "Rating of structural subdivisions and lecturers of Borys Grinchenko Kyiv University 2017-2021" was to use the data model in the form of "Star Scheme". This way of organizing data is based on the logical division of them into two types, for the storage of which dimension tables (a set of descriptive attributes characterizing the object) and fact tables (containing information about the events in which certain objects are involved) are used.

At the center of this model is the fact table ("direc"), around which the measurement tables ("stats2021", "stats2020", "stats2019", "stats2018" and "stats2017") are placed. Table "direc" contains data on lecturers: Full name, position, degree, rank, and place of work with the department and structural unit. Tables "stats2021", "stats2020", "stats2019", "stats2018" and "stats2017" contain summary data for all the defined rules of different years (2021-2017) of the "E-portfolio" system in accordance with the indicators of the annual rating assessment of the professional activity of scientific and pedagogical and scientific workers of Borys Grinchenko Kyiv University "Leader of the Year" (Fig. 2).

Fig. 2. The working window "Model" of the Power BI application

To process these data, the corresponding "Measures" were created using the syntax of the DAX data analysis expression language in accordance with the indicators of the annual rating assessment of scientific and pedagogical and scientific workers of Borys Grinchenko Kyiv University during 2017-2021 (Fig. 3).

Fig. 3. An example of calculating the formula for the average value for the indicator "Professional development" in 2021

Using the created "Measure" and standard functions of pivot tables in Power BI, visualizations of the input data of the rating of structural units and lecturers were implemented, details and filters were created for more convenient viewing and analysis of data. After completing all the settings of visualization and filters, the report was published using the Power BI report server.

All calculations of rating indicators were reduced to calculating the scores of employees and, accordingly, their ranking. The total score of the lecturers of the unit should be the rating score of the unit, but given that different structural units have different numbers of lecturers, this would be a false statement, so the ranking of the units was carried out on the basis of the average score of the lecturers of the unit. Thus, Fig. 4 shows the dynamics of the average score of the lecturers of the unit in the period 2017-2021.

Fig. 4. View of possible visualizations and filters

The report "Rating of structural units and lecturers of Borys Grinchenko Kyiv University 2017-2021" contains the following visualization (Fig. 5):

Fig. 5. Displaying details and filters in the published report

Dynamics of the average score of the university lecturer:

details by structural units with the maximum score; o filters - institute, department; academic degree, academic title, position;

contribution of each type of activity to the rating:

detail by subdivisions;

filter by the activities of the departments; displaying the activities of each department by certain performance indicators;

contribution of each lecturer to professional development;

contribution of each lecturer to the teaching activity;

the contribution of each lecturer to research activities;

dynamics of each lecturer's activity:

over the years; o by type of activity;

by contribution to research activities and its main indicators; o professional development in the context of the rules; o by all indicators of teaching activity.

Each visualization contains additional information that can be used to detail the data display and highlight the necessary components or elements using filters. Tools for in-depth detailing of the results are available for viewing (Fig. 5):

- 14 - return to the original summary charts;

- enable the detail mode and work in it (to disable any mode );

- U - go to the next level of detail;

- Л . use an even lower level of detail;

- V - select a filter;

- Е - focusing on the selected chart (increases to full screen).

In total, more than 1000 reports can be viewed on one dashboard.

Each page of the report contains details and filters (Fig. 5), with the help of which it is possible to analyze in detail the dynamics of the rating indicators of lecturers and university departments during 2017-2021.

Using the first level of detail, we get the dynamics of the average scores of lecturers of each structural unit and can compare them with the maximum score of the lecturer of the unit in 2021. The dynamics of the performance indicators of the Education Research Lab for 20172021 of the selected institute is formed using the filter of the institute and the next level of detail, reflecting the maximum score of lecturers of these departments (Fig. 6).

Going further into detail, we trace the dynamics of the rating scores of all university staff of the selected Education Research Lab and are able to carry out a comparative analysis of the activities of each of the lecturers during the specified period (Fig. 7). The formed visualization clearly demonstrates the improvement of the performance of the university stsff of the Digitization of Education Research Lab in 2021.

Fig. 6. An example of the published page of the report "General rating of structural subdivisions" using the filter "Institute, Faculty" (Education Research Lab)

Fig. 7. An example of a published report page with the use of filters "Institute, Faculty" (Education Research Lab) and "Department" (Digitization of Education Research Lab)

Figures 5-7 show examples ofjust a few fragments of the reports of one dashboard, which presents all possible variations of visualizations for analysis by average ratings of lecturers and units in general.

Since there is a need to track the dynamics of all the main activities, namely research, professional development and teaching activities, let's move on to the next created dashboard, which is based on the contribution of all activities to the overall rating.

By choosing the first level of detail, we observe the contribution of each structural unit to the relevant activity, which allows us to analyze the changes in the performance of the units based on the visualization obtained. Using the next level of detail, we get the contribution of the activities of each department to the overall rating. And even deeper detailing allows us to analyze the contribution of each lecturer in accordance with the defined activities (Fig. 8).

Fig. 8. Example of a published report page using different levels of detail

For a more thorough analysis of the activities of the departments of a particular structural unit, it is necessary to use the appropriate filter and obtain information on the dynamics of certain indicators, which will allow to identify weaknesses and strengths in the research activities of the department, professional development or teaching activities.

Each dashboard has the ability to analyze the contribution of lecturers in general and in the context of the positions they hold. In addition, we can form instant samples of performance indicators for all or individual lecturers with academic titles, academic degrees for the university or for a separate institute/faculty, or for individual departments (Fig. 9). This tool makes it possible to compare the activities of the departments not only within the same unit, but also different units. We can also track the dynamics of any university staff by typing only his surname.

Fig. 9. Example of the "Professional development" dashboard

Any visualization on the report page can be not only expanded and detailed for a more detailed review, but also presented in the form of a table (Figure 10).

Fig. 10. An example of detailing with the display of table data

The report traces in detail the dynamics of the performance of all lecturers in general and separately each. The dashboard (Fig.11) shows the dynamics of the lecturer's rating points for the period 2017-2021 both in general and by type of activity. This visualization allows you to analyze which activities the lecturer prefers.

Fig. 11. An example of displaying the dynamics of lecturer's activity in general and by type of activity

Going deeper into the analysis of research activities, we can see what indicators increased the lecturer's scores, what he focused on and what indicators still need to be worked on. To compare the rating indicators of the employees of the unit, you need to select the necessary unit in the filter and get a visualization of the activities of the employees of the unit by year.

We can track the dynamics of activity, compare indicators both within the unit and within the institute, faculty or university. Obtaining such instant data allows you to clearly track the activities of the lecturer in different years, to see the priorities of his activities, to identify weaknesses; to make balanced and fair decisions when extending contracts, holding competitions, etc.

The article presents only a small part of the reports, since the capabilities of the Power BI tool are quite extensive and indispensable in obtaining instant end-to-end analytics and reporting. A significant advantage when viewing reports is the distribution of access levels to the report - each university employee has the opportunity to view the report according to his position.

The published report "Rating of structural subdivisions and lecturers of Borys Grinchenko Kyiv University 2017-2021" (“Rating of university departments and lecturers of Borys Grinchenko Kyiv University 2017-2021”, 2022) is available at the link: https://eportfolio.kubg.edu.ua/rating/visualization

Conclusions and prospects for further research

In order to track the dynamics of the rating indicators of scientific and pedagogical, scientific workers of Grinchenko University, we have chosen a modern business intelligence tool Power BI. The use of the Power BI tool allows top managers of the university to have a holistic picture of the performance of employees and departments, and also eliminates the need to compare reports in order to track the dynamics of performance indicators.

The main advantage of using Power BI is the ability to build interactive dashboards using all possible filters and details with the display of key performance indicators, on the basis of which various visualizations for analysis are formed. The generated reports are available for viewing from any device connected to the Internet. A significant advantage when viewing reports is the distribution of access levels to the report - each employee of the university has the opportunity to view the report according to his position.

Creating reports using Power BI allows you to carry out detailed, in-depth analyses of the activities of lecturers and university departments, compare rating indicators for each of the defined rules both for the lecturer and for the university as a whole. Directors, deans, heads of departments can clearly track the activities of the lecturer in different years, see the priorities of his activities, identify weaknesses; make balanced and fair decisions when extending contracts, holding competitions, etc.

REFERENCES

Verkhovna Rada of Ukraine. Session 2. (2014, July. 1). Law №1556-VII, On Higher Education. October 11, 2022.

https://zakon.rada.gov.ua/laws/show/1556-18 (in Ukrainian)

Recommendations of National Agency for Higher Education Quality Assurance regarding the introduction of an internal quality assurance system, National Agency for Higher Education Quality Assurance (2019). October 15, 2022.

https://naqa.gov.ua/2019/07/рекомендації-національного-агентств/ (in Ukrainian)

Annual Reports. The International Network for Quality Assurance Agencies in Higher Education (2022). October 15, 2022. https://www.inqaahe.org/annual-reports

Heintze, R. (2017). Workshop 1. Internal & External Quality assurance for QA agencies, The International Network for Quality Assurance Agencies in Higher Education, Manama. October 15, 2022.

https://www.inqaahe.org/sites/default/files/IC2017 WS1 RH.pdf

Javed, S. (2017). Aligning the quality assurance of higher education with the education for sustainability requirements - a solution to future world challenges, The International Network for Quality Assurance Agencies in Higher Education, Manama. October 15, 2022. https://www.inqaahe.org/sites/default/files/IC2017 S4P4 0.pdf

Latif,_K., Latif, I., Sahibzada, U. & Ullah, M. (2019). In search of quality: measuring Higher Education Service Quality (HiEduQual), Total Quality Management & Business Excellence, vol. 30, no. 7-8, pp. 768-79. October 15, 2022. https://www.tandfonline.com/doi/full/10.1080/14783363.2017.1338133.

Pauflera, N. & Sloat, E. (2020). Using standards to evaluate accountability policy in context: School administrator and teacher perceptions of a teacher evaluation system, Studies in Educational Evaluation, vol. 64. October 15, 2022.

https://www.sciencedirect.com/science/article/pii/S0191491X18303006#bib0160

Ogneviuk, V. (2016). Education and competitiveness of society, OSWIATOLOGIA, Vol. 5, pp. 37-44. October 15, 2022.

http://osvitologia.kubg.edu.ua/en/numbers/27-archive-2016/51-5-

2016.html?showall=&start=6 (in Ukrainian)

Bykov, V., Spirin, O. & Pinchuk, O. (2020). Modern tasks of digital transformation of education, UNESCO Chair Journal "Lifelong Professional Education in the XXI Century", no.1, pp. 27-36. October 15, 2022. https://www.unesco-

iournal.com.ua/index.php/iournal/article/view/7 (in Ukrainian)

Bykov, V. , Spirin, O., Biloshchytskyi,_A., Kuchansky,_A., Dikhtiarenko, O. &

Novytskyi, O. (2020). Open digital systems for assessment of pedagogical research results, Information Technologies and Learning Tools, vol. 75, no.1, pp. 294-315. October 15, 2022.

https://iournal.iitta.gov.ua/index.php/itlt/article/view/3589 (in Ukrainian)

Lapinskyi, V., Hurzhii, A. & Kartashova, L. (2018). Ways and means of ensuring quality education for everyone, Materials of the II All-Ukrainian scientific and practical Internet conference "Resource-oriented learning in "3D": accessibility, dialogue, dynamics", Poltava, p. 11-16. October 15, 2022.

http://acup.poltava.ua/wp-content/uploads/2017/12/Збірник-2018-26.02-3.pdf (in Ukrainian) Karamyshev, D. (2020). Theoretical and methodological principles of ensuring the quality of education", Kharkiv, Ukraine: Publishing house KharRI NADU «Mahistr» (in Ukrainian)

Morze, N. & Buinytska, O. (2017). Rating indicators system of evaluation teacher's activities in modern universities, Scientific journal of National Pedagogical Dragomanov University: Kompiuterno-oriientovani systemy navchannia, Series 2, no. 19, pp. 34-44. (in Ukrainian)

Morze, N. & Varchenko-Trotsenko, L. (2016). Educator's e-Portfolio in the Modern University, ICTERI, Kyiv, p. 231-240. October 15, 2022. http://ceur-ws.org/Vol-1614/paper 68.pdf

Morze, N., Buinytska, O. & Kocharian, A. (2015). IC-competence of teachers and students as a way to form an information educational environment of the university, Competence-oriented education: qualitative measurements, V. Ogneviuk, L. Khoruzha, S. Sysoieva, N. Chernukha, N. Terentieva, Eds. Kyiv, Ukraine: Borys Grinchenko Kyiv University, pp. 151-196. October 15, 2022. http://elibrary.kubg.edu.ua/id/eprint/10019/ (in Ukrainian)

Morze, N. & Buinytska, O. (2019). Digital Competencies of University Teachers, Universities in the Networked Society. Cultural Diversity and Digital Competences in Learning Communities, vol. 10, pp. 19-37. October 15, 2022. https://link.springer.com/chapter/10.1007/978-3-030-05026-9 2

Morze, N. & Buinytska, O. (2017). Raising information and communication technologies competence of scientific and pedagogical employees - a key requirement of the quality of educational process, Information Technologies and Learning Tools, vol. 59, no.3, pp. 189-200. October 15, 2022.

https://journal.iitta.gov.ua/index.php/itlt/article/view/1667 (in Ukrainian)

Rayevnyeva, O. & Stepurina, S. (2017). Alternative rating system of scientific activity of ukrainian higher educational institutions: methodical bases and results, ScienceRise: Pedagogical Education, vol. 16, no.8, pp. 48-57. October 15, 2022. http://iournals.uran.ua/sr edu/article/view/109316 (in Ukrainian)

Buinytska, O. & Tiutiunnyk, A. (2022). Monitoring of the teacher's rating indicators for making management decisions using ICT. Information Technologies and Learning Tools, 87(1), 336-356. October 17, 2022. https://doi.org/10.33407/itlt.v87i 1.4051

E-portfolio (2022). October 21, 2022. https://eportfolio.kubg.edu.ua/ (in Ukrainian)

Regulations on the annual rating assessment of the professional activity of scientific- pedagogical and scientific employees of Borys Grinchenko Kyiv University "Leader of the year" (2020). October 23, 2022.

https://kubg.edu.ua/images/stories/Departaments/vdd/documenty/rozdil 7/nakaz 199 19.03.2

020 .pdf (in Ukrainian)

What is Power BI? (2022). October 23, 2022. https://powerbi.microsoft.com/en-us/what-is-power-bi/

Rating of university departments and lecturers of Borys Grinchenko Kyiv University 2017-2021 (2022). October 21, 2022.

https://eportfolio.kubg.edu.ua/rating/visualization (in Ukrainian)

Размещено на Allbest.ru