The monitoring of monitoring: what’s wrong with the ministry’s new approach to supervision of effectiveness of higher education institutions’ performance?

The analysis of the Monitoring of HEIs' effectiveness was based on the study of legislative and regulatory documents of the Ministry; the Monitoring indicators and methodology for their calculation; statistical data and decisions made on the results, and public opinion about the effectiveness and appropriateness of the procedure. A more flexible method for presenting the Monitoring results in the league table format is based on McKinsey-Abel's method and can be used as one of possible methods to enhance the performance of HEIs. The article presents the authors' expert opinion on the subject based on the personal experience in the sphere of quality assurance in education.

Monitoring as a tool of the state educational policy

In the Federal Law "On Education in the Russian Federation", the monitoring is defined as "systematic standardized supervision over the conditions of education and the dynamics of its outcomes, the conditions of implementing educational activities, student population, academic and extracurricular achievements of students and career development of graduates...". "The order., and the mandatory information to be presented is established by the Government of the Russian Federation" (RF Federal Law, 2012).

Based on the law monitoring can be defined as a specially organized procedure for information collecting and analysis with the aim of continuous standardized oversight of a process, its diagnostics and predicting its development. However, since 2012 monitoring has become something different from a tool for supervision, diagnostics and predicting. It has become an instrument for decision-making on ineffective higher educational institutions and their branches, which are to be reorganized.

Enhancing the quality of higher education provision, which in many cases does not meet the required standards, remains an urgent issue. However, tackling it by reorganisation of ineffective educational institutions is not necessarily the most appropriate way. Firstly, the concept of an effective educational institution can imply different features, and therefore needs in-depth discussion. Secondly, the focus should be on the quality of programs, but not the effectiveness of a HEI. In one and the same HEI there may be both popular up-to-date and adequately equipped programs, as well as programs which fail to meet the required standards.

Obviously, the introduction of the Law and outlined procedures was a response to the President's Decree of May 2012 and his direct order to carry out "the monitoring of higher education institutions with the purpose of assessment of their effectiveness and reorganization of ineffective state HEIs" until the end of December 2012 (Decree, 2012). Following this order, in the autumn of 2012 the Ministry of Education conducted the first "effectiveness" monitoring of the state HEIs, which caused controversy in the academic community. There were several reasons for such controversy.

First of all, the outcomes of the State Monitoring. There's nothing unusual in the monitoring procedure in itself. The governing body, in the case of the Russian Federation it's the Ministry of Education,, should have at its disposal full and credible information on the activity of subordinate organizations (Brennan, 1997), especially if there was "an order from above". The Ministry certainly collected various data before, however, in light of the new Law the monitoring procedures became highly burdensome for HEIs and, more over, turned into controlling and restructuring the HEI network.

The second reason is the shift in the direction of the state educational policy: from granting autonomy and rights to expand the higher education sector to tightening the state regulation and decreasing education expenditures (Forrat, 2012). Alongside with the Federal Law, a roadmap for education development has been worked out. According to this document, one of the ways for effectiveness enhancement of higher education is reduction of the number of HEIs: in the period from 2012 to 2017 twenty-eight HEIs have to be reduced annually (the total of 168 HEIs) (Order of the Gov., 2012).

Despite criticism these measures could be justified by the state of higher education sector by 2010. From the beginning of 1990-s until 2009, the higher education system greatly increased in scope: the number of students tripled - from 2.5 million to

7.5 million (Fig. 1); there was an increase in the number of educational programs (from 10.2 thousand to

32.5 thousand) and of the number of HEIs (without branches) - from 528 to 1,134 (Fig. 2). But already in 2010 there was a prominent tendency towards a sharp decrease in the number of students due to the demographic dive caused by the falling birth rates (10 % annually) during the perestroika period and dramatic deterioration of social economic conditions in the country (Fig. 3).

Fig. 1. Dynamics of the number of students enrolled in state and private HEIs (in thousands) (Isras.org, 2018)

Fig. 2. Dynamics of the number of HEIs (excluding branches) (Gks.ru, 2018)

Fig. 3. Dynamics of the number of births in Russia against the data of 1975,% (Gks.ru, 2018)

Moreover, the education boom at the turn of the century had a negative impact on the quality of graduates in HEIs: drastic increase in the number of students could hardly be supported by adequate material, human and financial resources. The Ministry set the task of enhancing education quality, first and foremost through the reduction of ineffective HEIs.

The concept of effectiveness and the criteria for its assessment had not been defined by any regulatory document. However, the monitoring procedure was launched in the autumn of 2012 (502 state educational institutions and 930 branches); and since 2013 it has been conducted annually as on April 1. All state HEIs (since 2013 all private HEIs as well) have to fill in specially developed forms with information on their activity.

The obtained information is systematized and compared against the threshold values. The results are reviewed by a special committee comprised of members from ministries and bodies representing higher education. The committee makes recommendations for the founders of HEIs and their branches which are found to be ineffective. The founder makes a decision either to close the HEI or the brunch (with the students being transferred to other HEIs) or to merge it with another HEI which is found effective.

Thus, the goal of enhancing the quality of education is considered to be achieved. However, until now there have not been any verified data that the administrative measures reducing the number of institutions have been really effective in achieving this goal.

The Monitoring indicators as a vector of the state educational policy

There are 154 criteria for collecting information, which are divided into seven groups (Indicators. miccedu.ru, 2018). The monitoring indicators, in fact, characterize the direction of the state educational policy and determine the developmental trends of Russia's higher school.

These indicators include the following:

Educational performance: the quality of admitted applicants. This indicator is calculated by the grade point average of the Unified State Exam, which is a high school final and university entrance exam taken upon completion of the 11th grade of the secondary comprehensive school. Only full-time students enrolled on the basis of general competition are taken into account. This aspect is indicative of the HEI prestige, the demand for the programs it delivers. But these are not the direct results of the institution's performance. Obviously, better prepared applicants make a prerequisite for a higher level of graduates; however, the HEI does not contribute to the training of applicants.

Research activity: the estimated indicator for allocating funds for research from all possible sources per one HEI teacher involved in teaching and research. This indicator can be regarded as a general performance indicator, but not as an indicator of educational activity as such. It is supposed to characterize the level of teachers, as a modern teacher should carry out research. However, it is impossible to estimate how much this or that researcher is involved in teaching.

International cooperation of a HEI is characterized by the number of admitted international students. It is calculated against the total number of students enrolled in a HEI or a branch. It is not clear why the share of international students is for the most part (99 %) comprised of compatriots from the former Soviet Union and developing countries from Asia and Africa. And these countries admittedly have a lower level of education in comparison with Russia. How can this indicator possibly determine the effectiveness or performance of a HEI activity? This might be a contribution to the solution of certain geopolitical issues, but they are beyond the scope of the statutory activities of HEIs. This indicator could be interpreted in different ways, and there is no reason to believe that the quality of education is directly related to the number of international students enrolled in a HEI.

Financial and economic activity is an indicator of the income received by a HEI from all possible sources: educational activity (fee-paying students), research, grants, sponsorship, publication activity etc. This indicator characterizes the ability of a HEI to draw budget and out of budget means. It can characterize the effectiveness of the institution, but not its educational activity. This indicator is more indicative of entrepreneurship activity.

Infrastructure of a HEI: the estimated floor space of academic buildings per one student. This indicator can be interpreted in different ways: on the one hand, the less the better, as the floor space should be used effectively. But how it is going to impact the educational and research activity is hard to tell. On the other hand, adequate laboratory facilities testify to a HEI's ability to offer quality educational services. But the issue whether there is a direct correlation between the size of laboratories and the quality of education needs to be further explored. Besides, many HEIs borrow material resources for their educational process. These are, first of all, medical and arts and culture HEIs. Since 2015, this indicator has been excluded from the monitoring list.

The average pay of the teaching staff as related to the average pay in the real economy of the region, where a HEI is located (this indicator was introduced in 2015). The average pay in Russia's regions is calculated by the Federal State Statistics Service. The correlation of the teachers' salary and the average pay in the region does not have any relevance either to educational or research activity. This is a vector of the state social policy, but unfortunately, it is not underscored by the budgetary funding. The indicator is determined by the administration of a HEI.

Employability of graduates: this indicator is calculated on the basis of the tax received by the Pension Fund of the Russian Federation from the graduates employed in the previous year. The information is provided by the Pension Fund of Russia. The employability of graduates should be one of the most important. But in fact, it is not because besides representing the average percentage of graduates' employability in a given HEI, it does not draw a complete and clear picture of the quality of graduates and their achievements.

Additional indicatorsis applicable to specific HEIs and their branches offering programs in arts, sports, military training, transportation, medicine, agriculture. For every group of HEIs, there are used specific indicators, which, as a rule, characterize the quality of the teaching staff.

Practically each of the indicators can be subjected to criticism by HEIs, and for a good reason. The competition based on the results of the Unified State Exams characterizes the quality of admission rather than a HEI's performance. It is also influenced by the location of a HEI and the labor market situation. Neither is it correct to evaluate research based on the allocated funds because technology and sciences research is much more substantially funded than research into humanities or economics. Especially problematic is this indicator for teacher training and culture and arts institutions.

The requirement to increase the admission of foreign students has drawn the strongest criticism: for regional HEIs and for the HEIs training specialists for the local regional labor market this requirement is perplexing, to say the least. Besides, the solution of this task makes HEIs minimize the admission requirements for foreign applicants; which means trying to meet the indicator regardless of the level of applicants and their readiness for education and training. This will decrease the quality of higher education still further.

The indicators of economic activity and teachers' pay rate reveals obvious imbalance when calculating the funding of HEIs, and the calculation of tax revenue characterizing employability of graduates iden- tifieds gaps in the calculation of tax returns.

Thus, the values of the majority of indicators do not depend on a HEI's performance, but are contingent on external factors; whereby these external factors operate in the conditions of different HEIs in different ways, which, under the current methodology, puts HEIs in unequal conditions. The selection of data should not be based on their accessibility, but on the indicators that reflect the quality and strong points of an institution. It should be made clear why this or that indicator is selected and what it shows. The suggested indicators characterize, in the first place, a HEI's potential, but not quality, because it is much more difficult to assess quality than resources (Marshall, 2016). This is also true regarding the accessibility of data.

To sum up, the weaknesses of the present Monitoring are: firstly, the indicators do not directly refer to the quality of education provision or the effectiveness of educational activities (only indirectly). Secondly, statistical information submitted by HEIs is not verified as is done during the accreditation procedure where there is always a site visit by experts. There have been instances of mistakes and misrepresentation of information when filling in statistical report forms. And finally, practically every year the "rules of the game" change: criteria, indicators and calculation methodology are reviewed and altered. These rules are not supported by any regulatory documents and are not communicated to HEIs, or are communicated very late (less than a month before the deadline of submitting information).

The dynamics of the Monitoring outcomes

It should be noted that the methodology of indicator calculation was based on the model of state accreditation, which was effectively used in 1999-2009 when determining the status of a HEI (Motova & Pykko, 2012). The criteria of state accreditation were determined by the value of the lower quartile in the sampling of HEI of one kind (institute, academy, university) and, as a rule, were set at a threshold value for the period of 5 years.

As previously mentioned, the decision was made on the basis of all indicators taken together with the consideration of the compensation mechanism.

But unlike the state accreditation, in the methodology of calculation of the Monitoring the threshold value is set as the median value of the previous year. The decision is made based on the threshold value, which divides all HEIs into effective and ineffective regardless of their kind (university, academy, institute). The profile of each HEI is taken into consideration (arts and culture, sports, transportation, medicine, agriculture etc.).

The threshold value is calculated for each indicator separately. A HEI's effectiveness is determined by the number of its indicators which are equal to the threshold values or are above them. A HEI is considered effective if out of seven indicators four or more are above the threshold values.

Simple calculation shows that if we apply this model, the probability of a HEI's being considered effective is 0.5.

It means that every other HEI will have values below threshold in every indicator. Besides, the threshold values will be different every year, as the indicator values of the institutions do not remain the same.

The statistics of HEIs' participation in the Monitoring procedures of 2012-2018 (Table 1) shows that since 2014 there has been a decrease in the number of HEIs submitting statistical information.

It is a direct consequence of introducing the procedure of monitoring, as based on its results the Ministry initiated the procedures of optimizing the higher school system. On the basis of the obtained data and the calculation of effectiveness, the founder is recommended that a decision concerning the future of its HEI or its branch should be made: either to close or reorganize it, i. e., in fact, to merge it with a more successful HEI.

The required number of indicators is distributed unevenly by year. 2014 was the year when the largest number of HEIs was recognized ineffective: 1,010 (43.46 %), including 238 HEIs (79 state and 159 private) and 772 branches (489 state and 283 private).

1,289 institutions of higher education (712 HEIs and 577 branches) took part in the monitoring procedures of 2017 (Table 2). 121 institutions (9.4 %) did not meet the necessary requirements.

In the last five years, as a result of the demographic crisis and the state educational policy, the national higher education experienced a dramatic decrease in the scope, and, as predicted in the officially published sources, this tendency is likely to persist. The number of students has decreased from

7.5 million to 5 million. The number of HEIs and their branches has decreased by 1,000.

This hard-hitting system of decision-making will undoubtedly turn the Monitoring procedure into a "high-stakes game", which is bound to raise a wave of indignation and criticism. On the other hand, these methods of education quality enhancement do not significantly reflect on the quality.

The reorganisation of HEIs by merger adds to the numerous problems connected with the redistribution of governance functions, merger of departments and faculties, reduction of the number of the teaching staff and increase of their workload. Besides, there is no guarantee that after the acquisition of a weaker institution the stronger HEI will retain its leading posi- tion. There are also doubts whether in this situation research schools will be preserved. Among HEIs, which lost their autonomy there are HEIs with long-standing traditions and achievements in research and education of highly-qualified graduates. The disappearance of these HEIs from the map of national higher education will cause confusion not only in Russian, but also in the international community.

Table 1