Automated system of fuzzy identification of expert’s competence for assessing the quality of pedagogical phenomena and processes

Размещено на http://www.allbest.ru/

Automated system of fuzzy identification of expert's competence for assessing the quality of pedagogical phenomena and processes

Hryhorii V. Tereshchuk

Doctor of Pedagogical Sciences, Professor, Vice-Rector

Ternopil V. Hnatiuk National Pedagogical University, Ternopil, Ukraine

Ivan M. Tsidylo

Doctor of Pedagogical Sciences, Associate Professor at the Department of Computer Technologies Ternopil V. Hnatiuk National Pedagogical University, Ternopil, Ukraine

Abstract. The article describes the problem of the application of the theory of fuzzy sets to describe the indicators of expert assessment by using linguistic variables instead of numerals or in addition to them. An automated fuzzy system of Sugeno has been developed for determining the competence of an expert. Its input variables are six factors: work experience, academic degree, if any, publications, justification of position as to the discussed problem, knowledge of the achievements in the relevant sphere, knowledge of the essence and objects of work of specialists of a certain area. Simple relations between variables are described using fuzzy statements, complex relationships are described by fuzzy algorithms. In the Simulink environment a fuzzy controller was built on the basis of Sugeno FIS-system, optimized using the anfis function. Testing the controller on a test selection proves the functional suitability of the developed model.

Keywords: fuzzy set theories; linguistic variable; competence of an expert; competence factors; Sugeno FIS-system; anfis function; pedagogical phenomena and processes.

Автоматизована система нечіткого визначення компетентності експерта для оцінювання якості педагогічних явищ і процесів

Терещук Г., Цідило І.

Анотація. У статті викладено проблему застосування теорії нечітких множин для опису показників оцінки компетентності експерта з використанням лінгвістичних змінних замість числових або в доповнення до них. Розроблено автоматизовану нечітку систему Суґено з визначення компетентності експерта вхідними змінними якої є шість чинників: стаж роботи, наявність вченого ступеня, наявність опублікованих робіт, обґрунтування думки з обговорюваної проблеми, знання досягнень відповідної галузі, знання змісту та об' єктів праці фахівців певної спеціальності. Прості відношення між змінними описуються за допомогою нечітких висловлювань, складні відношення описуються нечіткими алгоритмами. У середовищі Simulink побудовано нечіткий контролер на базі оптимізованої за допомогою функції anfis FIS-системи Суґено. Тестування контролера на тестовій вибірці доводить функціональну придатність розробленої моделі.

Ключові слова: теорії нечітких множин; лінгвістична змінна; компетентність експерта; чинники компетентності; FIS-система Суґено; функція anfis; педагогічні явища і процеси.

Автоматизированная система нечеткого определения компетентности экспертов для оценивания качества педагогических явлений и процессов

Терещук Г., Цидыло И.

Аннотация. В статье изложено проблему применения теории нечетких множеств для описания показателей оценки компетентности эксперта с использованием лингвистических переменных вместо числовых или в дополнение к ним. Разработана автоматизированная нечеткая система Суґено по определению компетентности эксперта, входными переменными которой есть шесть факторов: стаж работы, наличие ученой степени, наличие опубликованных работ, обоснование мнения по обсуждаемой проблеме, знание достижений соответствующей отрасли, знание содержания и объектов труда специалистов определенной специальности. Простые отношения между переменными описываются с помощью нечетких высказываний, сложные отношения описываются нечеткими алгоритмами. В среде Simulink построено нечеткий контроллер на базе оптимизированной с помощью функции anfis FIS-системы Суґено. Тестирование контроллера на тестовой выборке доказывает функциональную пригодность разработанной модели.

Ключевые слова: теории нечетких множеств; лингвистическая переменная; компетентность эксперта; факторы компетентности; FIS-система Суґено; функция anfis; педагогические явления и процессы.

This work is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Introduction. The problem setting. The intensity of the development of science in general leads to an increase in interest in the methodology and methods of pedagogical research, in which one has to deal with complex experiments in order to study various factors, which are not subjects to strict accounting and control. For the processing and interpretation of such experiments, methods of mathematical statistics are usually used. However, in pedagogical experiments, quite often, there is a need for quantitative and (or) ordinal evaluation of processes or phenomena, which are not directly measurable. In this case, researchers apply an expert assessment which is based on judgments of experts.

The use of expert assessments in pedagogical research requires the establishment of a general algorithm (procedure) for conducting an expertise, as well as the methodology for the formation of expert judgment. In addition, it is necessary to take into account one of the decisive factors of the expert procedure - the competence of an expert, which is determined on the basis of criteria for the selection of qualified experts, as well as consider the length of preparation, and the assessment of competence, etc.

The Analysis of Recent Studies and Publications. The analysis of modern research and scientific publications shows the activity of discussions on the problem of applying expert assessment not only in technical and economic sciences, but also in pedagogical research since they are the most common way of obtaining and analyzing qualitative information. Ukrainian and foreign scientists contributed to the development of the theory and practice of expert assessments: V. M. Heyets, V. M. Hlushkov, G. M. Hnatiyenko, B. Ye. Hrabovetskyi, G. M. Dobrov, M. V. Mikhalevich, F. M. Khilyuk, N. Z. Shor, S. M. Yampolskyi and others.

In particular, the involvement of experts in the assessment and certification of qualifications of service personnel - O. M. Kriukova, D. R. Makeeva [1] and others; use of the method of expert evaluation in designing the model of competencies of the heads of structural units for the sale of communication services and customer service as the basis for modeling the system for developing their professional competence - O. V. Borodienko [2] et al.; the assessment of the quality of an expert, which involves the implementation of several stages: the formation of initial parameters; the choice of the method for assessing the quality of an expert (a priori and a posteriori methods of evaluation); the selection of experts, which was investigated in the works of D. A. Gornitska, O. G. Korchenko, M. G. Lutskyi, I. M. Yarmosevich [3], and others.; an expert survey using the method of expert groups is proposed by B. S Gershunsky [4], Y. S. Grisyuk [5], O. G.Timinsky [6], and others; consistency of expert assessments on the basis of the mathematical theory of fuzzy sets - N.A. Brodyagina, V. V. Myakushko [7], O. M. Poleshchuk [8] and others; application of software tools for evaluating the competence of experts is proposed by P. F. Baranov, O.N. Velychko, A. A. Gaber, T. B. Gordienko [9] and others; organization of the doctrines of pedagogical phenomena and processes - V. K. Sydorenko, G. V. Tereshchuk [10] and others.

Unresolved aspects of the problem. In addition to the algorithm for assessing the competence of an expert, the reliability is also affected by the development of special automated systems for conducting expert assessment. Existing methods of obtaining expert assessments do not go beyond the so-called “point” assessments, the disadvantage of which is that it is difficult even for a skilled expert to tell a specific number, which will be valuation of a certain factor. It is much easier to give a qualitative assessment of the value of any indicator at the linguistic level (for example, low, below average, average, above average, high). Even minor mistakes in the assessment of indicators can lead to a significant change in the values of the complex indicators of an expert group.

At the moment, we have not met the work in which the mathematical apparatus of fuzzy logic would be combined with the representation of input data in the form of linguistic variables and the automation of computational processes for determining the competence of experts. The above mentioned determined the direction of our study.

The objective of this paper is designing and developing an automated system of fuzzy identification of expert's competence for assessing the quality of pedagogical phenomena and processes.

Research methods. Inaccuracy and uncertainty in the input data, the presence of a fuzzy statement of the problem at the initial stage of solving the problem leads to the need to use methods of fuzzy mathematics. The apparatus of fuzzy mathematics allows formalizing the concepts, used by experts in describing their ideas about the real system.

In the developed automated system, the quantitative assessment of the competence of experts is carried out in accordance with the methodology proposed in [4]. Thus, the competence of an expert depends on six factors: work experience, academic degree, if any, publications, justification of position as to the discussed problem, knowledge of the achievements in the relevant sphere, knowledge of the essence and objects of work of specialists of a certain area. The expert's answers to each question in a survey were evaluated by a certain numerical coefficient, which gave the opportunity to quantify the overall competence in the problem being discussed. The approximate numerical values from the relevant question were expressed by linguistic variables followed by fuzzification as input variables of the automated system. automated identification competence pedagogical

The process of formalization and integration of fuzzy input data to determine the level of representativeness of an expert we described in [11]. Fuzzy methods used in the derivation of the expert system (in which the formalized knowledge of an expert about a certain subject area is preserved) in the knowledge base can be characterized by three features: the use of linguistic variables instead of numeral or in addition to them; simple relations between variables are described by fuzzy statements; complex relationships are described by fuzzy algorithms.

The development of an automated system of fuzzy identification of expert's competence for assessing the quality of pedagogical phenomena and processes was carried out in the MATLAB software environment, in particular Fuzzy Logic Toolbox. The training of the developed system was carried out using the ANFIS algorithm. Based on this system, a fuzzy controller in the Simulink environment was built.

The results of the research. To construct a fuzzy system, we needed to construct a match matrix of possible score combinations of input variables and their corresponding representational values. Each of the six factors mentioned above is evaluated using scores. The values of the factors will be input to the fuzzy system. The formula for the calculation of representativeness is given in [4, p. 99]. The level of representativeness is the output of the fuzzy expert system.

The program of building a training sample and dividing it into input and outputs:

To develop an automated system of fuzzy identification of expert competence for assessing the quality of pedagogical phenomena and processes, with the help of fuzzy command in the MATLAB environment we run a FIS editor, which is designed to work with fuzzy systems. This editor has the ability to create and configure Sugeno type fuzzy logic output systems. We create a new fuzzy system of Sugeno, the structure of which is depicted in Figure 1, a.

The number of membership functions on the input variables is given in accordance with the score distribution of factors, which have influence on the representativeness (competence) of an expert, suggested in [4] and formalized in [11]. The graphs of the membership functions of input linguistic variables are given by the Gaussian type with the corresponding grouping according to the numerical values of the input factors.

Parameters of the membership functions of the input variables:

For the output variable, we assign five membership functions of the type of constants with the following parameters: verylow - 0.5625, low - 0.6667, midle - 0.7708, high - 0.8958, veryhigh - 1.

Due to the large number of input variables, it is difficult to manually create rules for all possible combinations of terms (maximum number of combinations is 216). But, as we know, in order for the system to be adequate, it is not necessary to have all possible rules. Sufficient will be the number of rules, which allows approximately to equalize the level of error with the level of plateau saturation [12].

Using the surface viewer of the FIS editor, you can view the surface of dependencies of the system output on the two input variables, with the rest of the values of the variable constants (see Figure 2, a). We save the created system and export it to the MATLAB work area called Sugeno. We read the data of the fuzzy logic output of the system, construct a diagram of system failures (Fig. 3, a) and find the system error.

As a result, a knowledge base was formed of the following nine rules:

The system error was 0.049117, which is more than 10% in percentage terms. Such a system is not able to adequately reproduce the given analytical dependence. The quality of the system can be improved by adding rules, but due to a large number of input variables and the cumbersome amount of training sample data, this path is not effective. It is necessary to use the possibilities of MATLAB, in particular the automatic adjustment (training) of Sugeno type fuzzy systems.

For training, we will generate a new fuzzy Sugeno system using the function genfis1. fis = genfis1(tr_data, [2 2 2 2 2 2], 'gaussmf').

The basic parameters of the generated system, as well as its structure are shown in Figure 1, b.

Fig. 1. The structure of the generic Sugeno fuzzy system to assess the competence of an expert: а - of a zero-order; б - with the help of anfis1 function

The generated system must be trained based on the data of the training sample tr_data.

System training is carried out using anfis function. Sugeno training program:

learn = anfis(tr_data, fis, 10)

As a result of this program, we have received a new fuzzy system learn with optimized parameters.

Parameters of the membership functions of the input linguistic variables of a Sugeno fuzzy system optimized using anfis:

Fragment of base of rules of automatically generated Sugeno fuzzy system:

Fig. 2. The surface of the response of the Sugeno fuzzy system: а - zero-order; b - optimized by anfis

Program of readout of fuzzy logic output and error estimation of optimized system:

Fig. 3. Diagram of Sugeno fuzzy system failures: а - zero-order; б - optimized by anfis

The error of this system was 6.8474e-008. This error is close to zero and fully meets our requirements for the adequacy of the decision-making by system. Thus, we use this system to build a fuzzy controller in the Simulink environment.

In the Simulink environment, we build a fuzzy controller based on the Sugeno FIS- system, optimized using the anfis function, because it showed the smallest error using the Fuzzy Logic Controller. The scheme of the fuzzy controller and the structure of the fuzzy output constructed from the Simulink blocks is shown in Figure 4.

Fig. 4. Scheme of fuzzy controller

To organize the testing of the controller, a group of seven experts people was selected, which was asked a questionnaire, which included the question of determining the above factors whose significance affects their overall representativeness (competence). The expert's response to each question was evaluated by a certain numerical coefficient, which gave the opportunity to quantify the overall competence of the problem under discussion.

The results of testing the controller shown in Table 1 confirm the high accuracy of the reproduction of a given dependence received as a result of the survey of the input vectors.

Table 1

Controller testing data

Conclusions and prospects for further research. Built with the help of an FIS-editor automated system of fuzzy identification of expert's competence for assessing the quality of pedagogical phenomena and processes adequately reflects the use of input data. The error of Sugeno fuzzy first-order system (trained using the ANFIS algorithm) is 6.8474e-008, which is an approximation to zero. Based on this system, a fuzzy controller was built in the Simulink environment, testing of which confirmed the high quality and reliability of the fuzzy system.

Further research needs to be carried out on the question of the automated formation of working expert groups and the implementation of on-line version of the automated system of fuzzy identification of competence of an expert. The results can be used by researchers in the field of Pedagogy to automate expert assessments of pedagogical phenomena and processes.

References

[1] E. M. Kryukova and D. R. Makeeva, "The role of experts in the assessment and certification of qualifications of the personnel of the service sector", Journal of the Association of Universities of Tourism and Services, No. 1, s. 9-14, 2013 (in Russian).

[2] O. V. Borodienko, "Method of expert evaluation in modeling the system of development of professional competence of managers", Scientific Bulletin of Uzhgorod National University, Series "Pedagogy, social work-", Issue 35, s. 35-38, 2015 (in Ukrainian).

[3] M. G. Luts'kiy, O. G. Korchenko, D. A. Gornits'ka and І. M. Yarmoshevich, "Model of expert quality assessment to increase the objectivity of expertise in the field of information security", Naukovo- tekhnichniy zhurnal "Zakhist informatsi'i", № 2, s. 115-120, 2011 (in Ukrainian).

[4] B. S. Gershunskiy, Forecasting the content of training in technical schools, M.: Vyssh.shkola, 1980 (in Ukrainian).

[5] Yu. S. Grisyuk and A. V. Labuta, "Formation of working groups and determination of an experts' competence in project and program management processes", Upravlinnya proektami, sistemniy analiz і logistika: Naukoviy zhurnal, Vyp. 9. s. 36-39, 2012 (in Ukrainian).

[6] O. G. Timins'kiy, "Certain aspects of the formation of the genetic code of the project manager in accordance with the terms of the project in portfolio management", Upravlinnya proektami ta rozvitok virobnitstva: Zb.nauk.pr., № 1(21), s. 49-57, 2007 (in Ukrainian).

[7] N. A. Brodyagina and V. V. Myakushko, "Consistency of expert assessment on the basis of the mathematical theory of fuzzy sets", Global'naya yadernaya bezopasnost', № 4(9), s. 23-28, 2013 (in Ukrainian).

[8] O. M. Poleshchuk, "System analysis and processing of group expert information on the basis of linguistic variables", Lesnoy vestnik, № 1, s. 65-74, 2015 (in Russian).

[9] O. N. Velichko, T. B. Gordienko, A. A. Gaber and P. F. Baranov, "The use of software to assess the competence of experts", Otkrytoe obrazovanie, № 4, s. 15-21, 2015 (in Russian).

[10] G. V. Tereshchuk and V. K. Sydorenko, The basics of pedagogical research, Ol'shtyn,: Vyssh.shkola Informatyki I Ekonomii TWP, 2010 (in Polish).

[11] І. M. Tsidylo, "The application of a fuzzy logic apparatus to assess the representativeness of an expert", Naukovi zapysky TNPU im. V. Gnatyuka. Seriya: Pedagogika, № 4. s. 171-177, 2011 (in Ukrainian).

[12] V. V. Kruglov, M. І. Dli and R. Yu. Golunov, Fuzzy logic and artificial neural networks: Tutorial, M. Izdatel'stvo fiziko-matematicheskoy literatury, 2001 (in Russian).

Размещено на Allbest.ru