Use of facial emotion recognition in e-learning systems

The concept of e-learning, its advantages, flexibility of time and place in comparison with formal learning. Creation of a hybrid information system that integrates computer vision and machine learning technologies for visual and interactive systems.

Рубрика Программирование, компьютеры и кибернетика
Вид статья
Язык английский
Дата добавления 10.05.2018
Размер файла 280,0 K

Отправить свою хорошую работу в базу знаний просто. Используйте форму, расположенную ниже

Студенты, аспиранты, молодые ученые, использующие базу знаний в своей учебе и работе, будут вам очень благодарны.

Размещено на http://www.allbest.ru/

Размещено на http://www.allbest.ru/

Use of facial emotion recognition in e-learning systems

The word e-learning was pronounced at the CBT Systems seminar in 1999 firstly. Elearning is called "blended learning” that picks all educational and technical items and brings them together in a rich and interactive environment [1]. As soon as the idea of e-learning was explained, expectations of converting the formal education to e-learning are increased. However, in the early 2000s, web technologies could not meet these high expectations in the sense of the e-learning system and course materials (Figure 1) [2]. Here, the preparation phase of the educational documents was an important problem, because it was a time consuming and expensive process. The contents of the e-learning course were not impressive and beneficial enough for keeping the students in the virtual environment for a long time [3]. In the last fifteen years, e-learning activities and web-based training documents have been getting common around the world with the increase of personal computers and the Internet usage.

E-learning enables to access educational materials easily at any time and from anywhere. In addition, with the personalized learning technologies, the productivity in education towards the student groups in the heterogeneous structure is also increasing. Today, e-learning is not limited to computer-aided systems. It is also accessible from mobile devices. [5]. Thus, students can take virtual courses whether from their personal computer or mobile devices, at home or in a cafe. These elasticities allow the student to feel comfortable. This situation has led to increasing efficiency and facilitating the learning process.

Recent researches show that emotional states and motivation of students directly or indirectly influence the learning process [6]. Isen [7] argues that a positive mood not only makes us feel good, but it also leads to more accurate decision-making, greater creativity, and flexibility toward problem-solving. According to Isen, the human brain is not completely cognitive information. The human brain is evaluated as a system that integrates emotional and cognitive functions inseparably. Individuals need to be physically, mentally, emotionally and socially ready to be able to comprehend the information easily [8]. Feidakis [9] has presented a review of emotional awareness in the context of e-learning.

One of the biggest problems of e-learning systems is to maintain the motivation of the students in the virtual classrooms. This situation is easier in formal classrooms because the educator is in face-to-face contact with the students in the same environment [10]. In this aspect, it is much easier to observe their emotional states and motivations.

Several face recognition studies have been utilized to evaluate the emotional state of participants in e-learning environments. Ashwin et al. [11] proposed multi-user face detection based e-learning system using SVM with different data sets (LFW, FDDB, and YFD). Al-Alwani et al. [12] performed mood extraction using a neural network, which classifies facial expressions according to facial features in order to improve students' learning tendencies in e-learning systems. Krithika et al. [13] developed a SERS using the local binary pattern (LBP) for facial recognition and detection. It finds exciting, disturbing and moving patterns and reveals meaningful information to understand the mood of the student studying in an e-learning environment. Recently, real-time emotion recognition studies have been encountered in e-learning. For instance, Magdin et al. [14] have designed software, which evaluates the emotional state of the user using a webcam. The system works in real time using neural networks. An overall accuracy of their software based on the requested emotions and the recognized emotions is 78%. As an adaptive e-learning study that appeals to a different student population, an emotion recognition system was proposed for students with high-functioning autism [15].

The purpose of the study is to develop a new Facial Emotion Recognition System (FERS), which recognize the emotional states and motivation levels of students in videoconference type e-learning systems. In order to create a more interactive educational environment, this system transfers the emotional states of the students to the educator instantaneously. Our study is supportive of the studies that make possible to observe the motivation level of both the individual and the virtual classroom in the e-learning systems.

In this section, the hardware infrastructure and the software development stages of the proposed system are described.

Hardware infrastructure

This work is suitable for e-learning applications through video-conference. For this reason, experimental work has been carried out through the Skype program. Once the virtual classroom in the Skype program is launched, the screenshots of the educator's computer involving frontal face images of participants are recorded in “.png” or “.jpg” format instantaneously. Each recorded image is read instantly and the image processing process begins. In this process, GPU is important in terms of the proposed system performance [16]. We used a computer with Nvidia GeForce GTX 860M graphics card. The proposed FERS was developed on Python platform. Python has various powerful software packages for both image processing (OpenCV, dlib, etc.) and machine learning (scikit-learn, numpy, pandas, etc.) processes.

Software development stages 2.2.1. Face Detection

Face detection is the first step in the FERS. Face detection can be realized on images and on real time video. The Viola-Jones algorithm [17] is preferred in facial detection in terms of real-time operation and fast detection. There are six categorized emotions (Happiness, Fear, Sadness, Anger, Surprise, and Disgust) that are universally accepted in FER [18].

Obtaining the Facial Landmarks and Attributes

Humans' faces give an idea about their emotional state. As the emotional moods change, there will also be changes in the sections of the face. The edges of these sections are determined and followed by facial landmarks. In this study, a facial landmarks detection algorithm proposed by Sagonas et al. [19] was used. Sagonas performed this algorithm on different face data sets [20] - [23] and aimed to obtain a standard in determining facial landmarks.

Displaying facial landmarks on multi-participants' faces in Skype video conference

This algorithm yields 68 facial landmarks on the detected face indicating the edges of sections of the face (Figure 2). In this study, the new distinctive attributes obtained using these facial landmarks (Figure 2).

Attribute Description

Ratio

Name

The ratio of face width to mouth width

dl/d2

al

The ratio of mouth width to month openness

d2/d3

a2

The ratio of eye width to eye openness

d4/d 5

a3

The ratio of eyebrow length to distance between two eyebrows

d6/d7

a4

The ratio of eyebrow length to distance betw een eye and eyebrow

d6/dS

a 5

The ratio of the length of the nose to the distance between the nose and the upper lip

d9/dl0

аб

The ratio of the distance between the extreme points of the eyebrows to the length of the nose

dl J/d9

a7

The ratio of distance between two irises to month width

d 12/d2

aS

Figure 2. Distinctive attributes extracted from facial expressions

Training Data Set

The training data set has been described in detail in our previous study [24]. The data set has the size of 11680x10 and consists of attributes obtained from the students' facial expressions. A section of this data set is given in Figure 3.

al

a2

a3

a4

a5

a6

a7

a3

gender

emotion a l_state |

2,2565

3,1314

3,4315

1,9529

2,6196

2,9567

1,1655

3,1065

1

Happiness

2,2213

3,3773

3,4315

2,0676

2,6196

2,9567

1,1665

3,0013

0

Happiness

2,3032

2,4596

2,9872

1,9702

2,6798

2,203

1,2244

3,0376

0

Surprise

2,2434

2,6616

2,7613

2,5336

1,5455

1,7676

1,2773

3,2375

1

Fear

2,213

2,5633

2,754

2,5903

1,6172

1,7676

1,2489

3,2367

1

Fear

1,9155

2,1664

3,2838

3,7272

1,4998

2,4759

1,0962

3,0643

1

Disgust

1,9576

2,2133

3,2338

3,2062

1,6106

2,5466

1,1184

3,0643

1

Disgust

2,3073

4,9497

5,6874

3,1226

2,8932

3,6409

1,312

2,693

0

Anger

2,457

2,1143

5,6374

2,6519

2,9019

3,4933

1,4337

2,3067

0

Anger

2,4334

2,9372

5,3413

1,7897

2,8635

2,4703

1,2279

3,3539

0

Sadness

2,5714

2,1523

3,4

2,3637

2,4434

2,6133

1,4233

3,2292

0

Neutral

2,5 26

2,1926

3,0922

2,3403

2,4086

2,4641

1,4213

3,2551

0

Neutral

2,2539

3,1786

3,4315

2,0155

2,6542

2,9995

1,1665

3,1055

0

Happiness

Figure 3. A section of training data set

Emotion Recognition

Each instance in training data set contains nine attributes (Figure 4). Eight of them are facial attributes and the other attribute is gender information represented numerically as "0" and "1" for "male" and "female. These instances are used in both training and testing process of emotional class prediction. The goal of this phase is to predict the emotional class of the each instance obtained after face detection and feature extraction phases. The prediction performances have been compared by using four different types of classification methods in order to determine emotional classes.

Support Vector Machine (SVM) has proven that core-based non-linear classifiers give good results in practical problems in machine learning and data mining [25]. SVM determines the boundary surface between the two classes, maximizes the distance between the training set and the nearest points to the discrimination surface.

The k-nearest neighbor (kNN) classification method is based on the similarity between the sample to be classified and the samples in the class. Training samples are defined by n- dimensional numerical qualities. Each sample points to n-dimensional space [26]. In this way, all training samples are stored in n-dimensional sample space. These k-education examples are the closest neighbors to the unknown sample A. Proximity is found by calculating Euclidean Distance as given in equation (1).

Random Forest (RF) is a classifier made by Breiman and Cutler, consisting of multiple trees produced from random samples of states or variables. In this method, the tree starts with a single node that all the data has formed, and if all of the samples belong to the same class, the node ends up as leaf and the class label is given. If the samples are not included in the same class, the samples are chosen to best fit the classes [27].

Classification & Regression Trees (CART) algorithm has a structure that forms binary decision trees by dividing the corresponding cluster into two sub-clusters that are more homogeneous at each step. The best independent variable is selected using the impurity and the variance in the variation measure. Here, the goal is to produce the most homogeneous data subgroups possible for the target variable [28].

Determining the Motivation Level

In this section, instant and weighted emotional states of students in a virtual class have been determined. From the beginning of the video conference, the emotional state of the students transferred to the educator. The emotional state of the majority of the learners indicates the emotional mood of the virtual class. Herein, some of the six universal emotions are considered as positive and some of them negative. For instance; happiness and surprise are considered as positive emotions; sadness, disgust, fear, anger are negative emotions. The instantaneous emotional states of the students are displayed with different emoji (Figure 4). The educator is also able to see the general motivation level of the virtual classroom with the chart of motivation level.

Figure 4. General motivation level of the virtual-classroom

A two-stage test process was performed to determine the best of the prediction algorithms given in Section 2.2.4. In the first stage, the entire data set was statically classified. The results obtained in the first stage are given in Table 1.

The classification results of machine learning algorithms

Tested Algorithms

Accuracy (%)

kNN

96.38

Random Forest

93.36

CART

89.11

SVM

97.15

The accuracy rates were sorted as CART, RF, kNN and SVM algorithms in ascending order. The reason for the higher performance of SVM and kNN algorithms is that the features are extracted from the Euclidean distance between two facial points. It is known that KNN and SVM algorithms are very successful in classifying data sets with such spatial values.

In the second stage, the emotional class of the instances is predicted with algorithms mentioned at Table 1. The emotional classes are happiness, surprise, sadness, anger, disgust, fear and neutral. They are enumerated 0 to 6 respectively. The instances whose actual emotional classes are already known were randomly selected from entire data set. A total of 1650 instances were predicted separately and the prediction results of the algorithms were recorded. Figure 6 presents the actual emotional class and prediction of the algorithms.

As shown in Figure 6, different prediction results can be obtained when distinct instances are tested with different algorithms. Prediction accuracy was obtained by calculating how many pieces of accurate predictions were made for each algorithm. Table 2 gives the prediction accuracies of these algorithms.

The prediction results of the algorithms

kNN

RF

CART

SVM

# of Actual Class/

# of Correctly Classified

1650 / 1574

1650 / 1534

1650 / 1585

1650 / 1621

Accuracy (%)

95.39

92.97

96.06

98.24

With the SVM algorithm, only 29 of 1650 different data samples were incorrectly estimated and a high accuracy value of 98.24% was obtained. The SVM algorithm with the RBF kernel function, which gives the best performance as a result of these two steps, has been preferred for instant emotion recognition process.

In this study, an FERS was developed that sends emotional states of the students in the virtual classroom to the educator. This supportive system makes possible to observe the motivation level of both the individual and the whole virtual classroom in the e-learning systems. In the experimental phase, SVM was chosen for emotional state classification because it gave the best prediction accuracy. In conclusion, evaluating the emotion of a learner in the virtual classroom can help in enhancing the learning experience. Skype is preferred as a test platform because it is free of charge. Moreover, it allows multi-user video conferencing. Nevertheless, this work can also be adapted for other video conferencing software in the future. As the camera becomes standard equipment for personal computers, the usability of this system increases.

This study is realized as part of the M.Sc. Thesis Project (16/129). The authors acknowledge the support from Scientific Research Project Office of Mugla Sitki Kocman University.

References

learning hybrid information

[1] A. Nagy, "The Impact of E-Learning", in: P.A. Bruck, A. Buchholz, Z. Karssen, A. Zerfass, (Eds). E- Content: Technologies and Perspectives for the European Market. Berlin: Springer-Verlag, pp. 79-96, 2005. (in English)

[2] M. D. Lytras and A. Pouloudi, "E-Learning: Just a Waste Of Time," in Seventh Americas Conference on Information Systems, America, 2001. (in English)

[3] S. Carliner and P. Shank, The e-Learning Handbook: Past Promises, Present Challenges, John Wiley & Sons, 2008. (in English)

[4] A. Hicken, "2016 Elearning Hype Curve Predictions'', 23.12.2015. [Online]. Available: http://www. webcourseworks. com/2016-elearning-hype-curve-predictions/. [Acces sed 06.07.2017].

(in English)

[5] K. I. Benta, C. Marcel and F. M. Vaida, "A multimodal affective monitoring tool for mobile learning," Institute of electrical and electronics engineers computer society, pp. 34-38, 2015. (in English)

[6] M. Lewis, H. Jones and F. Barrett, Handbook Of Emotions, New York, 2000. (in English)

[7] A. M. Isen, "Positive Affect Influences Decision Making," in Handbook of emotions, M. Lewis and J. Haviland, Eds., Guilford, New York, The Guilford Press, p. 720, 2000. (in English)

[8] N. Razon, "Okul Ba^arisini Etkileyen Faktorler”, [Online]. Available: http://www.ekipnormarazon.com/makale-detay/okul-basarisini-etkileyen-faktorler

[Accessed 06.07.2017]. (in Turkish)

[9] M. Feidakis, "A Review of Emotion-Aware Systems for e-Learning in Virtual Environments," in Formative Assessment, Learning Data Analytics and Gamification, S. Caballe and R. Clariso, Eds., Academic Press, Boston, p.217-242, 2016. (in English)

[10] W. K. Horton, Leading e-learning, American Society for Training and Development, 2001. (in English)

[11] T. S. Ashwin, J. Jose, G. Raghu and G. R. Reddy, "An E-learning System With Multifacial Emotion Recognition Using Supervised Machine Learning," in IEEE Seventh International Conference on Technology for Education, 2015. (in English)

[12] A. Al-Awni, "Mood Extraction Using Facial Features to ImproveLearning Curves of Students in ELearning Systems," International Journal of Advanced Computer Science and Applications, vol. 7, no. 11, pp. 444-453, 2016. (in English)

[13] L. B. Krithika and G. G. Lakshmi Priyya, "Student Emotion Recognition System (SERS) for e-learning," Procedia Computer Science, p. 767 - 776, 2016. (in English)

[14] M. Magdin, M. Turcani and L. Hudec, "Evaluating the Emotional State of a User Using a Webcam," Special Issue on Artificial Intelligence Underpinning, vol. 4, no. 1, pp. 61-68, 2016. (in English)

[15] C. H. Chu, W. J. Tsai, M. J. Liao and Y. M. Chen, "Facial emotion recognition with transition detection for students with high-functioning autism in adaptive e-learning," Soft Computing, pp. 1-27, 2017. (in English)

[16] A. W. Wai, S. M. Tahir and Y. C. Chang, "GPU Acceleration of Real Time Viola-Jones Face Detection," in IEEE International Conference on Control System, Computing and Engineering, 2015. (in English)

[17] P. A. Viola and M. J. Jones, "Rapid object detection using a boosted cascade," IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR, pp. 511-518, 2001. (in English)

[18] P. Ekman and D. Keltner, "Universal Facial Expression of Emotion:An Old Controversy and New Findings," Nonverbal Behav,, vol. 21, no. 1, pp. 3-21, 1997. (in English)

[19] C. Sagonas, G. Tzimiropoulos, S. Zafeiriou and M. Pantic, "300 Faces in-the-Wild Challenge: The first facial landmark localization Challenge," in The IEEE International Conference on Computer Vision, 2013. (in English)

[20] P. N. Belhumeur, D. W. Jacobs, D. J. Kriegman and N. Kumar, "Localizing parts of faces using a consensus of exemplars," in The IEEE International Conference on Computer Vision and Pattern Recognation (CVPR), 2011. (in English)

[21] V. Le, J. Brandt, L. Bourdev and T. S. Huang, "Interactive Facial Feature Localization," in European Conference on Computer Vision, 2012. (in English)

[22] X. Zhu and D. Ramanan, "Face detection, pose estimation, and landmark localization in the wild," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012. (in English)

[23] M. Kostinger, P. Wohlhart, P. M. Roth and H. Bischof, "Annotated facial landmarks in the wild: A large- scale, realworld database for facial landmark localization," in IEEE International Conference on ComputerVision Workshops (ICCV Workshops), 2011. (in English)

[24] U. Ayvaz and H. Guruler, "The Detection of Emotional Expression towards Computer Users," International Journal of Informatics Technologies, vol. 10, no. 2, pp. 231-239, 2017. (in Turkish)

[25] V. N. Vapnik, "An overview of statistical learning theory," IEEE transactions on neural networks, vol. 10, no. 5, pp. 988-999, 1999. (in English)

[26] X. Wu, V. Kumar, J. R. Quinlan, J. Ghosh, Q. Yang and H. Motoda, "Top 10 algorithms in data mining," Knowledge and information systems, vol. 14, no. 1, pp. 1-37, 2008. (in English)

[27] A. Liaw and M. Wiener, "Classification and regression by random Forest," R news, vol. 2, no. 3, pp. 1822, 2002. (in English)

[28] D. Steinberg and P. Colla, "CART: classification and regression trees," in The Top Ten Algorithms in Data Mining, X. Wu and V. Kumar, Eds., Chapman & Hall/CRC, 2009, p. 179. (in English)

Размещено на Allbest.ru

...

Подобные документы

  • Управление электронным обучением. Технологии электронного обучения e-Learning. Программное обеспечение для создания e-Learning решений. Компоненты LMS на примере IBM Lotus Learning Management System и Moodle. Разработка учебных курсов в системе Moodle.

    курсовая работа [146,6 K], добавлен 11.06.2009

  • The material and technological basis of the information society are all sorts of systems based on computers and computer networks, information technology, telecommunication. The task of Ukraine in area of information and communication technologies.

    реферат [29,5 K], добавлен 10.05.2011

  • Общие понятия об e-learning. Области применения продукта. Модели и технологии. Исследование и анализ программных инструментов. Создание учебного курса для преподавателей инженерно-экономического факультета. Оценка эффективности внедрения такого обучения.

    дипломная работа [4,7 M], добавлен 03.05.2018

  • Machine Learning как процесс обучения машины без участия человека, основные требования, предъявляемые к нему в сфере медицины. Экономическое обоснование эффективности данной технологии. Используемое программное обеспечение, его функции и возможности.

    статья [16,1 K], добавлен 16.05.2016

  • Machine Translation: The First 40 Years, 1949-1989, in 1990s. Machine Translation Quality. Machine Translation and Internet. Machine and Human Translation. Now it is time to analyze what has happened in the 50 years since machine translation began.

    курсовая работа [66,9 K], добавлен 26.05.2005

  • Интернет технологии как средство обучения. Cтaндaрт E-learning, web-прилoжeния для прoвeдeния прoцecca oбучeния. Шкoльнaя инфoрмaциoннo-aнaлитичecкaя cиcтeмa "Coнaтa", инcтрумeнтaльнaя cрeдa для её coздaния. Дoпуcтимыe урoвни звукa и звукoвoгo дaвлeния.

    дипломная работа [156,0 K], добавлен 10.07.2015

  • Вивчення особливостей використання всесвітньої мережі Інтернет, адресації інформації, вірусних загроз. Розробка та підготовка сайту до експлуатації за допомогою візуального редактора Front Page. Характеристика дизайну та структури створеного web-сайту.

    курсовая работа [1,4 M], добавлен 22.11.2012

  • Характеристика та основні напрями діяльності друкарні "Добробут". Особливості дистанційної системи навчання "Moodle", сутність програми "Learning Space 5.0". Основне призначення діаграми використання, її склад: блоки використання, зовнішні користувачі.

    дипломная работа [2,9 M], добавлен 12.04.2012

  • Функции, место и виды контроля в обучении. Тест как инструмент измерения качества знаний, формы тестов. Балльно-рейтинговая система оценивания студентов. Разработка компьютерных тестов по математике на базе Конструктора Distance Learning Studio.

    дипломная работа [2,2 M], добавлен 05.09.2011

  • Современные системы дистанционного обучения и анализ требований, предъявляемых при создании сетевых курсов и виртуальных университетов. Архитектура и ценообразование СДО для повышения квалификации и переподготовки специалистов транспортной отрасли РФ.

    курсовая работа [1,2 M], добавлен 28.07.2011

  • Система дистанционного обучения Distance Learning Belarus и лабораторный практикум курса "Разработка трансляторов для языков программирования", его перенос в интерактивную среду обучения. Описание работы программы и её взаимодействия с пользователями.

    курсовая работа [588,5 K], добавлен 03.11.2012

  • Система дистанционного обучения Distance Learning Belarus. Разработка лабораторного практикума по курсу "Разработка трансляторов для языков программирования". Базовые концепции разработки приложений для Интернет. Схема диалога пользователя с системой.

    курсовая работа [1,3 M], добавлен 03.11.2012

  • Обучение через Интернет. Обзор систем дистанционного обучения. Система дистанционного обучения Learning Space 5.0. Создание базы данных. Запуск установочного файла для создания файла config.php. Структура каталога CMS Moodle. Создание нового курса.

    курсовая работа [1,3 M], добавлен 27.04.2009

  • Information security problems of modern computer companies networks. The levels of network security of the company. Methods of protection organization's computer network from unauthorized access from the Internet. Information Security in the Internet.

    реферат [20,9 K], добавлен 19.12.2013

  • Consideration of a systematic approach to the identification of the organization's processes for improving management efficiency. Approaches to the identification of business processes. Architecture of an Integrated Information Systems methodology.

    реферат [195,5 K], добавлен 12.02.2016

  • Сравнение эталонных моделей OSI, TCP. Концепции OSI: службы; интерфейсы; протоколы. Критика модели, протоколов OSI. Теория стандартов Дэвида Кларка (апокалипсис двух слонов). Плохая технология как одна из причин, по которой модель OSI не была реализована.

    реферат [493,1 K], добавлен 23.12.2010

  • Creation of the graphic program with Visual Basic and its common interface. The text of program code in programming of Visual Basic language creating in graphics editor. Creation of pictures in Visual Basic, some graphic actions with graphic editor.

    лабораторная работа [1,8 M], добавлен 06.07.2009

  • Basic assumptions and some facts. Algorithm for automatic recognition of verbal and nominal word groups. Lists of markers used by Algorithm No 1. Text sample processed by the algorithm. Examples of hand checking of the performance of the algorithm.

    курсовая работа [22,8 K], добавлен 13.01.2010

  • History of development. Building Automation System (BMS) and "smart house" systems. Multiroom: how it works and ways to establish. The price of smart house. Excursion to the most expensive smart house in the world. Smart House - friend of elders.

    контрольная работа [26,8 K], добавлен 18.10.2011

  • Program game "Tic-tac-toe" with multiplayer system on visual basic. Text of source code for program functions. View of main interface. There are functions for entering a Players name and Game Name, keep local copy of player, graiting message in chat.

    лабораторная работа [592,2 K], добавлен 05.07.2009

Работы в архивах красиво оформлены согласно требованиям ВУЗов и содержат рисунки, диаграммы, формулы и т.д.
PPT, PPTX и PDF-файлы представлены только в архивах.
Рекомендуем скачать работу.