Holistic and analytic processing of facial expressions: a method of multidimentional scaling

Размещено на http://www.allbest.ru/

HOLISTIC AND ANALYTIC PROCESSING OF FACIAL EXPRESSIONS: A METHOD OF MULTIDIMENTIONAL SCALING

facial expression identification

YA.A. BONDARENKO, G.YA. MENSHIKOVA

Lomonosov Moscow State University, 1 Leninskie Gory, Moscow, 119991, Russian Federation b Russian Presidential Academy of National Economy and Public Administration (The Presidential Academy, RANEPA), 82-84, Prospect Vernadskogo, Moscow, 119571, Russian Federation

Abstract

Contemporary scientific literature is controversial about the two basic mechanisms that play a significant role in the identification and discrimination of facial expressions, that is, the analytic process (perceiving individual facial features) and the holistic process (perceiving the face as a Gestalt). Despite the large number of studies, the question of their relative contributions into the perception of facial expressions remains open. In the present study, we investigated the interaction of the analytic and holistic processing simultaneously using the methods of composite and inverted faces in the task of assessing the expression similarity To achieve this goal, the method of multidimensional scaling was used to enable the construction of a subjective space of expression similarities and then to analyze the clustering of expressions under up-right and inverted exposure conditions. The results showed that: 1. In up-right conditions of face presentations holistic processing plays a more important role; 2. Under the inverted conditions, the role of analytic processing is increased that is manifested in a change of expression clustering in the subjective space of expression similarities; 3. The method of multidimensional scaling is productive for investigating the interaction of analytic and holistic encoding mechanisms of facial expressions; 4. Sharing methods of the composite and inverted faces is an effective tool for changing the impact of analytic and holistic processes in assessing expression similarities.

Keywords: face recognition, basic and composite facial expressions, analytic and holistic processing, inverted faces, multidimensional scaling, diagnostic features.

Introduction

In 2018 Y. N. Harari1 suggested that many new professions would be associated with the understanding of human emotions. Despite the fact that the appeal to study the emotional sphere was announced in the 90s of the 20th century (Rothermund & Koole, 2018), until now many questions about the human affective sphere have yet remained little investigated. However, its importance in life is increasing (Damasio, 1994; Goleman, 2006). In modern cognitive psychology, there is a special branch of cognitive psychology named emotional cognition (Falikman, 2014). Within this framework, the problem of recognizing facial expressions occupies a special place, in particular, the question of its mechanisms and their contribution to the perception of emotional expressions (Young & Bruce, 2011).

Harari, Y. N. (2018). Davos 2018-Harari: An algorithm will be your best therapist, but it can be hacked too. Retrieved from https://www.youtube.com/watch?v=rIiHOu1tezIetc.) and analytic (Mauer et al., 2002; Donnelly & Davidoff, 1999; Rhodes, 1988; Diamond).

The fourth question concerns the debate about whether facial expressions processing is worth studying. So, many scientists do not take into account the specificity of face processing, reducing the mechanisms of face recognition to general mechanisms of perception, and thereby declaring the predominance of the Gestalt information processing. Nevertheless, a lot of evidence suggests the specificity of perceptual processing aimed at recognition of facial expressions. Here are some examples: processing of facial information can be localized in isolated brain areas (Kanwisher & Yovel, 2006); the technique of `composite' images does not work on other (not faces) objects; during the individual development the brain specialization in face recognition is very rapid; patients with prosopagnosia (Duchaine & Nakayama, 2006) are not able to recognize faces of familiar people, but at the same time they are able to identify any other objects, as well as individual facial features.

In our opinion, the main reason for the problems mentioned above is methodical. According to the authors (Richler & Gauthier, 2014) methods themselves are aimed not at some aspects of the process, but rather at different phenomena in general. Due to the very quick, therefore, partly unconscious process of face recognition a conscious part of processing only complicates (or impairs) the recognition (Calvo & Lundqvist, 2008). It leads to the question of conditions in a normal situation that contribute to the strengthening of one of the mechanisms.

Currently, some methods studying face processing are accepted that were defined as gold standards (Tanaka & Gordon, 2011), where the major methods are the method of composite faces and the method of inverted faces. The first one was used to investigate the recognition of faces reflecting composite facial expressions: a facial display conveying one expression in upper and the other - in lower half-faces of the same person (Menshikova, 2010). The second method showed decreasing in expression recognition efficiency for inverted faces (Tanaka et al., 2012) and was characterized by a longer time of identification. It was suggested that the results obtained by these two methods differ in nature (Rezlescu, Susilo, Wilmer, & Caramazza, 2017), However, at the moment there is no obvious evidence of this idea.

The question arises of the need to consider analytic and holistic processing separately. What is the role of each of the mechanisms? In our opinion, solving this problem requires a single instrument allowing a comparison in the functioning of both mechanisms in a common space. In this case, facial expressions recognition can be defined as a process of encoding and decoding emotional facial expressions based on sensory and non-sensory information. This process may be carried out via a single holistic-analytic continuum of mechanisms with their own characteristics aimed at solving a particular class of problems.

We suppose that, firstly, the analytic and holistic encoding of facial expressions comprises two independent but interacting basic processes; secondly, their interaction depends on face orientation: under normal conditions (upright faces) holistic processing of expressions plays a more important role, but under abnormal condition (inverted faces) the contribution of the analytic processing increases because of the increasing role of individual features.

The most effective way to investigate the relationship between analytic and holistic mechanisms is to combine a) the composite face effect; b) the face inversion effect; and c) the Method of Multidimensional Scaling (MDS) to analyze subjective spaces of similarities in facial expressions evaluation. E. N. Sokolov and his colleagues showed that MDS may be used as a universal way to model cognitive processes (Sokolov & Boucsein, 2000). They successfully applied it to study the structure of difference space in color perception, lines orientation, facial schematic expressions, etc. In the present study this method was applied for studying the structure of Space of Perceptual Similarities (SPS) in expression discrimination.

Our study is aimed at studying the impact of analytic and holistic processing in expression discrimination using the MDS method (Shepard, 1962). We hypothesize that the structure of SPS would change for inverted compared to upright faces bearing composite expressions. The analysis of the SPS structure was supposed to be performed according to several criteria: space grouping of basic expressions, space grouping of composite expressions, and clustering composite expressions on the basis of valence. In particular, we expected changes in (a) the space structure of basic expressions; (b) the space structure of composite expressions (grouping around diagnostic features) or (c) clustering by expression valence.

Method

Participants. 37 persons participated (20 females, 17 males), aged 18 to 25. All participants had normal or corrected to normal vision.

Stimuli. Stimuli to construct composite expressions were images of six basic emotional expressions plus Neutral of a male poser from Paul Ekman's atlas (Ekman, 1976). We used the only male face (PE2-21) because our earlier studies showed no significant difference in the evaluation of expression similarity in female and male composite faces (Menshikova, Bimler, Bondarenko, & Paramei, 2016). Images of faces bearing composite expressions combined Happiness, Anger and Fear in the upper or lower face with other five basic expressions. For example, using a basic expression of Happiness we created five images combining Happiness in the upper face + 5 other expressions in the lower face and five images of vice versa combinations. In total, 24 composite images were compiled. These three expressions (Happiness, Anger and Fear) were chosen because of the concentration of their distinguishing features in a single half-face (Happiness features in lower half-faces, Anger features - in upper half-faces, and Fear features - in both half-faces). To ensure that images did not cause an impression of unnatural “gluing”, we applied a special procedure for smoothing brightness differences using Photoshop CS5.

Figure 1 shows an example of composite faces for a male face, with a composite expression of “anger” (top) + “fear” (below) and a composite expression of “fear” (top) + “anger” (bottom). For illustration, a special black line is applied.

Equipment. We used a computer with the following properties: Processor Intel (R) Core (TM) i5-3470 CPU (3.20 GHz), RAM 4GB, System type 64-bit; 2) ZHFT Monitor Hp Compaq LA2306x: The screen resolution is 1920x1080 (60 Hz), the diagonal is 23 inches. The Microsoft Wired Desktop 600 keyboard was also used

Figure 1

Image in the Upright Orientation: Basic Expressions of Anger and Fear (A - fear, С - anger), to the Right a Composite Face (B - anger + fear; D - fear + anger).

for recording estimates of similarity degree between two images bearing different composite expressions.

Procedure. At the beginning, the participants were trained to estimate differences between pairs of faces bearing basic expressions. They were asked to evaluate similarity of those pairs. Then they took part in two experiments.

Experiment 1. The goal of this study was to reconstruct a space of perceptual similarities (SPS) for upright and inverted faces bearing basic expressions. The participants were asked to compare the pairs of upright images and then the same inverted images. They were asked to evaluate the degree of similarity between the two images using a scale from 1 (the minimum degree of similarity) to 9 (the maximum degree of similarity). The duration of images exposure was 2000 ms. After the face presentation an interface with numbers from 1 to 9 appeared on the monitor screen, and during 2000 ms the participants were asked to enter a number reflecting the degree of similarity between two expressions. The ISI was 500 ms. The number of pairs for comparison was 21.

Experiment 2.T he goal was to reconstruct a space of perceptual similarities (SPS) for upright and inverted faces bearing basic and composite expressions. The procedure was the same as in Experiment 1. The number of pairs for comparison was 930.

The average duration of the whole experimental session (training, Experiments 1 and 2) was from one hour to an hour and a half.

Estimates of expression differences were presented in four generalized matrices of similarities: (7X7) for Experiment 1 and (31X31) for Experiment 2, each for upright and inverted faces. The matrices were checked for random responses through the comparison of differences between individual matrices and the generalized matrix through the Pearson correlation. Each matrix was divided into two submatrices along the diagonal line. One of the parts was transposed. Later a statistical package of Excel was used. The correlation between the two submatrices was calculated using the standard procedure. If the correlation coefficient was less than 0.4, the matrices were excluded from further analysis. Then a generalized matrix for all subjects was created. The calculated correlation coefficients of the generalized and individual matrices were higher than 0.85, indicating a high consistency of the results. Results of the nine participants with low consistency with the generalized data were excluded from the analysis. Final matrices were processed with the PROXSCAL multidimensional scaling program to construct a geometrical model of the space of perceptual similarities (SPS) for basic and composite expressions. According to this model each facial expression is represented as a point within a space, and the similarities between different expressions are shown as distances between the points.

Results and Discussion

We analyzed the structure of the space of perceptual similarities for inverted and upright faces to estimate the impact of analytic and holistic processing on expression discrimination. The analysis of SPS structure was supposed to be carried out according to several criteria: space grouping of basic and composite expressions, grouping by expression valence. In particular, we expected changes in (a) the space structure of clustering basic expressions specific for the so called “Scholsberg circle” (Scholsberg, 1941); (b) the space structure of clustering composite expressions (grouping around diagnostic features or definite expressions); (c) valence expression grouping (clustering positive and negative expressions); (d) the degree of differentiation of the SPS structure (more differentiated structure with many separate small clusters or less differentiated with larger clusters). We supposed that clustering by the feature and the higher degree of SPS differentiation may be treated as strengthening of analytic processes, while clustering by valence, the lower degree of differentiation and the emergence of new Gestalts - as strengthening of holistic processes.

The structure of SPS for upright and inverted faces bearing basic expressions. Firstly, we analyzed the data of Experiment 1. We retained a 2D solution for both upright and inverted faces (Stress = 8.3% and 9.1%, respectively). The SPS structure for upright and inverted faces is showed in Figure 2. The space structure for upright faces was similar to the results obtained in previous studies (Scholsberg, 1941; Sokolov & Boucsein, 2000). First of all, the sequence of the cyclic organization is very close to the data obtained by H. Scholsberg (Scholsberg, 1941). Also the axes interpretation is very similar: for both figures one of the axes may be interpreted as a Pleasantness-Unpleasantness Scale and the other - as an Attention- Rejection Scale. So, the results demonstrated that our procedure was valid and reliable. As can be seen in the Figure 2, the structure for inverted faces was slightly changed, namely the distances between nearly all basic expressions were increased.

In our opinion, this result may be interpreted as a greater impact of analytic processing for inverted faces.

The structure of SPS for upright and inverted faces bearing basic and composite expressions. Using the MDS analysis of data obtained in Experiment 2, a 6D solution

Figure 2

The SPS Structure of Basic Expressions for Upright Faces (Upper) and Inverted Faces (Lower)

was revealed (Stress = 10.3%), reflecting a more complex perceptual space for composite expressions. It should be noted that earlier studies revealed less-dimensional spaces of expression (Sokolov & Boucsein, 2000; Bimler & Paramei, 2006). From our point of view, the higher dimensionality of space obtained in our study may be explained by the higher perceptual complexity of our stimuli, often reflecting hard- to-verbalise emotions. We retained D1 and D3 captured lower-face variations: D1 separated X-Happiness (smiling mouth) from other stimuli, with X-Disgust and X-Sadness (closed mouth) at the other extreme; D3 opposed X-Anger (compressed mouth) and X-Fear, X-Surprise (open mouth). Other dimensions were upper-face: D2 opposed Fear-X, Surprise-X (wide-opened eyes) and other expressions; D4 separated Fear-X from Surprise-X, and Sadness-X from others; D5 separated Anger-X from others.

Analysis of SPS clusters was carried out in D1, D2, D3 and D4 coordinate planes. It revealed that the structure of localization of composite expressions for upright faces reflected two types of grouping: first, a clustering on the basis of definite expressions (not on the basis of diagnostic features) for all types of composite faces; second, a clustering on the basis of expression valence. We believe that these types of clustering may specify holistic processing. An example of space structure for the D3 and D4 dimensions is shown in Figure 3 (the left figure for upright faces and the right figure for inverted faces), each point in space denoted by an acronym consisting of one or two letters identifying the basic or composite expression. For example, the acronym Su_A means a composite expression with Surprise in the upper half-face and Anger in the lower half-face. As can be seen in Figure 3 clustering of composite expressions for upright faces (the left figure) is observed on the basis of definite expressions while clustering for inverted faces (the right figure) occurred on the basis of diagnostic features. Also the comparison of SPS structures for upright and inverted faces showed differences in the degree of differentiation between expressions: the less differentiated structure with larger clusters was observed for upright faces while the space structure for inverted faces showed many separate small clusters. In Figure 3 clusters are highlighted with black contours. The analysis of the emergence of a “new Gestalt” of the expression not associated with the expressions reflected in the upper and lower parts of the face, is revealed. For example, the composite expression “a16_H”, consisting of Anger in upper and Happiness in lower half-faces, is localized in space by itself and is not surrounded by composite expressions with diagnostic features of Anger or Happiness.

The analysis of expressions grouping in D6 coordinate planes revealed the analytic type of clustering: composite expressions clustered around a diagnostic feature (e.g., a smile or eyes wide open). Factor analysis (exploratory), namely principal components analysis with varimax rotation, allowed the suggestion of a greater contribution of the first factors, in contrast to the rest of factors (this solution accounted for 41% of the total variance).

Figure 3

The SPS Structure of Composite Expressions for Upright (Upper) and Inverted (Lower) Faces

Upright faces bearing composite expressions

Note. Each point in both spaces denoted by the acronym consisting of one or two letters identifying the basic or composite expression. Analytic and holistic types of clusters are highlighted with black contours.

Conclusion

In general, the analysis of the structure of space of perceptual similarities between basic and composite expressions for upright and inverted faces enabled the following conclusions.

There are structural and topological changes in space structure of composite and basic expressions when the face orientation is altered.

It was revealed that for upright faces clustering of expressions on the basis of definite expressions and on the basis of expression valence is more often observed. The most illustrative example is grouping of composite expressions around the basic expression Happiness that is formed not in accordance with the diagnostic feature “smile”.

The effect was shown of emergence of a “new expression Gestalt”: in the SPS space the composite expression is not surrounded by the expressions reflected in the upper and lower parts of the face.

But for inverted faces clustering of expressions on the basis of diagnostic features is observed more frequently. Also more differentiated clustering prevailed.

Also it should be noted that the dimensionality of space of perceptual similarities increases when faces bearing composite expressions are used as stimuli. This result may be due to the complexity of the task to evaluate unrealistic composite expressions.

In our point of view our results showed that holistic processing plays a more important role in the evaluation of expressions similarity for upright faces. This is manifested in the fact that clustering specific for analytic processing (by diagnostic features) is observed only for two dimensions of the SPS space, while for the other four dimensions holistic clustering is more pronounced. When the natural conditions of perception are distorted (inverted faces), the role of the analytic processing increases.

Our study showed a high efficiency of the method of multidimensional scaling in studying the impact of analytic and holistic processing on expression discrimination. Also it should be mentioned that the simultaneous use of the composite face effect and the face inversion effect showed itself as productive for investigating the interaction between analytic and holistic processing of face expressions. Both analytic and holistic processing is involved in the encoding of facial expressions, but their impact depends on the natural conditions of perception (upright/inverted faces).

References

Bimler, D. L., & Paramei, G. V. (2006). Facial-expression affective attributes and their configural correlates: components and categories. The Spanish Journal of Psychology, 9(1), 19-31.

Bruce, V., & Young, A. (1998). In the eye of the beholder: the science of face perception. New York: Oxford University Press.

Calder, A., Rhodes, G., Johnson, M., & Haxby, J. (Eds.). (2011). Features, configuration and holistic face processing. In Oxford handbook offace perception (pp. 177-195). New York: Oxford University Press.

Calvo, M. G., & Lundqvist, D. (2008). Facial expressions of emotion (KDEF): Identification under different display-duration conditions. Behavior Research Methods, 40(1), 109-115.

Damasio, A. R. (1994). Descartes' error and the future of human life. Scientific American, 271(4), 144.

Donnelly, N., & Davidoff, J. (1999). The mental representations of faces and houses: Issues concerning parts and wholes. Visual Cognition, 6(3-4), 319-343.

Duchaine, B. C., & Nakayama, K. (2006). Developmental prosopagnosia: A window to content-specific face processing. Current Opinion in Neurobiology, 16(2), 166-173.

Ekman, P. (1976). Pictures of facial affect. Palo Alto, CA: Consulting Psychologists Press.

Falikman, M. (2014). Cognition and its master: New challenges for cognitive science. In A. Yasnitsky, R. van der Veer, & M. Ferrari (Eds.), Cambridge handbook of cultural-historical psychology (pp. 474-487). Cambridge: Cambridge University Press.

Goleman, D. (2006). Working with emotional intelligence (10th anniversary hardcover ed.). New York: Bantam Books.

Harari, Y. N. (2018). Davos 2018-Harari: An algorithm will be your best therapist, but it can be hacked too. Retrieved from https://www.youtube.com/watch?v=rIiHOu1tezI

Kanwisher, N., & Yovel, G. (2006). The fusiform face area: a cortical region specialized for the perception of faces. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 361(1476), 2109-2128.

Martinez, A., & Du, S. (2012). A model of the perception of facial expressions of emotion by humans: Research overview and perspectives. Journal of Machine Learning Research, 13(May), 1589-1608.

Maurer, D., Le Grand, R., & Mondloch, C. J. (2002). The many faces of configural processing. Trends in Cognitive Sciences, 6(6), 255-260.

Menshikova, G. Ya. (2010). Facial expression recognition with the use of chimeric face technique. Psychology in Russia: State of the Art, 3(1), 278-286.

Menshikova, G. Ya., Bimler, D., Bondarenko, Ya., & Paramei, G. (2016). Composite facial expressions: half-face diagnostic features dominate emotion discrimination. Perception, 45(2, Suppl.), 235-236.

Michel, C., Caldara, R., & Rossion, B. (2006). Same-race faces are perceived more holistically than other-race faces. Visual Cognition, 14(1), 55-73.

Piepers, D., & Robbins, R. (2012). A review and clarification of the terms “holistic,” “configural,” and “relational” in the face perception literature. Frontiers in Psychology, 3,559.

doi:10.3389/fpsyg.2012.00559

Rezlescu, C., Susilo, T., Wilmer, J. B., & Caramazza, A. (2017). The inversion, part-whole, and composite effects reflect distinct perceptual mechanisms with varied relationship to face recognition. Journal of Experimental Psychology: Human Perception and Performance, 43(12), 1961-1973.

Richler, J. J., & Gauthier, I. (2014). A meta-analysis and review of holistic face processing. Psychological Bulletin, 140(5), 1281-1302. doi:10.1037/a0037004

Rothermund, K., & Koole, S. L. (2018). Three decades of Cognition and Emotion: A brief review of past highlights and future prospects. Cognition and Emotion, 32(1), 1-12.

Scholsberg, H. (1941). A scale for the judgment of facial expressions. Journal of Experimental Psychology, 29(6), 497-510.

Shepard, R. N. (1962). The analysis of proximities: multidimensional scaling with an unknown distance function. Psychometrika, 27(2), 125-140.

Sokolov, E. N., & Boucsein, W. (2000). A psychophysiological model of emotion space. Integrative Physiological and Behavioral Science, 35(2), 81-119.

Tanaka, J. W., & Gordon, I. (2011). Features, configuration, and holistic face processing. In G. Rhodes, A. Calder, M. Johnson, & J. V. Haxby (Eds.), Oxford handbook of face perception (pp. 177-194). New York: Oxford University Press.

Tanaka, J. W., Kaiser, M. D., Butler, S., & Le Grand, R. (2012). Mixed emotions: Holistic and analytic perception of facial expressions. Cognition and Emotion, 26(6), 961-977.

doi: 10.1080/02699931.2011.630933

Young, A. W., & Bruce, V. (2011). Understanding person perception. British Journal of Psychology, 102(4), 959-974.

Yakov A. Bondarenko - PhD Student, Faculty of Psychology, Lomonosov Moscow State University; Senior Researcher, OOO LKL (F2FGroup); Lecturer, The Russian Presidential Academy of National Economy and Public Administration.

Research Area: general psychology, psychology of personality, methodology of psychology, cognitive psychology and neurosience, affective computing.

Galina Ya. Menshikova - Head of Laboratory, Laboratory “Perception”, Faculty of Psychology, Lomonosov Moscow State University, DSc.

Research Area: visual perception, space perception, facial recognition and facial expression, eye movement, application of virtual reality technologies in psychology.

Холистический и аналитический процессы при восприятии лицевых экспрессий: метод многомерного шкалирования

Я.А. Бондаренко, Г.Я. Меньшикова

Резюме

Современная научная литература характеризуется противоречивостью в отношении изучения двух основных механизмов (процессов) восприятия, которые играют значительную роль в процессах идентификации и распознавания выражений лица - аналитического (восприятие индивидуальных особенностей лица) и целостного (восприятие лица как гештальта). Несмотря на большое количество исследований, вопрос об их относительном вкладе в распознавание мимики остается открытым. В настоящей работе представлен анализ основных методологических трудностей изучения проблемы роли механизмов в восприятии экспрессий лица и результаты эмпирического исследования. Мы изучали взаимодействие аналитической и целостной обработки с использованием методов композитных и инвертированных изображений лица в решении задачи по оценке сходства экспрессий.

Для достижения этой цели использовался метод многомерного шкалирования, который позволяет построить субъективное пространство сходств выражений, а затем проанализировать кластеризацию выражений в нормальном и инвертированном условии. Результаты показали, что: 1) в правильных условиях презентации лица целостная обработка играет более важную роль; 2) в перевернутых условиях роль аналитической обработки возрастает, что проявляется в изменении кластеризации выражений в субъективном пространстве сходства выражений; 3) метод многомерного шкалирования полезен для исследования взаимодействия аналитических и целостных механизмов кодирования выражений лица; 4) методы одновременного использования композитных и инвертированных лиц являются эффективным инструментом для изменения влияния аналитического и целостного процессов при оценке сходства выражений.

Ключевые слова: распознавание лица, базовые и композитные лицевые экспрессии, аналитическая и холистическая обработка, ивертированные лица, метод многомерного шкалирования, ключевые признаки.

Бондаренко Яков Александрович - аспирант, кафедра психологии личности, факультет психологии, МГУ имени М.В. Ломоносова; старший научный сотрудник, ООО ЛКЛ (F2FGroup); преподаватель, Российская академия народного хозяйства и государственной службы при Президенте Российской Федерации.

Сфера научных интересов: общая психология, психология личности, методология психологии, когнитивная психология, affective computing.

Меньшикова Галина Яковлевна - заведующая лабораторией, лаборатория «Восприятие», факультет психологии, МГУ имени М.В. Ломоносова, доктор психологических наук. Сфера научных интересов: зрительное восприятие, восприятие пространства, зрительные иллюзии, распознавание лиц и лицевых экспрессий, использование технологий виртуальной реальности и айтрекинга в психологических исследованиях.

Размещено на Allbest.ru