Automated assessment of discourse coherence in schizophrenia and schizoaffective disorder

Generally, PCA with PANSS scores results in PANSS defining the first PC and a relatively well-separated sample. Maximal and average global and semi-local coherence projections are almost exactly opposite PANSS scores projections, and thus it is reasonable to expect a negative correlation between them. Education projection is usually directed opposite PANSS scores projections.

The second PC on both types of PCA is generally defined by the range of cosine-metrics (maximal projections being opposite to the minimal). Age projection usually lies opposite maximal global coherence projection.

Interestingly, PCA plots for ELMo and SIF are noticeably more similar to each other than to BERT. A possible reason for this is that BERT provides ready-made sentence embeddings, while SIF and ELMo require averaging of the word vectors.

Below I describe the quality of separation for each task and vectorization method, as well as quantitative analysis where applicable. It is probable that PANSS scores would still correlate with some of the untested measures, but due to the small sample size, this study lacks statistical power to test all the measures against all PANSS scores.

6.3.1 Procedural Discourse (Chair Task)

In the PCA biplot on SIF vectorization with PANSS scores, the groups are relatively well-separated on the first PC, with several exceptions (such as H01, H04, S12, and S20).

As for the analysis that excludes PANSS scores, the first PC still has some explanatory potential, as most of the patients score low on it as compared to controls, although with the same exceptions: H01 and H04 are very far in the TD group, while S12 and S20 are closer to the control group. S12 and S20 are, indeed, more well-structured than most TD texts. H04 is notably different from most texts in that the participant addresses an imaginary friend, who was unfortunate to buy a chair without the brochure, rather than giving a more general instruction.

Given the relatively high degree of separation between the groups in эFigure 6, the metrics were analyzed for being able to separate the groups and for correlation with PANSS scores. The results of an independent t-test with 33 degrees of freedom can be found in эTable 13 and эTable 14. All minimal measures differed significantly between groups. All maximal measures but cumulative coherence differed significantly between groups. On the other hand, among mean measures, only cumulative coherence differed significantly between the groups.

Table 13.

Independent t-test results for SIF vectorization on Chair task for group effect. * - significant at 0.05, ** - significant at 0.01, *** - significant at 0.001.

Table 14 presents the correlations of the SIF-vectorized metrics with PANSS scores. None of the correlations proved significant after corrections for multiple testing.

Table 14.

Correlation of SIF vectorization metrics on Chair task with PANSS scores. * - significant at 0.05, ** - significant at 0.01, *** - significant at 0.001

The PCA of ELMo vectorization on Chair task revealed a strong separation between the groups, most notably on local and semi-local coherence, with S06 and S12 being closer to controls than others from the TD group. Given the quality of separation, a quantitative analysis was performed. The results are presented in э0 and эTable 16 below.

Table 15.

Independent t-test results for ELMo vectorization on chair task for group Independent t-test results for ELMo vectorization on Chair task for group effect. * - significant at 0.05, ** - significant at 0.01, *** - significant at 0.001

All minimal ELMo measures but cumulative coherence proved to be significantly different between the groups, while cumulative coherence differed in maximal values.

Table 16.

Correlation of ELMo vectorization metrics on Chair task with PANSS scores. * - significant at 0.05, ** - significant at 0.01, *** - significant at 0.001

Mean and maximal values of ELMo local and semi-local coherence were strongly or moderately negatively correlated with all PANSS subscales. Additionally, mean scores on global and control global coherence were moderately correlated with general psychopathological symptoms, while maximal global coherence score was moderately correlated with the TD score.

The PCA on BERT vectorization did not reveal any explanatory potential, with the groups mixed on the biplot. Thus, the scores were not analyzed quantitatively.

6.3.2 Personal Story (Gift Task)

Across vectorization methods, on PCA with PANSS scores, S13, S15, and S18 had atypical stories, closer to those of the control group, most likely because they are significantly longer than other stories in the TD group. The PCA without PANSS scores did not reveal any pattern with regards to diagnosis, and the scores were not tested for group differences. As for alignment with PANSS, only SIF and ELMo vectorization methods had such alignment. Maximal global, control global and cumulative coherence, as well as mean local and semi-local coherence were selected for analysis with SIF vectorization. The results are presented in эTable 17 below. None of the measures were significantly correlated

Table 17.

Correlation of SIF vectorization metrics on Gift task with PANSS scores. * - significant at 0.05, ** - significant at 0.01, *** - significant at 0.001

As for ELMo vectorization, maximal local, semi-local, globalб and control global coherence were negatively aligned with PANSS scores on the PCA biplot. The results of a Spearman's correlation test are presented in эTable 18. None of the measures were significantly correlated.

Table 18.

Correlation of ELMo vectorization metrics on Gift task with PANSS scores. * - significant at 0.05, ** - significant at 0.01, *** - significant at 0.001

6.3.3 Picture Description (Child Task)

None of the PCA biplots of purely linguistic measures clustered the groups, therefore no group test analysis was performed. Maximal coherence scores were opposite PANSS scores for both BERT and ELMo vectorizations. SIF vectorization showed slight positive alignment of maximal global coherence with PANSS scores, but insufficient for any further analysis.

The results of a correlation test on ELMo vectorization with PANSS scores are shown in эTable 19. Both global coherence metrics turned out to be moderately negatively correlated with all PANSS scores.

Table 19.

Correlation of ELMo vectorization metrics on Child task with PANSS scores. * - significant at 0.05, ** - significant at 0.01, *** - significant at 0.001

The results of a correlation test on BERT vectorization with PANSS scores are shown in эTable 20. Both local coherence metrics turned out to be moderately negatively correlated with all PANSS scores.

Table 20.

Correlation of BERT vectorization metrics on Child task with PANSS scores. * - significant at 0.05, ** - significant at 0.01, *** - significant at 0.001

As for qualitative patterns, H01 and H02 display extreme values. H01 uses several neologisms, which is atypical of the control sample, and H02 produces an extremely succinct summary of the story. S13, S15, and S18 also can be regarded as outliers, but the reasons for it are unclear.

6.3.4 Picture Description (Suit Task)

On PCA plots with PANSS scores H01, S13, and S15 lie outside the general area of the respective group. In two cases the differences can be explained by the story length: H01 produces a story almost twice as short as the others in the control group, while S13 makes an unusually long story.

Neither SIF nor ELMo PCA biplots show a clear separation between the groups when PANSS scores are taken out. However, BERT vectorization does to some extent separate the groups on the second PC, which is not aligned with PANSS scores. This requires further investigation.

эTable 21 presents the results of a t-test for group effects on Suit task. All the measures, but minimum and mean cumulative coherence differed significantly between the groups. The largest effect size can be seen in maximal global coherence score.

Table 21.

Independent t-test results for BERT vectorization on Suit task for group effect. * - significant at 0.05, ** - significant at 0.01, *** - significant at 0.001

э

Table 22 presents the results of a correlation test for BERT vectorization maximal coherence measures with PANSS scores. Maximal global coherence proved to moderately negatively correlate with all the scores, while local and semi-local coherence were only connected with the positive subscale.

Table 22.

Correlation of BERT vectorization metrics on Suit task with PANSS scores. * - significant at 0.05, ** - significant at 0.01, *** - significant at 0.001

ELMo did not show any measure - PANSS alignment on PCA biplot. SIF vectorization showed negative alignment with PANSS scores on maximal global and control global coherence. However, the correlation was not significant, as shown in эTable 23.

Table 23.

Correlation of SIF vectorization metrics on Child task with PANSS scores. * - significant at 0.05, ** - significant at 0.01, *** - significant at 0.001

6.4 Discussion

All the tasks in the study, but Gift task, showed group differences. However, in many cases, the differences shown by the metrics were not connected with PANSS scores. This might mean that PANSS misses some relevant TD phenomena. It is possible, that a scale more specifically targeted at language phenomena in TD, such as the thought language and communication scale or TLC developed in Andreasen, 1986, could be more informative. Unfortunately, TLC is yet to be validated for the Russian language. It is also worth noting, that the neurotypical population is rarely tested with PANSS, and that introduces some unknown variables to the picture.

I would argue that the personal story task (or the Gift task) was unsuccessful, as many participants had trouble recalling a memorable gift, and the interviewer had to encourage them. In some cases, this resulted in more of a dialogue, than a monologue, which could negatively impact the results. Further research is needed to investigate other possibilities for eliciting personal discourse, as well as other types of elicitation, such as retelling a commonly known story (e.g. the story of Cinderella).

The tasks proved to differ in terms of diagnostic informativeness and connection with PANSS scores.

Chair task and Suit task were quite successful both for predicting the group differences and for correlation with PANSS scores. Picture description task on child story was good for approximating PANSS scores, but not for separating the groups based on coherence.

Below I provide a more detailed description of the two test types.

6.5 Group Differences

Procedural discourse (Chair task) was the best for separating the groups on SIF and ELMo vectorizations, while the picture description Suit task was very well-separated on BERT vectorization. The summary of the differences between the groups on the selected measures is presented in table эTable 24.

Table 24.

Coherence measures different in the groups across tasks. “+++” - measure differed between the groups, “-” - measure did not differ between the groups, NT - not tested

Quite unexpectedly, different averaging methods were more efficient for different vectorizations: SIF-vectorized Chair task was best separated on minimal scores, while ELMo-vectorized Chair task and BERT-vectorized Suit task showed better results on maximal scores.

There was almost no difference between the metrics tested. Cumulative coherence was less predictive than other metrics (as in 4 cases all the other metrics were, but it was not). Understandably, across the tests, global coherence was very similar to control global coherence, and local coherence to semi-local coherence, although with some exceptions. To reduce the number of tests performed, only one of each pair should be used in the future, and it might be a good idea to use the traditional metrics (global and local coherence).

6.6 PANSS Scores

The summary of PANSS correlation tests is presented in эTable 25. Surprisingly, the vectorization methods proved to be more different than one could expect.

The picture description Child task proved to be the best for correlation with PANSS scores, as both tested vectorization methods, ELMo and BERT, showed correlation with all PANSS scores on selected methods. There was a difference in terms of the metrics that showed this correlation. For ELMo vectorization, they were maximal and control global coherence, while for BERT vectorization they were maximal local and semi-local coherence.

None of the measures proved to be correlated with PANSS scores for the personal story task. On Chair task, only ELMo vectorization was correlated with PANSS scores, namely maximal local and semi-local coherence. On Suit task, only BERT vectorization was correlated with PANSS scores, and only maximal global coherence was correlated with all the scales.

Overall, for correlation with PANSS the maximal scores proved to be by far the most efficient method for all metrics but cumulative coherence. Possibly, this is because maximal coherence is the lowest in the stories of people with high TD, as shown by PANSS scores. As mentioned above, cumulative coherence being the exception from the rule naturally follows from the fact that cumulative coherence is defined through minimal coherence pairs in the story.

39

Table 25.

Measures correlated with PANSS scores across tasks. NT - not tested, “-” - none of the measures

6.6.1 Connection between Group Differences and PANSS

Just like for verbal fluency task, PANSS scores and t-test results on coherence metrics were not directly connected across tasks, which could mean that PANSS does not capture some of the TD phenomena.

7. Intra-Experimental Analysis

The coherence scores that were correlated with more than three of PANSS subscales, were analyzed for connection with verbal fluency metrics. The results are presented in эTable 26. Only unique word number on verbal fluency task correlated with selected coherence scores on Chair task, namely with mean and maximal local and semi-local coherence.

The connection between the tested coherence metrics and verbal fluency assessment metrics was only significant for unique word count - local & semi-local coherence on Chair task with ELMo vectorization. The evidence from the intra-experimental analysis is in favor of positive TD being the reason for incoherence in the psychotic discourse, yet the evidence is very weak. It is important to note that many measures remain untested in the intra-experimental analysis, as the sample size does not allow for more tests.

53

Table 26.

Correlation of ELMo vectorization metrics on Chair task with verbal fluency task metrics. * - significant at 0.05, ** - significant at 0.01, *** - significant at 0.001. unique_num - unique word count, repeat_num - number of repetitions, mean_cluster_len - mean cluster length, min_cluster_len - minimal cluster length, max_cluster_len - maximal cluster length; mean_cos_sim - mean cosine similarity between adjacent words, min_cos_sim - minimal cosine similarity between adjacent words, max_cos_sim - maximal cosine similarity between adjacent words

8. Limitations

The biggest limitation of the present paper is the sample size. Collecting a large sample is quite problematic, particularly when it comes to an infrequent and severe disease like schizophrenia. Another complication is that people in the state of psychosis may refuse to collaborate or may be unable to respond to the questions, as it was the case for several tasks. Thus, a serious drawback to this study is that its statistical power is undetermined. This is further complicated by the fact that the expected effect size is unclear - this might mean that the sample size is too small for detecting the effects. On top of that, even with the corrections in place, multiple testing on all the effects makes the study preliminary and exploratory, rather than strictly confirmatory.

We could not afford excluding all people showing schizo-spectrum tendencies in their thought patterns, but in some cases, they clearly were closer to the TD group than other controls. Luckily, this did not completely undermine the group differences.

There are two issues arising from the psychiatric setting that can influence the results and that cannot be easily accounted for. These are medication effects, the affective components in schizoaffective disorder, and possibly the overall low TD levels. Unfortunately, we could not control for either of these issues.

9. Conclusion

This study shows that automated coherence measures from Elvevеg et al. (2007) and Bedi et al. (2015) can be successfully applied to spoken Russian discourse. The metrics used in this paper proved to be able to discriminate between the texts produced by patients in thought disorder group and the control texts, although with moderate effect size.

The analysis revealed that in many cases the differences captured by coherence were not correlated with PANSS scores, which means they can become a useful tool in addition to the standard test.

On verbal fluency task, most measures used differed significantly between the groups. The clustering measures were successfully automatically approximated. The quality of the automated clustering was comparable to the inter-rater agreement level, which means the approximation tool can be used instead of the time-consuming manual clustering.

The connection between the number of unique words produced on verbal fluency task and the coherence scores on the procedural discourse task can be regarded as weak evidence in favor of the PTD as the cause of discourse incoherence seen in psychosis.

10. References

1. Abbe et al., 2016 - Abbe, A., Grouin, C., Zweigenbaum, P., and Falissard, B. (2016). Text mining applications in psychiatry: A systematic literature review. International Journal of Methods in Psychiatric Research, 25(2):86- 100.

2. Andreasen, 1986 - Andreasen, N. C. (1986). Scale for the Assessment of Thought, Language, and Communication (TLC). Schizophrenia bulletin., 12(3):473-482.

3. Arora et al., 2017 - Arora, S., Liang, Y., & Ma, T. (2017). A simple but tough-to-beat baseline for sentence embeddings. In International Conference on Learning Representations

4. Bedi et al., 2015 - Bedi, G., Carrillo, F., Cecchi, G. A., Slezak, D. F., Sigman, M., Mota, N. B., Ribeiro, S., Javitt, D. C., Copelli, M., and Corcoran, C. M. (2015). Automated analysis of free speech predicts psychosis onset in high-risk youths. npj Schizophrenia, 1(1).

5. Bleuler, 1911 - Bleuler, E. (1911). Dementia praecox: oder Gruppe der Schizophrenien. F. Deuticke.

6. Bokat and Goldberg, 2003 - Bokat, C. E. and Goldberg, T. E. (2003). Letter and category ?uency in schizophrenic patients: a meta-analysis. Schizophrenia Research, 64(1):73-78.

7. Bora and Baysan Arabaci, 2009 - Bora, E., & Baysan Arabaci, L. (2009). Effect of age and gender on schizotypal personality traits in the normal population. Psychiatry and clinical neurosciences, 63(5), 663-669.

8. Coelho, C. A., & Flewellyn, L. (2003). Longitudinal assessment of coherence in an adult with fluent aphasia: A follow-up study. Aphasiology, 17(2), 173-182.

9. Cohen and Elvevеg, 2014 - Cohen, A. S. and Elvevеg, B. (2014). Automated computerized analysis of speech in psychiatric disorders. Current opinion in psychiatry, 27(3), 203.

10. Cohen et al., 2017 - Cohen, A. S., Le, T. P., Fedechko, T. L., and Elvevеg, B. (2017). Can RDoC help ?nd order in thought disorder? Schizophrenia bulletin, 43(3), 503-508.

11. Derogatis and Fitzpatrick, 2004 - Derogatis, L. R. and Fitzpatrick, M. (2004). The SCL-90-R, the Brief Symptom Inventory (BSI), and the BSI-18. In Handbook of psychological assessment in primary care settings, pages 297- 334. Lawrence Erlbaum Associates Publishers.

12. Devlin et al., 2018 - Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805.

13. Ditman and Kuperberg, 2010 - Ditman, T., & Kuperberg, G. R. (2010). Building coherence: A framework for exploring the breakdown of links across clause boundaries in schizophrenia. Journal of Neurolinguistics, 23(3), 254-269

14. Elvevеg et al., 2007 - Elvevеg, B., Foltz, P. W., Weinberger, D. R., and Goldberg, T. E. (2007). Quantifying incoherence in speech: An automated methodology and novel application to schizophrenia. Schizophrenia Research, 93(1-3):304-316.

15. Foltz, 1996 - Foltz, P. W. (1996). Latent semantic analysis for text- based research. Behavior Research Methods, Instruments, and Computers, 28(2):197-202.

16. Glosser and Deser, 1991 - Glosser, G. and Deser, T. (1991). Patterns of discourse production among neurological patients with ?uent language disorders. Brain and Language, 40(1):67-88.

17. Hart and Lewine, 2017 - Hart, M. and Lewine, R. R. (2017). Rethinking thought disorder. Schizophrenia Bulletin, 43(3):514-522.

18. He, 2013 - He, Q. (2013). Text mining and IRT for psychiatric and psychological assessment. Master's thesis, University of Twente.

19. Holshausen et al., 2014 - Holshausen, K., Harvey, P. D., Elvevеg, B., Foltz, P. W., and Bowie, C. R. (2014). Latent semantic variables are associated with formal thought disorder and adaptive behavior in older inpatients with schizophrenia. Cortex, 55:88 -96.

20. Iter et al., 2018 - Iter, D., Yoon, J., and Jurafsky, D. (2018). Automatic Detection of Incoherent Speech for Diagnosing Schizophrenia. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, pages 136-146, Stroudsburg, PA, USA. Association for Computational Linguistics.

21. Jucker, 1997 - Jucker, A. H. (1997). The discourse marker well in the history of English. English Language and Linguistics, 1(1):91-110.

22. Just et al., 2019 - Just, S., Haegert, E., Koшбnovб, N., BrЁocker, A.-L., Nenchev, I., Funcke, J., Montag, C., and Stede, M. (2019). Coherence models in schizophrenia. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology, pages 126-136.

23. Juhasz et al., 2012 - Juhasz, B. J., Chambers, D., Shesler, L. W., Haber, A., & Kurtz, M. M. (2012). Evaluating lexical characteristics of verbal fluency output in schizophrenia. Psychiatry research, 200(2-3), 177-183.

24. Kay et al., 1987 - Kay, S. R., Fiszbein, A., and Opler, L. A. (1987). The Positive and Negative Syndrome Scale (PANSS) for Schizophrenia. Schizophrenia Bulletin, 13(2):261-276.

25. Keefe, 2004 - Keefe, R. (2004). The Brief Assessment of Cognition in Schizophrenia: reliability, sensitivity, and comparison with a standard neurocognitive battery. Schizophrenia Research, 68(2-3):283-297.

26. Kim et al., 2019 - Kim, N., Kim, J.-H., Wolters, M. K., MacPherson, S. E., and Park, J. C. (2019). Automatic Scoring of Semantic Fluency. Frontiers in Psychology, 10.

27. Kioseva, 2016 - Kioseva, O. V. (2016). Психопатологическая характеристика эмоциональной сферы у студентов младших курсов (Psychopathological Characteristics of an Emotional Sphere of University First Year Students). Диагностика и лечение психических и наркологических расстройств (diagnosis and treatment of psychological and narcological disorders, 1(86):1- 4.

28. Koшбnovб, 2017 - Koшбnovб, N. (2017). Analyzing coherence in spontaneous speech of schizophrenic patients. Master's thesis, University of Potsdam.

29. Kraepelin, 1919 - Kraepelin, E. (1919). Dementia Praecox and Paraphrenia. Translated by RM Barclay. Krieger Publishing Company. (1971)

30. Kutuzov and Kuzmenko, 2016 - Kutuzov, A. and Kuzmenko, E. (2016). WebVectors: a toolkit for building web interfaces for vector semantic models. In International Conference on Analysis of Images, Social Networks and Texts, pages 155-161. Springer.

31. Kuperberg, 2010a - Kuperberg, G. R. (2010). Language in schizophrenia part 1: an introduction. Language and linguistics compass, 4(8), 576-589.

32. Kuperberg, 2010b - Kuperberg, G. R. (2010). Language in schizophrenia Part 2: What can psycholinguistics bring to the study of schizophrenia… and vice versa? Language and linguistics compass, 4(8), 590-604.

33. Merriam-Webster, 2015 - Merriam-Webster (2015). De?nition of Discourse.

34. Mikolov et al., 2013 - Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimat...