Multi-modal cognitive language model

There are some available resources of recorded human signals while people are reading, e.g. various eye-tracking [1,2,3], EEG [4] and fMRI [5] datasets. The goal of this project is to bring these resources of language understanding together, learn representations for each of them and combine them into a multi-modal language model. Such a cognitive language model is expected to have some advantages compared to standard neural language models [6]. These advantages and the quality of a cognitive language model are to be explored.

Supervised by Nora Hollenstein


  1. [1] Cop, U., Dirix, N., Drieghe, D., & Duyck, W. (2017). Presenting GECO: An eye-tracking corpus of monolingual and bilingual sentence reading. Behavior research methods, 49(2).
  2. [2] Kennedy, A., Pynte, J., Murray, W. S., & Paul, S. A. (2013). Frequency and predictability effects in the Dundee Corpus: An eye movement analysis. The Quarterly Journal of Experimental Psychology, 66(3).
  3. [3] Papoutsaki, A., Gokaslan, A., Tompkin, J., He, Y., & Huang, J. (2018, June). The eye of the typer: a benchmark and analysis of gaze behavior during typing. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications.
  4. [4] Hollenstein N., Rotsztejn J., Troendle M., Pedroni A., Zhang C. and Langer N. “ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading.” Scientific Data. 2018.
  5. [5] Wehbe, L., Murphy, B., Talukdar, P., Fyshe, A., Ramdas, A., & Mitchell, T. (2014). Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PloS one, 9(11).
  6. [6] Kiros, R., Salakhutdinov, R., & Zemel, R. (2014, January). Multimodal neural language models. In International Conference on Machine Learning.