{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T16:32:16Z","timestamp":1753893136011,"version":"3.41.2"},"reference-count":47,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2022,6,22]],"date-time":"2022-06-22T00:00:00Z","timestamp":1655856000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Virtual Real."],"abstract":"<jats:p>This paper provides new research perspectives in the field of multimodal comprehension (auditory crossing visual information) by using immersion and incorporating eye tracking in a virtual reality environment. The objective is to investigate the influence of a change in narrative perspective (point of view) during the activation of a mental model underlying comprehension between visual and auditory modalities. Twenty-eight participants, equipped with a headset SMI HMD HTC eye-tracking 250\u00a0Hz watched 16 visual scenes in virtual reality accompanied by their corresponding auditory narration. The change in perspective may occur either in the visual scenes or in listening. Mean fixations durations on typical objects of the visual scenes (Area of Interest) that were related to the perspective shift were analyzed as well as the free recall of narratives. We split each scene into three periods according to different parts of the narration (Before, Target, After), the target was where a shift in perspective could occur. Results shown that when a visual change of perspective occurred, mean fixation duration was shorter (compared to no change) for both Target and After. However, when auditory change of perspective occurred, no difference was found on Target, although during After, mean fixation duration was longer (compared to no change). In the context of 3D video visualization, it seems that auditory processing prevails over visual processing of verbal information: The visual change of perspective induces less visual processing of the Area of Interest (AOIs) included in the visual scene, but the auditory change in perspective leads to increased visual processing of the visual scene. Moreover, the analysis showed higher recall of information (verbatim and paraphrase) when an auditory change in perspective was coupled with no visual change of perspective. Thus, our results indicate a more effective integration of information when there is an inconsistency between the narration heard and viewed. A change in perspective, instead of creating comprehension and integration difficulties, seems to effectively raise the attention and induce a shorter visual inspection. These results are discussed in the context of cross-modal comprehension.<\/jats:p>","DOI":"10.3389\/frvir.2022.874054","type":"journal-article","created":{"date-parts":[[2022,6,22]],"date-time":"2022-06-22T10:23:51Z","timestamp":1655893431000},"update-policy":"https:\/\/linproxy.fan.workers.dev:443\/https\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Eye Movements During Comprehension in Virtual Reality: The Influence of a Change in Point of View Between Auditory and Visual Information in the Activation of a Mental Model"],"prefix":"10.3389","volume":"3","author":[{"given":"V\u00e9ronique","family":"Drai-Zerbib","sequence":"first","affiliation":[]},{"given":"L\u00e9a","family":"Bernigaud","sequence":"additional","affiliation":[]},{"given":"Alexandre","family":"Gaston-Bellegarde","sequence":"additional","affiliation":[]},{"given":"Jean-Michel","family":"Boucheix","sequence":"additional","affiliation":[]},{"given":"Thierry","family":"Baccino","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2022,6,22]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"1061","DOI":"10.1037\/0278-7393.19.5.1061","article-title":"Updating a Mental Model: Maintaining Both Local and Global Coherence","volume":"19","author":"Albrecht","year":"1993","journal-title":"J. Exp. Psychol. Learn. Mem. Cognition"},{"key":"B2","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1027\/\/1016-9040.3.1.51","article-title":"Spatial Encoding and Referential Processing during Reading","volume":"3","author":"Baccino","year":"1998","journal-title":"Eur. Psychol."},{"key":"B3","doi-asserted-by":"crossref","DOI":"10.1145\/3313831.3376872","article-title":"Investigating Representation of Text and Audio in Educational VR Using Learning Outcomes and EEG","author":"Baceviciute","year":"2020"},{"key":"B4","doi-asserted-by":"publisher","first-page":"104122","DOI":"10.1016\/j.compedu.2020.104122","article-title":"Remediating Learning from Non-immersive to Immersive Media: Using EEG to Investigate the Effects of Environmental Embeddedness on Reading in Virtual Reality","volume":"164","author":"Baceviciute","year":"2021","journal-title":"Comput. Educ."},{"key":"B5","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1007\/s10339-018-0894-1","article-title":"Referential Processing during Reading: Concurrent Recordings of Eye Movements and Head Motion","volume":"20","author":"Ballenghein","year":"2019","journal-title":"Cogn. Process"},{"key":"B6","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1016\/s0022-5371(79)90118-x","article-title":"Point of View in Narrative Comprehension, Memory, and Production","volume":"18","author":"Black","year":"1979","journal-title":"J. Verbal Learn. Verbal Behav."},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.16910\/jemr.12.1.3","article-title":"Eye Tracking in Virtual Reality","volume":"12","author":"Clay","year":"2019","journal-title":"J. Eye Mov. Res."},{"volume-title":"Attention and Memory: An Integrated Framework","year":"1995","author":"Cowan","key":"B8"},{"key":"B9","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1016\/j.psfr.2014.12.002","article-title":"Effets On-Line d'un environnement musical dans la lecture de texte : analyse oculom\u00e9trique","volume":"62","author":"Drai-Zerbib","year":"2017","journal-title":"Psychol. Fran\u00e7aise"},{"key":"B10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3758\/bf03203630","article-title":"GPOWER: A General Power Analysis Program","volume":"28","author":"Erdfelder","year":"1996","journal-title":"Behav. Res. Methods, Instrum. Comput."},{"key":"B11","doi-asserted-by":"publisher","first-page":"1149","DOI":"10.3758\/brm.41.4.1149","article-title":"Statistical Power Analyses Using G*Power 3.1: Tests for Correlation and Regression Analyses","volume":"41","author":"Faul","year":"2009","journal-title":"Behav. Res. Methods"},{"key":"B12","article-title":"Aider les \u00e9l\u00e8ves \u00e0 comprendre, du texte au multim\u00e9dia","volume-title":"Editions Hachette Education","author":"Fayol","year":"2003"},{"key":"B40","first-page":"233","article-title":"Causality and the Allocation of Attention During Comprehension","volume-title":"J. Exp. Psychol.: Learn. Mem. Cogn.","author":"Fletcher","year":"1990"},{"key":"B13","first-page":"183","author":"Genette","year":"1972","journal-title":"Discours du R\u00e9cit: Mode"},{"key":"B43","doi-asserted-by":"crossref","DOI":"10.21236\/ADA221854","author":"Gernsbacher","year":"1990","journal-title":"Language Comprehension as Structure Building"},{"key":"B44","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1037\/h0084720","article-title":"The Mechanisms of Suppression and Enhancement in Comprehension","volume":"36","author":"Gernsbacher","year":"1995","journal-title":"Can. Psychol."},{"key":"B45","first-page":"289","article-title":"The Structure-Building Framework: What it is, what it Might Also be, and Why","author":"Gernsbacher","year":"1996","journal-title":"Models of Understanding Text"},{"key":"B14","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1146\/annurev.psych.48.1.163","article-title":"Discourse Comprehension","volume":"48","author":"Graesser","year":"1997","journal-title":"Annu. Rev. Psychol."},{"key":"B15","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1016\/j.concog.2016.09.001","article-title":"Narrative Perspective Shift at Retrieval: The Psychological-Distance-Mediated-Effect on Emotional Intensity of Positive and Negative Autobiographical Memory","volume":"45","author":"Gu","year":"2016","journal-title":"Conscious. cognition"},{"unstructured":"The Effects of Multimodality on L2 Learners: Implications for CALL Resource Design\n            GuichonN.\n            McLornanS.\n          2008","key":"B16"},{"key":"B42","doi-asserted-by":"publisher","first-page":"76","DOI":"10.2307\/747739","article-title":"Processing of Topic Shifts by Adults and Children","volume":"29","author":"Hy\u00f6n\u00e4","year":"1994","journal-title":"Read. Res. Q."},{"key":"B17","doi-asserted-by":"publisher","first-page":"1365","DOI":"10.1037\/0278-7393.21.5.a","article-title":"An Eye Movement Analysis of Topic-Shift Effect during Repeated Reading","volume":"21","author":"Hy\u00f6n\u00e4","year":"1995","journal-title":"J. Exp. Psychol. Learn. Mem. Cognition"},{"year":"1998","author":"Hy\u00f6n\u00e4","article-title":"Strategies for Processing an Expository Text for Recall: Evidence from Eye Fixation Patterns","key":"B18"},{"key":"B47","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1016\/0001-6918(90)90026-C","article-title":"Eye Movements During Repeated Reading of a Text","volume":"73","author":"Hy\u00f6n\u00e4","year":"1990","journal-title":"Acta Psychol."},{"key":"B19","doi-asserted-by":"publisher","first-page":"548","DOI":"10.3758\/pbr.15.3.548","article-title":"Characteristic Sounds Facilitate Visual Search","volume":"15","author":"Iordanescu","year":"2008","journal-title":"Psychonomic Bull. Rev."},{"key":"B20","doi-asserted-by":"publisher","first-page":"339","DOI":"10.1177\/0305735610387777","article-title":"Eye Movements and Reading Comprehension while Listening to Preferred and Non-preferred Study Music","volume":"40","author":"Johansson","year":"2012","journal-title":"Psychol. Music"},{"key":"B21","article-title":"La th\u00e9orie des mod\u00e8les mentaux","volume-title":"Les mod\u00e8les mentaux: Approche cognitive des repr\u00e9sentations","author":"Johnson-Laird","year":"1993"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog0401_4","article-title":"Mentel Models in Cognitive Science","volume":"4","author":"Johnson-Laird","year":"1980","journal-title":"Cognitive Sci. A Multidiscip. J."},{"key":"B23","doi-asserted-by":"publisher","first-page":"306","DOI":"10.3389\/fnhum.2016.00306","article-title":"Mobile Brain\/Body Imaging (MoBI) of Physical Interaction with Dynamically Moving Objects","volume":"10","author":"Jungnickel","year":"2016","journal-title":"Front. Hum. Neurosci."},{"year":"1973","author":"Kahneman","journal-title":"Attention and Effort","key":"B46"},{"volume-title":"Comprehension: A Paradigm for Cognition","year":"1998","author":"Kintsch","key":"B24"},{"key":"B25","doi-asserted-by":"publisher","first-page":"346","DOI":"10.1111\/j.1756-8765.2010.01107.x","article-title":"The Construction of Meaning","volume":"3","author":"Kintsch","year":"2011","journal-title":"Top. Cognitive Sci."},{"key":"B26","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1037\/0033-295X.95.2.163","article-title":"The Role of Knowledge in Discourse Comprehension: A Construction-Integration Model","volume":"95","author":"Kintsch","year":"1988","journal-title":"Psychol. Rev."},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295x.85.5.363","article-title":"Toward a Model of Text Comprehension and Production","volume":"85","author":"Kintsch","year":"1978","journal-title":"Psychol. Rev."},{"key":"B41","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1080\/01638538709544659","article-title":"Task Effects and Individual Differences in On-Line Processing of the Topic Structure of a Text","volume":"10","author":"Lorch","year":"1987","journal-title":"Discourse Process."},{"key":"B28","doi-asserted-by":"publisher","first-page":"e0214944","DOI":"10.1371\/journal.pone.0214944","article-title":"Equivalence of Using a Desktop Virtual Reality Science Simulation at Home and in Class","volume":"14","author":"Makransky","year":"","journal-title":"PLoS One"},{"key":"B29","doi-asserted-by":"publisher","first-page":"2079","DOI":"10.1111\/bjet.12954","article-title":"Can an Immersive Virtual Reality Simulation Increase Students' Interest and Career Aspirations in Science?","volume":"51","author":"Makransky","year":"2020","journal-title":"Br. J. Educ. Technol."},{"key":"B30","doi-asserted-by":"publisher","first-page":"937","DOI":"10.1007\/s10648-020-09586-2","article-title":"The Cognitive Affective Model of Immersive Learning (CAMIL): a Theoretical Research-Based Model of Learning in Immersive Virtual Reality","volume":"33","author":"Makransky","year":"2021","journal-title":"Educ. Psychol. Rev."},{"key":"B31","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1016\/j.learninstruc.2017.12.007","article-title":"Adding Immersive Virtual Reality to a Science Lab Simulation Causes More Presence but Less Learning","volume":"60","author":"Makransky","year":"","journal-title":"Learn. Instr."},{"key":"B32","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/9427.001.0001","volume-title":"Computer Games for Learning: An Evidence-Based Approach","author":"Mayer","year":"2014"},{"key":"B33","doi-asserted-by":"publisher","first-page":"837","DOI":"10.1007\/s11423-020-09749-6","article-title":"Five Ways to Increase the Effectiveness of Instructional Video","volume":"68","author":"Mayer","year":"2020","journal-title":"Educ. Tech Res. Dev"},{"key":"B34","doi-asserted-by":"publisher","first-page":"390","DOI":"10.3758\/s13421-015-0575-6","article-title":"Semantic Congruency but Not Temporal Synchrony Enhances Long-Term Memory Performance for Audio-Visual Scenes","volume":"44","author":"Meyerhoff","year":"2016","journal-title":"Mem. Cogn."},{"key":"B35","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1080\/01638539509544930","article-title":"Causal Connectives Increase Inference Generation","volume":"20","author":"Millis","year":"1995","journal-title":"Discourse Process."},{"key":"B36","doi-asserted-by":"publisher","first-page":"104429","DOI":"10.1016\/j.compedu.2021.104429","article-title":"A Study of How Immersion and Interactivity Drive VR Learning","volume":"179","author":"Petersen","year":"2022","journal-title":"Comput. Educ."},{"key":"B37","doi-asserted-by":"publisher","first-page":"372","DOI":"10.1037\/0033-2909.124.3.372","article-title":"Eye Movements in Reading and Information Processing: 20 Years of Research","volume":"124","author":"Rayner","year":"1998","journal-title":"Psychol. Bull."},{"key":"B38","doi-asserted-by":"publisher","first-page":"401","DOI":"10.1016\/j.tins.2008.05.002","article-title":"Crossmodal Binding through Neural Coherence: Implications for Multisensory Processing","volume":"31","author":"Senkowski","year":"2008","journal-title":"Trends Neurosci."},{"key":"B39","doi-asserted-by":"publisher","first-page":"1053","DOI":"10.1037\/0096-1523.34.5.1053","article-title":"Pip and Pop: Nonspatial Auditory Signals Improve Spatial Visual Search","volume":"34","author":"Van der Burg","year":"2008","journal-title":"J. Exp. Psychol. Hum. Percept. Perform."}],"container-title":["Frontiers in Virtual Reality"],"original-title":[],"link":[{"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/www.frontiersin.org\/articles\/10.3389\/frvir.2022.874054\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,22]],"date-time":"2022-06-22T10:24:15Z","timestamp":1655893455000},"score":1,"resource":{"primary":{"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/www.frontiersin.org\/articles\/10.3389\/frvir.2022.874054\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,22]]},"references-count":47,"alternative-id":["10.3389\/frvir.2022.874054"],"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/doi.org\/10.3389\/frvir.2022.874054","relation":{},"ISSN":["2673-4192"],"issn-type":[{"type":"electronic","value":"2673-4192"}],"subject":[],"published":{"date-parts":[[2022,6,22]]},"article-number":"874054"}}