Cognitive Models of Polycode Texts: A Comprehensive Analysis
Abstract
Polycode texts, which combine verbal and non-verbal elements, are increasingly prevalent in modern communication. This study investigates the cognitive models underlying the comprehension of polycode texts, aiming to elucidate the mental processes involved in integrating multiple semiotic systems. Using a mixed-methods approach, we conducted experiments with 120 participants (Mage = 28.5, SD = 4.2) to assess their comprehension of various polycode texts. Eye-tracking data and think-aloud protocols were collected and analyzed using both quantitative and qualitative methods. Results indicate that successful polycode text comprehension involves a complex interplay of visual attention, verbal processing, and cognitive integration. A novel "Integrated Polycode Comprehension Model" (IPCM) is proposed, synthesizing elements from dual coding theory and cognitive load theory. The IPCM suggests that comprehension is optimized when verbal and non-verbal elements are semantically congruent and spatially proximate. Furthermore, individual differences in cognitive styles significantly influenced comprehension patterns. These findings have important implications for the design of educational materials, user interfaces, and multimodal communication strategies. Future research directions and practical applications are discussed.
Keywords
Polycode texts, cognitive models, multimodal comprehensionHow to Cite
References
Ainsworth, S. (2006). DeFT: A conceptual framework for considering learning with multiple representations. Learning and Instruction, 16(3), 183-198.
Ainsworth, S., Prain, V., & Tytler, R. (2011). Drawing to learn in science. Science, 333(6046), 1096-1097.
Baddeley, A. D., & Hitch, G. (1974). Working memory. In G.H. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory (Vol. 8, pp. 47-89). Academic Press.
Bateman, J. A. (2014). Text and image: A critical introduction to the visual/verbal divide. Routledge.
Bertin, J. (1983). Semiology of graphics: Diagrams, networks, maps. University of Wisconsin Press.
Boucheix, J. M., & Lowe, R. K. (2010). An eye tracking comparison of external pointing cues and internal continuous cues in learning with complex animations. Learning and Instruction, 20(2), 123-135.
Carney, R. N., & Levin, J. R. (2002). Pictorial illustrations still improve students' learning from text. Educational Psychology Review, 14(1), 5-26.
Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8(4), 293-332.
Chandler, P., & Sweller, J. (1992). The split-attention effect as a factor in the design of instruction. British Journal of Educational Psychology, 62(2), 233-246.
Clark, J. M., & Paivio, A. (1991). Dual coding theory and education. Educational Psychology Review, 3(3), 149-210.
Eitel, A., & Scheiter, K. (2015). Picture or text first? Explaining sequence effects when learning with pictures and text. Educational Psychology Review, 27(1), 153-180.
Forceville, C. (2020). Visual and multimodal communication: Applying the relevance principle. Oxford University Press.
Hegarty, M. (2011). The cognitive science of visual-spatial displays: Implications for design. Topics in Cognitive Science, 3(3), 446-474.
Hegarty, M., & Just, M. A. (1993). Constructing mental models of machines from text and diagrams. Journal of Memory and Language, 32(6), 717-742.
Holsanova, J., Holmberg, N., & Holmqvist, K. (2009). Reading information graphics: The role of spatial contiguity and dual attentional guidance. Applied Cognitive Psychology, 23(9), 1215-1226.
Horz, H., & Schnotz, W. (2010). Cognitive load in learning with multiple representations. In J. L. Plass, R. Moreno, & R. Brünken (Eds.), Cognitive load theory (pp. 229-252). Cambridge University Press.
Jarodzka, H., Scheiter, K., Gerjets, P., & van Gog, T. (2010). In the eyes of the beholder: How experts and novices interpret dynamic stimuli. Learning and Instruction, 20(2), 146-154.
Jewitt, C., Bezemer, J., & O'Halloran, K. (2016). Introducing multimodality. Routledge.
Johnson, C. I., & Mayer, R. E. (2012). An eye movement analysis of the spatial contiguity effect in multimedia learning. Journal of Experimental Psychology: Applied, 18(2), 178-191.
Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23-31.
Kosslyn, S. M. (1989). Understanding charts and graphs. Applied Cognitive Psychology, 3(3), 185-225.
Kress, G., & Van Leeuwen, T. (2006). Reading images: The grammar of visual design (2nd ed.). Routledge.
Lowe, R. K. (2003). Animation and learning: Selective processing of information in dynamic graphics. Learning and Instruction, 13(2), 157-176.
Mayer, R. E. (2009). Multimedia learning (2nd ed.). Cambridge University Press.
Mayer, R. E., & Fiorella, L. (2014). Principles for reducing extraneous processing in multimedia learning: Coherence, signaling, redundancy, spatial contiguity, and temporal contiguity principles. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 279-315). Cambridge University Press.
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43-52.
Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The role of modality and contiguity. Journal of Educational Psychology, 91(2), 358-368.
Paivio, A. (1986). Mental representations: A dual coding approach. Oxford University Press.
Rau, M. A., Aleven, V., & Rummel, N. (2015). Successful learning with multiple graphical representations and self-explanation prompts. Journal of Educational Psychology, 107(1), 30-46.
Schnotz, W. (2014). Integrated model of text and picture comprehension. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 72-103). Cambridge University Press.
Schnotz, W., & Bannert, M. (2003). Construction and interference in learning from multiple representation. Learning and Instruction, 13(2), 141-156.
Schnotz, W., & Kürschner, C. (2008). External and internal representations in the acquisition and use of knowledge: Visualization effects on mental model construction. Instructional Science, 36(3), 175-190.
Sweller, J., van Merriënboer, J. J., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251-296.
Tufte, E. R. (2001). The visual display of quantitative information (2nd ed.). Graphics Press.
Tversky, B., Morrison, J. B., & Betrancourt, M. (2002). Animation: Can it facilitate? International Journal of Human-Computer Studies, 57(4), 247-262.
Van der Meij, H., & de Jong, T. (2006). Supporting students' learning with multiple representations in a dynamic simulation-based learning environment. Learning and Instruction, 16(3), 199-212.
Van Gog, T., Paas, F., Marcus, N., Ayres, P., & Sweller, J. (2009). The mirror neuron system and observational learning: Implications for the effectiveness of dynamic visualizations. Educational Psychology Review, 21(1), 21-30.
Vekiri, I. (2002). What is the value of graphical displays in learning? Educational Psychology Review, 14(3), 261-312.
Winn, W. (1991). Learning from maps and diagrams. Educational Psychology Review, 3(3), 211-247.
Wu, H. K., & Puntambekar, S. (2012). Pedagogical affordances of multiple external representations in scientific processes. Journal of Science Education and Technology, 21(6), 754-767.
Yoon, S. Y., & Narayanan, N. H. (2004). Mental imagery in problem solving: An eye tracking study. In Proceedings of the 2004 symposium on Eye tracking research & applications (pp. 77-84). ACM.
Zhang, J., & Norman, D. A. (1994). Representations in distributed cognitive tasks. Cognitive Science, 18(1), 87-122.
Zhu, L., & Grabowski, B. L. (2006). Web-based animation or static graphics: Is the extra cost of animation worth it? Journal of Educational Multimedia and Hypermedia, 15(3), 329-347.
License
Copyright (c) 2025 Odilova Kamola Takhirovna
This work is licensed under a Creative Commons Attribution 4.0 International License.