Polysemiotic communication: Integrating language, gesture and depiction in three cultural practices

1. Purpose and aims

This research project aims to potentiate insights into the polysemiotic nature of human communication by investigating the integration of three distinct semiotic systems – language, gesture, and depiction – as instantiated in specific semiotic resources such as speaking, bodily movements and drawings. We aim to fill theoretical, methodological and empirical gaps in the field by analysing the communicative strategies used in three different cultural practices to convey abstract meanings related to kinship structures, mythologies and scientific knowledge. The three cultural practices under study are Paamese sand drawings (Vanuatu), sand story telling (Central Australia), and classroom teaching (Sweden). Despite obvious differences, all three serve to convey abstract knowledge related to kinship, mythology and science, and all combine language, gesture and depiction in what we term polysemiotic communication.

The Paamese cultural practice of sand drawing consists of tracing complex geometrical patterns with a finger in the sand augmented with gestures and verbal narration. Once it is complete, the sand drawer and the audience engage in a discussion about the theme of the drawing, while pointing to specific parts of it so as to elaborate or comment on certain aspects (Devylder 2014). As many as 138 different languages are spoken in Vanuatu, which makes it the country with the highest linguistic density in the world (François et al. 2015). This exceptional density may have motivated the emergence of this practice, which “developed as a form of communication and symbolic exchange within this dynamic cross-cultural environment. They facilitated the exchange of ideas between different language groups” Zagala (2004, p. 33), allowing the transmission of knowledge about “local histories, indigenous cosmologies, kinship systems, and scientific knowledge” (ibid, p. 32).

Central Australian sand story telling is a traditional form of communication that incorporates speech, gesture and depiction (drawing) employed sequentially or simultaneously. A sand storyteller “builds up layers of real and imagined spaces, using drawing, signs and gesture and by moving objects around the story space” (Green 2014, p.2). In contrast to Paamese sand drawings, which are performed in one continuous go, and elaborated afterwards once the geometrical pattern is complete, Central Australian sand stories are episodic in nature. During a performance the storyteller regularly erases the drawings so as to “clean the slate” to leave space for following narrative items. The motivation behind using the ground as a semiotic resource is ecologically understandable with regards to the landscape of the region but it also has a particular cultural significance. Aboriginal Australians are preoccupied with interpreting the information encoded on its surface whether it is animal footprints or signs left by the ancestors of the Dreamtime, an important dimension of Aboriginal Australian mythology: “a kind of time out of time, a time hidden beyond or even within the evident, manifest presence of the land […] It is that time before the world itself was entirely awake (a time that still exists just below the surface of wakeful awareness)” (Abram 1997, p.100). See also Green (2012).

Although Paamese and Central Australian sand drawings may, from a Western perspective, first appear as “exotic” and isolated cases of polysemiotic communication, the cultural practice of combining speech, body movements and drawings to communicate abstract knowledge is also very common in the Western world, as in Swedish classrooms. Mathematics or physics teachers, for example, systematically recruit verbal descriptions, pointing and other gestures, and graphic resources to explain their topics.

The concrete aims of the project are (a) to identify patterns of combining the three distinct semiotic systems into an integrated message, (b) to determine which of these patterns are specific to individual traditions, or even contexts, and which may obey general principles of human polysemiotic communication, and (c) to investigate the potentials of individual systems by studying the effects of reducing polysemiotic communication to only two of the three semiotic systems.

2. State-of-the-art

In addition to the three examples introduced in Section 1, communicative traditions combining language, gestures, and depiction are found across many different cultures.  DeMarrais et al. (1992) documented how the Yup’ik use a combination of singing and depiction by carving into a palette of mud with knives to convey legends or moral values. Enfield (2005) studied the conceptualization of kinship structures in Laos, with special attention to the gestural diagrams “drawn” in the air while giving verbal descriptions. He pointed out that while linguistic representations of kinship have received much attention within anthropology and linguistics, diagrammatic representations in gestures and drawings is an “intriguing domain which has received little scientific attention is the practice – qua cultural practice¬ – of visually representing kinship not only by anthropologists but also by lay people” (Enfield 2005, p.53). Similarly, Gaby (2017), was able to reveal Thaayorre’s kinship “organizing principles which would otherwise be invisible” (p. 160) when combining linguistic analysis with gesture analysis of hand signs. Polysemiotic strategies recruited to communicate about kinship systems have also been documented by Dousset (2003) for Ngaatjatjarra sand drawings (Australia) and Widlock et al. (2008) for speakers of ǂAkhoe Haiǁom (Namibia). Ellis, Green, & Kral (2017) also show how Ngaatjarra children play a game called “mama mama ngunytju ngunytju" that combines speech, gesture, and depiction to offer clues about target kinship referents that players have to guess.

On the basis of an integrative approach studying the interaction of language, gesture, and depiction as distinct semiotic systems in the cultural practises of Indigenous Australians as those described in Section 1, Green (2014, 2017) documents how they use the ground as a surface for tallying or listing of kin relations. For example, groups of siblings can be represented by a set of parallel lines joined by a line at right angles.
To turn to a different cultural practice, when an abstract concept is to be explained in a Western classroom, it is quite common to harness multiple semiotic systems. For example, a mathematics teacher would not only write the symbolic formula h=√(a^2+b^2 ) but also use speech to explain the Pythagorean Theorem, augment the verbal explanation with pragmatic and deictic gestures and point to the triangle drawn on the white board. As shown by Ochs, Gonzales, and Jacoby (1996, p. 328), scientists “build meaning through routine interpretative activity involving talk, gesture, and graphic representation”. Similarly, when Larkin & Simon (1987, p.72) gave their participants a complex physics problem, they report that every participant “reaches for pencil and paper, and draws a sketch of the situation.” 

Polysemiotic communication is not only recruited within cultures and across levels of expertise on a given topic but also across different cultures to communicate about kinship systems. For instance, Ambrymese consultants (Vanuatu) and Hanunóo speakers (Philippines) explained their traditional rules of marriage to Deacon (1927, p.332) and Conklin (1969, p.109) respectively, by drawing diagrams to augment their verbal explanations, shown in Figure 1 and Figure 2. Both Deacon’s and Conklin’s consultants managed the challenging task of communicating about a complex system of intertribal alliances to complete outsiders by recruiting the semiotic resources made available by the environment of the exchange and spontaneously combined the three semiotic systems that we focus on (i.e. language, gesture, and depiction).

In sum, when it appears that we have to understand and communicate about an abstract concept, we tend to harness the available semiotic resources to organize our thoughts and display them to others. These polysemiotic combinations provide a multidimensional sketchpad that allows the transmission of abstract notions to be successful across cultures and/or levels of expertise on a given topic. It is increasingly being acknowledged that an approach with adequate theoretical and methodological tools is required in order to provide insights into the nature of this process, especially when what is communicated involves abstract knowledge such as kinship structures and scientific knowledge.

To some degree, the phenomenon of polysemiotic communication has been previously studied under the heading of multimodality in two quite distinct senses of the term, in two non-overlapping research traditions. On the one hand, social semiotics (Kress 2009; Van Leeuwen 2005) has acknowledged the combination of an indefinite number of “modes” such as text, image, colour, gaze, pointing etc. in studies of almost exclusively Western social settings. The approach has generated a considerable degree of research, for example on “semiotic genres” such as advertising, but is limited theoretically by notions that are not always clearly defined, and methodologically, relying predominantly on qualitative descriptions based on the researchers’ interpretations, and empirically, focusing, as mentioned, on Western “contemporary communication” (Kress 2009).

The other tradition is that of gesture studies (Goldin-Meadow 1998; Kendon 2004, 2014; McNeill 1985, 1992, 2005), where the “modalities” in question are those of “speech” and “gesture”, most often considered as two different sides of “language, as a multimodal phenomenon” (Vigliocco, Perniss & Vinson 2014). The close interaction between speech and gesture in everyday language use in all human cultures that have been studied, and throughout ontogenetic development (Zlatev 2015b) is indeed an important finding engendered by this tradition. At the same time, it appears somewhat dogmatic to define language as necessarily “multimodal” in this sense, with speech and gesture constituting a “single system” (McNeill 2005). Proportions of co-speech gestures vary with age and context, and the phenomenon of writing clearly demonstrates that core grammatical features of a given language remain the same when spoken and written, while admittedly, each medium has its specifics, and linguistics may have suffered from a certain “written language bias” (Linell 2004). Further, there is an unfortunate conflation in terminology between sensory modalities such as vision and hearing, and the so-called “language modalities” of gesture and speech, with only partial correspondence, given that gestures can be also heard (e.g. hand clapping) and felt (a pat on the back), while speaking is seen in face-to-face communication (Zlatev et al. 2017). Finally, we should note a critical limitation of the tradition from our perspective: it completely neglects drawing and other forms of depiction.

A theoretical approach that is arguably more fruitful for understanding polysemiotic communication, and in particular for understanding its origins, is the theory of the evolution of language in particular, and human cognitive-semiotic specificity in general, developed by Donald (1991, 1998, 2001, 2012), and elaborated by Zlatev (2008a, 2008b, 2014a). In brief, this holds that our ancestors took a different evolutionary trajectory from other primates when they evolved an advanced capacity for bodily mimesis, involving a cross-modal mapping between the sensory modalities of proprioception (bodily sense) and vision, as well as the ability to monitor and volitionally control first their bodies and then their voices to produce representations of external events for communicative purposes. This resulted in the humanly unique and universal capacity for pantomime, with a secondary role played by pre-linguistic vocalization (Zlatev et al. 2017). Such bodily representations can either be performed “in the air”, or by leaving traces on a surface such as the sand, and thus also gave rise to drawing. Thus, bodily mimesis can be seen as a cognitive-semiotic precursor, and still functional infrastructure of the three semiotic systems that we see tightly coupled in polysemiotic communication: gesture, language and depiction. Thus, it could at least in part help explain their integration.

But as pointed out, the three systems can also pull apart, and develop into unisemiotic modes of communication, as in e.g. silent speech (or even more, writing), silent pantomime, or drawing. Apart from theoretical semiotic analysis of these systems (Sonesson 1989) their communicative potentials can also be studied experimentally, as in the field of experimental semiotics which “focuses on the experimental investigation of novel forms of human communication […] which people develop when they cannot use pre-established communication systems” (Galantucci & Garrod 2011, p. 1). Such investigations typically ask dyads of participants to play a “semiotic game” where the task is to achieve successful communication, according to some criterion, but without using language. For example, Fay, Arbib, & Garrod (2013) showed that through its greater potential for iconic (i.e. resemblance-based) representation, gesture was more successful than vocalization. In another study, where the task was to communicate events comprised with different actions and agents, silent pantomime was more successful than pantomime with vocalization, thus challenging the simple idea that “multimodality” is always more functional than “unimodality” (Zlatev et al. 2017). Other studies, focusing on depiction, could show how communities of dyads could converge on a diagrammatic system of shared symbols, combing iconicity and conventionality (see Fay, Garrod, Roberts & Swoboda 2010).

In sum, various aspects of polysemiotic communication have been studied by anthropologists, linguists, semioticians and psychologists, but from rather complementary perspectives. Until recently (Devylder 2018; Ellis et al. 2017; Enfield 2005; Gaby 2017) anthropological studies of indigenous knowledge such as kinship focused only on language, and only recently has the need to integrate gesture and depiction been fully realized. Semioticians have provided theoretical analyses of such systems, though often biased by features of language when turning to the systems less studied, see (Sonesson 1989). Finally, experimental psychologists have begun to investigate the potentials of unisemiotic and polysemiotic communication in controlled settings, but with questionable ecological validity. Our project brings these scientific traditions together with the help of cognitive semiotics (Zlatev, Sonesson & Konderak 2016), the trans-disciplinary study of meaning making, and enriches the empirical database necessary for polysemiotic analysis, with data collected through ecologically valid methods.

3. Research questions

• (i) How do members of three distinct cultures communicate abstract knowledge within their own communities?

• (ii) What polysemiotic strategies are employed to communicate about such knowledge to a member of a different culture?

• (iii) How is the communicative potential of polysemiotic communication affected by selectively “removing” one of the three semiotic systems (language, gesture, and depiction)?

4. Theoretical framework, methods, and data collection

Theoretically, we adopt, and aim to elaborate, a consistent cognitive semiotic framework. Cognitive semiotics is a relatively new discipline, combining and merging concepts and methods from the systematic study of meaning (i.e. semiotics), mind (i.e. cognitive science) and language (i.e. linguistics) (Sonesson 2010; Zlatev 2015a; Zlatev, Sonesson & Konderak 2016). Unlike in much of traditional semiotics, we view meaning making as dynamic and value-based, implying a bi-directional interaction between an experiencing (intersubjective) subject and experienced world (Zlatev 2009). The sign is a special kind of meaning structure, requiring the experiencing subject to both link and distinguish it from it its referent; this link is based on a combination of grounds: iconic (i.e. resemblance-based), indexical (contiguity-based) and symbolic (convention-based), (Sonesson 2007). To the extent that one of these three grounds dominates (Jakobson 1965), we may speak of iconic/indexical/symbolic signs, but this is always a simplification. We distinguish between three general and universal semiotic systems language, gesture and depiction, and, on the one hand, their particular instantiations in sociocultural practices (e.g. sand drawings, graphic representations, white boards, or technological devices) and on the other hand, the sensory channels employed, which as pointed out in Section 2, do not correspond to the systems in a one-to-one fashion. Our guiding assumption is that all three semiotic systems emerge (in evolution and development) from the panhuman capacity for bodily mimesis, (Donald 1991, 2012; Zlatev 2008a, 2008b, 2014a) allowing the volitional use of the body for creating representations. In actual communication, the three systems (and possibly additional ones, like music) interact in complex and still poorly understood ways, depending also on specific factors. For example, we agree that “[t]he composite signal performance par excellence involves simultaneous integration of (symbolic) speech, (indexical) gesture, and (iconic/indexical) visual representations or artifacts” (Enfield 2005, p. 52), but warn against making simple connections between semiotic grounds and signs, and even more with the composite systems that they comprise.

Methodologically, in line with the centrality of methodological triangulation that is typical for cognitive semiotics (Zlatev 2009), we use first-person methods, explicating the intuitions and intentions of the sign-users, second-person methods, targeting the social interaction as such, and third-person methods, operationalizing and quantifying key variables, and subjecting these to statistical analysis.

• Study 1 consists of naturalistic participant observation of three culturally-specific situations of knowledge transmission in each of three sociocultural contexts: sand drawing performances in Vanuatu, sand storytelling in Central Australia, and classroom teaching in Sweden.

• Study 2 will elicit kinship narratives in the form of dyadic exchange between members of the three investigated cultures, and a researcher who is external to each community. Consultants in Vanuatu, Australia, and Sweden will be asked to explain the structure of their own family as well as culturally specific practices. They will be encouraged to use these descriptions as a starting point for personal stories about their family and its members. Consultants will be provided with the semiotic resources to not only speak and gesture in their descriptions but also draw in the sand (In Vanuatu and Central Australia) or write on a piece of paper (in Sweden).

• Study 3 will employ methods from experimental semiotics (e.g. Galantucci & Garrod, 2011). Pairs of participants from different fields of expertise (e.g. a linguistic student and a biology student) will be ask to convey complex notions from their field of expertise to each other (e.g. what is a conceptual metaphor? what is DNA?). This experiment will be time-limited and done in four conditions. The participants will be allowed to use (a) language and gesture, (b) language and depiction, (c) gesture and depiction and (d) all three semiotic systems.

All data from the three studies will be transcribed, annotated, and coded using ELAN with independent tiers for speech, gesture, and drawing. The signs in each semiotic system will be coded for the three semiotic grounds (iconicity, indexicality, conventionality), investigating all possible combinations, and interaction between system, perceptual modality and semiotic ground.

5. Project plan

The project runs from January 2019 to December 2021, and can be divided according to the following plan:

• Months 1-6: theoretical and fieldwork preparation; testing the elicitation procedures and the annotation scheme; ethics vetting and research permits applications.

• Months 6-12: fieldwork and data collection for studies 1 & 2 in Vanuatu and Australia; transcription of the data; fine-tuning of the annotation scheme.

• Months 12-18: data collection for studies 1 & 2 in Sweden; annotation and coding of the linguistic and gesture data from studies 1 & 2;  pilot experiments for Study 3.

• Months 18-24: data collection and annotation of study 3; annotation and coding of the depiction data;  compilation of a polysemiotic communication corpus; comparative analyses.

• Months 24-36: additional analyses, preparation/publication of results in at least four articles in high profile journals; dissemination of results at international conferences.

References

Deacon, A. B., & Wedgwood, C. H. (1934, [1927]). Geometrical drawings from Malekula and other islands of the New Hebrides. The Journal of the Royal Anthropological Institute of Great Britain and Ireland, 64, 129-175.

DeMarrais, K. B., Nelson, P. A., & Baker, J. H. (1992). Meaning in mud: Yup'ik Eskimo girls at play. Anthropology & Education Quarterly, 23(2), 120-144.

Devylder, S. (2014). Paamese Language and Culture  (Publication no. DOI: 10.4225/72/58ab0479d6eeb). PARADISEC  http://catalog.paradisec.org.au/collections/SD1

Devylder, S. (2018). Diagrammatic iconicity explains asymmetries in Paamese possessive constructions. Cognitive Linguistics, 29(2).

Donald, M. (1991). Origins of the moden mind: Three stages in the evolution of human culture. Cambridge, MA: Harvard University Press.

Donald, M. (1998). Mimesis and the executive suite: Missing links in language evolution. In J. R. Hurford, M. Studdert-Kennedy, & C. Knight (Eds.), Appoaches to the evolution of language: Social and cognitive biases (pp. 44-67). Cambridge: Cambridge University Press.

Donald, M. (2001). A mind so rare: The evolution of human consciousness. New York: Norton.

Donald, M. (2012). The mimetic origins of language The Oxford Handbook of Language Evolution (pp. 180-183). Oxford: OUP.

Ellis, E. M., Green, J. A.,  and Kral, I. (2017). Family in mind: Socio-spatial knowledge in a Ngaatjatjarra/Ngaanyatjarra children’s game. Research on Children and Social Interaction, 1(2), 164-198.

Enfield, N. J. (2005). The body as a cognitive artifact in kinship representations: Hand gesture diagrams by speakers of Lao. Current anthropology, 46(1), 51-81.

Fay, N., Arbib, M., & Garrod, D. (2013). How to bootstrapp a human communication system. Cognitive science, 37, 1356-1367.

Fay, N., Garrod, S., Roberts, L., & Swoboda, N. (2010). The interactive evolution of human communication systems. Cognitive science, 34(3), 351-386.

François, A., Franjieh, M., Lacrampe, S., & Schnell, S. (2015). The exceptional linguistic density of Vanuatu: Asia-Pacific Linguistics Open Access.

Gaby, A. (2017). Kinship Semantics: Culture in the Lexicon Advances in Cultural Linguistics (pp. 173-188). Berlin: Springer.

Galantucci, B., & Garrod, S. (2011). Experimental semiotics: a review. Frontiers in human neuroscience, 5, 11.

Geismar, H. (2014). Drawing it out. Visual Anthropology Review, 30(2), 97-113.

Gell, A. (1998). Art and agency: an anthropological theory: Clarendon Press.

Goldin-Meadow, S. (1998). The development of gesture and speech as an integrated system. San Francisco, CA: Jossey-Bass.

Green, J. (2012). The Altyerre Story—“Suffering Badly by Translation.” The Australian Journal of Anthropology 23 (2): 158–78.

Green, J. (2014). Drawn from the ground: Sound, sign and inscription in Central Australian sand stories (Vol. 13): Cambridge University Press.

Green, J. (2017). Multimodal complexity in sand story narratives. Narrative Inquiry, 26(2), 312-339.
Jakobson, R. (1965). Quest for the essence of language. Diogenes, 13, 21-38.

Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge: Cambridge University Press.

Kendon, A. (2014). The 'poly-modalic' nature of utterances and its implication. In D. Dor, C. Knight, & D.

Lewis (Eds.), The social origins of language (pp. 67-76). Oxford: OUP.

Kress, G. (2009). Multimodality: A social semiotic approach to contemporary communication. London: Routledge.

Lévi-Strauss, C. (1965). The future of kinship studies. Proceedings of the Royal Anthropological Institute of Great Britain and Ireland(1965), 13-22.

Linell, P. (2004). The written language bias in linguistics: Its nature, origins and transformations. London: Routledge.

McNeill, D. (1985). So you think gestures are nonverbal? Psychological Review, 92, 350-371.

McNeill, D. (1992). Hand and mind. Chicago: University of Chicago Press.

McNeill, D. (2005). Gesture and thought. Chicago: University of Chicago Press.

Ochs, E., Gonzales, P., & Jacoby, S. (1996). "When I come down I'm in the domain state": Grammar and graphic representation in the interpretive activity of physicists. Studies in Interactional Sociolinguistics, 13, 328-369.

Patterson, M. (2006). Agency, kinship, and history in North Ambrym. Journal of the Royal Anthropological Institute, 12(1), 211.

Rio, K. (2005). Discussion around a sand-drawing: creations of agency and society in melanesia. Journal of the Royal Anthropological Institute, 11(3), 401.

Sonesson, G. (1989). Pictorial concepts. Inquiries into the semiotic heritage and its relevance for the analysis of the visual world. Lund: Aris/Lund University Press.

Sonesson, G. (2007). From the meaning of embodiment to the embodiment of meaning: A study in phenomenological semiotics. In T. Ziemke, J. Zlatev, & R. Frank (Eds.), Body, Language and Mind. Vol 1: Embodiment (pp. 85-128). Berlin: DGM.

Sonesson, G. (2010). From mimicry to mime by way of mimesis: Reflections on a general theory of iconicity. Sign Systems Studies, 38(1/4), 18-65.

Taylor, J. (2005). Paths of relationship, spirals of exchange: Imag (in) ing north Pentecost kinship. The Australian journal of anthropology, 16(1), 76-94.

Van Leeuwen, T. (2005). Introducing social semiotics: Psychology Press.

Vigliocco, G., Perniss, P., & Vinson, D. (2014). Language as a multimodal phenomenon: implications for language learning, processing and evolution. Phil. Trans. R. Soc. B, 369(1651), 20130292.

Zagala, S. (2004). Vanuatu sand drawing. Museum International, 56(1‐2), 32-35.

Zlatev, J. (2008a). The coevolution of intersubjectivity and bodily mimesis The Shared Mind: Perspectives on intersubjectivity (pp. 215-244). Amsterdam: Benjamins.

Zlatev, J. (2008b). From proto-mimesis to language: Evidence from primatology and social neuroscience. Journal of Physiology – Paris, 102, 137-152.

Zlatev, J. (2009). The semiotic hierarchy: Life, consciousness, signs and language. Cognitive Semiotics, 4(Supplement), 169-200.

Zlatev, J. (2014a). Bodily mimesis and the transition to speech. In M. Pina & N. Gontier (Eds.), The Evolution of Social Communication in Primates (pp. 165-178). New York: Springer.

Zlatev, J. (2014b). Image schemas, mimetic schemas, and children's gestures. Cognitive Semiotics, 7, 3-30.

Zlatev, J. (2015a). Cognitive semiotics International handbook of semiotics (pp. 1043-1067): Springer.

Zlatev, J. (2015b). The emergence of gestures. In B. MacWhinney & W. O'Grady (Eds.), The Handbook of Language Emergence (pp. 458-477). New York: Wiley.

Zlatev, J., Sonesson, G., & Konderak, P. (2016). Meaning, mind and communication : explorations in cognitive semiotics: Frankfurt am Main : Peter Lang, 2016.

Zlatev, J., Wacewicz, S., Zywiczynski, P., & van de Weijer, J. (2017). Multimodal-first or pantomime-first? Communicating events through pantomime with and without vocalization. Interaction Studies, 18(3), 455-479.