Constituent order in non-verbal representations: introduction of new film stimuli and a pilot study with gestures

Anu Vastenius

The research on non-verbal representations in experimental settings is one way to address the possible effect of language on thought, which has independently been supported by many scholars (Slobin, 1996, Casasanto, 2008, Zlatev & Blomberg, 2013). Word-order is a fundamental typological feature in languages, which may have an impact on other cognitive processes as well. On the other hand, since many cognitive functions are universal, a cognitive bias on how people conceptualise and represent events can be expected. The order of constituents in non-verbal representations has been empirically investigated by using gestures or, to a lesser extent, pictures. Depending on the design and the semiotic modality in experiments different results have been obtained (cf. McNeill 2000; Kita and Özyürek 2003; Casasanto 2008; Goldin-Meadow et al. 2008; Langus and Nespor 2010; Meir et al 2010; Schouwstra 2012; Christensen and Tylen 2013; Hall et al 2013). Thus, the relation between semiotic vehicles, event types in the stimuli, the word-order in L1 and the structure of non-verbal representations has still remained unresolved.
The project addresses the following questions:
•    Does the type of events in the stimuli affect how they are represented non-verbally?
•    Does the type of non-verbal semiotic vehicle i.e. pictures, gestures have an impact on the constituent orders used in the representations?
•    Does the order of constituents in non-verbal representations overlap with the word order in the native language?
To address the research questions new video-stimuli has been designed and recorded during April- May 2017, and a pilot experiment with gestures has been conducted. In the stimulus used earlier in my MA Thesis addressing similar questions (Vastenius, 2014) certain types of contrasts, that have been shown to affect the order, were missing, such as manipulation events (with Patient already existing)) vs. construction events (Patient being created) (Schouwstra, 2012; Christensen & Tylen, 2013), thus they were added. New picture cards depicting the constituents on the video clips are being developed with more concrete depictions of Acts. Thus, their iconicity will be higher than in the previous design used in my MA-thesis, where the Acts were represented by plain arrows.
The two different vehicles differ in their semiotic properties: Gestures are a bodily vehicle, and also they have the “rapid fading” feature, meaning that the representation does not continue to exist after performing it (Hockett, 1960). Additionally, gestures are three-dimensional unlike pictures. According to Peirce (1958), Sonesson (1995) and Zlatev (2013), the relation between the referent and the vehicle i.e. the semiotic ground can be iconic, indexical or symbolic. In the two vehicles used in this study, the semiotic ground is different in both modalities so that the pictures and the gestures are iconic and symbolic, i.e. conventional, and the gestures sometimes also indexical. Considering the differences between the vehicles, it can be expected that the order used for retelling the events should vary between them.     

Sidansvarig: Goran.Sonessonsemiotik.luse | 2017-08-21