Short description: Recording a communication game with Kinect, we measured changes in gesture that are the hallmarks of conventionalization in sign language

Abstract:

Gestures produced by users of spoken languages differ from signs produced by users of sign languages in that gestures are more typically ad hoc and idiosyncratic, while signs are more typically conventionalized and shared within a language community. To study how gestures may change over time as a result of the process of conventionalization, we designed a social coordination game to elicit repeated silent gestures from hearing nonsigners, and used Microsoft Kinect to unobtrusively track the movement of their bodies as they gestured (following Lenzen, 2015). Our approach follows both a tradition of lab experiments designed to study social coordination and transmission in the emergence of linguistic structure (Schouwstra et al., 2014) and insights from sign language research on language emergence. Newly emerging sign languages are being discovered, and even established sign languages are relatively young; it is therefore possible to observe linguistic conventionalization as it happens naturally (Senghas et al., 2005). Working with silent gesture, we were able to simulate and quantify effects of conventionalization that have been described for sign languages (Frishberg, 1975), including changes in efficiency of communication and size of articulatory space, in the laboratory.

Participants took turns either giving clues about (the Communicator) or guessing (the Guesser) items from a set of English nouns. Items were presented on a screen visible only to the Communicator, and once the Communicator confirmed that the Guesser had guessed correctly, the Guesser pressed a button to advance to the next item. Trial length was recorded as the time (ms) between button presses. Participants switched roles halfway through each of four rounds. Each item appeared once per round. In Round 1, the Communicator could use gesture and speech to ensure that both participants were familiar with the entire set of items going into Rounds 2-4, which were gesture-only. The Communicator’s movements were recorded as sequences of locations in XYZ-space using the Kinect. 10 pairs of undergraduates received course credit for participating in the study. Participant pairs had never met before, and no participant reported knowledge of a sign language.

We examined Rounds 2-4, in which the Communicator was gesturing without speech about a set of items known to both participants. We observed rapid alignment between participant pairs across the rounds; as participants became familiar with the items in the game, they correctly guessed the items at faster rates. Trial lengths (s) started longer in Round 2 (M=11.54), and became shorter in Round 3 (M=5.65) and Round 4 (M=4.39). A linear mixed-effects model showed that ROUND significantly affected TRIAL LENGTH (𝜒2=87.09, p < 0.0001), reducing trial length by about 3.57s (S.E. +/-0.37) each round.

Two additional analyses concerned volume of gesture space and distance traveled by the hands. The Kinect measurements showed that gesture spaces started larger (m3) in Round 2 (M=0.15) and became smaller in Round 3 (M=0.11) and Round 4 (M=0.10). A linear mixed-effects model showed that ROUND significantly affected GESTURE SPACE (𝜒2=51.01, p < 0.0001), reducing the volume of the gesture space by about 0.03m3 (S.E. +/- 0.004) each round. The total distance traveled by the hands also started longer (m) in Round 2 (M=11.58) and became shorter in Round 3 (M=6.22) and Round 4 (M=5.14). A linear mixed-effects model showed that ROUND significantly affected HAND TRAVEL DISTANCE (𝜒2=75.85, p < 0.0001), reducing the distance that the hands traveled by about 3.22m (S.E. +/- 0.36) each round. (Figures associated with trial length, volume of gesture space, and distance traveled by the hand are included in the supplementary materials.)

We chose an experimental setup known to result in rapid conventionalization (Scott-Phillips & Kirby, 2010), and with Kinect we were able to measure changes in gesture that are also hallmarks of conventionalization in sign language. This approach opens the door for more direct future comparisons between ad hoc gestures produced in the lab and natural sign languages in the world. By operationalizing concepts like reduction and articulatory space, which, out of necessity have been typically discussed in vague terms, we anticipate that this approach will also be beneficial for future studies of (sign) language emergence.

Download Paper (pdf)

Supplementary Materials (zip)

Citation:

Namboodiripad S., Lenzen D., Lepic R. and Verhoef T. (2016). Measuring Conventionalization In The Manual Modality. In S.G. Roberts, C. Cuskley, L. McCrohon, L. Barceló-Coblijn, O. Fehér & T. Verhoef (eds.) The Evolution of Language: Proceedings of the 11th International Conference (EVOLANG11). Available online: http://evolang.org/neworleans/papers/107.html