Schrifttanz: Written Dance/Movement Poems

One of the most frustrating challenges facing practitioners of Dance is the need to use spoken/written language to reference non-verbal movement. The non-verbal to verbal, and vice versa, is not a challenge isolated to practitioners of dance, but is a frustration shared amongst researchers working to represent movement through technological means. Perhaps it is enough to allow the movement, non-verbal as it is, to speak for itself. Then again, it is often through verbal language that we can make meaning from movement. The intersection of language and movement is the point at which meaning-making enforces the mind-body connection, and it is often how embodied experience is transmitted. Two HCI research projects that have studied this connection between words and movement as a means to the classification and automatic recognition of movement, ARTeFACT and POEME, are now collaborating in a new project: Schrifttanz.

ARTeFACT seeks to enable automatic recognition, identification, annotation, and retrieval of movement-based data from streaming video. A lofty goal that has been approached through the generation and analysis of both codified and abstract movement using motion capture data collected at 120 Hz with a Vicon 8-camera motion capture system and a modified Plug-In Gait full body marker set with 38 infrared reflecting markers placed on performers. Early research included instances of iterative choreography created from accelerometer data and descriptions of movement through verbal language by observers (Coartney and Wiesner, 2009). The second phase of the project involved the capture of over 200 codified dance ‘steps’ in various genres, the development of an ontology, and the creation of IDMove, a tool through which we were able to automatically identify and name dance movement from single dancers (Wiesner et al., 2011, 2014). The third phase ventured into the realm of identifying abstract movement that represents the conceptual metaphor CONFLICT, as introduced by Lakoff and Johnson (2003).

This data, derived from 7 dances about conflict and 19 CONFLICT terms, 396 different sections of movement, each lasting from 2-120 seconds, were captured and categorised. Critical reviews (written) about the dance works were also collected. Through statistical analysis of written and danced texts about CONFLICT, we distinguished body parts frequently used, structural preferences (e.g. stage spacing), and choreographic time spent on the different CONFLICT terms in total and per dance (Wiesner et al., 2015). The findings were validated by a small ‘crowd-sourced’ experiment, and movement ‘rules’ were developed per term in order to enable identification of a concept through movement (terms include victim, attack, surrender, struggle, conquer, hero, victory, survive, etc.) (Wiesner and Stalnaker, 2015). Concordances and collocation studies aided in the investigation of the intersections of words and movement, through the phenomenological approach of dancers’ descriptions and perceptions of viewers in the form of the reviews written by dance critics. A final step has been to align the non-verbal and verbal output with concepts used in Laban Movement Analysis (Body, Effort, Shape, and Space). ARTeFACT has collected a wealth of data from captured movement, from written descriptions and articles based on the dances, and from a broad LMA perspective. In the future, various modes of comparison (e.g. the metaphor PEACE, general language studies, etc.) will further test the system.

Other researchers have incorporated LMA into their study as a means to identify movement (including studies on rats’ play) (Foroud and Pellis, 2003), yet their focus is on deconstruction. POEME and ARTeFACT coincide in that rather than provide a deconstruction of the movement they both seek to characterise the essence of that movement, an approach inspired by LMA. In the case of ARTeFACT, this is accomplished by describing the relations between parts of a movement that accompany the portrayal of abstract concepts in dance. For POEME, it means finding computational means to summarize a movement experience so that it can be studied as a whole.

Where ARTeFACT has taken a more actively analytical approach - which resulted in a set of rules that describe stereotyped motion consistent with conceptual metaphors - POEME has taken a more abstract approach, letting data collection from a large number of human responses gradually expose relationships between verse and movement. POEME (Portrayal of Ephemeral Movement Experiences) is a mobile website (http://poeme.iat.sfu.ca) that interprets movement data, captured through a MARG sensor array commonly found in smartphones, into insightful, witty and whimsical poetry. The goals of POEME are twofold: 1) to create a playful game that can inspire movement exploration and 2) to explore new methods of classifying the nuances of bodily experiences through poetry and in turn understand how bodily knowledge can more easily be transmitted and articulated through words.

POEME, inspired by the Japanese form of poetry known as haika no renga (comic, collaborative poetry), involves the social creation of poems through turn taking (Basho, 1998). Creating these linked poems requires participants to respond to previous stanzas with original verse creating a ‘movement poem’, an interleaved work of verses of words and phrases of movement. We believe, as Herbelot notes, that poetry can be “analysed along the usual dimensions of prosody, syntax, semantics, etc.” (2015). In POEME, each verse follows a rigid template that follows the form of a Parts of Speech Poem (PoSP). This PoSP template allows for very simple production of verses. Each poem begins with a noun, followed by two adjectives, two verbs, and an adverb. POEME composes verses from a large dictionary of English language words, based on measurements taken from prior movement responses. In an initial study, the system was trained by collecting movement responses to human- and machine-composed verses. In a later study, training was focused on two wordlists that depict different themes of movement: the ‘stillness wordlist’ (75 words that relate to little or no dynamic change in movement, e.g. frozen, still, serene); and a ‘motion wordlist’ (250 words related to the mechanics or physics of motion, e.g. buoyant, centripetal, accelerated). Using a Naive Bayes algorithm, POEME can differentiate between stillness and motion with 97% accuracy (Cuykendall et al., 2016).

Schrifttanz is a unique project that combines the scale of data collection, which POEME offers, with the nuanced language model of ARTeFACT. In Schrifttanz we explore if POEME can recognise conceptual differences in movement, specifically concepts related to CONFLICT, which have been studied in detail during the creation of the ARTeFACT system. Three dancers train the POEME agent by recording movement based on the rules defined in ARTeFACT. These recorded movement sessions are used to model conceptual relations to movement in POEME. This allows us to generalise previous findings from ARTeFACT into a new sensor modality. Also, a Laban/Bartenieff Movement Analysis using the rules-based movements establishes a secondary set of language elements and validates distinctions between movement rules. A computational agent in POEME learns to classify these movements according to the metaphor terms and their associated movements generated by the rules. We then compare POEME’s ability to classify these movement metaphors with existing capabilities in ARTeFACT to validate and generalise previous findings and establish the reliability of the POEME system.

Concurrently, POEME composes verses comprised of natural language words gathered through the written articles and reviews based on dances representing the conceptual metaphors identified in ARTeFACT. Free form responses to these verses are collected using POEME’s normal training procedure. A machine learning-process associates features of movement with the words. This allows POEME to generate relevant responses to movement. We explore methods of supplementing POEME’s verse generation by including the conceptual metaphor model.

Words, like movements, have layers of meaning and can be recombined to create a multitude of sentences or sequences that all vary in their underlying meaning. As philosopher Mark Johnson states, “It is true that when we read, we read words. But words have meanings, and meanings go far beyond words” (2008). In Schrifttanz, the same could be said for bodily movement. POEME’s use of current mobile technology and the powerful interactive engine the POEME project has developed enables us to draw from a broader population in order to study and validate the findings of ARTeFACT. Synergistically, ARTeFACT offers to POEME the ability to advance its training mechanisms while it supports the representational nature of the data generated. The two projects are pieces of a puzzle that seeks to establish a better understanding of the relationship between movement and language through the particular and the universal.