Special Issue on Gesture and Speech in Interaction


Following recent progress in multimodal systems and an increasing awareness of the important role 
of manual and facial gestures in communication, we invite submissions for a special issue of Speech
Communication entitled “Gesture and Speech in Interaction“.

With this issue we would like to promote interdisciplinary perspectives that facilitate an 
efficient understanding of the complexity of multimodal communication. We would like to invite 
linguists, phoneticians, speech technologists, computer and cognitive scientists as well as 
researchers from all other fields who share an interest in speech and gesture interfaces. The 
focus of the special issue lies on speech-accompanying gestures interacting temporally and 
functionally with the speech stream. The type of gestures addressed will be primarily manual 
and/or head gestures while not excluding additional analyses such as eyebrow movement and facial 
expressions. However, studies on articulatory gestures and sign language will not be considered 
because the former is an integral part of the speech stream and the latter is a separate linguistic
system. Also, papers on facial expression of emotion are not encouraged unless they have a direct 
(pragma-)prosodic rather than psychological focus.

Background

During the last years at conferences such as INTERSPEECH, the speech science community has 
witnessed a steadily growing interest in the topic of multimodality. Clear evidence of the growing 
need for understanding the speech-manual gesture interaction has been proved at recent workshops
on the topic such as GESPIN conferences in Poznan (2009) and Bielefeld (2011) which have attracted 
an international, interdisciplinary audience.

A similar scope was adopted by the Gesture Workshops (GW) with main focus on technical aspects of 
manual gesture modeling in human-machine interaction. Both GESPIN and GW have evolved in addition 
to the AVSP workshops (concentrating largely on technical aspects of multimodal facial 
communication) and ISGS meetings, most widely attended by gesture researchers, that include all 
classic gesture topics with specific themes changing at each meeting. However it is the two 
interdisciplinary GESPIN meetings that have explicitely encouraged a deeper understanding of the 
topics below that we would like submissions to this issue to address.

A growing number of gesture researchers have recently gained great insights about the role of 
manual gesture in communication. This invaluable experience has been gained using mainly 
psycholinguistic and discourse analytic methods but has so far rarely dealt with the gesture-speech
interface in a strictly formal manner, e.g.: applicable in technical systems.

On the other hand, the wider speech science community (concentrated around e.g. ISCA affiliated 
journals such as Speech Communication and ISCA conferences) with its command of formal tools and 
applications has already begun to address the challenges of multimodal communication. It is ready 
to concentrate specifically on the analysis and modeling of the interaction between speech and 
head/manual gesture. Both communities are ultimately interested in a joint account of the interplay
of the two modalities in a wide context of human communication. With the proposed special issue 
the two communities will receive an opportunity to combine their expertise.

Topics

We would like to see the following topics to be be addressed in the issue:

1. Mechanisms of temporal coordination of gesture and speech production. How is gesture form and 
type constrained to coordinate with speech?

locally: what are the anchor points in the speech stream ("affiliates") that the gesture is 
coordinated with, are they lexical, conceptual or prosodic?
globally: given that gesture is holistic and synthetic and speech is linear and segmented, how 
are different gesture forms (smooth, punctual, mixed) that arise in succession coordinated with 
continuous speech?
2. The functional interplay of gesture and speech in communication; the semantics and pragmatics 
of gesture-speech interaction

how do deictic and beat gestures express and modify communicative intention and dialogue structure?
how do iconic and metaphoric gestures express additional meaning and complement semantic content 
together with speech?
3. An integrative perceptual account of speech and gesture

the effects of gesture-speech timing mismatches on the perception of the message.
4. The interaction of manual and head gestures with perception and production of prosody in a 
simultaneous speech stream

how is prosodic emphasis modified by head movement and manual gesture?
relations between prominence and "intensity" of gesturing.
5. Application and modeling of the above in technical systems

dialogue systems involving head and manual gesture generation and recognition,
artificial agents and robots as implementations of speech-hand-head timing models,
in general, papers describing technical systems that are able to process both speech and head/manual
gesture (and other modalities) will be encouraged.
As regards the methodological angle that will be promoted, theory building, empirical 
investigations and gesture-speech modeling will define the general approach to the problems. 
At the same time, we would like to put special emphasis on technical applications (as in point 5 
above) as well as first steps towards tools and annotation standards, i.e., tools for integrative 
analyses of speech and gesture corpora and systems for joint gesture and speech annotation.

Important Dates

Submission deadline: 31 May 2012
Notification of acceptance: September 2012
Final manuscript due: January 2013
Tentative publication date: Summer 2013

Editors

Prof. Dr. Petra Wagner (Bielefeld University, Germany)
Dr Zofia Malisz (Bielefeld University, Germany)