Modeling Children's Nonverbal Behaviors.


Data collection for children's backchannel
behaviors in peer-to-peer storytelling and
personalized modeling for predicting
child's attentive state.


Screen Shot 2017-10-24 at 6.54.41 PM.png.1400x1400.png

Data Collection

Understanding social-emotional behaviors in storytelling interactions plays a critical role in the development of interactive and educational technologies for children. A challenge when designing for such interactions using technologies like social robots, virtual agents, and tablets is understanding the social-emotional behaviors pertinent to the storytelling context–especially when emulating a natural peer-to-peer relationship between the child and the technology. 

In this project, a dataset (P2PSTORY) of young children (5-6 years old) engaging in natural peer-to-peer storytelling interactions with fellow classmates was collected. 

The dataset contains 58 recorded storytelling sessions along with a diverse set of behavioral annotations as well as developmental and demographic profiles of each child participant. These are described in more detail below.

1. Audio and Video: Recordings were collected for each session from three time-sychronized cameras and a high quality microphone.

2. Behavioral features: Video recordings were coded for a wide range of behaviors including gaze, posture, nods, smiles & frowns, eyebrow movement, and backchannel utterances. In addition, interaction-level features were annotated including listener’s attention and whether dyads were on or off task.

3. Prosodic features: The storyteller’s use of prosodic cues including pitch, energy, pauses, filled pauses, and long utterances was annotated.

4. Personal features: Demographic and socio-emotional development parameters were collected for each participant.

5. Adult & Child Perceptions: To better understand how children perceived the effectiveness of their interaction partner, participants were asked to rate their partner on measures relating to attention and understanding. For comparison, we also collected similar ratings from adult coders.

Children participated in multiple interactions as either a storyteller or listener, allowing for the exploration of individual variations in behaviors across interactions of either roles. Additionally, these interactions took place with familiar peers as opposed to novel adults. In these three ways, the dataset is a unique contribution to the study of human-human interactions for the design and evaluation of interactive story-listening and story-telling technologies.

Perception of Attention Analysis

To build on this, we evaluated the attentiveness of a listener from two perspectives. First, a subjective metric based on adult annotations of attentive state. And secondly, a subjective rating provided by the child storyteller about the listener’s level of attention. Our noteworthy finding is demonstrating that behaviors relevant to the subjective perception of attention differ between a child and an adult. Brow raises and nods are salient listening behaviors related to adult’s perception of attention and conversely, gazes, leans, and smiles are salient to a child’s perception. Given this finding, it is relevant to consider using nonverbal language understood by young children when designing a natural peer-to-peer relationship between a child and an interactive technology.

Personalized Model of Attention

Building further on this, we designed a model to predict if a child (in the listener role) was not listening to the storyteller. To do this, we considered two modeling approaches: decision trees and neural nets. Furthering this, we evaluated feature augmentation and multi-task learning as strategies to enhance our ability to predict whether the listener was paying attention. Through this work, we were able to demonstrate that by knowing 70% of an interaction, we can predict the listener state in the remaining interaction. We also show how personal features such as the ASQ score for social-emotional development and gender, about subjects can be leveraged to build better models of the specific interactions. 


  • extensive data analysis for both perception analysis and personalized modeling
  • heuristic development, feature selection and data cleaning
  • design of modeling approaches using Gaussian Processes, Deep Learning, and Decision Trees with team


Python | Pandas | machine learning | neural nets | statistics | data structures