Vision-Based Learning Status Monitoring on Color and Depth Live Facial Videos

This study analyzes live facial videos for recognizing nonverbal learning-related facial movements and head poses to discover the learning status of students. First, color and depth facial videos captured by a Kinect are analyzed for face tracking using a three-dimensional （3D） active appearance model （AAM）. Second, the facial feature vector sequences are used to train hidden Markov models （HMMs） to recognize seven learning-related facial movements （smile, blink, frown, shake, nod, yawn, and talk）. The final stage involves the analysis of the facial movement vector sequence to evaluate three status scores （understanding, interaction, and consciousness）, each represents the learning status of a student and is helpful to both teachers and students for improving teaching and learning. Five teaching activities demonstrate that the proposed learning status analysis system promotes the interpersonal communication between teachers and students.