Virtual Environment (VE) based immersive training systems have been widely adopted by US military as an alternative to costly and time consuming live training exercises. Current VE based training systems lack an affective state detection component, which may lead to decrements in training outcomes. In this paper, we introduce the Realtime Affective State Detection and Induction System (RADIS); a novel system that incorporates affective state detection and induction capabilities into existing training and simulation frameworks. RADIS is capable of: 1) dynamically monitoring a trainee's facial and speech features through visual and auditory channels, 2) detecting the trainee's affective state based on multimodal information fusion at the decision level, and 3) driving the trainee's affective state towards a target affective state specified in the structured lesson plan. RADIS currently uses human facial expression and speech sound for affective state detection. The visual and auditory signals are non-intrusive and provide higher prediction and recognition accuracy compared with physiological and motion signals. We extracted pitch, energy, formants, Mel-Frequency Cepstral Coefficients (MFCC), and speech rate from the speech signal and geometric and holistic features from the real time video input. The extracted feature vector was classified by the Support Vector Machine (SVM) to detect the trainee's affective state. RADIS was designed using a data-driven approach to support training domain independence. It follows the Sharable Content Object Reference Model (SCORM) eLearning standard and isolates all domain specific information in data so that a single code base can be successfully reused across multiple domains. By encoding all of the information required for a training session within the Structured Lesson Plan (SLP) the code base remains independent of the training domain and can be used in multiple training scenarios. RADIS will enhance the VE based training systems to better approximate the real-world experiences for the trainees.
RADIS: Real Time Affective State Detection and Induction System
4 Views