EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset

• Main Authors: Lovegrove, S. (Author), Newcombe, R. (Author), Northcutt, C.G (Author), Zha, S. (Author)
• Format: Article
• Language: English
• Published: NLM (Medline) 2023
• Subjects: algorithm; Algorithms; artificial intelligence; Artificial Intelligence; Bayes theorem; Bayes Theorem; Communication; human; Humans; interpersonal communication; machine learning; Machine Learning
• Online Access: View Fulltext in Publisher; View in Scopus
• Physical Description: 11
• ISBN: 19393539 (ISSN)
• DOI: 10.1109/TPAMI.2020.3025105

Multi-modal datasets in artificial intelligence (AI) often capture a third-person perspective, but our embodied human intelligence evolved with sensory input from the egocentric, first-person perspective. Towards embodied AI, we introduce the Egocentric Communications (EgoCom) dataset to advance the state-of-the-art in conversational AI, natural language, audio speech analysis, computer vision, and machine learning. EgoCom is a first-of-its-kind natural conversations dataset containing multi-modal human communication data captured simultaneously from the participants' egocentric perspectives. EgoCom includes 38.5 hours of synchronized embodied stereo audio, egocentric video with 240,000 ground-truth, time-stamped word-level transcriptions and speaker labels from 34 diverse speakers. We study baseline performance on two novel applications that benefit from embodied data: (1) predicting turn-taking in conversations and (2) multi-speaker transcription. For (1), we investigate Bayesian baselines to predict turn-taking within 5 percent of human performance. For (2), we use simultaneous egocentric capture to combine Google speech-to-text outputs, improving global transcription by 79 percent relative to a single perspective. Both applications exploit EgoCom's synchronous multi-perspective data to augment performance of embodied AI tasks.