Designing an Interactive Communication Assistance System for Hearing-Impaired College Students Based on Gesture Recognition and Representation

• Main Authors: Clepper, G. (Author), Jia, J. (Author), Liu, W. (Author), Zhang, J. (Author), Zhang, Z. (Author), Zhu, Y. (Author)
• Format: Article
• Language: English
• Published: MDPI 2022
• Subjects: Audition; College students; Design; Efficiency; Gesture recognition; gesture recognition and representation; Gesture representations; Gestures recognition; Hearing impaired; hearing-impaired college student; Hearing-impaired college student; Learning efficiency; multimedia-equipped classroom; Multimedium-equipped classroom; Processing; qualitative analysis; Qualitative analysis; Smart classroom; Students; tracking processing; Tracking processing
• Online Access: View Fulltext in Publisher

 Developing a smart classroom can make the modern classroom more efficient and intelligent. Much research has been conducted pertaining to smart classrooms for hearing-impaired college students. However, there have been few significant breakthroughs in mobilizing students’ learning efficiency as measured by information transmission, communication, and interaction in class. This research collects data through nonparticipatory observation and in-depth interviews and analyzes available data on classroom interaction needs of these students. We found that diversified explanations, recordable interactive contents, and interaction between teachers and students could improve the learning effects in the classroom. We also propose a tracking-processing method based on gesture recognition and representation and present a design for a processing system based on AT89C52 microcontroller and Kinect. In this way, sign language can be translated into text and all students can receive the information and participate in the interaction, which greatly improves students’ autonomy and enthusiasm of learning. This design enables deaf students to fully use classroom learning resources, reduces learning time costs, and improves learning efficiency. It can assist teachers in teaching and tutoring students to enhance their experience. © 2022 by the authors. Licensee MDPI, Basel, Switzerland.