See-Through Captions

What it does

See-Through Captions is a system that displays captions in real-time on a transparent display, which was developed for deaf and hard-of-hearing people to communicate with hearing people.

Your inspiration

Our development team has a deaf member. While spending time with him, how to enhance communication with him has always been an important issue. When he first became a member of the laboratory, the challenge was how to participate in the lab seminars. After trying various techniques, he came up with a way to take advantage of automatic speech recognition technology. After that, due to the epidemic of COVID-19, we developed a voice captioning system that can be used in the online lab seminars, and as an application example of the further developed system, a face-to-face communication support system has been created.

How it works

See-Through Captions is a system that displays captions converted into text by real-time speech recognition on a transparent display. Two types of this system have been developed, a stationary type and a portable type. The stationary type is designed for one-on-one communication between the hearing person and the daef person, and is expected to be installed between the two people. The portable type is a system in which the hearing person displays captions while carrying a transparent display by hand.

Design process

The development process of this system can be divided into the following three stages. 【1. Development of online caption generation / display system】 An online caption generation / display system that runs on a web browser and was developed to communicate with a deaf student in online seminars in the laboratory. This system is open to the public on the web and can be used by anyone. 【2. Development of stationary See-Through Captions】 In order to improve the quality of face-to-face communication, we have developed a system that allows access to voice information while looking at the facial expressions and gestures of the conversation partner. This system is an improved version of the online caption generation / display system. We have conducted empirical usage tests at city residents' information sessions and receptions at a science museum. 【3. Development of portable See-Through Captions for guided tours in science museums】 After developing the stationary See-Through Captions, we developed a portable See-Through Captions with a small transparent display. We have actually conducted a trial introduction on a guided tour of the science museum.

How it is different

There are various methods to improve accessibility to voice information of hearing-impaired people. In written conversation, the problems are that it takes time and effort to convey information, and in oralism and gestures, it is difficult to confirm whether the content was accurately conveyed to the conversation partner. With See-Through Captions, captions can be easily displayed using only audio, and inconsistencies in information transmission can be prevented. When using the voice recognition app on a smartphone, there is a problem that the line of sight is nailed to the smartphone to check the recognition result, and the facial expression and body language of the partner are overlooked. When using the voice recognition app of the AR device, the hearing person cannot see the voice recognition result, and communication mistakes due to misrecognition occur. See-Through Captions can also solve these problems.

Future plans

The stationary See-Through Captions can be installed in places where anyone can use them as needed, such as reception desks and cash registers, and in workplaces where people with deafness work. The portable See-Through Captions are expected to be applied to guided tours. It is expected that it can contribute to increasing learning opportunities for children with deafness and hearing loss by using it for group tours of deaf schools. See-Through Captions is a system that can be deployed in a wide range of areas, from daily life to working environments and learning experiences.

Awards