GenAI Assistant to Bridge Communication Gaps for the Deaf and Hard of Hearing
Bridge
Project
Academic
Timeline
4 Weeks
The Team
2 UX Designers
2 UX Researchers
My Role
User Research
Competitive research
Data Synthesis
Conceptualization,
Sketches and Story boards
Visual Design
Usability testing
Context
Having personal experiences with people with hard of hearing, We wanted to understand problems that deaf students might face, attending regular school to see the issues students might be facing to enhance their schooling environment
Context
Overview of the project!
What is the problem?
"Deaf students often face isolation as not everyone understands sign language, highlighting the crucial need for interpreters everywhere they go. However, having an interpreter at all times proves challenging."
A Deaf student during one of our interviews
To understand more about about the problem space
Meet Jason
Sandy, a 65-year-old living alone near the suburbs of Indianapolis, relies on her pension fund. She's looking for nearby food banks to help with groceries but isn't very tech-savvy for online searches. Recently, she heard about Community Compass, a potential resource
What are the Jason's Pain-points
Google Live Transcribe
The existing transcription apps lacks intuitive features
While the apps aids one-on-one conversations, it struggles in group discussions, as they are not capable of identifying or segregating voices
The existing tools and apps lacks intuitive features
Current interactions rely on writing or texting using live transcription apps. While those apps aids one-on-one conversations, it struggles in group discussions, as they are not capable of identifying or segregating voices
Difficulty in Identifying and segregating voices in real-time
People with deafness cannot distinguish or identify individuals during group conversations when multiple people are speaking simultaneously through an interpreter.
Loss of technical information due to translation
Deaf students struggle to maintain academic pace due to the loss of information in translation as Interpreter might find it difficult to transcribe Technical components of the course, making it challenging for them to stay on top of all the information
How did I solve the problem?
User interviews with deaf students
Conducted 4 interviews: 2 with students with hearing impairments and 2 with peers who have interacted with deaf students to gain both perspectives.
Data Synthesis &
Conceptualization
Employed Affinity Diagrams for data synthesis and leveraged the Walt-Disney approach for ideation of solutions
Sketching & Prototyping
Executed Crazy 8's sketching method alongside storyboards to visualize potential solutions. Developed interactive prototypes using Figma.
Evaluation &
Re-iteration
Performed a cognitive walkthrough testing and think aloud methods to assess the solution, and redesigned key solutions based on user responses
The Result?
The Solution
Bridge App actively listens to ongoing conversations, converting speech to text and offering real-time subtitles to the user. The conversation is subsequently saved within the Bridge mobile application. This eliminates the need of an Interpreter at all times.
Identifying and segregating voices in real-time
All the voices in the vicinity are identified based on voice pitches and modulations and are segregated based on color to identify various speakers simultaneously.
My Impact:
Translated Business Requirements into more than 75 user scenarios
App and Watch
Voice Contacts
Voice Contacts
Users have the option to save conversations and categorize them according to voice contacts, simplifying the process of information management for them.
My Impact:
Translated Business Requirements into more than 75 user scenarios
GenAI Assistant
Users have the option to save conversations. However, To read through an entire conversation could be a tedious tasks, hence, Bridge will use GenAI to create summary of the conversation as well as provide suggested activites etc.
My Impact:
Translated Business Requirements into more than 75 user scenarios
Design Process
But wait, How did we get there?
Oopsie, This Project is optimized for desktops and tablets Only.
However, Feel free to explore!
Introduction
Data Gathering
Data gathering
For our academic project, We wanted to work towards UN Sustainable Development Goals. Hence, We chose Goal 4 which focusses on ensuring Equal access to all levels of education including persons with disabilities
We have started our Data gathering by conducting Interviews in the below organizations
This is what they had to say!
Why is this happening?
Interpreters are not able to comprehend technical components of their curriculum
Currently, there are applications that translate speech to sign language, but there are none that support the reverse.
Desk Research
What is the impact of this problem?
The above study suggests that deaf students using educational interpreters only comprehend approximately 60%–75% of the content from lectures
These students also struggle with sustained attention, which impacts academic performance. This puts deaf children 1–4 grade levels behind their hearing peers in school
of Deaf, hard of hearing, and disabled students with disabilities reported feeling that
Social isolation was the main cause of stress
Competitive Analysis
Google Transcribe excels in real-time transcription, while Rogervoice specializes in transcribing phone calls, and Ava is designed for meetings and discussions. However, as of now, there isn't a platform that provides real-time identification and segregation of various voices.
Google Transcribe
Speech-to-text
Multiple Languages
Paid Subscriptions
ASL to Speech/Text
Voice Detection
Caption Phone Calls
Ava
Speech-to-text
Multiple Languages
Paid Subscriptions
ASL to Speech/Text
Voice Detection
Caption Phone Calls
Roger Voice
Speech-to-text
Multiple Languages
Paid Subscriptions
ASL to Speech/Text
Voice Detection
Caption Phone Calls
However, as of now, there isn't a platform that provides real-time identification and segregation of various voices.
Insights and Takeaways
Insights from research
My Takeaways
The existing method of interaction lacks intuitiveness.
Current method of interactions includes writing on a paper or texting. Which might not be effective during group discussions involving multiple
The product should be able to convert ASL to Speech/Text
Since hearing users may not understand sign language, the product should have the capability to translate ASL into real-time speech or text.
Loss of information due to translation
People with deafness cannot distinguish or identify individuals during group conversations when multiple people are speaking simultaneously through an interpreter.
The system should be able to identify segregate voices
Users should have access to the entire conversation, including identifying voices part of the conversation.
Social Isolation
Relying on an interpreter may not be the most effective method for conveying personal information in all situations.
The product should integrate seamlessly into the user's existing lifestyle.
The product should prioritize compactness and user-friendliness without requiring significant changes in current style of interaction
Ideation
For Ideation we used the Walt Disney method
The Dreamer
Any and Every wild Idea goes here.
The Realist
Which of the ideas are actually possible or practical.
The Critic
Its real-world implications from a more critical viewpoint
Our two favorite Ideas
Smart Gloves with Haptic sensors to identify and convert Sign-Language to Speech/Text
<
Scenario 1
A deaf student named Ron, Wants to participate in a group discussion
Ron wants to convey his ideas as well but none of the group members know sign language
Ron then uses his Haptic gloves
Those haptic gloves recognizes the sign language using the sensors on the gloves
The sign language is now converted into audio and is played via speaker
Now everyone can listen to what Ron has to say.
This seems
interesting!
I don’t know sign Language
And that is how you get methionylthreonylthreonylglutaminyl
That’s interesting
Great Job Ron!
Speaker
We’re sorry
Now I can have a conversation without any problem
Interaction using AR glasses
The Audio is now converted into captions real time
<
Scenario
A deaf student named Patrick, is interested to attend an important lecture in advanced sciences
His Interpreter couldn’t understand the technical words and he couldn’t sign them properly
The AR glasses recognizes the voice using the microphone
CC: And that is how you get methionylthreonylthreonylglutaminyl
Patrick now can actively listen to the class without any disturbance
This seems
interesting!
Technology is soo cool!
Microphone
AR Glasses
Since Patrick couldn’t follow the lecture, now Patrick can simply use his AR Glasses
I’m clueless, is that even a word?
Chapter 11
Hippopotomonstrosesquippedaliophobia
User flow
Wireframes
1
Onboarding flow
2
Recording a conversation
3
Adding a Voice contact
4
Adding Custom gestures
Usability testing
For our evaluation, we created tasks and recruited 4 experts to evaluate using the think-aloud method and the error reported was captured by our team. Later they were interviewed based on a questionnaire to gather more insights from them
Issues reported by the users
Lack of privacy for hearing users during conversation mode
A crucial revelation from one of the testers highlighted the necessity for a mechanism that allows hearing users to be aware of being recorded during conversation mode. In response, our suggested resolution involves implementing a notification for the hearing user group when the recording mode is activated.
Some sign language intricacies not captured by fish eye lens
An ASL expert has highlighted that certain sign language expressions involve intricate body movements that may not be fully captured by a fish-eye lens. Conducting further research on the ASL language could aid in addressing these edge scenarios.
Inadequate error handling
If there is a failure in the sign language conversion system, it identifies the Deaf person, but the hearing user remains unidentified.
My learnings
With no prior AR experience, the project proved both intriguing and challenging. Here are some lessons
I learned throughout this endeavor
Augmented Reality paves the way for new avenues.
Augmented Reality has the potential to inspire us to rethink and envision innovative solutions.
Improve accessibility through collaborative design.
Consider and embrace the perspectives of all users with mindfulness and inclusivity.
Empathize with users at every stage
Understand and connect with users on a personal level throughout their entire journey
Designer is not the user
Designing based solely on assumptions can introduce bias. It's crucial to stay mindful and rely on data rather than presumptions when creating scenarios.
Coffee is Overrated,
Let’s Connect for a Pizza!
(That’s Just a joke, I love Coffee)
2024
© Chandan Jonnavithula
Coffee is Overrated,
Let’s Connect for a Pizza!
(That’s Just a joke, I love Coffee)
