Case Study
Lip Reading Model
How LiRA is succeeding with developing lifesaving lipreading technologies with the help of Covintus.
Industry
Health & Wellness
Category
Machine Learning
Overview
LiRA is an early-stage technology startup comprised of a multi-disciplinary group of professionals passionate about partnering with patients, caregivers, and health systems. Their overarching objective has been to provide lip-reading technology that empowers voiceless individuals and advances the standard of medical care. They entered Covintus’ accelerator program (Tech Tank) and won back in 2021.
Requirements
prototype for the smithsonian
Having identified the user’s need and developed an early proof of concept, Dr. Prince (CEO) felt it was time to expand into a prototype and, ultimately, a functional product. To accomplish this, he needed to engage professionals that could bring machine learning and video-to-speech technologies to the table in time for a scheduled conference at the Smithsonian Institution.
Conduct
Research
A level of initial research was necessary in order to determine the ideal model architecture.
Build
Pipeline
Needed a pipeline of data to auto ingest videos from YouTube as well as contributors.
develop
Model
Model needed to be capable of successfully predicting phrases with 70-80% accuracy.
Rebuild
Applications
Both their web and mobile apps needed to be rebuilt to work properly for the event.
The Challenges
Build a working prototype
- Needed to get a working prototype in less than 3 months
- needed a minimum of 70% accuracy when being demoed to participants
- Needed to rebuild their mobile application to make it usable for the covention
- Needed to overcome lack of training data for the model
our Approach
Build, Train, Iterate
- Conducting product strategy workshops
- UI/UX design and usability testing
- modular architecture based on microservices
- intruducing tailored SCRUM processes
- iterate fast for feedback looops
the Result
Huge milestone reached
- Delivered a working prototype in time for Smithsonian
- model exceeded the 70% accuracy required
- System can now detect basic patient requests
- Web app was ready to collect training video
- Mobile app was ready to successfully demo the product
- Working on building a huge library to extend capabilities
Technologies
here are some of the technologies used in developing the app
Angular
Angular is used to build out the frontend for web and mobile.
Python
Python is used to build out the backend of the web and mobile apps.
Azure
Azure is used to host the model, pipeline and both apps.
NLTK
Natural Language Toolkit is used for language processing.
PyTorch
Pytorch is the machine learning framework used for the app.
YouTube
YouTube is scraped to curate relevant speech videos for training.
the outcome
Thanks to the software development that Covintus has delivered, LiRA now has a functioning prototype that is being used to further develop the related technologies. “We’ve reached a huge milestone,” said Dr. Prince. “We have successfully created a sentence-based algorithm that has surpassed 70% functionality for short sentences. Now, our system can detect when a patient says they can’t breathe or their chest is hurting. This is a huge deal, and we all are very excited about what we will be able to achieve soon.”
Share some basic details and we can connect with you right away to discuss which package is right for you.
