Skip to content

Case Study

Lip Reading Model

How LiRA is succeeding with developing lifesaving lipreading technologies with the help of Covintus.


Health & Wellness


Machine Learning


LiRA is an early-stage technology startup comprised of a multi-disciplinary group of professionals passionate about partnering with patients, caregivers, and health systems. Their overarching objective has been to provide lip-reading technology that empowers voiceless individuals and advances the standard of medical care. They entered Covintus’ accelerator program (Tech Tank) and won back in 2021. 


prototype for the smithsonian

Having identified the user’s need and developed an early proof of concept, Dr. Prince (CEO) felt it was time to expand into a prototype and, ultimately, a functional product. To accomplish this, he needed to engage professionals that could bring machine learning and video-to-speech technologies to the table in time for a scheduled conference at the Smithsonian Institution.


A level of initial research was necessary in order to determine the ideal model architecture.


Needed a pipeline of data to auto ingest videos from YouTube as well as contributors.


Model needed to be capable of successfully predicting phrases with 70-80% accuracy.


Both their web and mobile apps needed to be rebuilt to work properly for the event.

The Challenges

Build a working prototype

our Approach

Build, Train, Iterate

the Result

Huge milestone reached

"I really appreciate the culture and engineering mindset at Covintus, their team takes an iterative approach that helped us truly think through how to develop a minimum viable product”
LiRA Logo
Dr Andrew Prince


here are some of the technologies used in developing the app


Angular is used to build out the frontend for web and mobile.


Python is used to build out the backend of the web and mobile apps.


Azure is used to host the model, pipeline and both apps.


Natural Language Toolkit is used for language processing.


Pytorch is the machine learning framework used for the app.


YouTube is scraped to curate relevant speech videos for training.

the outcome

Thanks to the software development that Covintus has delivered, LiRA now has a functioning prototype that is being used to further develop the related technologies. “We’ve reached a huge milestone,” said Dr. Prince. “We have successfully created a sentence-based algorithm that has surpassed 70% functionality for short sentences. Now, our system can detect when a patient says they can’t breathe or their chest is hurting. This is a huge deal, and we all are very excited about what we will be able to achieve soon.”

Contact Us To Book Your
Free Consultation Today

Share some basic details and we can connect with you right away to discuss which package is right for you.