1. Dataset Processing
GRID corpus video was processed using MediaPipe facial landmarks and OpenCV to isolate the mouth region from each speaker.
Senior Project
VSR on Embedded Microcontroller
A lightweight Visual Speech Recognition (commonly known as VSR or "lip reading") AI model designed to classify visually distinct speech sounds using a convolutional neural network running on constrained STM32H7 embedded hardware.
Project Overview
Visual Speech Recognition at the Edge
Modern visual speech recognition systems often rely on large ai models, large datasets, and GPU-class computing resources.
My goal was to determine whether the problem could be reduced enough to run meaningfully on a microcontroller. Instead of performing complete word or sentence-level lip reading, I developed a five-class visual phoneme classifier using short sequences of grayscale mouth images.
Design Challenge
Reducing a GPU-scale problem
Full-scale visual speech recognition systems typically use large convolutional or recurrent neural networks operating on complete video sequences.
My design instead focused on reducing both the input data and the neural network. Video was cropped to the mouth region, converted to grayscale, divided into short temporal windows, and represented as compact NumPy arrays.
This significantly reduced the amount of information that needed to be processed while preserving the mouth motion required for classification.
Approach
System Pipeline
The project combined computer vision, speech processing, machine learning, and embedded deployment.
2. Speech Alignment
Audio and text transcripts were processed with Montreal Forced Aligner to obtain phoneme-level timestamps for each utterance.
3. CNN Classification
Short grayscale frame sequences were classified using lightweight 2D convolutional AI neural networks trained with TensorFlow and Keras.
AI Neural Network
Lightweight CNN Architecture
The network used three convolutional stages followed by a compact dense classifier.
I tested multiple input resolutions, temporal depths, and model widths to evaluate the tradeoff between classification accuracy and embedded resource usage.
Five visually distinct output classes were used: M, SH, O, E, and FV.
Embedded Deployment
STM32H7
Selected trained networks were converted to INT8 TensorFlow Lite models and imported through X-CUBE-AI in STM32CubeIDE.
Quantization reduced the model memory requirements enough for inference to run directly on the STM32H7 without GPU acceleration or cloud computing.
Results
82.46% Best Offline Accuracy
The best offline CNN configuration achieved 82.46% test accuracy on the five-class visual speech classification task.
A 32x32x7 Medium model was successfully quantized and deployed to the STM32H7. The embedded model required approximately 76 KB of flash and 20 KB of RAM.
The results demonstrated that a carefully constrained visual speech recognition system could achieve useful classification performance while remaining small enough for microcontroller deployment.
Project Documents
Presentation & Report
Full documentation from the project is available below.