Reading lips through a face mask, with WiFi instead of a camera

Lip-reading technology usually means cameras: a system watches a person's mouth and tries to infer what's being said from the shape it makes. That works in good conditions, but face masks broke it almost completely — exactly the moment, during the pandemic, when lip-reading-dependent communication aids were needed most. It also raises an obvious privacy concern: a lip-reading camera is, by definition, recording your face.

This research swapped the camera for two kinds of radio sensor — WiFi and radar — and asked a simpler question: can the way radio waves bounce off a moving mouth and jaw, even through a fabric mask, carry enough information to tell sounds apart? The team built a dataset of five vowel sounds (A, E, I, O, U) plus silence, recorded with both WiFi and radar hardware, with and without a face mask on.

The results were striking. Neural network models reading the WiFi data classified the vowels correctly 95% of the time with no mask, and still 80% of the time with a mask on. Radar data, processed with a VGG16 deep learning model, did even better through the mask: 91% accuracy unmasked, 83% masked.

No image of the speaker's face is ever captured at any point — the system works entirely from radio reflections. That makes it a genuine candidate for next-generation hearing aid technology: something that could help disambiguate speech in noisy rooms or support lip-reading-based communication, in the dark, through obstruction, and without the privacy cost of a camera. The work was published in Nature Communications and picked up by TechXplore, Hearing Review and Healthcare-in-Europe.