Interspeech 2026: Efficient audio AI moves closer to the edge

The talk also treated deepfake detection as an MLOps problem. A new speech generator should not automatically trigger detector retraining. Retraining should follow a measurable shift in the data distribution.

For deployed systems, this replaces a reactive update cycle with monitoring based on observable drift.

Small audio-language models running on a phone

Paper: Samsone: A Family of Open Small Audio Language Models for On-Device Inference

Samsone is a family of audio-language models from Samsung and the University of Kraków. The models listen to audio and generate a text response on the device, without sending the recording to the cloud.

The architecture connects a Whisper-Tiny audio encoder to a lightweight projector and a SmolLM2 language decoder. Depth pruning and vocabulary reduction produce versions with 99 million, 134 million, and 356 million parameters.

The researchers tested Samsone on a Samsung Galaxy S25 Ultra using MMAU, a benchmark with 10,000 human-annotated questions about speech, sound, and music. Every Samsone variant outperformed Mellow, the previous leading small audio-language model, across all three domains.

Because the models are open source and tested on commercial hardware, Samsone offers a useful reference for private, responsive audio assistants at the edge.

Using Mamba for audio-language processing

Paper: https://www.isca-archive.org/interspeech_2026/lee26d_interspeech.pdf

SAM, developed at Sogang University, replaces the transformer backbone of an audio-language model with Mamba-2. An 88 million-parameter EAT-Base encoder converts audio into a sequence of representations. A connector then maps them into the format used by the Mamba-2 backbone.

State-space models process sequence length with linear computational complexity and maintain a fixed-size state. That makes them an interesting option for long recordings and streaming systems, where transformer attention can become expensive.

SAM shows that a state-space model can act as the language backbone, rather than only the audio encoder. The next test is how that efficiency translates into memory, latency, and power on deployed hardware.

Real-time enhancement for hearing aids

Paper: RT-Tango: Real-Time Distributed Binaural Speech Enhancement for Low-Power Hearing Aid Devices

RT-Tango targets a demanding audio environment: a pair of hearing aids with strict power and latency limits.

Each device processes audio locally and exchanges a compact representation with the other ear. Information from both sides estimates speech and noise masks that guide spatial filtering. The neural network does not generate the waveform itself.

Four design choices reduce the workload. ERB feature compression shrinks the input using a perceptual frequency scale. Grouped recurrent networks lower the cost of temporal modeling. Temporal sparsification reuses earlier masks instead of running the network on every frame. An asymmetric STFT preserves frequency resolution while reducing delay.

The complete system runs at 33.4 MMAC/s with 8 milliseconds of algorithmic latency. It is a strong example of neural inference and signal processing working together under tight hardware constraints.

Processing fewer audio frames

Paper: HALO: Half-Frame-Rate Adaptive Learnable Operator for Lightweight STFT-Based Speech Enhancement

Speech-enhancement networks often process overlapping STFT frames, even though adjacent frames contain repeated information. HALO cuts that workload without simply discarding every second frame.

Before the enhancement backbone, a dynamic convolution combines pairs of adjacent frames. After processing, a learned restoration stage expands each output back into two complex-spectral frames.

In one comparison, a wider GTCRN with HALO used similar compute to the baseline, 32.85 versus 33.83 MMAC/s, while improving PESQ from 2.10 to 2.20.

Because HALO surrounds the backbone rather than replacing it, engineers can test it with existing STFT-based enhancement models.

Smarter quantization for speech recognition

Paper: Not All Frames Are Equal: Difference-Aware Quantization for Ultra-Low-Bit ASR

DiffAQ improves post-training quantization for speech-recognition models such as Whisper.

Standard GPTQ calibration treats each audio frame equally. But stable or silent frames can dominate the data even though rapidly changing frames often contain more information about the current phoneme.

DiffAQ measures changes between consecutive frame representations and converts them into importance weights. Those weights affect calibration, while the standard GPTQ quantization process remains unchanged.

With 2-bit weights and 16-bit activations, DiffAQ reduced word error rate from 17.53% to 12.93%, a 26% relative reduction. It achieved this without retraining the model.

The result offers a useful lesson for edge AI: calibration should reflect which inputs carry information, not only how often they occur.

Smaller models through personalization and reuse

Several poster papers focused on reducing the cost of systems that must listen continuously.

QuadVAD detects speech boundaries at 4-millisecond resolution using about 25 KB of memory and 25 MFLOPs. It matched or exceeded Silero VAD and TEN VAD on TIMIT, Earnings21, and VoxConverse. Precise boundaries can help prevent downstream systems from clipping short utterances.

OnDA personalizes keyword spotting on the device. It pseudo-labels a user’s audio, prunes less important channels, and then fine-tunes the remaining model. The method delivered up to 9.63 times compression and about 1.5 times faster adaptation and inference.

Another Samsung paper divides a keyword-spotting CNN into sub-models that run at different rates and reuse buffered activations. On an Arm Cortex-M55, it maintained accuracy while cutting compute by as much as 82% compared with standard sliding-window inference.

Together, these papers show that efficiency can come from execution strategy and data handling, not only from designing a smaller neural network.

One encoder for speech, sound, and music

Paper: USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding

USAD 2.0, from MIT CSAIL, distills specialist teachers for speech, environmental sound, and music into one universal audio encoder.

Many perception systems ship separate encoders for different tasks. That duplicates parameters, memory transfers, and maintenance work. A shared encoder could support several downstream models within a fixed memory budget.

The practical value will depend on how closely it matches specialist models on each task. Still, USAD 2.0 points toward simpler and more efficient audio-perception stacks.

What we took away

The papers we selected begin with deployment constraints and design around them. They reduce repeated work, improve calibration, share representations, or combine neural inference with established signal processing.

That discipline will matter as audio AI moves closer to microphones, hearing devices, vehicles, and other sensors. Better edge audio requires more than smaller models. Engineers must decide what to compute, when to compute it, and which information should remain local.

Interspeech 2026 showed meaningful progress on all three questions.