A made-up drive-thru in the middle of Berlin

Sometime in July, our recording team spent an afternoon in the streets of Berlin ordering fast-food that nobody was going to cook. They took shifts. One person stood on the sidewalk, holding a microphone and a menu card up to a car window. One or two others sat in a stationary rented car, reading items off the menu and thinking out loud about their order, radio on. Traffic and city noises were all around. And just to make sure the background was noisy enough, a speaker was on the ground playing random engine noises into the scene.

Sasha, Yunjoo, Tif and a guest, Rex, were recording an evaluation set for our voice activity detection (VAD) models. And we decided to write a blog about it, since this step was as important as the rest of the machine learning lifecycle: it is what lets us trust the numbers that come out of it.

Why did we record this?

There is a much cheaper way to make noisy audio, and we use it constantly. Take clean speech, add noise recordings, convolve it with an impulse response, run it through some extra sound effects, and you can generate as many hours of degraded speech as you need. For training a model, this works well, and most of what our VADs learn from is built exactly this way.

As evaluation data, though, it doesn't work so well. A synthetic test set can only contain the degradations you thought to put in it, so it measures your model against your own imagination. If something breaks speech detection in the real world and it was not on your list, the test set will not catch it.

These recordings encode how hard reality can be. The cheap intercom setup degrades the audio quality. The customer doesn't always speak directly into the microphone, they might turn their head mid-sentence resulting in quieter audio. And on top of that, the unpredictable background city noise is constantly creeping into the recording. You can always approximate these situations, but getting all of the variables right can only happen with real hardware in a real street.

What we were testing

VAD decides, moment to moment, whether anyone is speaking. In a voice agent stack it runs first, which makes it the "ears of the conversation". If the VAD does not open, the Speech-To-Text model in charge of the transcription never receives the audio and the customer ends up repeating themselves to a system that has already stopped listening, which can be pretty frustrating.

Drive-thru scenarios are one of the hardest cases we've ever dealt with. There is one microphone at car window distance, an engine idling, wind and traffic behind it, and in many lanes a cheap intercom on top of all of it. It is also a real product problem rather than a hypothetical one, which is why it ended up as one of the two headline benchmarks in the VAD 2.1 launch post.

Two dataset iterations

The team first recorded a dataset using strangers as "customers" and didn't use the intercom system. But after evaluating the final sound of the first dataset, they decided that we could still push the quality degradation further to get closer to a real scenario. So we purchased an intercom microphone + amplifier system and they gave it another go on a different day, this time without strangers.

The first dataset captures a wider range of voices, windier conditions, and audio recorded directly on a handheld Tascam. As for the second one, it encompasses a wider range of voice emotions (Tif and Rex stole the show), audio recorded through an intercom setup, and extra engine noises coming from a portable speaker.

Having these two datasets helped evaluate our models in more diverse scenes, which showed us the conditions the model would struggle more or less in. And ultimately, allowed us to iterate with a clearer goal.

Labelling every voice

After recording, we had to mark, by hand, where the speech was present. So, continuously through each recording, we drew it out on the waveform in a labelling/review tool the recording team custom built.

Since a task like this can get tedious quite fast, we distributed the work among a couple of colleagues. In order to achieve homogeneous results, we agreed on a set of rules covering what counts as speech and the conditions a labeller has to meet before making that call. Having all the labellers on the same page is what makes the ground truth unambiguous.

The main rule: speech is speech, regardless of where it comes from. If you can make out the words at a normal listening volume, it gets labelled, whether someone said them into the microphone, or somebody else on the passenger seat wanted to contribute to the order. Labellers can replay a segment as many times as they want, but they can't turn the volume up to catch something, since that isn't what the model gets either.

Deciding which of those voices actually matters is a separate problem, and a separate model: VAD Voice Focus, which only triggers on the main speaker. We never leave a voice unlabelled just because it isn't the customer's.

What the recordings showed

We compared VAD 2.1 with Silero VAD. These figures are streaming probabilities at the default threshold, with no post-processing.

First dataset, no intercom

Second dataset, with intercom

The AUC gap is the smallest of the three, and it's the metric that treats Silero most fairly: AUC ignores where you set the decision threshold, so what it tells us is that Silero isn't really confused by this audio. It just doesn't cross its own threshold often enough to report that speech as speech, which for a model making a call every few milliseconds is the difference that matters. That is where the gap opens up: the first dataset already cost Silero close to half of the speech, and adding the intercom chain took it down to roughly a third detected, while our VAD gave up quite little in comparison.

The full benchmark, across both the drive-thru recordings and several hours of real phone calls, is in the VAD 2.1 launch post.

A big shoutout

Our recording team has been working on expanding our evaluation repertoire for years now. Their work has kept our models grounded in reality since day 1. They're the reason why we confidently talk so often about solving real-world audio issues, because we're always putting our models to the test with the latest data that they provide us. So this blog was meant to shine some well deserved spotlight on our super talented and hard working recording team.