A made-up drive-thru in the middle of Berlin

Sometime in July, our recording team spent a couple of afternoons in the streets of Berlin recording themselves ordering fast food. They took shifts: one person stood on the sidewalk holding a microphone and a menu card up to a car window, while one or two other people sat in a stationary rented car reading items off the menu and thinking out loud about their order with the radio on. The team was surrounded by traffic and city noises. But just to make sure that the noise bleeding into the recording was loud enough, a speaker was on the ground playing random engine noises into the scene.

Sasha, Yunjoo, Tif and a guest, Rex, spent those two afternoons recording evaluation sets for our voice activity detection (VAD) models. And we decided to write a blog about it. First of all, because we thought it was a good idea to share some insights on this valuable yet fun and somewhat bizarre sidequest, but mainly because this step was as important as the rest of the machine learning lifecycle: it’s essential for us to trust the performance numbers coming from our models.

But why did we record this in the first place?

There is definitely a much cheaper way to make noisy audio (which we use, just as everybody else): take clean speech, add noise recordings, convolve it with an impulse response, run it through a few sound effects if feeling fancy, and there you go. You can generate as many hours of degraded speech as you need. This does wonders for training a model, and most of what our VADs learn comes from this synthetic approach (well, perhaps slightly more detailed, but you get the gist).

As evaluation data, though, this data collage doesn’t work so well. A synthetic test set can only contain the degradations you thought to put in it, so it measures your model against your own imagination. If something breaks speech detection in the real world and it was not on your list, the test set will not catch it. A big limitation if you ask me.

These recordings encode how hard reality can be. For example, the cheap intercom setup degrades the recorded audio quality (I sadly won’t be speaking about microphones on this blog, but perhaps we do a microphone-focused one in the future). Another case that is super hard to replicate synthetically is the fact that the customer doesn’t always speak directly into the microphone, they might turn their head mid-sentence resulting in quieter audio. And don’t forget that on top of that, the unpredictable background city noise is constantly creeping into the recording from all directions. You can always approximate these situations, but getting all of the variables right can only happen with real hardware in a real street.

What we were testing

VAD decides, moment to moment, whether anyone is speaking. In a voice agent stack it runs first, which makes it sort of the “ears of the conversation”. If the VAD signal does not go above a certain threshold, the Speech-To-Text model (let’s say, a well known open source one, like Parakeet) in charge of the speech transcription never receives the audio. That snowballs into a user talking to a machine that isn’t hearing them, and having to repeat themselves until it does. A pretty frustrating scenario, speaking from experience.

Drive-thru scenarios are one of the hardest cases we’ve ever dealt with. There is one microphone at car window distance, an engine idling, wind, traffic, and whatever noise the city is emitting behind it, and in many lanes a cheap intercom on top of all of it. It’s also a real product problem rather than a hypothetical one, which is why it ended up as one of the two headline benchmarks in the VAD Multi Speaker 2.1 launch post, check that out if you haven’t already.

Two datasets, two attempts

The team first recorded a dataset using strangers as “customers” and didn’t use an intercom system. But after evaluating the first attempt’s final sound, they decided that they could still push the quality degradation further to get closer to a real drive-thru scenario. So we purchased an intercom microphone + amplifier system and they gave it another go on a different day, this time recording themselves, no strangers involved (which gave them more freedom in terms of acting).

The first dataset captures a wider range of voices, windier conditions, and audio recorded directly on a handheld Tascam. As for the second one, it encompasses a wider range of voice emotions (Tif and Rex stole the show), audio recorded through an intercom setup, and extra engine noises coming from a portable speaker.

Having these two datasets helped evaluate our models in more diverse scenes, which gave us better insight into which conditions the model struggled in more, and which less. And ultimately, this allowed us to iterate with a clearer goal.

Labelling every voice

After recording, we had to mark, by hand, where the speech was present. So, continuously through each recording, we drew it out on the waveform in a labelling/review tool the recording team custom-built.

Since a task like this can get tedious quite fast, we distributed the work among a couple of colleagues. In order to achieve homogeneous results, we agreed on a set of rules covering what counts as speech and the conditions a labeller has to meet before making that call. Having all the labellers on the same page is what makes the ground truth unambiguous.

The main rule: speech is speech, regardless of where it comes from. If you can make out the words at a normal listening volume, it gets labelled, whether someone said them directly into the microphone, or a random passenger-seat person yelling about wanting large fries. Labellers can replay a segment as many times as they want, but they can’t crank up the volume to catch something, since that isn’t what the model gets either.

Deciding which of those voices actually matters is a separate problem, and a separate model: VAD Voice Focus, which only triggers on the main speaker. We never leave a voice unlabelled just because it isn’t the customer’s. That’s an important distinction to make, since this conceptual difference requires a very different training approach.

What the recordings showed

We compared VAD Multi Speaker 2.1 with the well-known, open-source Silero VAD. These figures are streaming probabilities at the default threshold, with no post-processing.

First dataset, no intercom

Second dataset, with intercom

The AUC gap is the smallest of the three, and it’s the metric that treats Silero most fairly: AUC ignores where you set the “is-there-speech” decision threshold, so what it tells us is that Silero isn’t really confused by this audio. It just doesn’t cross its own threshold often enough to report speech as speech, which for a model making a call every few milliseconds is the difference that matters. That is where the gap opens up: the first dataset already cost Silero close to half of the speech, and adding the intercom chain took it down to roughly a third detected, while our VAD gave up quite little in comparison. Which is exactly what we were hoping to see from the latest training recipe.

The full benchmark, across both the drive-thru recordings and several hours of real phone calls, is in the VAD Multi Speaker 2.1 launch post.

A big shoutout

Our recording team has been working on expanding our evaluation repertoire for years now. Their work has kept our models grounded in reality since day 1. They’re the reason why we confidently talk so often about solving real-world audio issues, because we’re always putting our models to the test with the latest data that they provide us. So this blog was meant to shine some well deserved spotlight on our super talented and hard working recording team.