Waking to your voice, not to the room #92

New issue

Open

opened 2026-08-15 12:16:18 +00:00 by aiko · 1 comment

aiko commented

2026-08-15 12:16:18 +00:00

Owner

A wake word answers "was that the phrase". It does not answer "was that you". A television saying "Hey Denpa" wakes her; so does anyone else in the room, and so does a recording of you.

The want is real — hands-free is only comfortable if she answers her operator and not the room. But it is a second, separate model, and the tempting shortcut is the wrong one, so this issue exists mainly to record why.

Not by training the wake word on one voice

The obvious idea is to train #87's keyword head on recordings of one person instead of synthetic voices. It does not work, for two reasons worth stating once so nobody tries it twice:

The training set collapses. openWakeWord's positives are thousands of synthetic utterances precisely because a few dozen real ones overfit — to that microphone, that room, that day's voice. It would work beautifully in the chair it was recorded in and fail with a cold, at a distance, on the phone.
It conflates two questions. "Was the phrase said" and "who said it" have different right answers and different failure costs. Fused into one score there is no way to tell a missed phrase from a rejected speaker, which is exactly the invisible-failure problem #89 is already trying to avoid.

By a speaker embedding, after the phrase

The standard shape, and the one that fits where #89 already put the gate:

Enrol once. You say a handful of sentences. Each becomes a fixed-length embedding; the stored profile is their centroid. Not the audio — the vectors.
On each wake, embed the triggering audio and take the cosine similarity against the profile. Below a threshold, the utterance is dropped exactly as an unwoken one is.

The models are text-independent, so enrolment does not have to be the wake phrase and the profile keeps working if the phrase changes later. Current options that export ONNX and therefore run through the ort already in the tree (Cargo.toml:47): wespeaker's ECAPA-TDNN — including a 512-channel light variant meant for smaller deployments — and 3D-Speaker's CAM++ / ERes2NetV2. Any of them is a drop-in at the same point; the choice is a size and accuracy trade, not an architectural one.

Size is the open cost. These are tens of megabytes rather than #87's 1.7, which puts this in the same conversation as #44's 27 MB runtime rather than in the free-lunch category the wake word itself is in. Quantisation likely helps a lot. Measure before committing.

Two things it must not be sold as

It is not security. Text-independent verification on short audio has a real false-accept rate, and a recording of your voice passes trivially. It is a convenience filter that stops the television and the flatmate. If anything downstream ever treats "verified speaker" as authorisation, that is a different issue with a different bar.
It cannot run on the wake phrase alone. "Hey Denpa" is under a second, and verification accuracy falls off badly on clips that short — this is the single most likely way to build it and have it perform terribly. Score the whole utterance the phrase opened, or the phrase plus what follows. #89 already keeps that audio together, which is the reason to build this after it rather than beside it.

Open questions before building

What happens on rejection? Silently dropped is honest but indistinguishable from a missed wake phrase, and #15 is already about not being invisibly stuck. Some visible "heard, not recognised" may be needed.
More than one enrolled voice? A profile per person is barely more code than one. Worth deciding at enrolment rather than retrofitting.
Where does enrolment live? A settings flow that records and stores vectors, and can forget them. That is a genuine UI, not a checkbox — possibly its own issue once this is settled.
Does it drift? A profile recorded once, on one microphone, in one room. Whether it needs re-enrolment or gentle updating is unknown until it has been lived with.

#89 — the gate this hangs off, and the utterance audio it needs
#87 — why the wake model stays speaker-independent
#44 — the size budget this eats into
#15 — invisible failure states

A wake word answers "was that the phrase". It does not answer "was that you". A television saying "Hey Denpa" wakes her; so does anyone else in the room, and so does a recording of you. The want is real — hands-free is only comfortable if she answers her operator and not the room. But it is a **second, separate model**, and the tempting shortcut is the wrong one, so this issue exists mainly to record why. ## Not by training the wake word on one voice The obvious idea is to train `#87`'s keyword head on recordings of one person instead of synthetic voices. It does not work, for two reasons worth stating once so nobody tries it twice: - **The training set collapses.** openWakeWord's positives are thousands of synthetic utterances precisely because a few dozen real ones overfit — to that microphone, that room, that day's voice. It would work beautifully in the chair it was recorded in and fail with a cold, at a distance, on the phone. - **It conflates two questions.** "Was the phrase said" and "who said it" have different right answers and different failure costs. Fused into one score there is no way to tell a missed phrase from a rejected speaker, which is exactly the invisible-failure problem `#89` is already trying to avoid. ## By a speaker embedding, after the phrase The standard shape, and the one that fits where `#89` already put the gate: 1. **Enrol once.** You say a handful of sentences. Each becomes a fixed-length embedding; the stored profile is their centroid. Not the audio — the vectors. 2. **On each wake**, embed the triggering audio and take the cosine similarity against the profile. Below a threshold, the utterance is dropped exactly as an unwoken one is. The models are text-independent, so enrolment does not have to be the wake phrase and the profile keeps working if the phrase changes later. Current options that export ONNX and therefore run through the `ort` already in the tree (`Cargo.toml:47`): **wespeaker**'s ECAPA-TDNN — including a 512-channel light variant meant for smaller deployments — and **3D-Speaker**'s CAM++ / ERes2NetV2. Any of them is a drop-in at the same point; the choice is a size and accuracy trade, not an architectural one. **Size is the open cost.** These are tens of megabytes rather than `#87`'s 1.7, which puts this in the same conversation as `#44`'s 27 MB runtime rather than in the free-lunch category the wake word itself is in. Quantisation likely helps a lot. Measure before committing. ## Two things it must not be sold as - **It is not security.** Text-independent verification on short audio has a real false-accept rate, and a recording of your voice passes trivially. It is a convenience filter that stops the television and the flatmate. If anything downstream ever treats "verified speaker" as authorisation, that is a different issue with a different bar. - **It cannot run on the wake phrase alone.** "Hey Denpa" is under a second, and verification accuracy falls off badly on clips that short — this is the single most likely way to build it and have it perform terribly. Score the whole utterance the phrase opened, or the phrase plus what follows. `#89` already keeps that audio together, which is the reason to build this after it rather than beside it. ## Open questions before building - **What happens on rejection?** Silently dropped is honest but indistinguishable from a missed wake phrase, and `#15` is already about not being invisibly stuck. Some visible "heard, not recognised" may be needed. - **More than one enrolled voice?** A profile per person is barely more code than one. Worth deciding at enrolment rather than retrofitting. - **Where does enrolment live?** A settings flow that records and stores vectors, and can forget them. That is a genuine UI, not a checkbox — possibly its own issue once this is settled. - **Does it drift?** A profile recorded once, on one microphone, in one room. Whether it needs re-enrolment or gentle updating is unknown until it has been lived with. ## Related - `#89` — the gate this hangs off, and the utterance audio it needs - `#87` — why the wake model stays speaker-independent - `#44` — the size budget this eats into - `#15` — invisible failure states

aiko added the

ready-for-human

labels

2026-08-15 12:16:18 +00:00

aiko commented

2026-08-15 12:20:50 +00:00

Author

Owner

The "it is not security" warning above is sharper than it was written.

#87 now plans to generate wake-phrase positives with the Qwen3-TTS already running in Comfy, which clones a speaker from three seconds of reference audio. That capability sits in this project's own stack, on this machine.

So the replay concern is not theoretical and does not require an attacker with resources. Anything that has captured a few seconds of the operator speaking — a video call, a voice note, a clip posted anywhere — is enough input for a generator we already run to produce arbitrary new speech in that voice. A speaker-embedding gate compares timbre and prosody; that is precisely what the cloner reproduces.

This does not argue against building the gate. It filters the television, the flatmate, and the podcast playing in the next room, which is the entire job it was asked to do. It argues against ever letting it grow into something else: no downstream capability should be unlocked by "the speaker was verified". If a decision ever needs to be gated on identity rather than convenience, it needs a different mechanism and its own issue.

Worth keeping in mind when #91's answer arrives too — the friendlier a gate feels in daily use, the more tempting it becomes to lean on it.

**The "it is not security" warning above is sharper than it was written.** `#87` now plans to generate wake-phrase positives with the Qwen3-TTS already running in Comfy, which clones a speaker from **three seconds** of reference audio. That capability sits in this project's own stack, on this machine. So the replay concern is not theoretical and does not require an attacker with resources. Anything that has captured a few seconds of the operator speaking — a video call, a voice note, a clip posted anywhere — is enough input for a generator we already run to produce arbitrary new speech in that voice. A speaker-embedding gate compares timbre and prosody; that is precisely what the cloner reproduces. This does not argue against building the gate. It filters the television, the flatmate, and the podcast playing in the next room, which is the entire job it was asked to do. It argues against ever letting it grow into something else: **no downstream capability should be unlocked by "the speaker was verified".** If a decision ever needs to be gated on identity rather than convenience, it needs a different mechanism and its own issue. Worth keeping in mind when `#91`'s answer arrives too — the friendlier a gate feels in daily use, the more tempting it becomes to lean on it.