aiko/denpa

Fork 0

Before she can wake to her name, the name has to exist as a file #87

New issue

Open

opened 2026-08-15 12:16:07 +00:00 by aiko · 9 comments

aiko commented

2026-08-15 12:16:07 +00:00

Owner

Before anything can listen for the wake phrase, the wake phrase has to exist as a file.

"Hey Denpa" is not one of openWakeWord's released models, so it has to be trained. This is the only part of hands-free waking that cannot be written in-tree, and it is the part everything else waits on — so it goes first, and it is a human's job because it needs a GPU hour and a judgement call about what "good enough" sounds like.

What comes out

Three ONNX files, committed as runtime assets:

File	What it is	Rough size
`melspectrogram.onnx`	audio → mel frames	~200 KB
`embedding_model.onnx`	Google speech embedding, shared by every keyword	~1.3 MB
`hey_denpa.onnx`	the keyword head, the only trained part	~200 KB

The first two are stock openWakeWord and are downloaded, not trained. Only the third is ours.

About 1.7 MB total. Worth stating next to #55, which weighed whisper at 75–500 MB and #44, where ONNX Runtime's own 27 MB already dominates the bundle. Waking to a name costs roughly a fifteenth of the runtime that is already shipping. That asymmetry is the argument for doing this before on-device ASR, not after.

How it is trained

Positives are synthetic. piper-sample-generator speaks "Hey Denpa" across a few thousand voices, speeds, and pitches; negatives come from openWakeWord's published negative corpora plus room noise. The keyword head trains on the embeddings, not on audio, which is why it is small and why it takes under two hours rather than days.

The official notebook has been stale since 2023. Two current paths:

alfiedennen/openwakeword-colab-2026 — run-all-and-walk-away, 75–90 min on Colab Pro, two lines to edit.
LiveKit's one-command trainer — newer, exports the same openWakeWord-compatible ONNX.

Either is fine; they produce the same artefact. Use whichever runs without a fight.

Why "Hey Denpa" rather than "Denpa"

Two syllables is the floor for a keyword that is not constantly tripped, and "Denpa" alone sits right on it. The "Hey" prefix is not decoration — it doubles the phonetic evidence the model has to work with, and it is the reason nearly every shipped wake phrase has one. It also gives the speaker gate in #92 something longer than 400 ms to work with, which it badly needs.

What "done" means

Not "it fired once". A report committed beside the model with:

Detection rate across at least 20 held-out utterances of the phrase, spoken by an actual person, at conversational distance. The 2026 notebook's own example run reported a median score around 0.85 across 13 utterances — that is the shape of a healthy result, not a target to game.
False accepts over a continuous hour of ordinary room audio with the machine doing something else. Zero is the expectation; anything above it means the threshold recommendation changes.
A recommended threshold, which is what #90 ships as the default.

Record all three. #88 cannot tune what nobody measured, and a wake word that fires at the television is worse than a keypress.

#88 — where it runs. Can be built against a stock model (hey_jarvis) before this lands, so it is not blocked on this.
#89 — what a detection actually does
#90 — the threshold this report recommends
#92 — waking to a particular voice rather than any voice
#55 — on-device whisper, the expensive neighbour
#44 — ort on Android, arm64-only, NDK r28

Before anything can listen for the wake phrase, the wake phrase has to exist as a file. "Hey Denpa" is not one of openWakeWord's released models, so it has to be trained. This is the only part of hands-free waking that cannot be written in-tree, and it is the part everything else waits on — so it goes first, and it is a human's job because it needs a GPU hour and a judgement call about what "good enough" sounds like. ## What comes out Three ONNX files, committed as runtime assets: | File | What it is | Rough size | |---|---|---| | `melspectrogram.onnx` | audio → mel frames | ~200 KB | | `embedding_model.onnx` | Google speech embedding, shared by every keyword | ~1.3 MB | | `hey_denpa.onnx` | the keyword head, the only trained part | ~200 KB | The first two are stock openWakeWord and are downloaded, not trained. Only the third is ours. **About 1.7 MB total.** Worth stating next to `#55`, which weighed whisper at 75–500 MB and `#44`, where ONNX Runtime's own 27 MB already dominates the bundle. Waking to a name costs roughly a fifteenth of the runtime that is already shipping. That asymmetry is the argument for doing this before on-device ASR, not after. ## How it is trained Positives are synthetic. `piper-sample-generator` speaks "Hey Denpa" across a few thousand voices, speeds, and pitches; negatives come from openWakeWord's published negative corpora plus room noise. The keyword head trains on the embeddings, not on audio, which is why it is small and why it takes under two hours rather than days. The official notebook has been stale since 2023. Two current paths: - `alfiedennen/openwakeword-colab-2026` — run-all-and-walk-away, 75–90 min on Colab Pro, two lines to edit. - LiveKit's one-command trainer — newer, exports the same openWakeWord-compatible ONNX. Either is fine; they produce the same artefact. Use whichever runs without a fight. ## Why "Hey Denpa" rather than "Denpa" Two syllables is the floor for a keyword that is not constantly tripped, and "Denpa" alone sits right on it. The "Hey" prefix is not decoration — it doubles the phonetic evidence the model has to work with, and it is the reason nearly every shipped wake phrase has one. It also gives the speaker gate in `#92` something longer than 400 ms to work with, which it badly needs. ## What "done" means Not "it fired once". A report committed beside the model with: - **Detection rate** across at least 20 held-out utterances of the phrase, spoken by an actual person, at conversational distance. The 2026 notebook's own example run reported a median score around 0.85 across 13 utterances — that is the shape of a healthy result, not a target to game. - **False accepts** over a continuous hour of ordinary room audio with the machine doing something else. Zero is the expectation; anything above it means the threshold recommendation changes. - **A recommended threshold**, which is what `#90` ships as the default. Record all three. `#88` cannot tune what nobody measured, and a wake word that fires at the television is worse than a keypress. ## Related - `#88` — where it runs. Can be built against a stock model (`hey_jarvis`) before this lands, so it is not blocked on this. - `#89` — what a detection actually does - `#90` — the threshold this report recommends - `#92` — waking to a particular voice rather than any voice - `#55` — on-device whisper, the expensive neighbour - `#44` — ort on Android, arm64-only, NDK r28

aiko added the

ready-for-human

labels

2026-08-15 12:16:07 +00:00

aiko commented

2026-08-15 12:20:49 +00:00

Author

Owner

Positives can come from the Qwen3-TTS already running in Comfy, rather than from piper. Operator's note, 2026-08-15: Denpa already calls it, so the generator exists and does not need standing up.

That is a better starting point than the issue assumed, but only if it is driven for the right thing. The metric for wake-word positives is speaker diversity, not naturalness. A keyword head trained on nine beautiful voices overfits to nine timbres — the same failure #92 refuses for one voice, just less extreme. piper-sample-generator's value was never audio quality; it was ~900 speakers, cheaply, with augmentation on top.

Qwen3-TTS can beat that on the axis that matters, two ways:

VoiceDesign — a voice from a text description. This is the interesting one: the diversity axis becomes programmatic. Enumerate a grid of age, accent, pitch, pace, delivery and generate a speaker per cell. No corpus to source, no licences to read.
Zero-shot cloning from 3 s of reference. Point it at a few thousand short clips from a public corpus and every one becomes a distinct speaker with a real timbre and a real accent.

The nine preset voices are the trap. They are the path of least resistance and the one that produces a model that works for nobody.

What changes in the plan

Throughput is the new constraint. Cloning or designing per clip is far heavier than piper's small VITS, which emits thousands a minute. Tens of thousands of positives is plausibly an overnight job rather than a coffee break. Worth a timing probe on a hundred clips before committing to a target count — and worth keeping piper in reserve for bulk if it hurts.
Drive Comfy headlessly. Its HTTP API, a workflow parameterised by text and reference, in a loop. Not the GUI.
The eval matters more, not less. openWakeWord's published negative corpora and its threshold guidance were calibrated against piper-shaped positives. Changing the positive distribution is fine and probably an improvement, but it means the numbers other people report transfer less well to ours. The held-out real-voice detection rate and the hour of false-accept audio are now the only trustworthy signal, not a formality.
Still never evaluate on synthetic. Whatever generates the training set, the held-out utterances are recorded by an actual person on actual hardware. A model graded by the generator that trained it will always look excellent.

Augmentation does not change: room impulse responses, noise, gain and speed variation on top of whatever the generator produced.

**Positives can come from the Qwen3-TTS already running in Comfy, rather than from piper.** Operator's note, 2026-08-15: Denpa already calls it, so the generator exists and does not need standing up. That is a better starting point than the issue assumed, but only if it is driven for the right thing. **The metric for wake-word positives is speaker diversity, not naturalness.** A keyword head trained on nine beautiful voices overfits to nine timbres — the same failure `#92` refuses for one voice, just less extreme. piper-sample-generator's value was never audio quality; it was ~900 speakers, cheaply, with augmentation on top. Qwen3-TTS can beat that on the axis that matters, two ways: - **VoiceDesign** — a voice from a text description. This is the interesting one: the diversity axis becomes programmatic. Enumerate a grid of age, accent, pitch, pace, delivery and generate a speaker per cell. No corpus to source, no licences to read. - **Zero-shot cloning** from 3 s of reference. Point it at a few thousand short clips from a public corpus and every one becomes a distinct speaker with a real timbre and a real accent. The nine preset voices are the trap. They are the path of least resistance and the one that produces a model that works for nobody. ## What changes in the plan - **Throughput is the new constraint.** Cloning or designing per clip is far heavier than piper's small VITS, which emits thousands a minute. Tens of thousands of positives is plausibly an overnight job rather than a coffee break. Worth a timing probe on a hundred clips before committing to a target count — and worth keeping piper in reserve for bulk if it hurts. - **Drive Comfy headlessly.** Its HTTP API, a workflow parameterised by text and reference, in a loop. Not the GUI. - **The eval matters more, not less.** openWakeWord's published negative corpora and its threshold guidance were calibrated against piper-shaped positives. Changing the positive distribution is fine and probably an improvement, but it means the numbers other people report transfer less well to ours. The held-out real-voice detection rate and the hour of false-accept audio are now the only trustworthy signal, not a formality. - **Still never evaluate on synthetic.** Whatever generates the training set, the held-out utterances are recorded by an actual person on actual hardware. A model graded by the generator that trained it will always look excellent. Augmentation does not change: room impulse responses, noise, gain and speed variation on top of whatever the generator produced.

aiko commented

2026-08-15 12:22:12 +00:00

Author

Owner

On the timing probe: two clips is enough to start, not a hundred. The first pays model load, the second is steady-state — that is the number, and it costs twenty seconds to get.

Two conditions on it. Use two different voices, or it measures per-clip cost and misses per-speaker cost, and the grid is thousands of distinct speakers. And do not size an overnight run from it: two clips run at boost clocks and answer sequentially, so sustained thermals push the real number up while batching pushes it down. Those widen the error bar rather than cancelling.

If the warm clip lands near half a second, the whole positive set is an overnight job and there is nothing left to decide. If it lands near thirty, then a longer run at real concurrency earns its keep before a target count is committed to.

On the timing probe: two clips is enough to start, not a hundred. The first pays model load, the second is steady-state — that is the number, and it costs twenty seconds to get. Two conditions on it. **Use two different voices**, or it measures per-clip cost and misses per-speaker cost, and the grid is thousands of distinct speakers. And **do not size an overnight run from it**: two clips run at boost clocks and answer sequentially, so sustained thermals push the real number up while batching pushes it down. Those widen the error bar rather than cancelling. If the warm clip lands near half a second, the whole positive set is an overnight job and there is nothing left to decide. If it lands near thirty, then a longer run at real concurrency earns its keep before a target count is committed to.

aiko commented

2026-08-15 12:24:18 +00:00

Author

Owner

Refinement on where the diversity comes from, 2026-08-15.

Vary the VoiceDesign description, not the spoken phrase. The keyword head only ever sees "Hey Denpa" — the text is fixed by definition. Positives are one phrase across thousands of speakers, so a few thousand generated description strings, one clip each, is the whole grid. No corpus to source, no licences, no three-second references to collect. Write the descriptions programmatically.

Varied text does have a home, and it is the other half: negatives, especially near-misses. "hey then", "Denpa" bare, "hey, then pa-", ordinary speech that shares phonemes. That is what stops it firing at the television, and it is worth generating deliberately rather than relying on the published negative corpora alone.

piper can drop out entirely if the probe comes back fast; it stays only as a throughput fallback. Augmentation is not piper and does not drop out either way — room impulse responses, noise, gain and speed variation run on whatever clips exist.

The risk this path carries that cloning does not

Two thousand descriptions may not be two thousand timbres. Descriptions map into a learned voice space and can collapse: "warm older man" and "gravelly older man" may land almost on top of each other. Cloning from two thousand real references is distinct by construction; designed voices are only as distinct as the model decided to make them. A training set that looks diverse in the manifest and is not is exactly how the head overfits to timbre while every metric looks fine.

Cheap way to find out before training anything: embed a sample of the generated clips with the speaker-verification model from #92 and look at the pairwise cosine spread. A tight cluster means the descriptions are decorative and real references are needed after all. #92's model grading #87's data, before either ships.

Refinement on where the diversity comes from, 2026-08-15. **Vary the VoiceDesign description, not the spoken phrase.** The keyword head only ever sees "Hey Denpa" — the text is fixed by definition. Positives are one phrase across thousands of speakers, so a few thousand generated *description strings*, one clip each, is the whole grid. No corpus to source, no licences, no three-second references to collect. Write the descriptions programmatically. Varied text does have a home, and it is the other half: **negatives, especially near-misses.** "hey then", "Denpa" bare, "hey, then pa-", ordinary speech that shares phonemes. That is what stops it firing at the television, and it is worth generating deliberately rather than relying on the published negative corpora alone. **piper can drop out entirely** if the probe comes back fast; it stays only as a throughput fallback. Augmentation is not piper and does not drop out either way — room impulse responses, noise, gain and speed variation run on whatever clips exist. ## The risk this path carries that cloning does not **Two thousand descriptions may not be two thousand timbres.** Descriptions map into a learned voice space and can collapse: "warm older man" and "gravelly older man" may land almost on top of each other. Cloning from two thousand real references is distinct by construction; designed voices are only as distinct as the model decided to make them. A training set that looks diverse in the manifest and is not is exactly how the head overfits to timbre while every metric looks fine. Cheap way to find out before training anything: **embed a sample of the generated clips with the speaker-verification model from `#92` and look at the pairwise cosine spread.** A tight cluster means the descriptions are decorative and real references are needed after all. `#92`'s model grading `#87`'s data, before either ships.

aiko commented

2026-08-15 12:42:50 +00:00

Author

Owner

Probe run 2026-08-15 against the real server. Three corrections, one of them structural.

It is not ComfyUI. It is vllmn's vLLM-Omni stack: a Base server on :8091 (Qwen/Qwen3-TTS-12Hz-1.7B-Base) and a separate VoiceDesign server on :8092, both WSL2-hosted, OpenAI-compatible /v1/audio/speech. denpatou/src/open_llm_vtuber/tts/vllmn_tts.py:9 is the client. Started only by the systemd user units vllmn-tts.service / vllmn-tts-design.service — scripts/tts-serve.sh:11 records why nothing else may launch them: vLLM sets SO_REUSEPORT, so a second launcher does not fail to bind, it silently shares the port and answers a random half of requests.

Warm render cost: 0.38–0.68 s for "Hey Denpa", four runs, two voices, first discarded. Output is 24 kHz mono WAV — openWakeWord wants 16 kHz, so a resample sits in the pipeline, which is nothing.

The structural correction

Base renders are deterministic. The same (voice, text) returns a byte-identical file — md5 matched across repeat runs for both voices tested. Rendering "Hey Denpa" fifty thousand times produces as many copies of however many pool voices exist, and nothing else.

So the plan two comments up — thousands of VoiceDesign description strings as the cheap diversity axis — does not describe this server. voice is not free text and not a description; it is the id of a speaker already minted into the pool (data/voices.json currently holds three: mai and char_sage_948468cf from VoiceDesign, camila cloned from a reference clip). A new speaker is a mint round: wake VoiceDesign, render a reference clip from the description, hand the card back, upload the speaker to Base. That is a GPU handoff, a file-based advisory lock, and a serialised critical section — per vllmn/src/vllmn/tts/sleep.py and minting.py.

Per-clip cost is therefore irrelevant and per-speaker cost is the entire question, which is the thing the earlier "use two different voices" note was reaching for without knowing how right it was. Render time does not scale the corpus; mint rounds do. A mint round has not been timed yet — that is the probe that actually matters and it needs vllmn's own coordinated path rather than raw HTTP, because doing the handoff by hand can leave both servers awake (~19 GB of a 24 GB card that ComfyUI is also on).

What this does to the plan

piper-sample-generator comes back as the bulk source, not the fallback. It emits ~900 speakers with no minting, no lock, no GPU handoff, and no contention with her actual voice. Reversing what the comment two up said.
Qwen3-TTS becomes a quality slice, if it is used at all: a few hundred minted speakers of much higher naturalness layered onto piper's bulk, rather than the whole corpus.
A full-corpus run monopolises the card. Base, VoiceDesign and image gen share one 24 GB 4090 with exactly one awake at a time. An overnight mint campaign means her voice and image gen are both down for its duration. That is an operator decision, not a technical one.
The sleep hazard is the real danger to a long run. sleep.py documents that a render against a stale speaker embedding does not error — it comes back audibly wrong, measured at 3.5 s of speech collapsing to 1.2 s or ballooning to 15.5 s. A sleep landing mid-campaign yields thousands of silently corrupt positives, and a wake-word model trained on wrong-duration audio would look fine in every log. Any batch generator must go through ensure_fresh(), and should duration-check its own output as it writes.

The cosine-spread check on the generated set matters more given all of the above, not less.

Probe run 2026-08-15 against the real server. Three corrections, one of them structural. **It is not ComfyUI.** It is vllmn's vLLM-Omni stack: a **Base** server on `:8091` (`Qwen/Qwen3-TTS-12Hz-1.7B-Base`) and a separate **VoiceDesign** server on `:8092`, both WSL2-hosted, OpenAI-compatible `/v1/audio/speech`. `denpatou/src/open_llm_vtuber/tts/vllmn_tts.py:9` is the client. Started only by the systemd user units `vllmn-tts.service` / `vllmn-tts-design.service` — `scripts/tts-serve.sh:11` records why nothing else may launch them: vLLM sets `SO_REUSEPORT`, so a second launcher does not fail to bind, it silently shares the port and answers a random half of requests. **Warm render cost: 0.38–0.68 s** for "Hey Denpa", four runs, two voices, first discarded. Output is 24 kHz mono WAV — openWakeWord wants 16 kHz, so a resample sits in the pipeline, which is nothing. ## The structural correction **Base renders are deterministic.** The same `(voice, text)` returns a byte-identical file — md5 matched across repeat runs for both voices tested. Rendering "Hey Denpa" fifty thousand times produces as many copies of however many pool voices exist, and nothing else. So the plan two comments up — thousands of VoiceDesign description strings as the cheap diversity axis — does not describe this server. `voice` is not free text and not a description; it is the id of a speaker **already minted into the pool** (`data/voices.json` currently holds three: `mai` and `char_sage_948468cf` from VoiceDesign, `camila` cloned from a reference clip). A new speaker is a **mint round**: wake VoiceDesign, render a reference clip from the description, hand the card back, upload the speaker to Base. That is a GPU handoff, a file-based advisory lock, and a serialised critical section — per `vllmn/src/vllmn/tts/sleep.py` and `minting.py`. **Per-clip cost is therefore irrelevant and per-speaker cost is the entire question**, which is the thing the earlier "use two different voices" note was reaching for without knowing how right it was. Render time does not scale the corpus; mint rounds do. **A mint round has not been timed yet** — that is the probe that actually matters and it needs vllmn's own coordinated path rather than raw HTTP, because doing the handoff by hand can leave both servers awake (~19 GB of a 24 GB card that ComfyUI is also on). ## What this does to the plan - **piper-sample-generator comes back as the bulk source**, not the fallback. It emits ~900 speakers with no minting, no lock, no GPU handoff, and no contention with her actual voice. Reversing what the comment two up said. - **Qwen3-TTS becomes a quality slice**, if it is used at all: a few hundred minted speakers of much higher naturalness layered onto piper's bulk, rather than the whole corpus. - **A full-corpus run monopolises the card.** Base, VoiceDesign and image gen share one 24 GB 4090 with exactly one awake at a time. An overnight mint campaign means her voice and image gen are both down for its duration. That is an operator decision, not a technical one. - **The sleep hazard is the real danger to a long run.** `sleep.py` documents that a render against a stale speaker embedding does not error — it comes back audibly wrong, measured at 3.5 s of speech collapsing to 1.2 s or ballooning to 15.5 s. A sleep landing mid-campaign yields thousands of silently corrupt positives, and a wake-word model trained on wrong-duration audio would look fine in every log. Any batch generator must go through `ensure_fresh()`, and should duration-check its own output as it writes. The cosine-spread check on the generated set matters more given all of the above, not less.

aiko commented

2026-08-15 12:52:57 +00:00

Author

Owner

Mint round timed 2026-08-15, and the answer settles the generator question.

One mint round: 78.3 s, end to end through vllmn's own design_session.

step	time	scales with
card handoff to VoiceDesign	21.1 s	round trips
VoiceDesign render	24.5 s	speakers
card handoff back to Base	13.4 s	round trips
upload to Base	1.0 s	speakers
first render (embedding)	16.2 s	speakers, overstated
warm render	2.2 s	clips

The reference clip was 11.9 s of audio, not the 30 s ceiling — design() stops at MAX_S but the three fallback lines never reached it. So VoiceDesign runs about 2x realtime. The 16.2 s embedding figure includes Base settling after its wake: the warm render immediately after was 2.2 s against the 0.38–0.68 s measured once it had settled, so real embedding cost is a few seconds.

44 % of the round is card handoff, and that is per round trip rather than per voice. Batched — card handed over once, N references designed, handed back once — marginal cost per speaker is about 28 s, and a shorter reference clip would take it to roughly 12 s. Nine hundred speakers is then ~7 h batched, ~3 h with short clips, against ~19.5 h at the measured unbatched rate.

So piper is the source, and the minted slice is not being built

Nine hundred piper speakers cost minutes and no card. Nine hundred minted ones cost hours and monopolise a GPU that also carries her voice and image gen. That question is now closed: piper generates the corpus.

The remaining idea was a small slice of minted voices layered on for naturalness. On reflection it should not be built yet, because there is no observed gap it fixes:

openWakeWord is designed around piper-shaped synthetic positives; the released models are trained that way.
The keyword head trains on embeddings, not audio. That embedding model is frozen and pretrained on real speech, so most timbre and synthesis-artifact detail is abstracted away before the trainable part sees anything.
The eval is real recordings regardless. If piper-only passes it, the slice bought nothing measurable.

Plan: train piper-only. Treat a minted slice as a contingency whose trigger is a failed eval — specifically a detection rate that is acceptable for other people and poor for the operator, which is the signature of a corpus that does not sound like the person using it. The numbers above are what that contingency would cost, which is the reason to have measured them.

Pre-flight before any of it: how does piper say "Denpa"

It is a Japanese word going through English-phonetic voices. If the positives all say DEN-puh and the operator says でんぱ, every positive is subtly the wrong phrase and the one person the model exists for is the one it serves worst — while the held-out eval, if read by the same English-speaking assumption, would never show it.

Generate the phrase across several piper speakers and spellings before generating a corpus. If the pronunciation is wrong, the first fix is a phonetic spelling of the input text, not a second TTS engine.

Mint round timed 2026-08-15, and the answer settles the generator question. **One mint round: 78.3 s**, end to end through vllmn's own `design_session`. | step | time | scales with | |---|---|---| | card handoff to VoiceDesign | 21.1 s | round trips | | VoiceDesign render | 24.5 s | speakers | | card handoff back to Base | 13.4 s | round trips | | upload to Base | 1.0 s | speakers | | first render (embedding) | 16.2 s | speakers, overstated | | warm render | 2.2 s | clips | The reference clip was 11.9 s of audio, not the 30 s ceiling — `design()` stops at `MAX_S` but the three fallback lines never reached it. So VoiceDesign runs about **2x realtime**. The 16.2 s embedding figure includes Base settling after its wake: the warm render immediately after was 2.2 s against the 0.38–0.68 s measured once it had settled, so real embedding cost is a few seconds. **44 % of the round is card handoff**, and that is per round trip rather than per voice. Batched — card handed over once, N references designed, handed back once — marginal cost per speaker is about **28 s**, and a shorter reference clip would take it to roughly 12 s. Nine hundred speakers is then ~7 h batched, ~3 h with short clips, against ~19.5 h at the measured unbatched rate. ## So piper is the source, and the minted slice is not being built Nine hundred piper speakers cost minutes and no card. Nine hundred minted ones cost hours and monopolise a GPU that also carries her voice and image gen. **That question is now closed: piper generates the corpus.** The remaining idea was a small slice of minted voices layered on for naturalness. On reflection it should not be built yet, because there is no observed gap it fixes: - openWakeWord is *designed* around piper-shaped synthetic positives; the released models are trained that way. - The keyword head trains on **embeddings**, not audio. That embedding model is frozen and pretrained on real speech, so most timbre and synthesis-artifact detail is abstracted away before the trainable part sees anything. - The eval is real recordings regardless. If piper-only passes it, the slice bought nothing measurable. **Plan: train piper-only. Treat a minted slice as a contingency whose trigger is a failed eval** — specifically a detection rate that is acceptable for other people and poor for the operator, which is the signature of a corpus that does not sound like the person using it. The numbers above are what that contingency would cost, which is the reason to have measured them. ## Pre-flight before any of it: how does piper say "Denpa" It is a Japanese word going through English-phonetic voices. If the positives all say `DEN-puh` and the operator says でんぱ, every positive is subtly the wrong phrase and the one person the model exists for is the one it serves worst — while the held-out eval, if read by the same English-speaking assumption, would never show it. Generate the phrase across several piper speakers and spellings before generating a corpus. If the pronunciation is wrong, the first fix is a phonetic spelling of the input text, not a second TTS engine.

aiko commented

2026-08-15 12:59:40 +00:00

Author

Owner

Pronunciation pre-flight run 2026-08-15, and the concern was real.

piper's LibriTTS-R voice (904 speakers, 22.05 kHz, espeak phonemes) was installed and the phrase phonemised across candidate spellings. This is objective rather than a listening test — espeak's own output says what will be synthesised.

spelling	phonemes	verdict
`Hey Denpa`	`hˈeɪ dˈɛnpə`	schwa — "DEN-puh"
`Hey Denpah`	`hˈeɪ dˈɛnpə`	identical; a trailing h does nothing
`Hey Den-pah`	`hˈeɪ dˈɛnpˈɑː`	right vowel, but a word break and a second stress
`Hey Dempa`	`hˈeɪ dˈɛmpə`	/m/, still schwa
`Hey Denpaa`	`hˈeɪ dˈɛnpɑː`	open vowel, no break
`Hey Dempaa`	`hˈeɪ dˈɛmpɑː`	/m/ and open vowel

espeak's Japanese for でんぱ is de̞ˈũpä — an open /ä/, with ん as a nasalised vowel rather than a clean /n/ or /m/.

So the naive spelling produces /ə/ where Japanese has /ä/. A corpus generated from Hey Denpa trains a detector for a phrase the operator may not be saying, and a held-out eval read by an English-speaking assumption would never reveal it — which is the failure this pre-flight existed to catch.

Two further findings worth keeping:

Hyphenation is not a pronunciation hint. Den-pah gets the vowel right but espeak treats the hyphen as a word boundary, which showed up as 0.94–1.45 s renders against ~0.7 s for every other spelling. It inserts a pause real speech does not have, so it is the wrong fix even though its phonemes look closest.
Doubling the vowel is the clean fix. Denpaa gives dˈɛnpɑː with no break and a duration in line with the rest.

What is still open

Which spelling is right depends on how the operator actually says it, not on what is correct Japanese. Anglicised "DEN-puh" and plain Denpa is fine and there is no problem to fix; でんぱ wants Denpaa or Dempaa. That decision sets the entire positive corpus and it is the operator's to make from their own speech.

If no spelling matches, piper accepts raw espeak phonemes, so the pronunciation can be specified directly rather than by finding a spelling that happens to trick the phonemiser. That is the exact-control fallback and it means this can always be made right.

Samples for the six spellings across three speakers were generated for the listening comparison.

Pronunciation pre-flight run 2026-08-15, and the concern was real. piper's LibriTTS-R voice (904 speakers, 22.05 kHz, **espeak** phonemes) was installed and the phrase phonemised across candidate spellings. This is objective rather than a listening test — espeak's own output says what will be synthesised. | spelling | phonemes | verdict | |---|---|---| | `Hey Denpa` | `hˈeɪ dˈɛnpə` | **schwa** — "DEN-puh" | | `Hey Denpah` | `hˈeɪ dˈɛnpə` | identical; a trailing h does nothing | | `Hey Den-pah` | `hˈeɪ dˈɛnpˈɑː` | right vowel, but a **word break and a second stress** | | `Hey Dempa` | `hˈeɪ dˈɛmpə` | /m/, still schwa | | `Hey Denpaa` | `hˈeɪ dˈɛnpɑː` | **open vowel, no break** | | `Hey Dempaa` | `hˈeɪ dˈɛmpɑː` | /m/ and open vowel | espeak's Japanese for でんぱ is `de̞ˈũpä` — an open `/ä/`, with ん as a nasalised vowel rather than a clean /n/ or /m/. **So the naive spelling produces `/ə/` where Japanese has `/ä/`.** A corpus generated from `Hey Denpa` trains a detector for a phrase the operator may not be saying, and a held-out eval read by an English-speaking assumption would never reveal it — which is the failure this pre-flight existed to catch. Two further findings worth keeping: - **Hyphenation is not a pronunciation hint.** `Den-pah` gets the vowel right but espeak treats the hyphen as a word boundary, which showed up as 0.94–1.45 s renders against ~0.7 s for every other spelling. It inserts a pause real speech does not have, so it is the wrong fix even though its phonemes look closest. - **Doubling the vowel is the clean fix.** `Denpaa` gives `dˈɛnpɑː` with no break and a duration in line with the rest. ## What is still open Which spelling is right depends on **how the operator actually says it**, not on what is correct Japanese. Anglicised "DEN-puh" and plain `Denpa` is fine and there is no problem to fix; でんぱ wants `Denpaa` or `Dempaa`. That decision sets the entire positive corpus and it is the operator's to make from their own speech. If no spelling matches, piper accepts raw espeak phonemes, so the pronunciation can be specified directly rather than by finding a spelling that happens to trick the phonemiser. That is the exact-control fallback and it means this can always be made right. Samples for the six spellings across three speakers were generated for the listening comparison.

aiko commented

2026-08-15 13:01:14 +00:00

Author

Owner

Operator says it Japanese-ish, so Denpaa/Dempaa (dˈɛnpɑː / dˈɛmpɑː) is the target pronunciation, not Denpa.

Question raised: does training on both pronunciations make the detector too sensitive? Reasoned answer below — from principles, not measured, and the last paragraph is how to settle it properly.

Mixing is mostly safe. The head is a binary classifier over frozen embeddings; a second pronunciation widens the positive region, but false accepts depend on whether that wider region overlaps things that actually occur in room audio. /dɛnpə/ and /dɛnpɑː/ are both rare sequences in English and neither is a word, so the space between them contains very little natural speech. The widening goes into empty territory.

Some spread is wanted anyway. Nobody says a phrase identically every time — tired, at a distance, mid-sentence, with a cold. A model trained on one narrow pronunciation is brittle in exactly the conditions hands-free exists for.

The two real risks are not about the vowel:

Dilution rather than sensitivity. A 50/50 split halves the effective data per pronunciation and yields two mediocre clusters instead of one sharp one — unless the corpus grows to match, which piper makes nearly free.
A wider positive region needs more negatives near its boundary. Hard negatives should be generated deliberately — "den", "Denver", "then pa", "empathy", "deep art" — rather than trusting openWakeWord's published negative corpora to happen to cover a phrase that did not exist when they were assembled.

Suggested split: about 75 % Denpaa/Dempaa, 25 % Denpa. The anglicised minority is insurance for a lazy delivery, not a co-equal target, and it keeps the model centred where the operator's mouth actually is.

Settle it by measurement, not argument. A piper corpus costs minutes, so generate both mixes, train both heads, and compare on this issue's eval — detection across held-out recordings of the operator, false accepts over an hour of room audio. That A/B is affordable because nothing is being minted, which is a better argument for piper than the speed one.

Sensitivity is also not the corpus's job alone: #90 ships a threshold control with a live score meter. A slightly wide model at a well-set threshold beats a narrow one that has to be shouted at.

Operator says it Japanese-ish, so `Denpaa`/`Dempaa` (`dˈɛnpɑː` / `dˈɛmpɑː`) is the target pronunciation, not `Denpa`. Question raised: does training on **both** pronunciations make the detector too sensitive? Reasoned answer below — from principles, not measured, and the last paragraph is how to settle it properly. **Mixing is mostly safe.** The head is a binary classifier over frozen embeddings; a second pronunciation widens the positive region, but false accepts depend on whether that wider region overlaps things that actually occur in room audio. `/dɛnpə/` and `/dɛnpɑː/` are both rare sequences in English and neither is a word, so the space between them contains very little natural speech. The widening goes into empty territory. **Some spread is wanted anyway.** Nobody says a phrase identically every time — tired, at a distance, mid-sentence, with a cold. A model trained on one narrow pronunciation is brittle in exactly the conditions hands-free exists for. **The two real risks are not about the vowel:** 1. **Dilution rather than sensitivity.** A 50/50 split halves the effective data per pronunciation and yields two mediocre clusters instead of one sharp one — unless the corpus grows to match, which piper makes nearly free. 2. **A wider positive region needs more negatives near its boundary.** Hard negatives should be generated deliberately — "den", "Denver", "then pa", "empathy", "deep art" — rather than trusting openWakeWord's published negative corpora to happen to cover a phrase that did not exist when they were assembled. **Suggested split: about 75 % `Denpaa`/`Dempaa`, 25 % `Denpa`.** The anglicised minority is insurance for a lazy delivery, not a co-equal target, and it keeps the model centred where the operator's mouth actually is. **Settle it by measurement, not argument.** A piper corpus costs minutes, so generate both mixes, train both heads, and compare on this issue's eval — detection across held-out recordings of the operator, false accepts over an hour of room audio. That A/B is affordable *because* nothing is being minted, which is a better argument for piper than the speed one. Sensitivity is also not the corpus's job alone: `#90` ships a threshold control with a live score meter. A slightly wide model at a well-set threshold beats a narrow one that has to be shouted at.

aiko commented

2026-08-15 17:29:51 +00:00

Author

Owner

First model trained 2026-08-15. It is not shippable, and the reason is the corpus split recommended earlier in this issue.

Numbers

Sliding-window peaks over held-out clips, which is how #88 actually scores audio -- a rolling window, detection is the peak:

spelling	n	median peak	>=0.5	>=0.9
`Hey Denpaa`	241	0.9675	75.1%	67.2%
`Hey Dempaa`	215	0.9628	68.4%	62.8%
`Hey Denpa`	144	0.0252	41.0%	33.3%

False positives against openWakeWord's own validation corpus -- 10.7 hours of real recorded speech, none of it ours -- are fine: 0.19/hour at threshold 0.70, zero at 0.90. The false-positive budget is met comfortably. Recall is the problem.

What went wrong

The anglicised minority was not learned. Its median peak is 0.025 against ~0.96 for the two Japanese-ish spellings. A quarter of the positive corpus taught the model nothing and diluted the three quarters that worked.

That is the opposite of the failure this issue reasoned about. The earlier comment weighed whether mixing pronunciations would make the detector too sensitive and concluded it was safe because the phonetic space between /dɛnpə/ and /dɛnpɑː/ is empty. The space being empty is exactly why it went the other way: with nothing to bridge them the head had no reason to cover both, so it took the majority and dropped the rest. The 75/25 recommendation was wrong, and the argument that produced it was addressing a risk that did not materialise while missing the one that did.

Even the majority spellings only reach ~70% at 0.5, with medians at 0.96. So each variant is itself bimodal -- most clips score near-perfect, a quarter score near zero. Pronunciation explains the big split but not this residual one. Untested candidates, in the order worth trying:

Clip length against the window. total_length is derived from the median clip duration (train.py:747) and floors at 32000 samples -- 2 s. Our clips average 0.81 s, so each sits in a 2 s window with over a second of padding whose position varies per clip.
max_negative_weight: 1500 is aggressive, and the training explicitly trades recall for the 0.2 FP/hour target. The false-positive result suggests there is room to spend.
augmentation_rounds: 1 -- more rounds is the cheapest way to raise effective positive count.

Retrain on a single pronunciation family (Denpaa/Dempaa only, since the operator says it Japanese-ish) rather than a weighted mix, and treat the anglicised form as out of scope unless it earns its way back in on measured evidence. Then move max_negative_weight and augmentation_rounds if recall is still short.

Two measurement errors worth recording, because both flattered or maligned the model wrongly

Scoring one stored feature window per clip understates recall badly (41% vs 65% for the same model). The training set is laid out one window per clip; detection is a rolling peak. They are not the same measurement.
Reusing one Model instance across clips contaminates every clip with the tail of the previous one. It carries rolling audio and embedding buffers by design. reset() between clips moved Denpaa from 62% to 75%. Any future eval must reset per clip.

First model trained 2026-08-15. **It is not shippable, and the reason is the corpus split recommended earlier in this issue.** ## Numbers Sliding-window peaks over held-out clips, which is how `#88` actually scores audio -- a rolling window, detection is the peak: | spelling | n | median peak | >=0.5 | >=0.9 | |---|---|---|---|---| | `Hey Denpaa` | 241 | 0.9675 | 75.1% | 67.2% | | `Hey Dempaa` | 215 | 0.9628 | 68.4% | 62.8% | | `Hey Denpa` | 144 | **0.0252** | 41.0% | 33.3% | False positives against openWakeWord's own validation corpus -- 10.7 hours of real recorded speech, none of it ours -- are fine: **0.19/hour at threshold 0.70, zero at 0.90**. The false-positive budget is met comfortably. Recall is the problem. ## What went wrong **The anglicised minority was not learned.** Its median peak is 0.025 against ~0.96 for the two Japanese-ish spellings. A quarter of the positive corpus taught the model nothing and diluted the three quarters that worked. That is the opposite of the failure this issue reasoned about. The earlier comment weighed whether mixing pronunciations would make the detector **too sensitive** and concluded it was safe because the phonetic space between `/dɛnpə/` and `/dɛnpɑː/` is empty. The space being empty is exactly why it went the other way: with nothing to bridge them the head had no reason to cover both, so it took the majority and dropped the rest. **The 75/25 recommendation was wrong**, and the argument that produced it was addressing a risk that did not materialise while missing the one that did. **Even the majority spellings only reach ~70% at 0.5**, with medians at 0.96. So each variant is itself bimodal -- most clips score near-perfect, a quarter score near zero. Pronunciation explains the big split but not this residual one. Untested candidates, in the order worth trying: - **Clip length against the window.** `total_length` is derived from the median clip duration (train.py:747) and floors at 32000 samples -- 2 s. Our clips average 0.81 s, so each sits in a 2 s window with over a second of padding whose position varies per clip. - **`max_negative_weight: 1500`** is aggressive, and the training explicitly trades recall for the 0.2 FP/hour target. The false-positive result suggests there is room to spend. - **`augmentation_rounds: 1`** -- more rounds is the cheapest way to raise effective positive count. ## Next Retrain on a single pronunciation family (`Denpaa`/`Dempaa` only, since the operator says it Japanese-ish) rather than a weighted mix, and treat the anglicised form as out of scope unless it earns its way back in on measured evidence. Then move `max_negative_weight` and `augmentation_rounds` if recall is still short. ## Two measurement errors worth recording, because both flattered or maligned the model wrongly - **Scoring one stored feature window per clip understates recall badly** (41% vs 65% for the same model). The training set is laid out one window per clip; detection is a rolling peak. They are not the same measurement. - **Reusing one `Model` instance across clips contaminates every clip with the tail of the previous one.** It carries rolling audio and embedding buffers by design. `reset()` between clips moved `Denpaa` from 62% to 75%. Any future eval must reset per clip.

aiko commented

2026-08-15 18:52:02 +00:00

Author

Owner

A trained model exists and meets the bar. hey_denpa.onnx, 205 KB, trained on the Japanese-ish corpus only.

threshold	detection	hard-neg	FP/hour
0.50	98.75%	8.75%	0.37
0.70	98.75%	8.00%	0.19
0.90	98.25%	6.25%	0.09
0.95	97.25%	3.25%	0.09

Operating point for the 0.2 FP/hour budget: threshold 0.68-0.70, 98.75% detection. That is the default #90 should ship.

Correction: the previous comment's diagnosis was a measurement artifact

The comment above reported that the anglicised Denpa was "essentially not learned" (median peak 0.025 against 0.96) and recommended retraining without it. That measurement was wrong, and the conclusion drawn from it should not be trusted.

The evaluation fed each test clip as a raw wav. The detection peak lands after the phrase has flushed through the model's 1.28 s window, so a clip that ends where the phrase ends never presents its own peak. Padding 0.5 s of trailing silence moves the same model from 64% to 98.7% detection. A live microphone always supplies that trailing audio, so the unpadded figure measured a property of finite test files and nothing about the model.

That artifact sits on a cliff edge, and the three spellings differ in duration by about 64 ms -- Denpa 0.773 s against Denpaa 0.837 s, with all three over 99% shorter than the window. Well under one 80 ms chunk of difference, but enough to tip clips across the edge systematically. So the apparent collapse of one spelling is explainable by length alone, and there is no evidence left that it was not learned. The first model has been overwritten, so this cannot be retested.

Dropping the anglicised spelling remains correct -- the operator does not say it -- but it is correct on preference, not on the evidence this issue previously recorded.

Also worth noting the retrain did not improve recall on the same broken measurement (68-75% before, 62-64% after); it only removed 7527 clips. Both numbers were measuring padding, not pronunciation.

Evaluation rules, since three separate errors were made getting here

Pad the clip with trailing audio. Otherwise recall is understated by ~35 points.
reset() the model between clips. It carries rolling audio and embedding buffers by design; without a reset each clip is scored with the tail of the previous one. Worth 13 points on its own.
Score a sliding peak, not the single stored training window. The corpus is laid out one window per clip; detection is a rolling maximum.

What this still does not prove

Every positive above came from the same piper generator as the training data. This says training converged and generalises across speaker, pace and jitter -- it says nothing about whether the model hears the operator. The false-positive half is trustworthy (10.7 h of real recorded speech, none of it ours); the detection half is not yet.

The eval this issue actually asks for still needs recordings: 20-30 utterances at conversational distance, tired, across a room, mid-sentence, plus an hour of ordinary room audio. That is the outstanding work here, and it is the only thing that can confirm the pronunciation choice was right.

Note for `#88` / `#89`

The detection peak arriving after the phrase completes is a property of the chain, not of this corpus. The Rust side already streams continuously so it sees it naturally, but #89's arming window must be measured from the peak, not from where the operator stopped speaking.

**A trained model exists and meets the bar.** `hey_denpa.onnx`, 205 KB, trained on the Japanese-ish corpus only. | threshold | detection | hard-neg | FP/hour | |---|---|---|---| | 0.50 | 98.75% | 8.75% | 0.37 | | **0.70** | **98.75%** | 8.00% | **0.19** | | 0.90 | 98.25% | 6.25% | 0.09 | | 0.95 | 97.25% | 3.25% | 0.09 | Operating point for the 0.2 FP/hour budget: **threshold 0.68-0.70, 98.75% detection**. That is the default `#90` should ship. ## Correction: the previous comment's diagnosis was a measurement artifact The comment above reported that the anglicised `Denpa` was "essentially not learned" (median peak 0.025 against 0.96) and recommended retraining without it. **That measurement was wrong**, and the conclusion drawn from it should not be trusted. The evaluation fed each test clip as a raw wav. **The detection peak lands *after* the phrase has flushed through the model's 1.28 s window**, so a clip that ends where the phrase ends never presents its own peak. Padding 0.5 s of trailing silence moves the same model from **64% to 98.7%** detection. A live microphone always supplies that trailing audio, so the unpadded figure measured a property of finite test files and nothing about the model. That artifact sits on a cliff edge, and the three spellings differ in duration by about 64 ms -- `Denpa` 0.773 s against `Denpaa` 0.837 s, with all three over 99% shorter than the window. Well under one 80 ms chunk of difference, but enough to tip clips across the edge systematically. So the apparent collapse of one spelling is explainable by length alone, and there is no evidence left that it was not learned. The first model has been overwritten, so this cannot be retested. Dropping the anglicised spelling remains correct -- the operator does not say it -- but it is correct on preference, not on the evidence this issue previously recorded. Also worth noting the retrain **did not improve recall** on the same broken measurement (68-75% before, 62-64% after); it only removed 7527 clips. Both numbers were measuring padding, not pronunciation. ## Evaluation rules, since three separate errors were made getting here - **Pad the clip with trailing audio.** Otherwise recall is understated by ~35 points. - **`reset()` the model between clips.** It carries rolling audio and embedding buffers by design; without a reset each clip is scored with the tail of the previous one. Worth 13 points on its own. - **Score a sliding peak, not the single stored training window.** The corpus is laid out one window per clip; detection is a rolling maximum. ## What this still does not prove Every positive above came from the same piper generator as the training data. This says training converged and generalises across speaker, pace and jitter -- it says nothing about whether the model hears **the operator**. The false-positive half is trustworthy (10.7 h of real recorded speech, none of it ours); the detection half is not yet. The eval this issue actually asks for still needs recordings: 20-30 utterances at conversational distance, tired, across a room, mid-sentence, plus an hour of ordinary room audio. That is the outstanding work here, and it is the only thing that can confirm the pronunciation choice was right. ## Note for `#88` / `#89` The detection peak arriving **after** the phrase completes is a property of the chain, not of this corpus. The Rust side already streams continuously so it sees it naturally, but `#89`'s arming window must be measured from the peak, not from where the operator stopped speaking.

aiko referenced this issue from a commit

2026-08-17 11:38:31 +00:00

feat(audio): score openWakeWord beside silero on the same stream

aiko referenced this issue from a commit

2026-08-17 11:38:31 +00:00

feat(audio): wake to her own name, not to hey jarvis

aiko referenced this issue from a commit

2026-08-17 11:38:31 +00:00

feat(receiver): three ways to listen, and a threshold you can see