openWakeWord runs beside silero, in the ort that is already there #88

New issue

Open

opened 2026-08-15 12:16:15 +00:00 by aiko · 0 comments

aiko commented

2026-08-15 12:16:15 +00:00

Owner

The wake word needs no new dependency, no new thread, and no new audio path. Everything it wants, silero already has.

src-tauri/src/audio/engine.rs resamples capture to 16 kHz mono (Resampler, engine.rs:107; MODEL_SAMPLE_RATE, engine.rs:31) and pushes it through an ort session in fixed frames (Detector::probability, engine.rs:188). openWakeWord wants exactly that stream — 16 kHz, mono, f32 — and ort is already a dependency on both desktop and Android (Cargo.toml:47). So this is a second consumer of a buffer that is already being produced, not a new pipeline.

This issue is inference only. Nothing changes about how the microphone behaves — the score is computed, published, and otherwise ignored. #89 is where it starts mattering. Splitting it this way means the detector can be tuned and watched on real audio before it has the power to swallow an utterance.

Shape

A WakeWord struct beside Detector (engine.rs:158), holding three sessions rather than one, because openWakeWord is a chain and not a model:

frames (16 kHz f32) -> melspectrogram.onnx -> embedding_model.onnx -> hey_denpa.onnx -> score

Two things differ from silero and will bite whoever assumes otherwise:

The chunk size is not 512. Silero's FRAME_SAMPLES (engine.rs:32) is the silero model's number. openWakeWord steps 1280 samples — 80 ms. The two consumers want different windows over the same stream, so Worker::feed (engine.rs:312) needs its own accumulator for this one rather than reusing silero's frame loop.
The stages are stateful across calls. The embedding stage runs over a rolling window of mel frames; feeding it disjoint chunks and expecting a score is the standard way to get a detector that never fires. The ring buffers live in WakeWord, not in the caller.

Tee the resampled samples in Worker::feed after resampling and before — or alongside — the silero path. Publish the score the way the meter level already travels (publish_level, engine.rs:292, and denpa://mic-level): a denpa://wake-score event, rate-limited the same way, so the renderer can draw it while tuning. Nothing consumes it yet.

It is not blocked on the model

#87 produces hey_denpa.onnx, but openWakeWord ships pretrained heads — hey_jarvis is the usual one. Build and test against that, swap the file when #87 lands. The two stages in front of the head are identical either way.

Cost to watch

Silero scores a 512-sample frame; this chain runs three graphs every 80 ms, forever, whenever the mic is open. On the desktop that is noise. On the phone it is the whole question, and it is #91's to answer — but leave the measurement reachable here rather than making someone add it later: log or expose the per-chunk inference time.

While you are in here

package.json:24 still lists @ricky0123/vad-web, and onnxruntime-web with it. ADR-0002 moved capture to Rust and the renderer no longer opens a device — those two are dead weight, and worse, they are the first thing anyone reading this repo for "where does the wake word go" will find. They point at a JS answer that is wrong for this codebase. Removing them is not this issue's job, but somebody should.

Done means

A score that tracks the phrase on real audio, visible without a debugger.
Unit tests in the style of the ones already at the foot of engine.rs — the chain loads, silence never scores high, a held-out clip of the phrase does.
No change to when the microphone opens, closes, or transmits.

#87 — the model file
#89 — the first thing that consumes the score
#91 — whether the phone can afford it
ADR-0002 — capture in Rust, why the renderer has no microphone

The wake word needs no new dependency, no new thread, and no new audio path. Everything it wants, silero already has. `src-tauri/src/audio/engine.rs` resamples capture to 16 kHz mono (`Resampler`, engine.rs:107; `MODEL_SAMPLE_RATE`, engine.rs:31) and pushes it through an `ort` session in fixed frames (`Detector::probability`, engine.rs:188). openWakeWord wants exactly that stream — 16 kHz, mono, `f32` — and `ort` is already a dependency on both desktop and Android (`Cargo.toml:47`). So this is a second consumer of a buffer that is already being produced, not a new pipeline. This issue is inference only. **Nothing changes about how the microphone behaves** — the score is computed, published, and otherwise ignored. `#89` is where it starts mattering. Splitting it this way means the detector can be tuned and watched on real audio before it has the power to swallow an utterance. ## Shape A `WakeWord` struct beside `Detector` (engine.rs:158), holding three sessions rather than one, because openWakeWord is a chain and not a model: ``` frames (16 kHz f32) -> melspectrogram.onnx -> embedding_model.onnx -> hey_denpa.onnx -> score ``` Two things differ from silero and will bite whoever assumes otherwise: - **The chunk size is not 512.** Silero's `FRAME_SAMPLES` (engine.rs:32) is the silero model's number. openWakeWord steps 1280 samples — 80 ms. The two consumers want different windows over the same stream, so `Worker::feed` (engine.rs:312) needs its own accumulator for this one rather than reusing silero's frame loop. - **The stages are stateful across calls.** The embedding stage runs over a rolling window of mel frames; feeding it disjoint chunks and expecting a score is the standard way to get a detector that never fires. The ring buffers live in `WakeWord`, not in the caller. Tee the resampled samples in `Worker::feed` after resampling and before — or alongside — the silero path. Publish the score the way the meter level already travels (`publish_level`, engine.rs:292, and `denpa://mic-level`): a `denpa://wake-score` event, rate-limited the same way, so the renderer can draw it while tuning. Nothing consumes it yet. ## It is not blocked on the model `#87` produces `hey_denpa.onnx`, but openWakeWord ships pretrained heads — `hey_jarvis` is the usual one. Build and test against that, swap the file when `#87` lands. The two stages in front of the head are identical either way. ## Cost to watch Silero scores a 512-sample frame; this chain runs three graphs every 80 ms, forever, whenever the mic is open. On the desktop that is noise. On the phone it is the whole question, and it is `#91`'s to answer — but leave the measurement reachable here rather than making someone add it later: log or expose the per-chunk inference time. ## While you are in here `package.json:24` still lists `@ricky0123/vad-web`, and `onnxruntime-web` with it. ADR-0002 moved capture to Rust and the renderer no longer opens a device — those two are dead weight, and worse, they are the first thing anyone reading this repo for "where does the wake word go" will find. They point at a JS answer that is wrong for this codebase. Removing them is not this issue's job, but somebody should. ## Done means - A score that tracks the phrase on real audio, visible without a debugger. - Unit tests in the style of the ones already at the foot of `engine.rs` — the chain loads, silence never scores high, a held-out clip of the phrase does. - No change to when the microphone opens, closes, or transmits. ## Related - `#87` — the model file - `#89` — the first thing that consumes the score - `#91` — whether the phone can afford it - ADR-0002 — capture in Rust, why the renderer has no microphone

aiko added the

ready-for-agent

labels

2026-08-15 12:16:15 +00:00

aiko referenced this issue from a commit

2026-08-17 11:38:31 +00:00

feat(audio): score openWakeWord beside silero on the same stream

aiko referenced this issue from a commit

2026-08-17 11:38:31 +00:00

feat(audio): wake to her own name, not to hey jarvis

aiko referenced this issue from a commit

2026-08-17 11:38:31 +00:00