aiko/denpa

Fork 0

Hands-free should drop what she was not called for #89

New issue

Open

opened 2026-08-15 12:16:16 +00:00 by aiko · 2 comments

aiko commented

2026-08-15 12:16:16 +00:00

Owner

Hands-free today means every utterance in the room is transmitted. That is why hands-free is not what anyone actually uses — pttDown/pttUp (src/renderer/src/components/receiver/transmission.ts:41, :61) exist because holding a key is the only way to say "this one was for you". The wake phrase is the way to stop paying that price.

With #88 landed the score exists and does nothing. This issue gives it consequences: an utterance the detector did not hear a wake phrase in is dropped, not sent.

Where the decision goes

In Rust, in Worker::emit_utterance (src-tauri/src/audio/engine.rs:385). Not in the renderer.

That placement matters more than it looks. If the gate lived in the webview, the audio would already have crossed the IPC boundary and been assembled into a SpeechEnd payload before anything decided it was nobody's business. Dropping it in the worker means unwanted speech never leaves the audio thread. It is also the only placement that keeps #92 honest later — a speaker gate that runs after the audio has been handed around is a gate in name.

The rule

Arm a window when the score crosses the threshold. An utterance emits if the window was open when it started; otherwise it is discarded and the buffer cleared.

Three cases that are easy to get wrong, and each is a test:

A press is not gated. from_press (engine.rs:98) is an explicit human signal — #33 settled that already, and handleSpeechEnd (vad-context.tsx:358) still honours it even when the detector saw nothing. The wake phrase is an alternative to the press, not a condition on it. Someone holding the key must never be told they were not heard.
The phrase and the sentence are usually one utterance. "Hey Denpa, what's the weather" is a single unbroken stretch of speech; silero opens it at "Hey" and closes it after "weather", so the trigger lands inside the utterance it is meant to admit. Arming on utterance boundaries only will silently reject exactly the normal case. The window has to be checked against the utterance's start, and the pre-roll kept.
The window closes. A trigger does not arm her forever. Some seconds after the wake phrase, with nothing said, it lapses back to deaf. Otherwise the first "Hey Denpa" of the day makes every later sentence transmissible and the whole feature is a no-op with extra steps.

The renderer has to show it

AiStateEnum (src/renderer/src/context/ai-state-context.tsx:18) has idle, listening, thinking-speaking, interrupted, waiting, loading. None of those is "the microphone is open and she is waiting to be called", which is a different thing from idle and a different thing from listening.

Without a visible distinction the failure mode is silent and infuriating: you speak, nothing happens, and there is no way to tell whether the phrase missed, the mic is shut, or the connection is down. #15 is already the issue about failure states being invisible; do not add a seventh way to be invisibly stuck.

handleSpeechStart (vad-context.tsx:344) currently goes straight to listening on any speech. It should only do that once armed.

Done means

Speech in an unarmed room is never transmitted, and never reaches the renderer.
"Hey Denpa, " transmits as one utterance, with the phrase included rather than clipped off — she should hear her own name.
A press transmits regardless, exactly as now.
The armed state is distinguishable at a glance from both idle and listening.
The tests at the foot of engine.rs cover the three cases above.

#88 — the score this consumes
#90 — the switch that turns this on, and the threshold it arms at
#92 — the second gate, which hangs off the same point
#15 — the WAITING state and failure screens
#33 — a press is an explicit human signal

Hands-free today means every utterance in the room is transmitted. That is why hands-free is not what anyone actually uses — `pttDown`/`pttUp` (`src/renderer/src/components/receiver/transmission.ts:41`, `:61`) exist because holding a key is the only way to say "this one was for you". The wake phrase is the way to stop paying that price. With `#88` landed the score exists and does nothing. This issue gives it consequences: **an utterance the detector did not hear a wake phrase in is dropped, not sent.** ## Where the decision goes In Rust, in `Worker::emit_utterance` (`src-tauri/src/audio/engine.rs:385`). Not in the renderer. That placement matters more than it looks. If the gate lived in the webview, the audio would already have crossed the IPC boundary and been assembled into a `SpeechEnd` payload before anything decided it was nobody's business. Dropping it in the worker means unwanted speech never leaves the audio thread. It is also the only placement that keeps `#92` honest later — a speaker gate that runs after the audio has been handed around is a gate in name. ## The rule Arm a window when the score crosses the threshold. An utterance emits if the window was open when it started; otherwise it is discarded and the buffer cleared. Three cases that are easy to get wrong, and each is a test: - **A press is not gated.** `from_press` (engine.rs:98) is an explicit human signal — `#33` settled that already, and `handleSpeechEnd` (`vad-context.tsx:358`) still honours it even when the detector saw nothing. The wake phrase is an alternative to the press, not a condition on it. Someone holding the key must never be told they were not heard. - **The phrase and the sentence are usually one utterance.** "Hey Denpa, what's the weather" is a single unbroken stretch of speech; silero opens it at "Hey" and closes it after "weather", so the trigger lands *inside* the utterance it is meant to admit. Arming on utterance boundaries only will silently reject exactly the normal case. The window has to be checked against the utterance's start, and the pre-roll kept. - **The window closes.** A trigger does not arm her forever. Some seconds after the wake phrase, with nothing said, it lapses back to deaf. Otherwise the first "Hey Denpa" of the day makes every later sentence transmissible and the whole feature is a no-op with extra steps. ## The renderer has to show it `AiStateEnum` (`src/renderer/src/context/ai-state-context.tsx:18`) has `idle`, `listening`, `thinking-speaking`, `interrupted`, `waiting`, `loading`. None of those is "the microphone is open and she is waiting to be called", which is a different thing from `idle` and a different thing from `listening`. Without a visible distinction the failure mode is silent and infuriating: you speak, nothing happens, and there is no way to tell whether the phrase missed, the mic is shut, or the connection is down. `#15` is already the issue about failure states being invisible; do not add a seventh way to be invisibly stuck. `handleSpeechStart` (`vad-context.tsx:344`) currently goes straight to `listening` on any speech. It should only do that once armed. ## Done means - Speech in an unarmed room is never transmitted, and never reaches the renderer. - "Hey Denpa, <sentence>" transmits as one utterance, with the phrase included rather than clipped off — she should hear her own name. - A press transmits regardless, exactly as now. - The armed state is distinguishable at a glance from both idle and listening. - The tests at the foot of `engine.rs` cover the three cases above. ## Related - `#88` — the score this consumes - `#90` — the switch that turns this on, and the threshold it arms at - `#92` — the second gate, which hangs off the same point - `#15` — the WAITING state and failure screens - `#33` — a press is an explicit human signal

aiko added the

ready-for-agent

labels

2026-08-15 12:16:16 +00:00

aiko commented

2026-08-17 10:41:14 +00:00

Author

Owner

The rule above is wrong. Correcting it before it gets built, on evidence measured in #87.

What was missed

The rule says: an utterance emits if the window was open when it started. That assumes the wake trigger arrives at or before the start of the utterance it admits. It does not.

The detection peak lands after the phrase has finished, because the score is computed over a rolling 16-embedding window and the phrase has to flush through it. Measured in #87: padding half a second of trailing silence onto a held-out clip moved detection from 64% to 98.7% for the same model. The peak is roughly half a second late by construction.

So for "Hey Denpa, what's the weather" as one unbroken utterance:

t	event
0.0 s	silero opens the utterance on "Hey"
~0.8 s	the phrase itself ends
~1.3 s	the wake score peaks
~2.5 s	silero closes the utterance after "weather"

The trigger lands 1.3 seconds after the utterance started. A rule that asks "was the window open when this utterance began" answers no, and drops the one case the feature exists for. The issue already warned that arming on utterance boundaries rejects the normal case; it then specified a rule with the same flaw one level down.

The rule that works

Keep the timestamp of the last trigger. At emit time:

Admit the utterance if a trigger occurred at or after .

One comparison, and it covers all three shapes:

Trigger inside the utterance ("Hey Denpa, what's the weather") — the trigger is later than the start, so it passes trivially. This is the case the old rule broke.
Trigger just before the utterance ("Hey Denpa" ... pause ... "what's the weather", which silero splits in two) — the first utterance is dropped, because at its end no trigger had arrived yet. The trigger lands during the pause, and the second utterance starts inside , so it is admitted.
Trigger long ago — falls outside the window, dropped, which is the lapse the issue already asked for.

therefore does double duty: it is both the follow-up grace after a bare "Hey Denpa" and the expiry. A few seconds.

Consequence worth accepting deliberately

In the split case the wake phrase itself is dropped, so "she should hear her own name" does not hold there — the audio carrying it belonged to an utterance that was discarded before the trigger existed. It still holds for the single-utterance case, which is the common one. Buffering the previous utterance in case a trigger shows up shortly after would fix it and is not worth the complexity now; noted so nobody reads the difference as a bug.

Also

The trigger must NOT be consumed on use, or the second sentence of a two-part request is deaf. It expires on time, not on a single admission.

**The rule above is wrong.** Correcting it before it gets built, on evidence measured in `#87`. ## What was missed The rule says: *an utterance emits if the window was open when it started.* That assumes the wake trigger arrives at or before the start of the utterance it admits. It does not. **The detection peak lands after the phrase has finished**, because the score is computed over a rolling 16-embedding window and the phrase has to flush through it. Measured in `#87`: padding half a second of trailing silence onto a held-out clip moved detection from 64% to 98.7% for the same model. The peak is roughly half a second late by construction. So for "Hey Denpa, what's the weather" as one unbroken utterance: | t | event | |---|---| | 0.0 s | silero opens the utterance on "Hey" | | ~0.8 s | the phrase itself ends | | **~1.3 s** | **the wake score peaks** | | ~2.5 s | silero closes the utterance after "weather" | The trigger lands **1.3 seconds after the utterance started**. A rule that asks "was the window open when this utterance began" answers no, and drops the one case the feature exists for. The issue already warned that arming on utterance boundaries rejects the normal case; it then specified a rule with the same flaw one level down. ## The rule that works Keep the timestamp of the last trigger. At emit time: > **Admit the utterance if a trigger occurred at or after .** One comparison, and it covers all three shapes: - **Trigger inside the utterance** ("Hey Denpa, what's the weather") — the trigger is later than the start, so it passes trivially. This is the case the old rule broke. - **Trigger just before the utterance** ("Hey Denpa" ... pause ... "what's the weather", which silero splits in two) — the first utterance is dropped, because at its end no trigger had arrived yet. The trigger lands during the pause, and the second utterance starts inside , so it is admitted. - **Trigger long ago** — falls outside the window, dropped, which is the lapse the issue already asked for. therefore does double duty: it is both the follow-up grace after a bare "Hey Denpa" and the expiry. A few seconds. ## Consequence worth accepting deliberately In the split case the wake phrase itself is dropped, so "she should hear her own name" does not hold there — the audio carrying it belonged to an utterance that was discarded before the trigger existed. It still holds for the single-utterance case, which is the common one. Buffering the previous utterance in case a trigger shows up shortly after would fix it and is not worth the complexity now; noted so nobody reads the difference as a bug. ## Also The trigger must NOT be consumed on use, or the second sentence of a two-part request is deaf. It expires on time, not on a single admission.

aiko commented

2026-08-17 10:41:44 +00:00

Author

Owner

The comment above lost three expressions to shell interpolation when it was posted, including the rule itself. Restating it here; this version is the one to build against.

The rule

Keep the timestamp of the last wake trigger. At emit time:

Admit the utterance if a trigger occurred at or after utterance_start - ARM_WINDOW.

That is, the trigger may land anywhere from ARM_WINDOW before the utterance began right through to the moment it ends. One comparison, three shapes covered:

Trigger inside the utterance — "Hey Denpa, what's the weather" as one unbroken stretch. The peak lands about 1.3 s in, later than the start, so it passes trivially. This is the case the original rule broke.
Trigger just before the utterance — "Hey Denpa", a pause, then "what's the weather", which silero splits into two utterances. The first is dropped: at its end no trigger had arrived yet, because the peak is half a second late. The trigger lands during the pause, and the second utterance begins inside ARM_WINDOW, so it is admitted.
Trigger long ago — outside the window, dropped. This is the lapse the issue already asked for.

ARM_WINDOW does double duty: the follow-up grace after a bare "Hey Denpa", and the expiry that stops one greeting admitting everything said afterwards. A few seconds.

Why the original rule fails

It asked whether the window was open when the utterance started, which assumes the trigger arrives at or before the start. It does not. The score is computed over a rolling 16-embedding window, so the phrase has to flush through before the peak appears — #87 measured this as the difference between 64% and 98.7% detection on the same model, purely from padding half a second of trailing silence onto the test clips. The peak is late by construction, roughly half a second.

The comment above lost three expressions to shell interpolation when it was posted, including the rule itself. Restating it here; this version is the one to build against. ## The rule Keep the timestamp of the last wake trigger. At emit time: > **Admit the utterance if a trigger occurred at or after `utterance_start - ARM_WINDOW`.** That is, the trigger may land anywhere from `ARM_WINDOW` before the utterance began right through to the moment it ends. One comparison, three shapes covered: - **Trigger inside the utterance** — "Hey Denpa, what's the weather" as one unbroken stretch. The peak lands about 1.3 s in, later than the start, so it passes trivially. This is the case the original rule broke. - **Trigger just before the utterance** — "Hey Denpa", a pause, then "what's the weather", which silero splits into two utterances. The first is dropped: at its end no trigger had arrived yet, because the peak is half a second late. The trigger lands during the pause, and the second utterance begins inside `ARM_WINDOW`, so it is admitted. - **Trigger long ago** — outside the window, dropped. This is the lapse the issue already asked for. `ARM_WINDOW` does double duty: the follow-up grace after a bare "Hey Denpa", and the expiry that stops one greeting admitting everything said afterwards. A few seconds. ## Why the original rule fails It asked whether the window was open *when the utterance started*, which assumes the trigger arrives at or before the start. It does not. The score is computed over a rolling 16-embedding window, so the phrase has to flush through before the peak appears — `#87` measured this as the difference between 64% and 98.7% detection on the same model, purely from padding half a second of trailing silence onto the test clips. The peak is late by construction, roughly half a second.

aiko referenced this issue from a commit

2026-08-17 11:38:31 +00:00

feat(audio): score openWakeWord beside silero on the same stream

aiko referenced this issue from a commit

2026-08-17 11:38:31 +00:00

feat(audio): drop the speech she was never called for