aiko/denpa

Fork 0

What crosses the Rust/webview boundary for audio? #43

New issue

Closed

opened 2026-07-30 13:15:04 +00:00 by aiko · 3 comments

aiko commented

2026-07-30 13:15:04 +00:00

Owner

Question

denpa#35 moves mic capture and VAD into Rust on both desktop (cpal + silero
via ort) and Android (Kotlin AudioRecord). Neither half is written. #36
states the rule that binds this one: the command signature must be the one
Android can implement — do not design around what happens to be easy on Windows.

So the interface has to be decided before either path is built. Specifically:

The carrier meter's data source moves. denpa#10's meter binds to
previousTriggeredProbability — a peak-hold that never falls mid-utterance and
resets on speech end or stopMic. What does Rust publish so that shape
survives, and at what rate?
Speech-start / speech-end semantics, including the positive/negative
threshold pair the readout displays. Events, or polled state?
Press-to-talk ownership. denpa#33 exists because stopMic() destroys the
VAD instance without flushing, so a short press sends nothing. Under the new
boundary, who holds the recorded buffer, and what does release emit?
Who owns mic lifecycle across the overlay service, the full receiver, and
the desktop shell — one recorder or several.

The answer is a command-and-event surface, not an implementation. Use
/codebase-design for the seam.

denpa#35, denpa#33, denpa#36, denpa#10
denpa/docs/adr/0002-hardware-goes-through-rust.md
src/renderer/src/context/vad-context.tsx — what exists today

Map: aiko/denpa#41

## Question `denpa#35` moves mic capture and VAD into Rust on both desktop (`cpal` + silero via `ort`) and Android (Kotlin `AudioRecord`). Neither half is written. `#36` states the rule that binds this one: *the command signature must be the one Android can implement — do not design around what happens to be easy on Windows.* So the interface has to be decided before either path is built. Specifically: - **The carrier meter's data source moves.** `denpa#10`'s meter binds to `previousTriggeredProbability` — a peak-hold that never falls mid-utterance and resets on speech end or `stopMic`. What does Rust publish so that shape survives, and at what rate? - **Speech-start / speech-end semantics**, including the positive/negative threshold pair the readout displays. Events, or polled state? - **Press-to-talk ownership.** `denpa#33` exists because `stopMic()` destroys the VAD instance without flushing, so a short press sends nothing. Under the new boundary, who holds the recorded buffer, and what does release emit? - **Who owns mic lifecycle** across the overlay service, the full receiver, and the desktop shell — one recorder or several. The answer is a command-and-event surface, not an implementation. Use `/codebase-design` for the seam. ## Related - `denpa#35`, `denpa#33`, `denpa#36`, `denpa#10` - `denpa/docs/adr/0002-hardware-goes-through-rust.md` - `src/renderer/src/context/vad-context.tsx` — what exists today --- Map: `aiko/denpa#41`

aiko added the

wayfinder:ticket

wayfinder:grilling

labels

2026-07-30 13:15:04 +00:00

aiko added a new dependency

2026-07-30 13:15:06 +00:00

#44 Does silero-via-ort run on Android, and does cpal hold up on Windows?

aiko commented

2026-07-30 13:28:34 +00:00

Author

Owner

Unblocked — #44 is resolved. Read its resolution comment before starting; findings in .denpa-work/research/ort-android-audio.md.

Two things it changes about this ticket's scope:

denpa#35's premise that "cpal under Tauri's Android target is unreliable" is refuted by primary sources. cpal's Android backend is first-class and Tauri does initialise ndk_context. So Kotlin AudioRecord is no longer the assumed Android path — it is now a live choice against cpal.
The choice turns on acoustic echo cancellation, not reliability. cpal exposes no input_preset, so no AEC and no VOICE_COMMUNICATION preset (PR #995, open). Android also returns zeroes rather than an error when it arbitrates the mic away.

So this ticket now has to answer a question it did not start with: does VAD have to work while she is speaking aloud? If yes, AEC matters and Android needs AudioRecord; if no, one cpal backend spans both platforms and the boundary is far simpler.

Windows-side constraint for the same decision: WASAPI never rebinds IAudioClient, so whatever owns the stream must rebuild it on StreamInvalidated / DeviceNotAvailable, and timeout is ignored so it needs its own watchdog.

Unblocked — [#44](https://git.aiko.works/aiko/denpa/issues/44) is resolved. Read its resolution comment before starting; findings in `.denpa-work/research/ort-android-audio.md`. Two things it changes about this ticket's scope: - **`denpa#35`'s premise that "cpal under Tauri's Android target is unreliable" is refuted by primary sources.** cpal's Android backend is first-class and Tauri does initialise `ndk_context`. So Kotlin `AudioRecord` is no longer the assumed Android path — it is now a live choice against cpal. - **The choice turns on acoustic echo cancellation**, not reliability. cpal exposes no `input_preset`, so no AEC and no `VOICE_COMMUNICATION` preset ([PR #995](https://github.com/RustAudio/cpal/pull/995), open). Android also returns **zeroes rather than an error** when it arbitrates the mic away. So this ticket now has to answer a question it did not start with: **does VAD have to work while she is speaking aloud?** If yes, AEC matters and Android needs `AudioRecord`; if no, one cpal backend spans both platforms and the boundary is far simpler. Windows-side constraint for the same decision: WASAPI never rebinds `IAudioClient`, so whatever owns the stream must rebuild it on `StreamInvalidated` / `DeviceNotAvailable`, and `timeout` is ignored so it needs its own watchdog.

aiko referenced this issue

2026-07-30 13:28:51 +00:00

Move mic capture and VAD into Rust, out of the webview #35

aiko referenced this issue

2026-07-30 13:58:58 +00:00

Is the Android overlay pal part of v1? #47

aiko referenced this issue

2026-07-30 15:11:14 +00:00

Move mic capture and VAD into Rust, out of the webview #35

aiko commented

2026-07-30 15:21:30 +00:00

Author

Owner

Resolution

Barge-in is supported, and it is a setting the operator can turn off.

You can talk over her while she is speaking and voice-activity detection will pick you up — but it is togglable, not unconditional. That answers the question this ticket was reduced to by #44, and it decides the backend.

What follows immediately

Android takes the Kotlin path. AudioRecord with MediaRecorder.AudioSource.VOICE_COMMUNICATION, for the platform AEC. #44 refuted the idea that cpal is unusable on Android, so this is chosen for echo cancellation — the capability — and not for reliability. #35's scope note should be read with that correction: the conclusion survives, the reason changes.

The boundary carries two modes, not one. With barge-in off, the mic can be closed while she speaks and no echo exists to cancel. With it on, capture runs concurrently with playback and the stream must be echo-cancelled. So the command surface has to express:

a barge-in mode that can be changed at runtime, not only at startup;
whether the current backend can actually deliver echo cancellation, because the UI must not offer a toggle the platform cannot honour;
speech-start and speech-end that mean the same thing in both modes.

The carrier meter's shape is unchanged by this. It still binds to a peak-hold that never falls mid-utterance and resets on speech end or stopMic (#10, via previousTriggeredProbability). Whatever Rust publishes preserves that, in both modes.

Press-to-talk is unaffected. On release it emits what was captured during the press regardless of what VAD thought — that is #33's fix and it does not depend on barge-in.

What is deliberately not decided here

The Windows echo-cancellation path is unknown. #44 established only that cpal cannot ask for an echo-cancelled stream; it never looked at what Windows offers instead. Filed as #54, which now blocks the Windows half of #35.

That research also has to answer a question with reach beyond audio: if Windows needs a software AEC, it needs a reference signal of what is being played — and Denpa plays TTS through Web Audio in the renderer. ADR-0002 names playback "the open edge" and lists exactly this as the condition that would move it into Rust. So the Windows AEC answer may enlarge #35 from capture to capture-and-playback.

The toggle's default is not fixed by this ticket. Recommendation, revisable: off by default, because barge-in cannot be honoured until the Windows path exists, and a setting that silently does nothing on the surface you use most is worse than one you turn on. The toggle belongs in 調整 (#29).

Sequencing this hands `#35`

Build the boundary and the desktop capture path with barge-in off — cpal, silero via ort, no echo cancellation required. This is buildable today.
Windows barge-in waits on #54.
The Android half waits on NDK r28 being installed — ort's Android prebuilts target it, and r27 fails at runtime with a dlopen libc++ symbol error rather than at build time. Only 27.2.12479018 is present on this machine.

## Resolution **Barge-in is supported, and it is a setting the operator can turn off.** You can talk over her while she is speaking and voice-activity detection will pick you up — but it is togglable, not unconditional. That answers the question this ticket was reduced to by [#44](https://git.aiko.works/aiko/denpa/issues/44), and it decides the backend. ### What follows immediately **Android takes the Kotlin path.** `AudioRecord` with `MediaRecorder.AudioSource.VOICE_COMMUNICATION`, for the platform AEC. `#44` refuted the idea that cpal is unusable on Android, so this is chosen for echo cancellation — the capability — and not for reliability. `#35`'s scope note should be read with that correction: the conclusion survives, the reason changes. **The boundary carries two modes, not one.** With barge-in off, the mic can be closed while she speaks and no echo exists to cancel. With it on, capture runs concurrently with playback and the stream must be echo-cancelled. So the command surface has to express: - a barge-in mode that can be changed at runtime, not only at startup; - whether the current backend can actually deliver echo cancellation, because the UI must not offer a toggle the platform cannot honour; - speech-start and speech-end that mean the same thing in both modes. **The carrier meter's shape is unchanged by this.** It still binds to a peak-hold that never falls mid-utterance and resets on speech end or `stopMic` (`#10`, via `previousTriggeredProbability`). Whatever Rust publishes preserves that, in both modes. **Press-to-talk is unaffected.** On release it emits what was captured during the press regardless of what VAD thought — that is `#33`'s fix and it does not depend on barge-in. ### What is deliberately not decided here **The Windows echo-cancellation path is unknown.** `#44` established only that `cpal` cannot ask for an echo-cancelled stream; it never looked at what Windows offers instead. Filed as [#54](https://git.aiko.works/aiko/denpa/issues/54), which now blocks the Windows half of `#35`. That research also has to answer a question with reach beyond audio: if Windows needs a *software* AEC, it needs a reference signal of what is being played — and Denpa plays TTS through Web Audio in the renderer. ADR-0002 names playback "the open edge" and lists exactly this as the condition that would move it into Rust. So the Windows AEC answer may enlarge `#35` from capture to capture-and-playback. **The toggle's default is not fixed by this ticket.** Recommendation, revisable: **off by default**, because barge-in cannot be honoured until the Windows path exists, and a setting that silently does nothing on the surface you use most is worse than one you turn on. The toggle belongs in 調整 (`#29`). ### Sequencing this hands `#35` 1. Build the boundary and the desktop capture path with barge-in **off** — cpal, silero via `ort`, no echo cancellation required. This is buildable today. 2. Windows barge-in waits on [#54](https://git.aiko.works/aiko/denpa/issues/54). 3. The Android half waits on **NDK r28** being installed — `ort`'s Android prebuilts target it, and r27 fails at runtime with a `dlopen` libc++ symbol error rather than at build time. Only 27.2.12479018 is present on this machine.

aiko closed this issue

2026-07-30 15:21:30 +00:00

aiko referenced this issue

2026-07-30 15:21:31 +00:00

Move mic capture and VAD into Rust, out of the webview #35

aiko referenced this issue

2026-07-30 15:21:31 +00:00

Wayfinder map — the conversation works (v1) #41

aiko commented

2026-07-30 15:37:37 +00:00

Author

Owner

Echo cancellation is needed on exactly the surfaces that have it natively. Operator account, 2026-07-30:

Desktop is driven on headphones. No acoustic path from her voice back into the microphone, so there is nothing to cancel. Barge-in is correct there with no AEC at all.
Tablet and phone play through their speakers. A real echo path, so AEC is genuinely required — and Android has it natively through AudioRecord's MediaRecorder.AudioSource.VOICE_COMMUNICATION, which is what #43 already chose.

So the platform that cannot echo-cancel is the one that does not need to, and the one that needs it gets it from the API already selected. That is a happier position than the research implied.

Consequence for the Windows work. #54 found the supported Windows AEC path (eCommunications role + SetClientProperties(AudioCategory_Communications) + IAudioEffectsManager / IAcousticEchoCancellationControl), and that reaching it means abandoning cpal for wasapi — cpal hardcodes eConsole and exposes no SetClientProperties. That swap is deferred, not cancelled. It becomes necessary only if the desktop is ever driven on speakers.

What this changes in the implementation: set_barge_in no longer refuses when the backend cannot echo-cancel. Refusing would have blocked the case that actually works — headphones — while the case that does not (desktop speakers) is what the echo_cancellation capability flag exists to let the UI describe. Capability informs; it does not forbid.

This is recorded because it is not derivable from the code: a future reader finds a capture path with no echo cancellation and a barge-in toggle that permits itself anyway, and the reason is a fact about hardware rather than about software.

**Echo cancellation is needed on exactly the surfaces that have it natively.** Operator account, 2026-07-30: - **Desktop is driven on headphones.** No acoustic path from her voice back into the microphone, so there is nothing to cancel. Barge-in is correct there with no AEC at all. - **Tablet and phone play through their speakers.** A real echo path, so AEC is genuinely required — and Android has it natively through `AudioRecord`'s `MediaRecorder.AudioSource.VOICE_COMMUNICATION`, which is what [#43](https://git.aiko.works/aiko/denpa/issues/43) already chose. So the platform that cannot echo-cancel is the one that does not need to, and the one that needs it gets it from the API already selected. That is a happier position than the research implied. **Consequence for the Windows work.** [#54](https://git.aiko.works/aiko/denpa/issues/54) found the supported Windows AEC path (`eCommunications` role + `SetClientProperties(AudioCategory_Communications)` + `IAudioEffectsManager` / `IAcousticEchoCancellationControl`), and that reaching it means abandoning `cpal` for `wasapi` — cpal hardcodes `eConsole` and exposes no `SetClientProperties`. **That swap is deferred, not cancelled.** It becomes necessary only if the desktop is ever driven on speakers. **What this changes in the implementation:** `set_barge_in` no longer *refuses* when the backend cannot echo-cancel. Refusing would have blocked the case that actually works — headphones — while the case that does not (desktop speakers) is what the `echo_cancellation` capability flag exists to let the UI describe. Capability informs; it does not forbid. This is recorded because it is not derivable from the code: a future reader finds a capture path with no echo cancellation and a barge-in toggle that permits itself anyway, and the reason is a fact about hardware rather than about software.

aiko referenced this issue

2026-07-30 15:37:37 +00:00

What is the acoustic echo cancellation path on Windows? #54

aiko referenced this issue

2026-07-30 15:37:38 +00:00

Move mic capture and VAD into Rust, out of the webview #35

aiko referenced this issue from a commit

2026-07-30 17:28:45 +00:00

docs(adr): record that hardware goes through Rust

aiko referenced this issue from a commit

2026-07-30 17:28:45 +00:00

feat(audio): capture and detect speech in Rust, not the webview