Move mic capture and VAD into Rust, out of the webview #35
Labels
No labels
needs-info
needs-triage
ready-for-agent
ready-for-human
v2
wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wayfinder:task
wayfinder:ticket
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Depends on
Reference
aiko/denpa#35
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Move microphone capture and voice-activity detection out of the webview and into Rust, so
getUserMediais never called and the browser never prompts.Decided by the human on 2026-07-30, in preference to auto-allowing the WebView2 permission request from Rust. The reasoning: the app already has its own OS-level permission screens, so a second browser-level prompt on every launch is noise — and WebView2 does not persist the grant, prompts every launch, and offers no way to re-prompt once blocked (tauri#8979, tauri#5042, WebView2Feedback#2930).
What is there today
src/renderer/src/context/vad-context.tsxusesMicVADfrom@ricky0123/vad-web, which opens the mic withgetUserMediaitself and runs silero throughonnxruntime-web. Two consequences:aiko/denpa#33is a direct consequence of that ownership. Press-to-talk drivesstartMic()/stopMic(), but the only path that transmits is VAD's ownonSpeechEnd, andstopMic()destroys the instance without flushing — so a short press sends nothing. This issue supersedes#33's fix approach; a Rust recorder can flush on release, which is what press-to-talk actually needs.Scope
cpal), exposed as Tauri commands.ortcrate), replacingMicVAD. Speech-start and speech-end semantics must match what the UI already expects, including the positive/negative threshold pair the readout displays.#10's meter binds topreviousTriggeredProbability— a peak-hold that never falls mid-utterance and resets on speech end orstopMic. Whatever Rust publishes has to preserve that shape, or the meter changes behaviour as a side effect. See the ledger entry on it in.denpa-work/QUESTIONS.md.#33fix.cpalunder Tauri's Android target is unreliable; expectAudioRecordin Kotlin, alongside theRECORD_AUDIOruntime permission.aiko/denpa#23already had to reason aboutRECORD_AUDIOinteracting withstartForegroundfor the overlay service — read its notes before starting.Known consequence: the web target loses audio
npm run dev:webhas no Rust, so a browser-hosted client will have no microphone at all — no press-to-talk, no hands-free. This matters more than it looks: browser-over-LAN is currently the only way the client has been verified on a real tablet (see.denpa-work/tablet-client.png), and that verification path will keep working for rendering but not for voice.Decide deliberately whether to keep a JS fallback for the web target or accept that voice is desktop-and-Android only. Accepting it is defensible — the design treats a phone and an Android overlay as first-class, and neither is a browser — but it should be a decision rather than a discovery.
Acceptance
getUserMediacall remains in the renderer.#33).Scope narrowed by human decision, 2026-07-30.
The "web target loses audio" section above is no longer a concern to design around. Android is an installed APK — one app that switches between the overlay pal and the full receiver — and desktop is the Tauri shell. The browser-over-LAN path was only an orchestrator test harness for rendering; it is not a usage mode.
So: no JS fallback is needed.
getUserMediacan disappear outright, and a browser-hosted client having no microphone is acceptable rather than a regression. The tablet screenshots in.denpa-work/were taken through Chrome purely because the APK had not been built yet; that is a verification detail, not a supported surface.This also means the Android capture path is not optional extra work — it is the primary mobile path.
AudioRecordin Kotlin plus theRECORD_AUDIOruntime permission, alongside whateveraiko/denpa#23already established for the overlay service and itsstartForegroundinteraction.Reinforced by a real-device observation, and now covered by a rule.
On the installed Android APK (universal debug, built from
mainat8237457), the OS-levelRECORD_AUDIOpermission was granted through the app's own permission screen — andgetUserMediastill failed withNotAllowedError. Pressing 押して送信 advanced the UI to 送信中 and transmitted nothing; the go-between log shows no voice frame arriving. The cause is a second permission gate inside the Android WebView, which onlyWebChromeClient.onPermissionRequestcan open.So this issue is not only about removing a launch-time prompt on desktop. Android voice does not work at all through the webview, regardless of
aiko/denpa#33.The general rule is now recorded as
docs/adr/0002-hardware-goes-through-rust.md: anything touching a device goes through Rust and is exposed to the renderer as a command. This issue is that rule applied to the microphone; screen capture gets its own issue for the same reason.Correction to this issue's scope section. It states "
cpalunder Tauri's Android target is unreliable; expectAudioRecordin Kotlin." Researched against primary sources in #44 — that claim is not supported. cpal lists Android as a first-class platform (AAudio; the oboe C++ dependency has been gone since 0.16.0, 2025-06-07), input is fully implemented, no open issue reports Android audio broken, and Tauri'smobile_entry_point->android_binding!-> taoonCreatechain does initialisendk_context, which is cpal's one Tauri-sensitive requirement.What is true, and is probably what the claim was formed from: cpal exposes no
input_preset, so no acoustic echo cancellation and noVOICE_COMMUNICATIONpreset (PR #995, open since 2025-07-31); and Android returns zeroes rather than an error when it arbitrates the mic away. Together those read as unreliability.So the Kotlin
AudioRecordpath is a live choice rather than a foregone one, and the thing it buys is AEC. That choice is being decided in #43; do not start this issue against the old premise.Also binding on this issue, from #44:
orthas an Android prebuilt for arm64-v8a only, so builds must pin--target aarch64(Tauri builds all four ABIs by default and will fail to link on three), the NDK must be r28 (r27 produces a runtimedlopenlibc++ symbol failure), and the x86_64 emulator is unusable — testing is on real hardware. Full findings:.denpa-work/research/ort-android-audio.md.Observed on Windows desktop, 2026-07-30 — this issue's premise is now confirmed on the desktop surface, not just Android.
Walked onboarding to completion on the Tauri desktop app against a live Hermes (gateway connected, adapter attached, a real turn crossed). In the finished receiver chrome, pressing either mic key raises a VAD permission-denied dialog:
key-row.tsx:62—hands-free(集音)key-row.tsx:86—ptt(押して送信)Both drive
VADContext, which callsgetUserMediaitself, so they fail the same way.interrupt(:114) is unaffected because it never touches the mic.Until now this issue argued the WebView2 microphone problem from upstream Tauri issue links, and
#36recorded it observed only on Android. It reproduces on Windows.The consequence that is not yet written down anywhere
There is no text input in the receiver chrome. The only
<Input>elements in the entire client are onboarding's address field (onboarding.tsx:349) and the settings sheet (settings-sheet.tsx:435). The receiver is voice-only.So with
getUserMediadenied, there is currently no way to send a turn from the desktop app at all. The only turn that has ever crossed this wire is onboarding's automated pairing probe (きこえてる?, relayed and answered in 10.4s at 17:05:15).That makes this issue the sole unblock for the desktop surface rather than the largest of several. v1's proof — a real conversation on desktop, tablet and phone — cannot begin on any surface until it lands.
Also relevant to the boundary decision in #43
#51was filed today: the desktop push-to-talk global hotkey never registers on Windows either, becauselib.rs:43asks for a bareAltLeft, whichRegisterHotKeywill not accept. So both routes to the mic on desktop — the on-screen key and the OS-level hotkey — are currently dead, for two unrelated reasons.The boundary is decided — see #43's resolution before starting.
Three things it fixes for this issue:
AudioRecordwithVOICE_COMMUNICATION, for its acoustic echo cancellation. Note this issue's stated reason — "cpalunder Tauri's Android target is unreliable" — is refuted by #44. The conclusion stands; the reason is AEC, not reliability.Buildable today: the boundary plus desktop capture with barge-in off — cpal, silero via
ort, no echo cancellation needed.Blocked: Windows barge-in on #54; the Android half on NDK r28, which is not installed here (only 27.2.12479018, and r27 fails at runtime with a
dlopenlibc++ symbol error, not at build time).Echo cancellation is needed on exactly the surfaces that have it natively. Operator account, 2026-07-30:
AudioRecord'sMediaRecorder.AudioSource.VOICE_COMMUNICATION, which is what #43 already chose.So the platform that cannot echo-cancel is the one that does not need to, and the one that needs it gets it from the API already selected. That is a happier position than the research implied.
Consequence for the Windows work. #54 found the supported Windows AEC path (
eCommunicationsrole +SetClientProperties(AudioCategory_Communications)+IAudioEffectsManager/IAcousticEchoCancellationControl), and that reaching it means abandoningcpalforwasapi— cpal hardcodeseConsoleand exposes noSetClientProperties. That swap is deferred, not cancelled. It becomes necessary only if the desktop is ever driven on speakers.What this changes in the implementation:
set_barge_inno longer refuses when the backend cannot echo-cancel. Refusing would have blocked the case that actually works — headphones — while the case that does not (desktop speakers) is what theecho_cancellationcapability flag exists to let the UI describe. Capability informs; it does not forbid.This is recorded because it is not derivable from the code: a future reader finds a capture path with no echo cancellation and a barge-in toggle that permits itself anyway, and the reason is a fact about hardware rather than about software.
The desktop half works, verified by conversation
Voice now travels cpal -> 16 kHz resample -> silero in Rust -> press flush ->
mic-audio-data-> wav -> adapter -> Hermes -> answer.getUserMediaandMicVADare gone fromvad-context.tsx;VADStateis unchanged, so the ten components consuming it did not move.Branch
issue/denpa-35-mic-capture-into-rust, commits0da3018(boundary) andbffad16(renderer + picker). Not merged, not pushed.The bug that made this hard, recorded because it will recur
The first working build produced this:
Correct duration, correct sample count, every sample zero. cpal's
default_input_device()on this machine returns digital silence, and two other devices share its name (Microphone). Hermes received a well-formed wav of nothing, transcribed it to an empty string, and answered using earlier context -- which presents as her replying to something you never said, in a language you did not use.Nothing in the client, the go-between or the gateway reported a problem. It took instrumenting the go-between (
aiko/ollvt-hermes-bridge4a11fe9) to separate "captured nothing" from "transcription failed".So the input picker is load-bearing, not a convenience, and it is in this commit: every input listed as name and manufacturer, "follow the system default" kept as a real first choice, and an open microphone cycled on selection so the change is audible rather than deferred to the next launch.
Review found six defects in the first renderer draft
All fixed before commit. Worth listing because four of them fail silently and none is caught by a test suite:
micOnagainst a microphone that is always closed at launch -- every press failed with "mic is not open", permanently, until hands-free was toggled off and onmic_start, leaving Rust holding a press that never ends, after whichemit_utterancesuppresses every later utteranceupdateSettingsrestarting capture with the previous thresholds forever, through a closure that never refreshedstopMicno-opping while a start was in flight, leaving the device recordingtAn earlier round on the Rust side found six more, including ONNX inference and whole-utterance JSON serialisation inside the cpal callback while holding a lock.
Still open on this issue
set_barge_inormic_capabilities, so the mode stays at Rust's default of off and the microphone always pauses while she speaks. The toggle belongs in the 調整 sheet (#29) beside the capability it depends on.AudioRecordwithVOICE_COMMUNICATION, and NDK r28, which is not installed here.Merged to
mainata3b782e—Merge #35's microphone into Rust. Pushed and verified against the remote.742 vitest in 48 files and 18 Rust tests are green on the merge commit itself, not only on the branch, and the typecheck delta against the 585-error baseline is zero.
Landed:
7e604c3ADR-0002 ·0da3018the Rust audio boundary ·bffad16the renderer swap and input picker ·3aa1437the live station meter.Staying open, because the issue is not finished:
set_barge_inormic_capabilities, so the mode sits at Rust's default of off and the microphone always pauses while she speaks. The toggle belongs in the 調整 sheet (#29), beside the capability it depends on.AudioRecordwithVOICE_COMMUNICATION, and NDK r28, which is not installed on this machine. r27 fails at runtime with adlopenlibc++ symbol error rather than at build time, so it will look like a working build.Correction to the note above: "barge-in has no UI" understates what exists.
There is no barge-in toggle — nothing calls
set_barge_inormic_capabilities, so the mode stays at Rust's default of off. That part stands. But the operator is not stuck waiting for her to finish, because 割り込み already does the job by a different route.key-row.tsx:114-124— the acid key (unprompted,rgba(234,255,60,…)) is 割り込み. Pressing it movesaiStateoffthinking-speaking, which firesset_output_active(false), which lifts the pause on capture. So the loop that works today is:What barge-in would add is removing the press: you simply talk over her and VAD picks you up. That is a convenience over an existing path, not a missing capability — which makes the toggle lower priority than the earlier note implied.
Worth keeping in mind when it is built: on the desktop it is safe with no echo cancellation because the operator wears headphones, and the toggle belongs in the 調整 sheet (
#29) beside the capability flag, so it can say what it will actually do on a surface that cannot echo-cancel.Merged to
mainat458f0c6— Merge #35's remaining half. Pushed, and verified by reading the remote rather than trusting the push.808 vitest across 56 files and 26 Rust tests, green on the merge commit itself. Typecheck delta zero.
Landed:
3219320the 割り込み許可 switch ·a4376a1the speech-start edit nothing emitted ·c739e70Android capture and the engine split ·d43a8a5a half-failed open.1. The barge-in toggle
The row is in 電波 beside the capability it depends on, and it is absent entirely until a backend answers — an affordance reaching nothing is the thing this sheet already refuses elsewhere. Where the backend cannot echo-cancel it says so and works anyway, which is
#43's "capability informs, it never forbids": on the desktop the echo path is removed rather than cancelled, because it is driven on headphones.Two silent disagreements closed with it.
MicStatestarts every launch with barge-in off, so a persistedtruewould have drawn the switch on over a microphone that still paused — reconciled at mount, and committed locally only once Rust has taken it. Andset_barge_in(false)mid-sentence left the pause lifted for the rest of her turn:pausedwas the conjunction of two facts with only one of them stored, so it could not be un-applied.2.
denpa://speech-startwas emitted by nobodyvad-context.tsx:414had listened for it since capture moved to Rust. Nothing sent it. Capture, detection and the transmitted audio were all unaffected and both suites were green; the symptoms were a receiver that never said 聞いている, andhandleSpeechStart's interrupt never firing.That is the whole of barge-in. The mode keeps the mic open through her turn so you can talk over her — without this edge she simply never stops, and the row would have promised a conversation it could not deliver. Found while wiring the toggle, not by a test.
The test added is the general form: every
denpa://the renderer listens for must be emitted somewhere insrc-tauri/src/audio, every command it invokes must exist and be registered. Checked against a deleted emit before committing — it fails on exactly this defect, and it caught the engine split later the same hour.3. The NDK, cleared
r28.2.13676358 installed,
sdkmanagerexit 0. Not taken on trust: a standalone aarch64 probe pushed to the Pixel 7 Pro loads silero and scores a frame —probability=0.044262677, and 0.234 ms per 32 ms frame, 137× realtime.One correction to this issue's premise. r27 also runs fine in that standalone form on the same phone. So the
dlopenlibc++ failure is about what ends up in the APK, not about the compiler. r28 is still what to build with, anddocs/android-build.mdsays why.4. Android capture
The shared half of the microphone moved to
audio/engine.rs— detector, hysteresis, pre-roll, press machinery. cpal andAudioRecordare now two sample sources for one engine, so#43's "speech start and end mean the same thing" holds across platforms instead of being written twice and drifting. KotlinMicRecorderopensVOICE_COMMUNICATIONand pushes PCM over JNI, not a Tauri channel — a channel serialises to JSON, and 16 kHz through JSON is paid on every buffer forever.Three things the build could not have told me
ndk_contextis never initialised in a Tauri app.#44concluded that themobile_entry_point→android_binding!→ taoonCreatechain populates it. It does not, andandroid_context()is anexpect— so the firstmic_startdid not fail, itSIGABRTed the process. Kotlin now hands the JVM over atonCreate, which is needed regardless: a class looked up from a natively-attached thread resolves against the system classloader, which has never heard of an app class. Worth correcting on#44.denpa://mic-levelfired per frame, 31/s; on Android an emit crosses JNI into the webview and costs more than the 32 ms of audio it represents. The channel grew for as long as the mic was open, andmic_stopblocked draining it — over 60 s after three minutes of capture. It is now sent only when the value changes, which is all a peak-hold ever has to say; the same values arrive in the same order, only the repeats are gone.mic_stopmeasured at 362 ms after the fix. Desktop was never near the limit.mic_startcould return an error with the recorder still open — one?after the device was started, leaving an open microphone with no handle to close it.Verified on a Pixel 7 Pro
mic_capabilities→{echo_cancellation: true, sample_rate: 16000}·mic_start→ 16000 in 327 ms ·AudioRecordonVOICE_COMMUNICATIONat 16 kHz · silero scoring the frames · press/release ·mic_stopin 421 ms ·set_barge_inboth directions · and, on the OnePlus Pad where the permission is denied,mic_startrejecting with 「マイクの許可がいる。許可してからもう一度。」 rather than opening a silent device. That last one is this issue's sixth acceptance line, on real hardware.Driven through the debug webview's devtools, because the receiver chrome needs a live go-between before any mic key exists.
docs/android-build.mdrecords the method along with the--target aarch64pin.What is not done —
#61No sample above digital silence has ever gone through the Android path. Not a suspected defect; an unverified one, which on this project is the more expensive kind.
Both devices defeated an unattended check. The Pixel takes
pm grantbut sits behind a keyguard — andRECORD_AUDIOis a foreground app-op, so a locked screen means the op is rejected andAudioRecordreturns full buffers of zeros with no error and no short read, indistinguishable from a quiet room. Confirmed byappops, with arejectTimeseconds old at every capture. The Pad is unlocked but refusespm grant, so its permission needs a finger.Because that failure is invisible,
MicRecordernow states a verdict on the opening seconds of every capture: either the peak it heard, or a warning naming the app-op as the first thing to check. It needs a person, an unlocked device and one spoken sentence — filed as#61, labelled ready-for-agent.Closing this one: every line of the acceptance list is built and merged, and the remaining work is a verification that needs hands rather than a change to the code.