aiko/denpa
1
0
Fork 0
Code Issues 20 Pull requests Projects Releases Packages Wiki Activity Actions

Does silero-via-ort run on Android, and does cpal hold up on Windows? #44

New issue
Closed
opened 2026-07-30 13:15:04 +00:00 by aiko · 2 comments
aiko commented 2026-07-30 13:15:04 +00:00
Owner
Copy link

Question

Facts the audio boundary decision waits on, all outside this working directory:

  • Does the ort crate build and run inference on Android under Tauri's
    Android target — architectures, ONNX Runtime binary availability, model
    loading from assets, and what the binary costs? denpa#35 assumes silero moves
    into Rust; if ort cannot reach Android, the Android VAD story is different
    from the desktop one and the boundary must absorb that.
  • Does cpal behave on Windows for a long-lived capture stream — device
    selection, what happens when the default input device changes or disappears
    mid-session, and whether exclusive mode is ever needed?
  • denpa#35 asserts "cpal under Tauri's Android target is unreliable".
    Verify or refute that against primary sources rather than inheriting it.
  • What does Kotlin AudioRecord require alongside RECORD_AUDIO when a
    foreground service is already running — denpa#23 had to reason about this
    once already.

Answer from primary sources: crate docs, upstream issue trackers, Android
developer documentation. Cite everything.

Related

  • denpa#35, denpa#23

Map: aiko/denpa#41

## Question Facts the audio boundary decision waits on, all outside this working directory: - Does the **`ort` crate** build and run inference on Android under Tauri's Android target — architectures, ONNX Runtime binary availability, model loading from assets, and what the binary costs? `denpa#35` assumes silero moves into Rust; if `ort` cannot reach Android, the Android VAD story is different from the desktop one and the boundary must absorb that. - Does **`cpal`** behave on Windows for a long-lived capture stream — device selection, what happens when the default input device changes or disappears mid-session, and whether exclusive mode is ever needed? - `denpa#35` asserts "`cpal` under Tauri's Android target is unreliable". **Verify or refute that** against primary sources rather than inheriting it. - What does **Kotlin `AudioRecord`** require alongside `RECORD_AUDIO` when a foreground service is already running — `denpa#23` had to reason about this once already. Answer from primary sources: crate docs, upstream issue trackers, Android developer documentation. Cite everything. ## Related - `denpa#35`, `denpa#23` --- Map: `aiko/denpa#41`
aiko commented 2026-07-30 13:28:33 +00:00
Author
Owner
Copy link

Resolution

Findings: .denpa-work/research/ort-android-audio.md — 659 lines, primary sources only, every claim carries a URL and anything unsourced is marked unverified.

The Rust audio boundary is viable on both targets, but it is not symmetric.

1. ort on Android — yes, arm64-v8a only. pyke has shipped a prebuilt static ONNX Runtime for aarch64-linux-android (NNAPI EP) since ort 2.0.0-rc.11, 2026-01-07. There is no prebuilt for armv7, x86_64 or i686. Tauri v2 builds all four ABIs by default, so an unconstrained tauri android build fails to link on three of four legs — pin --target aarch64. The x86_64 emulator is out; testing is on real hardware only, which makes #48 (get a phone attached) load-bearing rather than convenient.

2. The NDK version is load-bearing. pyke builds against NDK r28 / API 24. ort issue #514 was precisely an NDK mismatch, surfacing as a runtime dlopen failed: cannot locate symbol _ZTT...__cxx11.... Maintainer, verbatim: "The binaries target r28, so NDK 27 definitely won't work." r28 also brings 16 KB page alignment, required by Play for API 35+ since 2025-11-01.

3. Size: the runtime dominates the model tenfold. ONNX Runtime arm64 is 27.3 MiB shared / ~119 MiB static archive; Silero VAD itself is 1.2–2.2 MiB. Contribution after --gc-sections/LTO is unverified.

4. Model loading is not std::fs. Tauri resources on Android live behind asset://localhost/. The documented path is include_bytes! plus commit-from-memory; Rust-side AAssetManager access under Tauri is undocumented.

5. cpal on Windows capture is sound, but you own the lifecycle. WASAPI in cpal never rebinds IAudioClient, so a default-device change or removal yields StreamInvalidated / DeviceNotAvailable and the stream must be rebuilt by you. timeout is ignored on WASAPI — write a watchdog. Exclusive mode is unsupported by cpal (open since 2016) and Microsoft steers away from it for capture anyway; shared mode is correct.

6. Version-pin risk. Three fixes that bear directly on long-lived capture — #1264 panic on device loss, #1267 correct error kind, #1281 false first-buffer xrun — landed on master after 0.18.1 (2026-06-07) and are in no published release. Also note 0.18 removed Device::name() and StreamError.

7. "cpal under Tauri's Android target is unreliable" — refuted as phrased. denpa#35 asserts this; primary sources do not support it. cpal lists Android as first-class (AAudio; the oboe C++ dependency has been gone since 0.16.0, 2025-06-07), input is fully implemented, no open issue reports Android audio broken, and Tauri's mobile_entry_point → android_binding! → tao onCreate chain does initialise ndk_context, which is cpal's one Tauri-sensitive requirement. Tauri's own repos are silent on the claim. Likely formed against a pre-0.16 oboe-era cpal.

8. What is genuinely true is narrower, and worth putting in the ADR. API 26 floor. The app must declare and runtime-request RECORD_AUDIO itself — cpal does nothing and Tauri adds nothing. No acoustic echo cancellation: there is no input_preset control (PR #995, open since 2025-07-31). And Android hands you zeroes rather than an error when it arbitrates the mic away. Those last two produce exactly the "unreliable" symptom that was being attributed to cpal itself.

9. The foreground-service rules are the hardest constraint, and they bite the overlay specifically. Mic needs FOREGROUND_SERVICE_MICROPHONE (API 34+), android:foregroundServiceType="microphone" in the manifest, and the microphone bit in the startForeground() mask. SYSTEM_ALERT_WINDOW exempts you from the background-start restriction only — it is not on the while-in-use exemption list. So an already-running overlay FGS does not let you start or promote to a microphone-typed FGS from the background; the docs require a visible activity, not a visible overlay. Android 15+ narrows the SYSTEM_ALERT_WINDOW exemption further. Workable shape: one service carrying specialUse|microphone, promoted by a second startForeground() at a moment when a real activity is visible.

What this hands #43: the interface can be one Rust command surface across both platforms, but it must tolerate an asymmetric backend — a supervisor on Windows, and on Android either cpal-with-no-AEC or a Kotlin AudioRecord path. The real argument for AudioRecord is acoustic echo cancellation and the VOICE_COMMUNICATION preset, not reliability — and whether AEC matters depends on whether VAD has to work while she is speaking out loud. That is now a sharp question #43 has to answer.

What this hands #47: the overlay's foreground service does not help microphone access, and cannot promote to it from the background. It is a cost, not a shared asset.

## Resolution Findings: [`.denpa-work/research/ort-android-audio.md`](../../../../.denpa-work/research/ort-android-audio.md) — 659 lines, primary sources only, every claim carries a URL and anything unsourced is marked **unverified**. **The Rust audio boundary is viable on both targets, but it is not symmetric.** **1. `ort` on Android — yes, arm64-v8a only.** pyke has shipped a prebuilt static ONNX Runtime for `aarch64-linux-android` (NNAPI EP) since `ort` 2.0.0-rc.11, 2026-01-07. There is **no** prebuilt for `armv7`, `x86_64` or `i686`. Tauri v2 builds all four ABIs by default, so an unconstrained `tauri android build` fails to link on three of four legs — pin `--target aarch64`. **The x86_64 emulator is out; testing is on real hardware only**, which makes `#48` (get a phone attached) load-bearing rather than convenient. **2. The NDK version is load-bearing.** pyke builds against **NDK r28 / API 24**. `ort` issue [#514](https://github.com/pykeio/ort/issues/514) was precisely an NDK mismatch, surfacing as a runtime `dlopen failed: cannot locate symbol _ZTT...__cxx11...`. Maintainer, verbatim: *"The binaries target r28, so NDK 27 definitely won't work."* r28 also brings 16 KB page alignment, required by Play for API 35+ since 2025-11-01. **3. Size: the runtime dominates the model tenfold.** ONNX Runtime arm64 is 27.3 MiB shared / ~119 MiB static archive; Silero VAD itself is 1.2–2.2 MiB. Contribution after `--gc-sections`/LTO is **unverified**. **4. Model loading is not `std::fs`.** Tauri resources on Android live behind `asset://localhost/`. The documented path is `include_bytes!` plus commit-from-memory; Rust-side `AAssetManager` access under Tauri is **undocumented**. **5. `cpal` on Windows capture is sound, but you own the lifecycle.** WASAPI in cpal never rebinds `IAudioClient`, so a default-device change or removal yields `StreamInvalidated` / `DeviceNotAvailable` and the stream must be rebuilt by you. `timeout` is ignored on WASAPI — write a watchdog. Exclusive mode is unsupported by cpal (open since 2016) and Microsoft steers away from it for capture anyway; shared mode is correct. **6. Version-pin risk.** Three fixes that bear directly on long-lived capture — [#1264](https://github.com/RustAudio/cpal/pull/1264) panic on device loss, [#1267](https://github.com/RustAudio/cpal/pull/1267) correct error kind, [#1281](https://github.com/RustAudio/cpal/pull/1281) false first-buffer xrun — landed on master **after** 0.18.1 (2026-06-07) and are in no published release. Also note 0.18 removed `Device::name()` and `StreamError`. **7. "`cpal` under Tauri's Android target is unreliable" — refuted as phrased.** `denpa#35` asserts this; primary sources do not support it. cpal lists Android as first-class (AAudio; the oboe C++ dependency has been gone since 0.16.0, 2025-06-07), input is fully implemented, no open issue reports Android audio broken, and Tauri's `mobile_entry_point` → `android_binding!` → tao `onCreate` chain **does** initialise `ndk_context`, which is cpal's one Tauri-sensitive requirement. Tauri's own repos are silent on the claim. Likely formed against a pre-0.16 oboe-era cpal. **8. What is genuinely true is narrower, and worth putting in the ADR.** API 26 floor. The app must declare *and runtime-request* `RECORD_AUDIO` itself — cpal does nothing and Tauri adds nothing. **No acoustic echo cancellation**: there is no `input_preset` control ([PR #995](https://github.com/RustAudio/cpal/pull/995), open since 2025-07-31). And Android hands you **zeroes rather than an error** when it arbitrates the mic away. Those last two produce exactly the "unreliable" symptom that was being attributed to cpal itself. **9. The foreground-service rules are the hardest constraint, and they bite the overlay specifically.** Mic needs `FOREGROUND_SERVICE_MICROPHONE` (API 34+), `android:foregroundServiceType="microphone"` in the manifest, and the `microphone` bit in the `startForeground()` mask. **`SYSTEM_ALERT_WINDOW` exempts you from the background-*start* restriction only — it is not on the while-in-use exemption list.** So an already-running overlay FGS does **not** let you start or promote to a microphone-typed FGS from the background; the docs require a visible **activity**, not a visible overlay. Android 15+ narrows the `SYSTEM_ALERT_WINDOW` exemption further. Workable shape: one service carrying `specialUse|microphone`, promoted by a second `startForeground()` at a moment when a real activity is visible. --- **What this hands `#43`:** the interface can be one Rust command surface across both platforms, but it must tolerate an asymmetric backend — a supervisor on Windows, and on Android either cpal-with-no-AEC or a Kotlin `AudioRecord` path. **The real argument for `AudioRecord` is acoustic echo cancellation and the `VOICE_COMMUNICATION` preset, not reliability** — and whether AEC matters depends on whether VAD has to work while she is speaking out loud. That is now a sharp question `#43` has to answer. **What this hands `#47`:** the overlay's foreground service does not help microphone access, and cannot promote to it from the background. It is a cost, not a shared asset.
aiko closed this issue 2026-07-30 13:28:34 +00:00
aiko commented 2026-07-30 21:42:42 +00:00
Author
Owner
Copy link

Two of this issue's findings need correcting, from building the Android microphone in #35 (merged 458f0c6) and running it on a Pixel 7 Pro, 2026-07-30.

1. ndk_context is not initialised. This issue says Tauri's mobile_entry_point → android_binding! → tao onCreate chain "does initialise ndk_context, which is cpal's one Tauri-sensitive requirement." It does not — verified by calling it.

And the failure is not a graceful one. ndk_context::android_context() is ANDROID_CONTEXT.expect("android context was not initialized") (ndk-context-0.1.1/src/lib.rs:72), so the first call aborts the process — SIGABRT, a Rust panic unwinding across the extern "C" boundary of Tauri's JNI IPC handler, panic_cannot_unwind → abort. No error to handle, no log to read except a tombstone.

This matters beyond #35: it was the stated basis for cpal being viable on Android under Tauri, and anything else reaching for the Activity or the JavaVM from Rust will hit the same wall. Tauri's own run_on_android_context exists but lives on the runtime handle, which is not public API. What works is Kotlin handing the JVM over at onCreate — needed regardless, because a class looked up from a natively-attached thread resolves against the system classloader and will never find an app class.

2. NDK r27's failure is narrower than recorded. This issue says r27 "produces a runtime dlopen libc++ symbol failure". A standalone aarch64 binary built against r27 — linking the same ort prebuilt, loading silero from memory and scoring a frame — runs fine on the Pixel 7 Pro. So the failure is about what ends up in the APK rather than about the compiler; r27's libc++_shared.so is 1.79 MB against r28's 9.24 MB.

The conclusion is unchanged — build with r28 (r28.2.13676358 is installed) — but "a clean build proves nothing, and neither does a clean standalone run" is the more accurate warning.

Confirmed, for the record: the arm64-v8a-only prebuilt is real, and --target aarch64 must be pinned or Tauri fails to link the other three ABIs.

A third thing worth adding to the file, because it is the same class of trap and cost more than either of the above: RECORD_AUDIO is a foreground app-op. Granted, on a locked or sleeping screen, AudioRecord returns full buffers of zeros — no error, no short read, indistinguishable from a quiet room. That is the same shape as the default_input_device() silence that cost #35 an evening on Windows. docs/android-build.md records all of it.

**Two of this issue's findings need correcting, from building the Android microphone in `#35` (merged `458f0c6`) and running it on a Pixel 7 Pro, 2026-07-30.** **1. `ndk_context` is *not* initialised.** This issue says Tauri's `mobile_entry_point` → `android_binding!` → tao `onCreate` chain "does initialise `ndk_context`, which is cpal's one Tauri-sensitive requirement." It does not — verified by calling it. And the failure is not a graceful one. `ndk_context::android_context()` is `ANDROID_CONTEXT.expect("android context was not initialized")` (`ndk-context-0.1.1/src/lib.rs:72`), so the first call **aborts the process** — `SIGABRT`, a Rust panic unwinding across the `extern "C"` boundary of Tauri's JNI IPC handler, `panic_cannot_unwind` → `abort`. No error to handle, no log to read except a tombstone. This matters beyond `#35`: it was the stated basis for cpal being viable on Android under Tauri, and anything else reaching for the Activity or the JavaVM from Rust will hit the same wall. Tauri's own `run_on_android_context` exists but lives on the runtime handle, which is not public API. What works is Kotlin handing the JVM over at `onCreate` — needed regardless, because a class looked up from a natively-attached thread resolves against the **system** classloader and will never find an app class. **2. NDK r27's failure is narrower than recorded.** This issue says r27 "produces a runtime `dlopen` libc++ symbol failure". A standalone aarch64 binary built against r27 — linking the same `ort` prebuilt, loading silero from memory and scoring a frame — **runs fine** on the Pixel 7 Pro. So the failure is about what ends up in the APK rather than about the compiler; r27's `libc++_shared.so` is 1.79 MB against r28's 9.24 MB. The conclusion is unchanged — build with **r28** (r28.2.13676358 is installed) — but "a clean build proves nothing, and neither does a clean standalone run" is the more accurate warning. **Confirmed, for the record:** the arm64-v8a-only prebuilt is real, and `--target aarch64` must be pinned or Tauri fails to link the other three ABIs. A third thing worth adding to the file, because it is the same class of trap and cost more than either of the above: **`RECORD_AUDIO` is a *foreground* app-op.** Granted, on a locked or sleeping screen, `AudioRecord` returns full buffers of zeros — no error, no short read, indistinguishable from a quiet room. That is the same shape as the `default_input_device()` silence that cost `#35` an evening on Windows. `docs/android-build.md` records all of it.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/88-openwakeword-beside-silero
prototype/seam-and-pending
issue/denpa-49-onboarding-fills-viewport
issue/denpa-35-mic-capture-into-rust
issue/denpa-32-proactive-break-in
issue/denpa-28-onboarding
issue/denpa-29-settings-sheet
issue/denpa-19-approval-surface
issue/denpa-31-screen-share-grant
issue/denpa-27-two-hop-status
issue/denpa-17-log-relay-rows
issue/denpa-23-android-overlay-pal
issue/denpa-9-cut-group-camera-browser
integrate/denpa-18-pal
integrate/denpa-16-tablet
issue/denpa-12-mobile-chrome
issue/denpa-18-desktop-transparent-pal
issue/denpa-16-tablet-landscape
issue/denpa-10-design-tokens
docs/denpa-receiver-handoff
build
dev
R4-test
neo-mic
mic-test
close-button
pre-autonomous-run-2026-07-30
v1.2.1
v1.2.0
v1.1.0
v1.0.0
Labels
Clear labels   
needs-info
Waiting on reporter

  
needs-triage
Maintainer needs to evaluate

  
ready-for-agent
Fully specified, AFK-ready

  
ready-for-human
Needs human implementation

  
v2
Out of scope for v1; see the wayfinder map

  
wayfinder:grilling

   
wayfinder:map

   
wayfinder:prototype

   
wayfinder:research

   
wayfinder:task

   
wayfinder:ticket

   
wontfix
Will not be actioned

No labels
needs-info
needs-triage
ready-for-agent
ready-for-human
v2
wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wayfinder:task
wayfinder:ticket
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#43 What crosses the Rust/webview boundary for audio?
aiko/denpa
Reference
aiko/denpa#44
No description provided.