aiko/denpa

Fork 0

What is the acoustic echo cancellation path on Windows? #54

New issue

Closed

opened 2026-07-30 15:21:30 +00:00 by aiko · 2 comments

aiko commented

2026-07-30 15:21:30 +00:00

Owner

Question

#43 decided that barge-in — talking over her while she speaks — is supported and togglable. That requires acoustic echo cancellation, or her own output feeds straight back into the voice-activity detector.

Android is answered: Kotlin AudioRecord with MediaRecorder.AudioSource.VOICE_COMMUNICATION, which gives the platform AEC.

Windows is not. #44 established only the negative: cpal exposes no input_preset, so it cannot ask for an echo-cancelled stream (PR #995, open since 2025-07-31). What Windows offers instead was never investigated.

Answer from primary sources — Microsoft Learn, the Windows SDK, crate docs, upstream issue trackers. Cite everything; mark anything unsourced as unverified.

What is the supported AEC path on Windows today? Cover the Voice Capture DSP (IMediaObject / the media object for audio capture), the AUDIO_EFFECT_TYPE_ACOUSTIC_ECHO_CANCELLATION audio effects API, and Communications-category capture streams. Which of these still work on Windows 11, and which need the app to opt in at stream creation?
Can any of it be reached alongside cpal, or does an AEC-enabled capture stream have to be opened by WASAPI directly, bypassing cpal on Windows? If bypassing, what does that cost — a second capture implementation, or a thin WASAPI path with cpal retained for enumeration?
Are there maintained Rust crates for this — WASAPI bindings exposing the Communications category or the Voice Capture DSP, or a portable AEC (webrtc-audio-processing bindings, speexdsp)? Report maintenance state and licence, not just existence.
What does a software AEC need from us if the platform path is unavailable — a reference stream of what is being played, sample-clock alignment, latency bounds? Denpa plays TTS through Web Audio in the renderer (ADR-0002 calls playback "the open edge"), so the reference signal may not be reachable from Rust without moving playback too. Say plainly whether software AEC forces playback into Rust.

#43 — the boundary decision this serves
#35 — the capture work it unblocks on Windows
#44 — the research that established the gap
denpa/docs/adr/0002-hardware-goes-through-rust.md — playback is explicitly the open edge

## Question `#43` decided that barge-in — talking over her while she speaks — is supported and **togglable**. That requires acoustic echo cancellation, or her own output feeds straight back into the voice-activity detector. **Android is answered:** Kotlin `AudioRecord` with `MediaRecorder.AudioSource.VOICE_COMMUNICATION`, which gives the platform AEC. **Windows is not.** `#44` established only the negative: `cpal` exposes no `input_preset`, so it cannot ask for an echo-cancelled stream ([PR #995](https://github.com/RustAudio/cpal/pull/995), open since 2025-07-31). What Windows offers instead was never investigated. Answer from primary sources — Microsoft Learn, the Windows SDK, crate docs, upstream issue trackers. Cite everything; mark anything unsourced as unverified. 1. **What is the supported AEC path on Windows today?** Cover the Voice Capture DSP (`IMediaObject` / the media object for audio capture), the `AUDIO_EFFECT_TYPE_ACOUSTIC_ECHO_CANCELLATION` audio effects API, and Communications-category capture streams. Which of these still work on Windows 11, and which need the app to opt in at stream creation? 2. **Can any of it be reached alongside `cpal`,** or does an AEC-enabled capture stream have to be opened by WASAPI directly, bypassing `cpal` on Windows? If bypassing, what does that cost — a second capture implementation, or a thin WASAPI path with `cpal` retained for enumeration? 3. **Are there maintained Rust crates for this** — WASAPI bindings exposing the Communications category or the Voice Capture DSP, or a portable AEC (`webrtc-audio-processing` bindings, `speexdsp`)? Report maintenance state and licence, not just existence. 4. **What does a software AEC need from us** if the platform path is unavailable — a reference stream of what is being played, sample-clock alignment, latency bounds? Denpa plays TTS through Web Audio in the renderer (ADR-0002 calls playback "the open edge"), so the reference signal may not be reachable from Rust without moving playback too. Say plainly whether software AEC forces playback into Rust. ## Related - `#43` — the boundary decision this serves - `#35` — the capture work it unblocks on Windows - `#44` — the research that established the gap - `denpa/docs/adr/0002-hardware-goes-through-rust.md` — playback is explicitly the open edge

aiko added the

wayfinder:ticket

wayfinder:research

labels

2026-07-30 15:21:30 +00:00

aiko referenced this issue

2026-07-30 15:21:30 +00:00

What crosses the Rust/webview boundary for audio? #43

aiko added a new dependency

2026-07-30 15:21:30 +00:00

#35 Move mic capture and VAD into Rust, out of the webview

aiko referenced this issue

2026-07-30 15:21:31 +00:00

Move mic capture and VAD into Rust, out of the webview #35

aiko referenced this issue

2026-07-30 15:21:31 +00:00

Wayfinder map — the conversation works (v1) #41

aiko referenced this issue

2026-07-30 15:37:37 +00:00

What crosses the Rust/webview boundary for audio? #43

aiko commented

2026-07-30 15:37:37 +00:00

Author

Owner

Echo cancellation is needed on exactly the surfaces that have it natively. Operator account, 2026-07-30:

Desktop is driven on headphones. No acoustic path from her voice back into the microphone, so there is nothing to cancel. Barge-in is correct there with no AEC at all.
Tablet and phone play through their speakers. A real echo path, so AEC is genuinely required — and Android has it natively through AudioRecord's MediaRecorder.AudioSource.VOICE_COMMUNICATION, which is what #43 already chose.

So the platform that cannot echo-cancel is the one that does not need to, and the one that needs it gets it from the API already selected. That is a happier position than the research implied.

Consequence for the Windows work. #54 found the supported Windows AEC path (eCommunications role + SetClientProperties(AudioCategory_Communications) + IAudioEffectsManager / IAcousticEchoCancellationControl), and that reaching it means abandoning cpal for wasapi — cpal hardcodes eConsole and exposes no SetClientProperties. That swap is deferred, not cancelled. It becomes necessary only if the desktop is ever driven on speakers.

What this changes in the implementation: set_barge_in no longer refuses when the backend cannot echo-cancel. Refusing would have blocked the case that actually works — headphones — while the case that does not (desktop speakers) is what the echo_cancellation capability flag exists to let the UI describe. Capability informs; it does not forbid.

This is recorded because it is not derivable from the code: a future reader finds a capture path with no echo cancellation and a barge-in toggle that permits itself anyway, and the reason is a fact about hardware rather than about software.

**Echo cancellation is needed on exactly the surfaces that have it natively.** Operator account, 2026-07-30: - **Desktop is driven on headphones.** No acoustic path from her voice back into the microphone, so there is nothing to cancel. Barge-in is correct there with no AEC at all. - **Tablet and phone play through their speakers.** A real echo path, so AEC is genuinely required — and Android has it natively through `AudioRecord`'s `MediaRecorder.AudioSource.VOICE_COMMUNICATION`, which is what [#43](https://git.aiko.works/aiko/denpa/issues/43) already chose. So the platform that cannot echo-cancel is the one that does not need to, and the one that needs it gets it from the API already selected. That is a happier position than the research implied. **Consequence for the Windows work.** [#54](https://git.aiko.works/aiko/denpa/issues/54) found the supported Windows AEC path (`eCommunications` role + `SetClientProperties(AudioCategory_Communications)` + `IAudioEffectsManager` / `IAcousticEchoCancellationControl`), and that reaching it means abandoning `cpal` for `wasapi` — cpal hardcodes `eConsole` and exposes no `SetClientProperties`. **That swap is deferred, not cancelled.** It becomes necessary only if the desktop is ever driven on speakers. **What this changes in the implementation:** `set_barge_in` no longer *refuses* when the backend cannot echo-cancel. Refusing would have blocked the case that actually works — headphones — while the case that does not (desktop speakers) is what the `echo_cancellation` capability flag exists to let the UI describe. Capability informs; it does not forbid. This is recorded because it is not derivable from the code: a future reader finds a capture path with no echo cancellation and a barge-in toggle that permits itself anyway, and the reason is a fact about hardware rather than about software.

aiko referenced this issue

2026-07-30 15:37:38 +00:00

Move mic capture and VAD into Rust, out of the webview #35

aiko commented

2026-07-30 16:57:06 +00:00

Author

Owner

Resolution

Research delivered: .denpa-work/research/windows-aec.md, primary sources only, every claim carrying a URL.

Windows has a supported, non-deprecated AEC path. Capture endpoint under the eCommunications role, SetClientProperties with AudioCategory_Communications between Activate and Initialize, then IAudioEffectsManager to verify and IAcousticEchoCancellationControl::SetEchoCancellationRenderEndpoint to aim it. Microsoft's own sample and Chromium both do exactly this. IAudioEffectsManager needs build 22000, IAcousticEchoCancellationControl 22621 — both current on 24H2.

cpal cannot reach it. No SetClientProperties anywhere in its WASAPI backend, eConsole hardcoded, no public IMMDevice accessor. Reaching AEC on Windows means leaving cpal for wasapi 0.23.0 (MIT), which already ships the entire recipe including an in-tree examples/aec.rs.

Software AEC does not force playback out of the webview. WASAPI loopback supplies the reference signal on the same sample clock, and can reference the WebView2 process tree alone. ADR-0002's "playback is the open edge" survives; the trigger to move playback into Rust would be wanting a non-default output device, not echo cancellation.

Why the work it was blocking is deferred rather than done

#43 established that the desktop runs on headphones, which removes the echo path rather than cancelling it, while the tablet and phone run on speakers and get AEC natively from Android's VOICE_COMMUNICATION. So the platform that cannot echo-cancel is the one that does not need to.

The wasapi swap therefore becomes necessary only if the desktop is ever driven on speakers. Recorded on #35, which no longer blocks on this.

## Resolution Research delivered: `.denpa-work/research/windows-aec.md`, primary sources only, every claim carrying a URL. **Windows has a supported, non-deprecated AEC path.** Capture endpoint under the `eCommunications` role, `SetClientProperties` with `AudioCategory_Communications` between `Activate` and `Initialize`, then `IAudioEffectsManager` to verify and `IAcousticEchoCancellationControl::SetEchoCancellationRenderEndpoint` to aim it. Microsoft's own sample and Chromium both do exactly this. `IAudioEffectsManager` needs build 22000, `IAcousticEchoCancellationControl` 22621 — both current on 24H2. **cpal cannot reach it.** No `SetClientProperties` anywhere in its WASAPI backend, `eConsole` hardcoded, no public `IMMDevice` accessor. Reaching AEC on Windows means leaving cpal for `wasapi` 0.23.0 (MIT), which already ships the entire recipe including an in-tree `examples/aec.rs`. **Software AEC does not force playback out of the webview.** WASAPI loopback supplies the reference signal on the same sample clock, and can reference the WebView2 process tree alone. ADR-0002's "playback is the open edge" survives; the trigger to move playback into Rust would be wanting a non-default output device, not echo cancellation. ## Why the work it was blocking is deferred rather than done `#43` established that **the desktop runs on headphones**, which removes the echo path rather than cancelling it, while the tablet and phone run on speakers and get AEC natively from Android's `VOICE_COMMUNICATION`. So the platform that cannot echo-cancel is the one that does not need to. The `wasapi` swap therefore becomes necessary only if the desktop is ever driven on speakers. Recorded on `#35`, which no longer blocks on this.

aiko closed this issue

2026-07-30 16:57:08 +00:00

aiko referenced this issue from a commit

2026-07-30 17:28:45 +00:00

feat(audio): capture and detect speech in Rust, not the webview