Move screen capture into Rust, out of the webview #36

New issue

Open

opened 2026-07-30 09:23:10 +00:00 by aiko · 2 comments

aiko commented

2026-07-30 09:23:10 +00:00

Owner

画面共有 captures with navigator.mediaDevices.getDisplayMedia (src/renderer/src/context/screen-capture-context.tsx:32). Per the newly recorded docs/adr/0002-hardware-goes-through-rust.md, that has to move to the Rust layer.

This is follow-up work to aiko/denpa#31, which shipped and works on desktop. Not a revert.

Why the browser path cannot stay

Android has no usable getDisplayMedia. Screen capture there is MediaProjection — a system consent dialog plus a foreground service. The design treats a phone and an Android overlay as first-class surfaces, so a capture path that only exists on desktop is not a capture path.
A webview media API is a third permission layer, stacked on top of the OS grant and our own permission screens, with its own prompts and its own persistence rules. The microphone taught this the hard way on the same day: the Android app held RECORD_AUDIO at OS level, granted through our own screen, and getUserMedia still failed with NotAllowedError because the Android WebView keeps a separate gate. Pressing 押して送信 on a real tablet advanced the UI to 送信中 and transmitted nothing.

Scope

A Rust command that captures the screen and returns image bytes, replacing getDisplayMedia in screen-capture-context.tsx.
Android: MediaProjection in Kotlin, with the consent dialog and the foreground service it requires. Coordinate with aiko/denpa#23, which already runs a foreground service for the overlay and had to reason about service types — a second one, or a second type on the same one, needs thought rather than a copy.
Keep the renderer-facing shape: #31's grant model, its two-key hold, and the screen-capture message to the go-between all stay. Only the acquisition changes.
Desktop first is acceptable, but the command signature must be the one Android can implement — do not design around what happens to be easy on Windows.

Acceptance

No getDisplayMedia call remains in the renderer.
Screen capture works on desktop through a Rust command.
Screen capture works on the Android app, with the system consent dialog appearing once and the grant honoured after.
Revoking the 画面共有 grant stops capture, not merely the UI affordance — #31's existing test for this must still hold.
No browser permission prompt appears on any platform.

docs/adr/0002-hardware-goes-through-rust.md — the rule.
aiko/denpa#35 — the same move for microphone capture and VAD.
aiko/denpa#31 — the grant surface this preserves.
aiko/denpa#23 — the Android overlay and its existing foreground service.

`画面共有` captures with `navigator.mediaDevices.getDisplayMedia` (`src/renderer/src/context/screen-capture-context.tsx:32`). Per the newly recorded `docs/adr/0002-hardware-goes-through-rust.md`, that has to move to the Rust layer. This is follow-up work to `aiko/denpa#31`, which shipped and works on desktop. Not a revert. ## Why the browser path cannot stay - **Android has no usable `getDisplayMedia`.** Screen capture there is `MediaProjection` — a system consent dialog plus a foreground service. The design treats a phone and an Android overlay as first-class surfaces, so a capture path that only exists on desktop is not a capture path. - **A webview media API is a third permission layer**, stacked on top of the OS grant and our own permission screens, with its own prompts and its own persistence rules. The microphone taught this the hard way on the same day: the Android app held `RECORD_AUDIO` at OS level, granted through our own screen, and `getUserMedia` still failed with `NotAllowedError` because the Android WebView keeps a separate gate. Pressing 押して送信 on a real tablet advanced the UI to 送信中 and transmitted nothing. ## Scope - A Rust command that captures the screen and returns image bytes, replacing `getDisplayMedia` in `screen-capture-context.tsx`. - Android: `MediaProjection` in Kotlin, with the consent dialog and the foreground service it requires. Coordinate with `aiko/denpa#23`, which already runs a foreground service for the overlay and had to reason about service types — a second one, or a second type on the same one, needs thought rather than a copy. - Keep the renderer-facing shape: `#31`'s grant model, its two-key hold, and the `screen-capture` message to the go-between all stay. Only the acquisition changes. - Desktop first is acceptable, but the command signature must be the one Android can implement — do not design around what happens to be easy on Windows. ## Acceptance - No `getDisplayMedia` call remains in the renderer. - Screen capture works on desktop through a Rust command. - Screen capture works on the Android app, with the system consent dialog appearing once and the grant honoured after. - Revoking the `画面共有` grant stops capture, not merely the UI affordance — `#31`'s existing test for this must still hold. - No browser permission prompt appears on any platform. ## Related - `docs/adr/0002-hardware-goes-through-rust.md` — the rule. - `aiko/denpa#35` — the same move for microphone capture and VAD. - `aiko/denpa#31` — the grant surface this preserves. - `aiko/denpa#23` — the Android overlay and its existing foreground service.

aiko added the

ready-for-agent

label

2026-07-30 09:23:10 +00:00

aiko commented

2026-07-30 17:02:25 +00:00

Author

Owner

Observed working on Windows, 2026-07-30 — 画面共有 captures and sends today, through the getDisplayMedia path in screen-capture-context.tsx:32.

Worth stating plainly, because it cuts both ways: it works because of the browser media API that ADR-0002 forbids. So "screen capture works" and "this issue is still needed" are both true at once, and the reasons are the ones this issue already gives — Android has no usable getDisplayMedia at all, and a webview media API is a third permission layer that #35 proved unreliable on exactly this machine.

The open question is not whether 画面共有 belongs in v1 — it is working and wanted — but whether the move into Rust is v1 or follows the Android capture path. That is #45.

**Observed working on Windows, 2026-07-30** — 画面共有 captures and sends today, through the `getDisplayMedia` path in `screen-capture-context.tsx:32`. Worth stating plainly, because it cuts both ways: it works **because of** the browser media API that ADR-0002 forbids. So "screen capture works" and "this issue is still needed" are both true at once, and the reasons are the ones this issue already gives — Android has no usable `getDisplayMedia` at all, and a webview media API is a third permission layer that `#35` proved unreliable on exactly this machine. The open question is not whether 画面共有 belongs in v1 — it is working and wanted — but whether the **move into Rust** is v1 or follows the Android capture path. That is [#45](https://git.aiko.works/aiko/denpa/issues/45).

aiko added the

label

2026-07-30 17:04:53 +00:00

aiko commented

2026-07-30 17:04:53 +00:00

Author

Owner

Deferred to v2 by #45, operator's call 2026-07-30.

画面共有 itself stays in v1 — it works on desktop and is wanted. What waits is this migration, which travels with the Android capture path rather than ahead of it.

Carried forward as accepted cost: the renderer keeps a getDisplayMedia handle, which ADR-0002 forbids, and screen sharing does not exist on the tablet or the phone until this lands.

**Deferred to v2** by [#45](https://git.aiko.works/aiko/denpa/issues/45), operator's call 2026-07-30. 画面共有 itself stays in v1 — it works on desktop and is wanted. What waits is this migration, which travels with the Android capture path rather than ahead of it. Carried forward as accepted cost: the renderer keeps a `getDisplayMedia` handle, which ADR-0002 forbids, and screen sharing does not exist on the tablet or the phone until this lands.

aiko referenced this issue from a commit

2026-07-30 17:28:45 +00:00

docs(adr): record that hardware goes through Rust