aiko/denpa
1
0
Fork 0
Code Issues 20 Pull requests Projects Releases Packages Wiki Activity Actions

画面共有 — a grant, not a panel #31

New issue
Closed
opened 2026-07-30 00:03:50 +00:00 by aiko · 3 comments
aiko commented 2026-07-30 00:03:50 +00:00
Owner
Copy link

What to build

画面共有, a grant rather than a panel. Replaces #21, and it is no longer blocked.

It was cut because HermesAgent never read input_data.images and /v1/runs could not accept them. Under ADR-0001 we are in-process: a captured frame is a MessageType.PHOTO MessageEvent, Hermes routes it through its own vision pipeline, and there is nothing to wait for upstream.

Per the design: an acid indicator that never hides while sharing (blinking dot, 見えている, which display, inline やめる), a display and window picker, and every frame she takes landing in the log with a thumbnail - so "when did she look" is answerable after the fact rather than a matter of trust.

Acceptance criteria

  • A captured frame reaches the agent and she can describe it
  • Sharing cannot be active without a visible indicator
  • Every frame taken appears in the log with a thumbnail and a timestamp
  • Revoking is one tap from the indicator itself
  • Nothing is captured before the grant, and capture stops the instant it is revoked

Reference

ADR-0001. Design: 5d. The screen-capture path was deliberately kept in the tree for exactly this.

## What to build 画面共有, a grant rather than a panel. Replaces #21, and it is **no longer blocked**. It was cut because `HermesAgent` never read `input_data.images` and `/v1/runs` could not accept them. Under ADR-0001 we are in-process: a captured frame is a `MessageType.PHOTO` MessageEvent, Hermes routes it through its own vision pipeline, and there is nothing to wait for upstream. Per the design: an acid indicator that never hides while sharing (blinking dot, 見えている, which display, inline やめる), a display and window picker, and every frame she takes landing in the log with a thumbnail - so "when did she look" is answerable after the fact rather than a matter of trust. ## Acceptance criteria - [ ] A captured frame reaches the agent and she can describe it - [ ] Sharing cannot be active without a visible indicator - [ ] Every frame taken appears in the log with a thumbnail and a timestamp - [ ] Revoking is one tap from the indicator itself - [ ] Nothing is captured before the grant, and capture stops the instant it is revoked ## Reference ADR-0001. Design: 5d. The screen-capture path was deliberately kept in the tree for exactly this.
aiko commented 2026-07-30 08:31:08 +00:00
Author
Owner
Copy link

Branch: issue/denpa-31-screen-share-grant (denpa-client).

What landed. 画面共有 as a grant under client ADR-0001, not a panel. 見せる / やめて, with 交信中はずっと and これから毎回 one level down behind a press-and-hold on 見せる and a corner notch as the tell. The reducer in components/receiver/screen-share.ts is what refuses a standing scope that no hold opened — the timer only opens the reveal, so a caller wiring the keys differently still cannot widen the grant.

The grant is the only thing that decides whether a capture stream exists. Nothing is held before it; やめる on the indicator stops capture rather than hiding a button; 今回だけ expires on the frame it paid for; and leaving window mode revokes, because the receiver chrome is the only surface that draws the acid indicator and a grant in force behind a surface that cannot show it is exactly criterion 2's failure.

A frame goes up as {"type": "screen-capture", "image_b64", "mime"} — bytes inline, no device_id, no callback id. Both refusals the go-between sends (That capture is too large to send; nothing was sent. and No adapter is attached, so the captured screen was not delivered.) are matched verbatim and shown on the indicator, not only in a toast. Every frame lands in 受信記録 as a new screen_capture row with a thumbnail, a timestamp and the grant it was taken under.

Tests. npm test: 418 passed in 34 files (main was 352 in 30). 63 new across four files. The load-bearing ones are in screen-share.test.ts: a tap only ever gives 今回だけ, twenty taps never reach a standing scope, choose('always') with no reveal open is refused, and opening the reveal grants nothing by itself. Verified red for the right reason first — removing the reveal guard fails exactly those four on assertions.

npm run typecheck: 585 error TS, byte-identical to main's count, none in any file this branch touches.

Deliberately left out. No frame-request listener: no such message exists in either direction (websocket_handler.py:64-83), so the client takes a frame when the grant lands and at the head of each turn — building a listener would be another #27. No indicator on the pal or the Android overlay; §5d draws no band for a 40px bar, so the mode switch revokes instead. これから毎回 is not persisted across a relaunch. No camera, no browser panel, no ScreenPanel resurrection.

Assumed. The ask is reached from a 画面 key in the station bar, because §5d's home for it is the 電波 sheet and that sheet is not built — a grant nobody can reach is a grant nobody can revoke. All three scopes are always offered here, since unlike an approval there is no options array on the wire to gate them. ScreenCaptureContext.startCapture now returns Promise<MediaStream | null>: it swallowed a dismissed OS picker, which would have left the indicator reading 見えている over no stream.

Not verified. No rendered output was seen and no live capture was driven end to end — the OS picker, the real getDisplayMedia stream, the thumbnail as pixels and a frame actually reaching Hermes' vision pipeline are all unexercised. vitest runs in node with no jsdom, so the effects that start and stop the stream are pinned by source assertions in screen-share-wired.test.ts rather than by running them.

Eight ledger entries appended to .denpa-work/QUESTIONS.md.

Branch: `issue/denpa-31-screen-share-grant` (denpa-client). **What landed.** 画面共有 as a grant under client ADR-0001, not a panel. `見せる` / `やめて`, with `交信中はずっと` and `これから毎回` one level down behind a press-and-hold on `見せる` and a corner notch as the tell. The reducer in `components/receiver/screen-share.ts` is what refuses a standing scope that no hold opened — the timer only opens the reveal, so a caller wiring the keys differently still cannot widen the grant. The grant is the only thing that decides whether a capture stream exists. Nothing is held before it; `やめる` on the indicator stops capture rather than hiding a button; `今回だけ` expires on the frame it paid for; and leaving window mode revokes, because the receiver chrome is the only surface that draws the acid indicator and a grant in force behind a surface that cannot show it is exactly criterion 2's failure. A frame goes up as `{"type": "screen-capture", "image_b64", "mime"}` — bytes inline, no `device_id`, no callback id. Both refusals the go-between sends (`That capture is too large to send; nothing was sent.` and `No adapter is attached, so the captured screen was not delivered.`) are matched verbatim and shown on the indicator, not only in a toast. Every frame lands in 受信記録 as a new `screen_capture` row with a thumbnail, a timestamp and the grant it was taken under. **Tests.** `npm test`: **418 passed in 34 files** (main was 352 in 30). 63 new across four files. The load-bearing ones are in `screen-share.test.ts`: a tap only ever gives `今回だけ`, twenty taps never reach a standing scope, `choose('always')` with no reveal open is refused, and opening the reveal grants nothing by itself. Verified red for the right reason first — removing the reveal guard fails exactly those four on assertions. `npm run typecheck`: 585 `error TS`, byte-identical to main's count, none in any file this branch touches. **Deliberately left out.** No frame-request listener: no such message exists in either direction (`websocket_handler.py:64-83`), so the client takes a frame when the grant lands and at the head of each turn — building a listener would be another #27. No indicator on the pal or the Android overlay; §5d draws no band for a 40px bar, so the mode switch revokes instead. `これから毎回` is not persisted across a relaunch. No camera, no browser panel, no `ScreenPanel` resurrection. **Assumed.** The ask is reached from a `画面` key in the station bar, because §5d's home for it is the 電波 sheet and that sheet is not built — a grant nobody can reach is a grant nobody can revoke. All three scopes are always offered here, since unlike an approval there is no `options` array on the wire to gate them. `ScreenCaptureContext.startCapture` now returns `Promise<MediaStream | null>`: it swallowed a dismissed OS picker, which would have left the indicator reading `見えている` over no stream. **Not verified.** No rendered output was seen and no live capture was driven end to end — the OS picker, the real `getDisplayMedia` stream, the thumbnail as pixels and a frame actually reaching Hermes' vision pipeline are all unexercised. `vitest` runs in `node` with no jsdom, so the effects that start and stop the stream are pinned by source assertions in `screen-share-wired.test.ts` rather than by running them. Eight ledger entries appended to `.denpa-work/QUESTIONS.md`.
aiko commented 2026-07-30 08:52:06 +00:00
Author
Owner
Copy link

画面共有 — branch issue/denpa-31-screen-share-grant, pushed to aiko/denpa

Branched from main at 0e91bd9 ("Merge #27's two-hop connection model"). npm test: 443 passed in 34 files (main baseline per the brief: 352 in 30). No pre-existing test deleted or skipped.

 RUN  v4.1.10 F:/Projects/denpa-repos/denpa-client
 Test Files  34 passed (34)
      Tests  443 passed (443)
   Duration  7.63s

npm run typecheck reports 585 error TS lines — the same count the brief and the ledger record for main. Grepping that output for every path this branch touches (screen-share*, log-rail, log-rows, station-bar, receiver-chrome, chat-history-context, screen-capture-context, websocket-service, use-receiver-chrome, use-screen-share) returns zero matches. No new errors.

What was built

A grant model with three scopes — 今回だけ / 交信中はずっと / ずっと — governing a real getDisplayMedia stream, plus the surfaces that make the grant visible and revocable.

New:

  • src/renderer/src/components/receiver/screen-share.ts — the reducer, the wire builders, the copy.
  • src/renderer/src/components/receiver/screen-share-grant.tsx — the two-key ask with the notch, and the acid indicator.
  • src/renderer/src/hooks/receiver/use-screen-share.ts — the grant governing the stream, the capture path, the socket binding.
  • Four test files: screen-share.test.ts (reducer and wire), screen-share-grant.test.tsx, screen-share-log.test.tsx, screen-share-wired.test.ts.

Changed: receiver-chrome.tsx (indicator + ask over the stage, share key through to the station bar), station-bar.tsx (an optional 画面 key at a 44px target), log-rail.tsx (CaptureRow — thumbnail, timestamp, grant label), log-rows.ts (a fourth row kind, capture), chat-history-context.tsx (appendScreenCaptureMessage), screen-capture-context.tsx (startCapture returns the stream or null), websocket-service.tsx (Message.type gains screen_capture; Message.image), use-receiver-chrome.ts (mounts useScreenShare; takes a frame at the head of a turn). Eight entries appended to .denpa-work/QUESTIONS.md.

What the tests cover

The reducer end to end: every scope transition, the hold gesture as the reducer can see it, self-revocation of 今回だけ on captured, revoke from every entry point, and the negative property that no sequence of taps the reducer can see produces a standing grant. The wire builders: exact key set, the two refusal strings, absence of device_id, absence of any denpa.* string in client code. The log: the capture row's shape, its filtering under 中継のみ, its exclusion from relay-chain counting, and the thumbnail cap. The ask's markup: the notch, the three keys, the keyboard routes, the ARIA. The wiring file asserts over source text — see "could not verify".

Wire claims, each with its citation

Verified against the go-between source on main, not against issue comments.

  • Up-message name and shape: {"type": "screen-capture", "image_b64": ..., "mime": ...}. Handler map entry at denpa/src/open_llm_vtuber/websocket_handler.py:77 ("screen-capture": self._handle_screen_capture), dispatched at :468-477. Payload fields read at denpa/src/open_llm_vtuber/conversations/conversation_handler.py:122 (image_b64) and :138 (mime, defaulting to image/png).
  • The client sends bytes, never a path: conversation_handler.py:106-120 — the docstring says protocol also accepts a path, but only the inlined image is relayed, because a path the receiver names would be read on the machine running her (ADR-0004). captureMessage emits exactly three keys and a test asserts Object.keys(...).sort() === ['image_b64','mime','type'].
  • No device_id is sent: the go-between resolves the device from the connection at websocket_handler.py:474 (self.device_ids.get(client_uid, client_uid)), and the ledger entry "an ask is owned by the connection, not the device id" records that _route_message no longer rebinds it from a message body (adapter_handler.py:205-213). Nothing here sends one; a test greps for it.
  • Over-size refusal, verbatim: That capture is too large to send; nothing was sent. at conversation_handler.py:135, guarded by len(image_b64) > protocol.MAX_PHOTO_B64_LENGTH at :126, constant 8 * 1024 * 1024 at denpa/hermes_plugin/denpa/protocol.py:42.
  • Undelivered refusal, verbatim: No adapter is attached, so the captured screen was not delivered. — composed by report_undelivered at conversation_handler.py:67-72 (f"No adapter is attached, so {what}."), called with "the captured screen was not delivered" at :148.
  • Both refusals arrive as {"type": "error", "message": ...} — built inline at conversation_handler.py:130-137 and :70-72. The client already routes error at websocket-handler.tsx:225; this branch additionally subscribes to wsService.onMessage, checks type === 'error', and matches only those two strings, leaving every other error to the toast.
  • No message in either direction lets her request a frame. The complete client-facing handler map is websocket_handler.py:64-83: interrupt-signal, mic-audio-data, mic-audio-end, raw-audio-data, text-input, the *-response answers, screen-capture, fetch-configs, switch-config, fetch-backgrounds, request-init-config, heartbeat. Nothing is a frame request, so the client decides when to capture rather than listening for one.
  • The client never sees denpa.*. denpa.photo is minted on the far side of the translation, in adapter.send_up(protocol.FRAME_PHOTO_IN, ...) at conversation_handler.py:139-143. A test strips comments and asserts no denpa. string appears in screen-share.ts, use-screen-share.ts or receiver-chrome.tsx.
  • Answering by choice rather than callback id — relied on only as the pattern this grant follows, not as a message this branch sends: AskKind("approval", "approval_id", "approval-request", "approval-response") at adapter_handler.py:60, with the appr:<id>:<choice> id minted go-between-side per the ledger entry at adapter_handler.py:141-215. This branch sends no approval message at all — the screen-share grant is the client's own decision, which is why all three scopes are always offered.

How the two-key hold is enforced, and tested

Nothing standing (交信中はずっと, ずっと) can be reached by a tap. The 見せる key dispatches press on pointerdown; the reveal that exposes the two standing scopes only opens after the hold timer fires. release on pointerup inside the hold window grants only 今回だけ. pointercancel dispatches a distinct cancel action (pressedAt: null, revealed: false, no grant) — a press the browser aborts (scroll takeover, palm rejection, window losing the pointer) grants nothing, and a later release cannot cash the cancelled press in. press, release, choose and activate all require state.asking, so a stray dispatch with no ask on screen cannot mint a grant or open the OS picker.

Keyboard and AT: Space/Enter keydown/keyup drive the same press/release pair (repeat ignored, so a held key reaches the reveal), ArrowDown/ArrowUp dispatch reveal, and a bare click dispatches activate → 今回だけ. A clickSpent flag swallows exactly one trailing click after release/cancel/grant, so the click the browser fires after a hold's pointerup cannot answer the options the hold just opened. aria-haspopup/aria-expanded/aria-keyshortcuts announce the hidden options.

Tested: the whole transition table, twelve reducer tests specifically on the hold and cancel paths (press+cancel, cancel after the reveal opened, a later release not cashing in a cancelled press, the trailing click being swallowed, a click with no gesture behind it granting 今回だけ), and the negative property above. The gesture timing on a real touch surface is not exercised — see below.

Revocation: captureNow mirrors the grant and the stream into refs, and after await captureAllMedia() re-checks both before sendMessage/appendScreenCaptureMessage, so a やめる landing during the await drops the frame instead of putting it on the wire. A capturing ref makes one grab at a time. A new effect revokes whenever wsState !== 'OPEN' (deliberately including CONNECTING, so a reconnect blip costs the grant rather than carrying sight of the screen into a session nobody granted it for), per ADR-0004's connection binding. Leaving window mode also revokes, routed through onScreenShareAction so the pending hold timer is cleared.

Deliberately left out

  • No frame-request listener — no such message exists in either direction; building one would be a third UI wired to nothing.
  • No clarify or slash-confirm surface.
  • No indicator on the desktop pal (1b) or the Android overlay (2a). §5d draws no share band for a 40px bar, so rather than leave a grant in force behind a surface that cannot show it, switching out of window mode revokes. Drawing a pal band is design work this issue does not cover.
  • ずっと is not persisted across a relaunch. A standing grant to see the screen that returns silently on the next launch is exactly the invisible-grant case ADR-0001 warns about.
  • No client-side pre-check of the go-between's 8 MiB ceiling — a second copy of a number the server owns can drift; the refusal is surfaced instead.
  • The 電波 settings row §5d specifies is not built, so the ask is reached from a station-bar 画面 key. It renders only when a caller passes onShareKey, so existing station-bar callers are unchanged.
  • screen-panel.tsx, use-capture-screen.ts and the bottom-tab.tsx ScreenPanel entry are untouched — cut-surfaces.test.ts pins their existence and deleting the upstream sidebar is #9's recorded follow-up.
  • Nothing toward #30.

What could not be verified

I could not see rendered output and could not drive a live capture end to end. Specifically unverified: the OS display/window picker (that is getDisplayMedia's own picker — the client draws none), a real MediaStream and its track label as the indicator's display name, the thumbnail as actual pixels, and a frame arriving at Hermes' vision pipeline.

Acceptance criterion 1 ("a captured frame reaches the agent and she can describe it") is unproven by me. Only the message shape is proven, against the go-between source.

vitest runs environment: 'node' with no jsdom and no testing-library, so the effects that start and stop the stream, the hold timer, the socket-revoke effect and the refusal subscription cannot be exercised by rendering. They are pinned by source assertions in screen-share-wired.test.ts, in the shape chrome-is-mounted.test.ts established. That is weaker than running them and I am not claiming otherwise: those assertions pass on the presence of a string, would stay green if a branch became unreachable, and break on an unrelated reformat.

The press-and-hold gesture itself (pointerdown/pointerup timing on a real touch surface) is unexercised. What is tested is the reducer's response to every event sequence.

I also did not run a review subagent — no agent-spawn tool is available in this harness. The two defects I fixed before review (the swallowed picker refusal, and startCapture's unstable identity re-prompting every render) came from my own reading.

Assumptions

  • Worktree isolation. My worktree is a checkout of the go-between (denpa), not of denpa-client where this issue lives, and the isolation guard refuses any git operation aimed at the shared denpa-client checkout. I made a local clone at <worktree>/client, branched from its main at 0e91bd9, junctioned node_modules per the brief, and pushed to https://git.aiko.works/aiko/denpa.git. The branch is on the real remote and is what should be merged; nothing of value lives only in the worktree. I removed the junction afterwards and confirmed the shared install is intact, per the #9 ledger hazard.
  • All three scopes are always offered and the notch is never suppressed. Unlike an approval there is no options array on the wire here, because there is no ask on the wire at all. ADR-0001's "suppress the notch when only once/deny are offered" belongs to #19's approval card.
  • A frame is taken when the grant lands and at the head of each turn the operator starts, since nothing lets her ask for one. A standing grant yields one frame per turn, not a feed.
  • 今回だけ means one frame: the scope self-revokes on captured, which the stream effect then acts on. 交信中 and ずっと differ today in what the log records and in surviving turn boundaries, not in surviving a relaunch.
  • A frame she was shown is a client-side-only Message of type: 'screen_capture' carrying an image data URL, in the same shape reasoning took. Not counted in a relay chain, filtered out by 中継のみ.
  • ScreenCaptureContext.startCapture returning Promise<MediaStream | null> is in scope: as Promise<void> it swallowed a dismissed OS picker, which would have left the acid indicator reading 見えている over no stream. The one other caller ignores the return.
  • The record is session-scoped. websocket-handler.tsx:174 does setMessages(message.messages) on history-data, which would wipe capture rows. I verified that history-data currently has no producer: ADR-0002 deleted the go-between's conversation store and denpa/src/open_llm_vtuber/chat_history_manager.py no longer exists (only a stale .pyc). So rather than build a parallel client-owned message list for a wipe that cannot currently happen, the comment in websocket-service.tsx states plainly that the record is session-scoped and names 記録 (#30, held) as where a durable one would come from. AC 3 holds for the session, not across a reload.
  • Main-branch baselines (352 tests in 30 files, 585 typecheck errors) are the brief's figures. My counts reconcile (30 + 4 = 34 files; 585 unchanged) but I could not run the suite on main itself from this isolation, and my test count is 443 rather than the 418 I first reported because the review fixes added 25 tests.

Review findings and how they were handled

Twelve findings, six major. Nothing was skipped outright. Two were fixed differently from the literal fix text:

  • pointercancel granted 今回だけ (major) — fixed as written: a distinct cancel action, four new reducer tests.
  • 見せる was pointer-only (major) — fixed, but I rejected the proposed onClick → choose {scope:'once'} as a bug in its own right: the browser fires click after the pointerup that ends a hold, so that wiring would have granted 今回だけ and answered the very options the hold just opened. Replaced with the guarded activate action, the clickSpent flag, and Space/Enter/Arrow handlers. Eight new reducer tests plus markup and source assertions.
  • captureNow raced a revoke (major) — fixed with the sharingNow/streamNow/capturing refs described above, plus an early return after the frame when the scope is once, because the ref only catches up on the next render.
  • Nothing watched the socket (major) — fixed with the wsState !== 'OPEN' revoke effect.
  • Full JPEG stored and rendered at 64x40 (major) — fixed: thumbnailFrom downscales to a 128px-wide q=0.6 JPEG for the stored row, and appendCapture clears image on every capture row past the newest CAPTURE_THUMBNAIL_LIMIT (40) while keeping the row, its time and its grant. captureRow now takes string | null and returns a Message, so a frame the browser cannot decode yields a row with no picture rather than no row. Seven new tests.
  • The survives-a-reload claim (major) — I took the finding's second option (correct the claim) rather than the first (a parallel client-owned list), for the reason in the assumptions above. Justification: the wipe has no producer under ADR-0002, so the list would be dead code guarding an impossible event.
  • Minors, all fixed: the mode-switch revoke now routes through onScreenShareAction so the hold timer is cleared; the refusal subscription now guards on message.type === 'error'; the 画面 key drops the misleading aria-pressed for aria-expanded + aria-label="画面を見せる" and a non-colour sharing cue; press/release/choose now require state.asking; frames is now rendered on the indicator rather than kept for its own sake.
  • The source-grep testing finding is acknowledged, not fixed. I did not add jsdom + testing-library to the vitest env — that changes the test environment for all 34 files and is not this issue's call. The wiring remains grepped rather than exercised. This is the weakest part of the branch.

Both refusal strings, the wire shape and the 8 MiB constant are cited above rather than restated from memory; if any of them has drifted since I read it, the corresponding test is what will catch it.

## 画面共有 — branch `issue/denpa-31-screen-share-grant`, pushed to `aiko/denpa` Branched from `main` at `0e91bd9` ("Merge #27's two-hop connection model"). `npm test`: **443 passed in 34 files** (main baseline per the brief: 352 in 30). No pre-existing test deleted or skipped. ``` RUN v4.1.10 F:/Projects/denpa-repos/denpa-client Test Files 34 passed (34) Tests 443 passed (443) Duration 7.63s ``` `npm run typecheck` reports 585 `error TS` lines — the same count the brief and the ledger record for `main`. Grepping that output for every path this branch touches (`screen-share*`, `log-rail`, `log-rows`, `station-bar`, `receiver-chrome`, `chat-history-context`, `screen-capture-context`, `websocket-service`, `use-receiver-chrome`, `use-screen-share`) returns zero matches. No new errors. ### What was built A grant model with three scopes — 今回だけ / 交信中はずっと / ずっと — governing a real `getDisplayMedia` stream, plus the surfaces that make the grant visible and revocable. New: - `src/renderer/src/components/receiver/screen-share.ts` — the reducer, the wire builders, the copy. - `src/renderer/src/components/receiver/screen-share-grant.tsx` — the two-key ask with the notch, and the acid indicator. - `src/renderer/src/hooks/receiver/use-screen-share.ts` — the grant governing the stream, the capture path, the socket binding. - Four test files: `screen-share.test.ts` (reducer and wire), `screen-share-grant.test.tsx`, `screen-share-log.test.tsx`, `screen-share-wired.test.ts`. Changed: `receiver-chrome.tsx` (indicator + ask over the stage, share key through to the station bar), `station-bar.tsx` (an optional 画面 key at a 44px target), `log-rail.tsx` (`CaptureRow` — thumbnail, timestamp, grant label), `log-rows.ts` (a fourth row kind, `capture`), `chat-history-context.tsx` (`appendScreenCaptureMessage`), `screen-capture-context.tsx` (`startCapture` returns the stream or `null`), `websocket-service.tsx` (`Message.type` gains `screen_capture`; `Message.image`), `use-receiver-chrome.ts` (mounts `useScreenShare`; takes a frame at the head of a turn). Eight entries appended to `.denpa-work/QUESTIONS.md`. ### What the tests cover The reducer end to end: every scope transition, the hold gesture as the reducer can see it, self-revocation of 今回だけ on `captured`, revoke from every entry point, and the negative property that **no sequence of taps the reducer can see produces a standing grant**. The wire builders: exact key set, the two refusal strings, absence of `device_id`, absence of any `denpa.*` string in client code. The log: the capture row's shape, its filtering under 中継のみ, its exclusion from relay-chain counting, and the thumbnail cap. The ask's markup: the notch, the three keys, the keyboard routes, the ARIA. The wiring file asserts over source text — see "could not verify". ### Wire claims, each with its citation Verified against the go-between source on `main`, not against issue comments. - **Up-message name and shape**: `{"type": "screen-capture", "image_b64": ..., "mime": ...}`. Handler map entry at `denpa/src/open_llm_vtuber/websocket_handler.py:77` (`"screen-capture": self._handle_screen_capture`), dispatched at `:468-477`. Payload fields read at `denpa/src/open_llm_vtuber/conversations/conversation_handler.py:122` (`image_b64`) and `:138` (`mime`, defaulting to `image/png`). - **The client sends bytes, never a path**: `conversation_handler.py:106-120` — the docstring says `protocol` also accepts a `path`, but only the inlined image is relayed, because a path the receiver names would be read on the machine running her (ADR-0004). `captureMessage` emits exactly three keys and a test asserts `Object.keys(...).sort() === ['image_b64','mime','type']`. - **No `device_id` is sent**: the go-between resolves the device from the connection at `websocket_handler.py:474` (`self.device_ids.get(client_uid, client_uid)`), and the ledger entry "an ask is owned by the connection, not the device id" records that `_route_message` no longer rebinds it from a message body (`adapter_handler.py:205-213`). Nothing here sends one; a test greps for it. - **Over-size refusal, verbatim**: `That capture is too large to send; nothing was sent.` at `conversation_handler.py:135`, guarded by `len(image_b64) > protocol.MAX_PHOTO_B64_LENGTH` at `:126`, constant `8 * 1024 * 1024` at `denpa/hermes_plugin/denpa/protocol.py:42`. - **Undelivered refusal, verbatim**: `No adapter is attached, so the captured screen was not delivered.` — composed by `report_undelivered` at `conversation_handler.py:67-72` (`f"No adapter is attached, so {what}."`), called with `"the captured screen was not delivered"` at `:148`. - **Both refusals arrive as `{"type": "error", "message": ...}`** — built inline at `conversation_handler.py:130-137` and `:70-72`. The client already routes `error` at `websocket-handler.tsx:225`; this branch additionally subscribes to `wsService.onMessage`, checks `type === 'error'`, and matches only those two strings, leaving every other error to the toast. - **No message in either direction lets her request a frame.** The complete client-facing handler map is `websocket_handler.py:64-83`: `interrupt-signal`, `mic-audio-data`, `mic-audio-end`, `raw-audio-data`, `text-input`, the `*-response` answers, `screen-capture`, `fetch-configs`, `switch-config`, `fetch-backgrounds`, `request-init-config`, `heartbeat`. Nothing is a frame request, so the client decides when to capture rather than listening for one. - **The client never sees `denpa.*`.** `denpa.photo` is minted on the far side of the translation, in `adapter.send_up(protocol.FRAME_PHOTO_IN, ...)` at `conversation_handler.py:139-143`. A test strips comments and asserts no `denpa.` string appears in `screen-share.ts`, `use-screen-share.ts` or `receiver-chrome.tsx`. - **Answering by choice rather than callback id** — relied on only as the pattern this grant follows, not as a message this branch sends: `AskKind("approval", "approval_id", "approval-request", "approval-response")` at `adapter_handler.py:60`, with the `appr:<id>:<choice>` id minted go-between-side per the ledger entry at `adapter_handler.py:141-215`. This branch sends no approval message at all — the screen-share grant is the client's own decision, which is why all three scopes are always offered. ### How the two-key hold is enforced, and tested Nothing standing (交信中はずっと, ずっと) can be reached by a tap. The 見せる key dispatches `press` on `pointerdown`; the reveal that exposes the two standing scopes only opens after the hold timer fires. `release` on `pointerup` inside the hold window grants only 今回だけ. `pointercancel` dispatches a distinct `cancel` action (`pressedAt: null`, `revealed: false`, no grant) — a press the browser aborts (scroll takeover, palm rejection, window losing the pointer) grants nothing, and a later `release` cannot cash the cancelled press in. `press`, `release`, `choose` and `activate` all require `state.asking`, so a stray dispatch with no ask on screen cannot mint a grant or open the OS picker. Keyboard and AT: Space/Enter `keydown`/`keyup` drive the same press/release pair (`repeat` ignored, so a held key reaches the reveal), ArrowDown/ArrowUp dispatch `reveal`, and a bare `click` dispatches `activate` → 今回だけ. A `clickSpent` flag swallows exactly one trailing click after release/cancel/grant, so the click the browser fires after a hold's pointerup cannot answer the options the hold just opened. `aria-haspopup`/`aria-expanded`/`aria-keyshortcuts` announce the hidden options. Tested: the whole transition table, twelve reducer tests specifically on the hold and cancel paths (press+cancel, cancel after the reveal opened, a later release not cashing in a cancelled press, the trailing click being swallowed, a click with no gesture behind it granting 今回だけ), and the negative property above. The gesture *timing* on a real touch surface is not exercised — see below. Revocation: `captureNow` mirrors the grant and the stream into refs, and after `await captureAllMedia()` re-checks both before `sendMessage`/`appendScreenCaptureMessage`, so a やめる landing during the await drops the frame instead of putting it on the wire. A `capturing` ref makes one grab at a time. A new effect revokes whenever `wsState !== 'OPEN'` (deliberately including `CONNECTING`, so a reconnect blip costs the grant rather than carrying sight of the screen into a session nobody granted it for), per ADR-0004's connection binding. Leaving window mode also revokes, routed through `onScreenShareAction` so the pending hold timer is cleared. ### Deliberately left out - **No frame-request listener** — no such message exists in either direction; building one would be a third UI wired to nothing. - **No clarify or slash-confirm surface.** - **No indicator on the desktop pal (1b) or the Android overlay (2a).** §5d draws no share band for a 40px bar, so rather than leave a grant in force behind a surface that cannot show it, switching out of window mode revokes. Drawing a pal band is design work this issue does not cover. - **`ずっと` is not persisted across a relaunch.** A standing grant to see the screen that returns silently on the next launch is exactly the invisible-grant case ADR-0001 warns about. - **No client-side pre-check of the go-between's 8 MiB ceiling** — a second copy of a number the server owns can drift; the refusal is surfaced instead. - **The 電波 settings row §5d specifies is not built**, so the ask is reached from a station-bar 画面 key. It renders only when a caller passes `onShareKey`, so existing station-bar callers are unchanged. - **`screen-panel.tsx`, `use-capture-screen.ts` and the `bottom-tab.tsx` ScreenPanel entry are untouched** — `cut-surfaces.test.ts` pins their existence and deleting the upstream sidebar is #9's recorded follow-up. - **Nothing toward #30.** ### What could not be verified I could not see rendered output and could not drive a live capture end to end. Specifically unverified: the OS display/window picker (that is `getDisplayMedia`'s own picker — the client draws none), a real `MediaStream` and its track label as the indicator's display name, the thumbnail as actual pixels, and a frame arriving at Hermes' vision pipeline. **Acceptance criterion 1 ("a captured frame reaches the agent and she can describe it") is unproven by me.** Only the message shape is proven, against the go-between source. `vitest` runs `environment: 'node'` with no jsdom and no testing-library, so the effects that start and stop the stream, the hold timer, the socket-revoke effect and the refusal subscription cannot be exercised by rendering. They are pinned by source assertions in `screen-share-wired.test.ts`, in the shape `chrome-is-mounted.test.ts` established. That is weaker than running them and I am not claiming otherwise: those assertions pass on the presence of a string, would stay green if a branch became unreachable, and break on an unrelated reformat. The press-and-hold gesture itself (pointerdown/pointerup timing on a real touch surface) is unexercised. What is tested is the reducer's response to every event sequence. I also did not run a review subagent — no agent-spawn tool is available in this harness. The two defects I fixed before review (the swallowed picker refusal, and `startCapture`'s unstable identity re-prompting every render) came from my own reading. ### Assumptions - **Worktree isolation.** My worktree is a checkout of the **go-between** (`denpa`), not of `denpa-client` where this issue lives, and the isolation guard refuses any git operation aimed at the shared `denpa-client` checkout. I made a local clone at `<worktree>/client`, branched from its `main` at `0e91bd9`, junctioned `node_modules` per the brief, and pushed to `https://git.aiko.works/aiko/denpa.git`. The branch is on the real remote and is what should be merged; nothing of value lives only in the worktree. I removed the junction afterwards and confirmed the shared install is intact, per the #9 ledger hazard. - **All three scopes are always offered and the notch is never suppressed.** Unlike an approval there is no `options` array on the wire here, because there is no ask on the wire at all. ADR-0001's "suppress the notch when only once/deny are offered" belongs to #19's approval card. - **A frame is taken when the grant lands and at the head of each turn the operator starts**, since nothing lets her ask for one. A standing grant yields one frame per turn, not a feed. - **`今回だけ` means one frame**: the scope self-revokes on `captured`, which the stream effect then acts on. `交信中` and `ずっと` differ today in what the log records and in surviving turn boundaries, not in surviving a relaunch. - **A frame she was shown is a client-side-only `Message` of `type: 'screen_capture'`** carrying an `image` data URL, in the same shape `reasoning` took. Not counted in a relay chain, filtered out by 中継のみ. - **`ScreenCaptureContext.startCapture` returning `Promise<MediaStream | null>` is in scope**: as `Promise<void>` it swallowed a dismissed OS picker, which would have left the acid indicator reading 見えている over no stream. The one other caller ignores the return. - **The record is session-scoped.** `websocket-handler.tsx:174` does `setMessages(message.messages)` on `history-data`, which would wipe capture rows. I verified that `history-data` currently has **no producer**: ADR-0002 deleted the go-between's conversation store and `denpa/src/open_llm_vtuber/chat_history_manager.py` no longer exists (only a stale `.pyc`). So rather than build a parallel client-owned message list for a wipe that cannot currently happen, the comment in `websocket-service.tsx` states plainly that the record is session-scoped and names 記録 (#30, held) as where a durable one would come from. **AC 3 holds for the session, not across a reload.** - **Main-branch baselines** (352 tests in 30 files, 585 typecheck errors) are the brief's figures. My counts reconcile (30 + 4 = 34 files; 585 unchanged) but I could not run the suite on `main` itself from this isolation, and my test count is 443 rather than the 418 I first reported because the review fixes added 25 tests. ### Review findings and how they were handled Twelve findings, six major. **Nothing was skipped outright.** Two were fixed differently from the literal fix text: - **`pointercancel` granted 今回だけ** (major) — fixed as written: a distinct `cancel` action, four new reducer tests. - **見せる was pointer-only** (major) — fixed, but I **rejected the proposed `onClick → choose {scope:'once'}`** as a bug in its own right: the browser fires `click` after the `pointerup` that ends a hold, so that wiring would have granted 今回だけ and answered the very options the hold just opened. Replaced with the guarded `activate` action, the `clickSpent` flag, and Space/Enter/Arrow handlers. Eight new reducer tests plus markup and source assertions. - **`captureNow` raced a revoke** (major) — fixed with the `sharingNow`/`streamNow`/`capturing` refs described above, plus an early return after the frame when the scope is `once`, because the ref only catches up on the next render. - **Nothing watched the socket** (major) — fixed with the `wsState !== 'OPEN'` revoke effect. - **Full JPEG stored and rendered at 64x40** (major) — fixed: `thumbnailFrom` downscales to a 128px-wide q=0.6 JPEG for the stored row, and `appendCapture` clears `image` on every capture row past the newest `CAPTURE_THUMBNAIL_LIMIT` (40) while keeping the row, its time and its grant. `captureRow` now takes `string | null` and returns a `Message`, so a frame the browser cannot decode yields a row with no picture rather than no row. Seven new tests. - **The survives-a-reload claim** (major) — I took the finding's **second** option (correct the claim) rather than the first (a parallel client-owned list), for the reason in the assumptions above. Justification: the wipe has no producer under ADR-0002, so the list would be dead code guarding an impossible event. - Minors, all fixed: the mode-switch revoke now routes through `onScreenShareAction` so the hold timer is cleared; the refusal subscription now guards on `message.type === 'error'`; the 画面 key drops the misleading `aria-pressed` for `aria-expanded` + `aria-label="画面を見せる"` and a non-colour sharing cue; `press`/`release`/`choose` now require `state.asking`; `frames` is now rendered on the indicator rather than kept for its own sake. - **The source-grep testing finding is acknowledged, not fixed.** I did not add jsdom + testing-library to the vitest env — that changes the test environment for all 34 files and is not this issue's call. The wiring remains grepped rather than exercised. This is the weakest part of the branch. Both refusal strings, the wire shape and the 8 MiB constant are cited above rather than restated from memory; if any of them has drifted since I read it, the corresponding test is what will catch it.
aiko commented 2026-07-30 13:32:56 +00:00
Author
Owner
Copy link

Shipped. Merged to main at 8237457 (phase C2).

Closed as part of reconciling the board with main: the autonomous run's policy was not to auto-close, so shipped work stayed open and the tracker read as though nothing had landed. That policy is now reversed — issues close when they land.

Shipped. Merged to `main` at [`8237457`](https://git.aiko.works/aiko/denpa/commit/8237457) (phase C2). Closed as part of reconciling the board with `main`: the autonomous run's policy was not to auto-close, so shipped work stayed open and the tracker read as though nothing had landed. That policy is now reversed — issues close when they land.
aiko closed this issue 2026-07-30 13:32:56 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/88-openwakeword-beside-silero
prototype/seam-and-pending
issue/denpa-49-onboarding-fills-viewport
issue/denpa-35-mic-capture-into-rust
issue/denpa-32-proactive-break-in
issue/denpa-28-onboarding
issue/denpa-29-settings-sheet
issue/denpa-19-approval-surface
issue/denpa-31-screen-share-grant
issue/denpa-27-two-hop-status
issue/denpa-17-log-relay-rows
issue/denpa-23-android-overlay-pal
issue/denpa-9-cut-group-camera-browser
integrate/denpa-18-pal
integrate/denpa-16-tablet
issue/denpa-12-mobile-chrome
issue/denpa-18-desktop-transparent-pal
issue/denpa-16-tablet-landscape
issue/denpa-10-design-tokens
docs/denpa-receiver-handoff
build
dev
R4-test
neo-mic
mic-test
close-button
pre-autonomous-run-2026-07-30
v1.2.1
v1.2.0
v1.1.0
v1.0.0
Labels
Clear labels   
needs-info
Waiting on reporter

  
needs-triage
Maintainer needs to evaluate

  
ready-for-agent
Fully specified, AFK-ready

  
ready-for-human
Needs human implementation

  
v2
Out of scope for v1; see the wayfinder map

  
wayfinder:grilling

   
wayfinder:map

   
wayfinder:prototype

   
wayfinder:research

   
wayfinder:task

   
wayfinder:ticket

   
wontfix
Will not be actioned

No labels
needs-info
needs-triage
ready-for-agent
ready-for-human
v2
wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wayfinder:task
wayfinder:ticket
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
aiko/denpa#31
No description provided.