Give the transcript its own frame instead of parsing it out of her prose #56

New issue

Closed

opened 2026-07-30 16:27:01 +00:00 by aiko · 1 comment

aiko commented

2026-07-30 16:27:01 +00:00

Owner

The transcript of what you said should travel as its own frame, instead of being recovered by pattern-matching her reply.

Split from #55 so it is not held behind the on-device-whisper decision. This is the piece that answers "what is actually being transmitted", and it needs no model and no new dependency.

What happens today

Hermes owns the ASR — #21 stripped the go-between of it, and hermes_plugin/denpa/adapter.py:432 sends voice up as a MessageType.VOICE .wav. The transcript only comes back if she echoes it in prose, where hermes_plugin/denpa/protocol.py:175 regex-matches it out:

_TRANSCRIPT_ECHO = re.compile(r'^\s*🎙️?\s*"(.*)"\s*$', re.DOTALL)
...
echo = _TRANSCRIPT_ECHO.match(body)
if echo:
    return FRAME_TRANSCRIPTION, echo.group(1)

So whether the operator can see what was heard depends on her choosing to repeat it, in that exact shape, with that exact emoji. If she paraphrases, or the platform hint's "answer first" instruction wins, the transcript never appears — and nothing reports that it did not.

This is the same failure class ADR-0011 removed from pairing: something load-bearing recovered by parsing her prose. Every authentication defect in this project came out of that pattern. It is smaller here — a missing transcript is not a security hole — but it is the same shape, and it is the last place the pattern survives.

What is needed

A first-class transcription frame carrying what the upstream ASR heard, keyed to the turn it belongs to. FRAME_TRANSCRIPTION already exists as a name; it needs a real producer rather than a regex.
Delete _TRANSCRIPT_ECHO and the prose branch in classify_outbound once the frame exists. Leaving both means two sources for one fact, and they will disagree.
Somewhere to show it. The 受信記録 log is the natural home — #17 owns the row design, and a transcript is a row about what you said rather than what she did.
If the upstream sends no transcript, that is visible as absence rather than as silence.

Acceptance

Speaking produces a transcript row without her having to echo anything.
A reply that happens to be a quoted sentence in her own voice is not mistaken for a transcript — the regex being gone is the point.
Verified by speaking and seeing the transcript, not by a test asserting the frame shape.

#55 — on-device whisper, which would change where the transcript comes from but not that it needs a frame
#17 — the log and its row design
#23 (ollvt-hermes-bridge) — the other classifier that guesses intent from her text; being deleted under ADR-0011

The transcript of what you said should travel as its own frame, instead of being recovered by pattern-matching her reply. Split from [#55](https://git.aiko.works/aiko/denpa/issues/55) so it is not held behind the on-device-whisper decision. **This is the piece that answers "what is actually being transmitted", and it needs no model and no new dependency.** ## What happens today Hermes owns the ASR — `#21` stripped the go-between of it, and `hermes_plugin/denpa/adapter.py:432` sends voice up as a `MessageType.VOICE` `.wav`. The transcript only comes back if she echoes it in prose, where `hermes_plugin/denpa/protocol.py:175` regex-matches it out: ```python _TRANSCRIPT_ECHO = re.compile(r'^\s*🎙️?\s*"(.*)"\s*$', re.DOTALL) ... echo = _TRANSCRIPT_ECHO.match(body) if echo: return FRAME_TRANSCRIPTION, echo.group(1) ``` So whether the operator can see what was heard depends on her choosing to repeat it, in that exact shape, with that exact emoji. If she paraphrases, or the platform hint's "answer first" instruction wins, the transcript never appears — and nothing reports that it did not. **This is the same failure class ADR-0011 removed from pairing:** something load-bearing recovered by parsing her prose. Every authentication defect in this project came out of that pattern. It is smaller here — a missing transcript is not a security hole — but it is the same shape, and it is the last place the pattern survives. ## What is needed - A first-class transcription frame carrying what the upstream ASR heard, keyed to the turn it belongs to. `FRAME_TRANSCRIPTION` already exists as a name; it needs a real producer rather than a regex. - Delete `_TRANSCRIPT_ECHO` and the prose branch in `classify_outbound` once the frame exists. Leaving both means two sources for one fact, and they will disagree. - Somewhere to show it. The 受信記録 log is the natural home — `#17` owns the row design, and a transcript is a row about what *you* said rather than what she did. - If the upstream sends no transcript, that is visible as absence rather than as silence. ## Acceptance - Speaking produces a transcript row without her having to echo anything. - A reply that happens to be a quoted sentence in her own voice is **not** mistaken for a transcript — the regex being gone is the point. - Verified by speaking and seeing the transcript, not by a test asserting the frame shape. ## Related - `#55` — on-device whisper, which would change where the transcript comes from but not that it needs a frame - `#17` — the log and its row design - `#23` (ollvt-hermes-bridge) — the other classifier that guesses intent from her text; being deleted under ADR-0011

aiko added the

ready-for-agent

label

2026-07-30 16:27:01 +00:00

aiko referenced this issue

2026-07-30 16:27:02 +00:00

Transcribe on the device, so you can see what is being transmitted #55

aiko referenced this issue from a commit

2026-07-30 20:42:04 +00:00

Merge #56's client half: 受信記録 draws what was heard

aiko commented

2026-07-30 20:43:15 +00:00

Author

Owner

The transcript travels as its own frame now, and the prose parse is gone.

Go-between — ollvt-hermes-bridge@16fb05d (merge of e3de73c)

_TRANSCRIPT_ECHO and its branch in classify_outbound are deleted (hermes_plugin/denpa/protocol.py). A reply of hers that happens to be one quoted sentence is spoken, which is what the regex got wrong.
The producer is the ASR call itself. _install_transcript_relay() (hermes_plugin/denpa/adapter.py:822) wraps tools.transcription_tools.transcribe_audio at register() time; the adapter recorded which device each wav belongs to when it wrote it (_dispatch_media -> remember_voice), and the path travels unchanged to the ASR (gateway/run.py:12665-12680), so the frame is keyed to the turn without guessing. FRAME_TRANSCRIPTION finally has a real producer.
Why a wrap and not a hook: there is no hook between the ASR and the prompt. pre_gateway_dispatch fires before transcription (gateway/run.py:11018) and pre_llm_call is handed the already-enriched user message (agent/turn_context.py:1054), not the transcript. hermes-agent is read-only, so the call site is the only seam. Other platforms are unaffected: a clip no adapter owns emits nothing, and the ASR's result is returned untouched.
A failed clip emits text: "" rather than nothing, so absence is a row instead of silence.
The echo had to be suppressed, or she would read it aloud. stt_echo_transcripts is gateway-wide (gateway/run.py:15813) and cannot be turned off for one platform; with the regex gone, 🎙️ "..." would classify as speech and be synthesised. send() drops it by identity against a transcript this adapter emitted (protocol.stt_echo), never by shape — that is the trap the regex was.

Client — denpa@cdc719a (merge of b936bf7)

user-input-transcription was a console.log plus an append only when text was present. It now always draws a 自分 row in 受信記録: the words when the ASR made them out, 聞き取れず (faint) when it did not. The unheard flag is client-side only.
The go-between needed no change — adapter_handler.py:482 already forwards denpa.transcription verbatim, empty text included; a test now pins that.

Gates

go-between: 236 passed (uv run python -m pytest tests/ -q), 6 new in tests/test_transcript_frame.py plus 1 in test_adapter_frames.py.
client: 55 files, 799 passed (npm test), 4 new in transcript-row.test.tsx; 23 passed for Rust; npx tsc --noEmit adds no error in the touched files.

What I did not do

The acceptance asks for verification by speaking, and I did not speak to the stack — no audio in this session. I synced hermes_plugin/denpa into %LOCALAPPDATA%\hermes\plugins\denpa (it was one commit stale) so the wrap is installed, but the gateway has to be restarted before it takes effect. Until someone speaks a turn, the frame is proven by tests and by reading gateway/run.py, not by a transcript on screen. If it does not appear, the first thing to check is that register() ran after the sync — the wrap is installed there and nothing else reports it.

Closing on the merges above; reopen if speaking a turn draws no row.

The transcript travels as its own frame now, and the prose parse is gone. **Go-between** — `ollvt-hermes-bridge@16fb05d` (merge of `e3de73c`) - `_TRANSCRIPT_ECHO` and its branch in `classify_outbound` are deleted (`hermes_plugin/denpa/protocol.py`). A reply of hers that happens to be one quoted sentence is spoken, which is what the regex got wrong. - The producer is the ASR call itself. `_install_transcript_relay()` (`hermes_plugin/denpa/adapter.py:822`) wraps `tools.transcription_tools.transcribe_audio` at `register()` time; the adapter recorded which device each wav belongs to when it wrote it (`_dispatch_media` -> `remember_voice`), and the path travels unchanged to the ASR (`gateway/run.py:12665-12680`), so the frame is keyed to the turn without guessing. `FRAME_TRANSCRIPTION` finally has a real producer. - **Why a wrap and not a hook:** there is no hook between the ASR and the prompt. `pre_gateway_dispatch` fires before transcription (`gateway/run.py:11018`) and `pre_llm_call` is handed the already-enriched user message (`agent/turn_context.py:1054`), not the transcript. `hermes-agent` is read-only, so the call site is the only seam. Other platforms are unaffected: a clip no adapter owns emits nothing, and the ASR's result is returned untouched. - **A failed clip emits `text: ""`** rather than nothing, so absence is a row instead of silence. - **The echo had to be suppressed, or she would read it aloud.** `stt_echo_transcripts` is gateway-wide (`gateway/run.py:15813`) and cannot be turned off for one platform; with the regex gone, `🎙️ "..."` would classify as speech and be synthesised. `send()` drops it *by identity* against a transcript this adapter emitted (`protocol.stt_echo`), never by shape — that is the trap the regex was. **Client** — `denpa@cdc719a` (merge of `b936bf7`) - `user-input-transcription` was a `console.log` plus an append only when text was present. It now always draws a 自分 row in 受信記録: the words when the ASR made them out, `聞き取れず` (faint) when it did not. The `unheard` flag is client-side only. - The go-between needed no change — `adapter_handler.py:482` already forwards `denpa.transcription` verbatim, empty text included; a test now pins that. **Gates** - go-between: `236 passed` (`uv run python -m pytest tests/ -q`), 6 new in `tests/test_transcript_frame.py` plus 1 in `test_adapter_frames.py`. - client: `55 files, 799 passed` (`npm test`), 4 new in `transcript-row.test.tsx`; `23 passed` for Rust; `npx tsc --noEmit` adds no error in the touched files. **What I did not do** The acceptance asks for verification by speaking, and I did not speak to the stack — no audio in this session. I synced `hermes_plugin/denpa` into `%LOCALAPPDATA%\hermes\plugins\denpa` (it was one commit stale) so the wrap is installed, but **the gateway has to be restarted before it takes effect**. Until someone speaks a turn, the frame is proven by tests and by reading `gateway/run.py`, not by a transcript on screen. If it does not appear, the first thing to check is that `register()` ran after the sync — the wrap is installed there and nothing else reports it. Closing on the merges above; reopen if speaking a turn draws no row.

aiko closed this issue

2026-07-30 20:43:22 +00:00