Transcribe on the device, so you can see what is being transmitted #55

New issue

Open

opened 2026-07-30 16:25:10 +00:00 by aiko · 1 comment

aiko commented

2026-07-30 16:25:10 +00:00

Owner

Transcribe on the device, so the operator sees what was actually heard before it is acted on.

Raised 2026-07-30 as a long-term want, with a specific reason: there is currently no way to see what your voice became.

Where transcription happens today

Nowhere local. #21 stripped the go-between to TTS and Live2D — there is no src/open_llm_vtuber/asr. The client captures audio and sends it up as denpa.voice; hermes_plugin/denpa/adapter.py:432 dispatches it to Hermes as a MessageType.VOICE .wav. Hermes owns the ASR.

The transcript only returns by accident. hermes_plugin/denpa/protocol.py:175 runs a regex over her reply text looking for an echoed transcript, and promotes a match to FRAME_TRANSCRIPTION:

echo = _TRANSCRIPT_ECHO.match(body)
if echo:
    return FRAME_TRANSCRIPTION, echo.group(1)

So whether you can see what you said depends on whether she happened to repeat it back in prose. That is the same "read something load-bearing out of her prose" pattern ADR-0011 removed from pairing, surviving in a different corner. Worth recording even if on-device ASR never happens.

Two separable pieces

1. A real transcription frame (cheap, and useful on its own). The transcript travels as its own frame rather than being parsed out of her sentences. No model, no new dependency — a wire change plus somewhere in the log or the transmission band to show it. This alone answers "what is being transmitted", and it removes the prose-parsing.

2. Whisper on the device (the actual ask). Capture already lives in Rust (src-tauri/src/audio), and ort is already a dependency for silero, so an ONNX whisper has a path in-tree. Consequences worth deciding before building:

The frame changes. If the device transcribes, it sends denpa.text rather than denpa.voice, and Hermes stops doing ASR. That is a real shift in where the work happens, and it means her upstream never receives the audio — which may be a feature (privacy) or a loss (her ASR may be better, and prosody is gone).
Or both. Send text and audio, and let the transcript be a preview the operator can correct before it goes. That is the version that actually delivers "see what is being transmitted", rather than "see what was transmitted".
Size. Silero is 2.3 MB. Whisper tiny/base/small run roughly 75 MB / 150 MB / 500 MB. #44 already found the ONNX Runtime binary is 27 MB and dominates the app; whisper would dominate the runtime in turn.
Android. The tablet and phone are first-class surfaces. whisper.cpp has strong ARM support, but ort's Android prebuilt is arm64-only and needs NDK r28 (#44), and a phone transcribing in real time is a battery and thermal question nobody has measured.
Latency and streaming. Silero scores a 512-sample frame; whisper wants a whole utterance and takes real time over it. Where that time is spent — before transmission, or in parallel with it — changes how the transmission band behaves.

Suggested shape

Do piece 1 regardless; it is small, it kills a prose-parse, and it gives the visibility that prompted this. Treat piece 2 as its own decision with the model choice, the frame question, and the Android cost settled first.

#35 — capture in Rust, which is what makes local ASR reachable at all
#44 — ort on Android, arm64-only, NDK r28
#17 / #34 — the log, where a transcript would most naturally show
denpa/docs/adr/0011-devices-pair-to-the-go-between.md — the decision that removed the other prose-parse

Transcribe on the device, so the operator sees what was actually heard before it is acted on. Raised 2026-07-30 as a long-term want, with a specific reason: **there is currently no way to see what your voice became.** ## Where transcription happens today Nowhere local. `#21` stripped the go-between to TTS and Live2D — there is no `src/open_llm_vtuber/asr`. The client captures audio and sends it up as `denpa.voice`; `hermes_plugin/denpa/adapter.py:432` dispatches it to Hermes as a `MessageType.VOICE` `.wav`. **Hermes owns the ASR.** The transcript only returns by accident. `hermes_plugin/denpa/protocol.py:175` runs a regex over *her reply text* looking for an echoed transcript, and promotes a match to `FRAME_TRANSCRIPTION`: ```python echo = _TRANSCRIPT_ECHO.match(body) if echo: return FRAME_TRANSCRIPTION, echo.group(1) ``` So whether you can see what you said depends on whether she happened to repeat it back in prose. **That is the same "read something load-bearing out of her prose" pattern ADR-0011 removed from pairing**, surviving in a different corner. Worth recording even if on-device ASR never happens. ## Two separable pieces **1. A real transcription frame (cheap, and useful on its own).** The transcript travels as its own frame rather than being parsed out of her sentences. No model, no new dependency — a wire change plus somewhere in the log or the transmission band to show it. This alone answers "what is being transmitted", and it removes the prose-parsing. **2. Whisper on the device (the actual ask).** Capture already lives in Rust (`src-tauri/src/audio`), and `ort` is already a dependency for silero, so an ONNX whisper has a path in-tree. Consequences worth deciding before building: - **The frame changes.** If the device transcribes, it sends `denpa.text` rather than `denpa.voice`, and Hermes stops doing ASR. That is a real shift in where the work happens, and it means her upstream never receives the audio — which may be a feature (privacy) or a loss (her ASR may be better, and prosody is gone). - **Or both.** Send text *and* audio, and let the transcript be a preview the operator can correct before it goes. That is the version that actually delivers "see what is being transmitted", rather than "see what was transmitted". - **Size.** Silero is 2.3 MB. Whisper tiny/base/small run roughly 75 MB / 150 MB / 500 MB. `#44` already found the ONNX Runtime binary is 27 MB and dominates the app; whisper would dominate the runtime in turn. - **Android.** The tablet and phone are first-class surfaces. whisper.cpp has strong ARM support, but `ort`'s Android prebuilt is arm64-only and needs NDK r28 (`#44`), and a phone transcribing in real time is a battery and thermal question nobody has measured. - **Latency and streaming.** Silero scores a 512-sample frame; whisper wants a whole utterance and takes real time over it. Where that time is spent — before transmission, or in parallel with it — changes how the transmission band behaves. ## Suggested shape Do piece 1 regardless; it is small, it kills a prose-parse, and it gives the visibility that prompted this. Treat piece 2 as its own decision with the model choice, the frame question, and the Android cost settled first. ## Related - `#35` — capture in Rust, which is what makes local ASR reachable at all - `#44` — ort on Android, arm64-only, NDK r28 - `#17` / `#34` — the log, where a transcript would most naturally show - `denpa/docs/adr/0011-devices-pair-to-the-go-between.md` — the decision that removed the other prose-parse

aiko added the

ready-for-agent

labels

2026-07-30 16:25:10 +00:00

aiko referenced this issue

2026-07-30 16:27:01 +00:00

Give the transcript its own frame instead of parsing it out of her prose #56

aiko commented

2026-07-30 16:27:02 +00:00

Author

Owner

The cheap half is split out as #56 and is going ahead now. Operator's call, 2026-07-30: do the transcription frame regardless, and leave on-device whisper as its own decision.

This issue keeps the whisper question — the frame change (text instead of voice, or both), the model size against ONNX Runtime's already-dominant 27 MB, the Android battery and thermal cost, and whether the transcript becomes a preview the operator can correct before sending rather than a record of what already went.

**The cheap half is split out as [#56](https://git.aiko.works/aiko/denpa/issues/56) and is going ahead now.** Operator's call, 2026-07-30: do the transcription frame regardless, and leave on-device whisper as its own decision. This issue keeps the whisper question — the frame change (text instead of voice, or both), the model size against ONNX Runtime's already-dominant 27 MB, the Android battery and thermal cost, and whether the transcript becomes a preview the operator can correct *before* sending rather than a record of what already went.

aiko referenced this issue from a commit

2026-08-02 20:57:25 +00:00

fix(commands): make provenance a claim the frame carries, not one its type implies