The Receiver never reconnects: a dropped socket needs the app relaunched #57

Closed
opened 2026-07-30 16:53:15 +00:00 by aiko · 1 comment
Owner

The Receiver never reconnects. When the socket drops, nothing retries — the app sits on error.websocketNotOpen until it is relaunched.

Split from #15, which describes this as a reconnect toast. That framing understates it: there is no reconnect underneath the toast to announce.

Observed

2026-07-30. The go-between was restarted while the desktop app was running. The client's socket closed and never came back; the operator saw error.websocketNotOpen and the app was unusable until it was killed and started again. The go-between was healthy the entire time.

This happens on every go-between restart, which during development is many times an hour, and in ordinary use is every reboot of the machine she runs on.

Why it is worse than it looks

#15's reconnect toast is specified as "cyan, three seconds, no button, and it names the gap length because that is the only useful fact". A gap length can only be named by something that knows when the gap started and when it ended — which is the reconnect logic. The toast is the visible tail of a behaviour nobody built.

Note also that the adapter side already does this properly: hermes_plugins.denpa_platform.adapter logs "route dropped (no close frame received or sent); will reconnect" and backs off — 30s, 60s, 120s. The client half of the same wire has nothing.

What is needed

  • The client reconnects on its own after the socket closes, with a backoff, indefinitely. A receiver that gives up is a receiver you have to notice has given up.
  • The gap length is known, so #15's toast has something true to say when it lands.
  • A reconnect that restores the device's session rather than presenting as a new device — worth checking against the enrolment work in #40, which makes every connect present a token.
  • Reconnecting must not resend anything. A turn interrupted by a dropped socket is lost, not replayed.

Deliberately not here

  • The toast itself, and its acid variant for missed unprompted turns — those stay in #15, and the acid variant additionally waits on ollvt-hermes-bridge#22.

Acceptance

  • Restarting the go-between while the app is open results in the app working again, without being relaunched.
  • Verified by restarting the go-between and then speaking, not by a test that simulates a close event.
  • #15 — the toast, and the rest of the failure taxonomy
  • #40 — enrolment; a reconnect presents a token under ADR-0011
  • .denpa-work/denpa-stack.ps1 — currently warns the operator that a restart drops every device, because of this
The Receiver never reconnects. When the socket drops, nothing retries — the app sits on `error.websocketNotOpen` until it is relaunched. Split from `#15`, which describes this as a **reconnect toast**. That framing understates it: there is no reconnect underneath the toast to announce. ## Observed 2026-07-30. The go-between was restarted while the desktop app was running. The client's socket closed and never came back; the operator saw `error.websocketNotOpen` and the app was unusable until it was killed and started again. The go-between was healthy the entire time. This happens on **every** go-between restart, which during development is many times an hour, and in ordinary use is every reboot of the machine she runs on. ## Why it is worse than it looks `#15`'s reconnect toast is specified as *"cyan, three seconds, no button, and it names the **gap length** because that is the only useful fact"*. A gap length can only be named by something that knows when the gap started and when it ended — which is the reconnect logic. The toast is the visible tail of a behaviour nobody built. Note also that the adapter side already does this properly: `hermes_plugins.denpa_platform.adapter` logs *"route dropped (no close frame received or sent); will reconnect"* and backs off — 30s, 60s, 120s. The client half of the same wire has nothing. ## What is needed - The client reconnects on its own after the socket closes, with a backoff, indefinitely. A receiver that gives up is a receiver you have to notice has given up. - The gap length is known, so `#15`'s toast has something true to say when it lands. - A reconnect that restores the device's session rather than presenting as a new device — worth checking against the enrolment work in `#40`, which makes every connect present a token. - Reconnecting must not resend anything. A turn interrupted by a dropped socket is lost, not replayed. ## Deliberately not here - The toast itself, and its acid variant for missed unprompted turns — those stay in `#15`, and the acid variant additionally waits on `ollvt-hermes-bridge#22`. ## Acceptance - Restarting the go-between while the app is open results in the app working again, without being relaunched. - Verified by **restarting the go-between and then speaking**, not by a test that simulates a close event. ## Related - `#15` — the toast, and the rest of the failure taxonomy - `#40` — enrolment; a reconnect presents a token under ADR-0011 - `.denpa-work/denpa-stack.ps1` — currently warns the operator that a restart drops every device, because of this
Author
Owner

Shipped in 329795b (merge of d34d0aa), src/renderer/src/services/websocket-service.tsx.

What changed

  • The socket now retries on its own after any close it did not ask for, indefinitely. Backoff is the adapter's shape from run_forever — 1s doubling, capped at 30s — kept in RECONNECT_BASE_DELAY_MS / RECONNECT_MAX_DELAY_MS.
  • disconnect() detaches the old socket's handlers before closing it, so an address change or an operator-initiated close does not leave a second socket retrying. That was also a live bug before this: connect() on an open socket left the old onclose wired.
  • The gap is measured. wsService.getLastGapMs() is the time between the drop and the reconnect that closed it, null until a drop has actually been closed, so #15's toast has something true to name.
  • A reconnect resumes rather than presents as new: create-new-history is sent on a first connect only. The three fetches are idempotent reads. Nothing is replayed — a turn interrupted by a drop stays lost.

One thing the issue did not anticipate, found by running it rather than by the tests: resetting the backoff when the socket opens is not enough. A go-between that accepts the TCP connection and then refuses the device closes within milliseconds, and every retry "succeeds" — the delay never grows and the client hot-loops at one attempt per second. The reset now requires a session that held for at least the base delay. Verified live: with the reset-on-open version the go-between log showed roughly one accept/close per second; with this one it settles at one per 30s.

Tests: src/renderer/src/services/websocket-service.test.ts, 8 cases against a fake socket (backoff ladder to the 30s cap, reset after a session that held, no reset when the route flaps, gap length, no create-new-history on reconnect, one CLOSED per drop rather than one per retry, stays down after disconnect(), old socket does not retry after an address change). Gate: 761 vitest / 50 files, 18 Rust, tsc --noEmit adds nothing in the touched file.

What I could not verify, and it matters

The acceptance asks for a restart followed by speaking. I restarted the go-between for real (.denpa-work/denpa-stack.ps1 restart) with the client running, and the reconnect works — the client dials again and the backoff behaves. But no client can complete a connection to a restarted go-between at all right now: ollvt-hermes-bridge#26 landed the device gate (src/open_llm_vtuber/device_gate.py:45, called unconditionally from routes.py:44), which closes any socket whose first frame is not authenticate or enroll, and this repo sends neither — the client half is #40, still open. The observed symptom is auth-result frames the client logs as an unknown message type, then a close. So the reconnect loop is verified; "and then speaking" is blocked behind #40, not behind this.

The warning in .denpa-work/denpa-stack.ps1:107-110 ("restarting drops connected devices — relaunch the app afterwards") is now stale as far as reconnect goes, but I left it alone: .denpa-work/ is the operator's, and until #40 lands a restart really does still cost a relaunch, for a different reason.

Shipped in `329795b` (merge of `d34d0aa`), `src/renderer/src/services/websocket-service.tsx`. **What changed** - The socket now retries on its own after any close it did not ask for, indefinitely. Backoff is the adapter's shape from `run_forever` — 1s doubling, capped at 30s — kept in `RECONNECT_BASE_DELAY_MS` / `RECONNECT_MAX_DELAY_MS`. - `disconnect()` detaches the old socket's handlers before closing it, so an address change or an operator-initiated close does not leave a second socket retrying. That was also a live bug before this: `connect()` on an open socket left the old `onclose` wired. - The gap is measured. `wsService.getLastGapMs()` is the time between the drop and the reconnect that closed it, `null` until a drop has actually been closed, so #15's toast has something true to name. - A reconnect resumes rather than presents as new: `create-new-history` is sent on a first connect only. The three fetches are idempotent reads. Nothing is replayed — a turn interrupted by a drop stays lost. **One thing the issue did not anticipate**, found by running it rather than by the tests: resetting the backoff when the socket *opens* is not enough. A go-between that accepts the TCP connection and then refuses the device closes within milliseconds, and every retry "succeeds" — the delay never grows and the client hot-loops at one attempt per second. The reset now requires a session that **held** for at least the base delay. Verified live: with the reset-on-open version the go-between log showed roughly one accept/close per second; with this one it settles at one per 30s. **Tests:** `src/renderer/src/services/websocket-service.test.ts`, 8 cases against a fake socket (backoff ladder to the 30s cap, reset after a session that held, no reset when the route flaps, gap length, no `create-new-history` on reconnect, one `CLOSED` per drop rather than one per retry, stays down after `disconnect()`, old socket does not retry after an address change). Gate: **761 vitest / 50 files**, **18 Rust**, `tsc --noEmit` adds nothing in the touched file. **What I could not verify, and it matters** The acceptance asks for a restart followed by *speaking*. I restarted the go-between for real (`.denpa-work/denpa-stack.ps1 restart`) with the client running, and the reconnect works — the client dials again and the backoff behaves. But **no client can complete a connection to a restarted go-between at all right now**: `ollvt-hermes-bridge#26` landed the device gate (`src/open_llm_vtuber/device_gate.py:45`, called unconditionally from `routes.py:44`), which closes any socket whose first frame is not `authenticate` or `enroll`, and this repo sends neither — the client half is #40, still open. The observed symptom is `auth-result` frames the client logs as an unknown message type, then a close. So the reconnect loop is verified; "and then speaking" is blocked behind #40, not behind this. The warning in `.denpa-work/denpa-stack.ps1:107-110` ("restarting drops connected devices — relaunch the app afterwards") is now stale as far as reconnect goes, but I left it alone: `.denpa-work/` is the operator's, and until #40 lands a restart really does still cost a relaunch, for a different reason.
aiko closed this issue 2026-07-30 18:16:17 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
aiko/denpa#57
No description provided.