wiki(audit): 2026-10-29 window — Conduit voice fault documented and repaired; radar two-stage split resolved both incidents (20 pages, 2 letters, Notion 455->475); cron 176/0

This commit is contained in:
Tony0410
2026-10-30 00:21:21 +08:00
parent cc95dcc54d
commit 1d335ca8e6
10 changed files with 404 additions and 82 deletions
@@ -1,11 +1,11 @@
---
title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
type: incident
status: active
status: resolved
created: 2026-10-28
updated: 2026-10-29
verified_on: 2026-10-29
review_after: 2026-11-04
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-13
confidence: high
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
@@ -101,6 +101,26 @@ The **first real two-stage cycle ran on 2026-10-28** and answered the open quest
- **Open question 4 is answered: the row did not close.** `dca848_e93a538ade6b` stayed open because the 10-28
slot was not clean, and a second row (`dca848_0a2f03b5952b`) opened — open incident rows **2 → 4**.
## Follow-up — the 2026-10-29 slot: the split held (recorded 2026-10-30)
**Resolved.** Stage 1 (`dca8482e4f76`) ran **2026-10-29 07:00:59 → 10:16:12** — 3 h 15 m 13 s — and
closed **`completed`**. It wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` (24.8 KB):
`run_date 2026-10-29T08:23:51+08:00`, **170 discovered / 33 duplicates avoided / 37 verified**,
sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Both ledger rows this incident opened —
`dca848_e93a538ade6b` and `dca848_0a2f03b5952b` — closed **`resolved` at 10:16:12**, and the run
opened no new row. Open cron incident rows went **4 → 1**; the survivor is the Facebook monthly's
`0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot.
Stage 2 (`5fc292ff0f3b`) fired at **10:13:46**, consumed the handoff at **10:40:59**, finished
**10:44:48 `completed`** and **delivered** to topic 1030 in the same second (delivery `delivered`).
Its `stage2_result`: **20 pages written / 20 `page_ids` / 2 cover letters / 2 same-day strong matches
/ `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went
**455 → 475**. The later slots no-oped as designed (11:30 → 10 m 46 s, 14:30 → 1 m 51 s, already-consumed
handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns the report.
The questions this page left open are answered for this run: the handoff **was** written inside budget and
consumed, and **no cover letter went missing on the first real run of the split**.
## Related
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
@@ -1,11 +1,11 @@
---
title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
type: incident
status: active
status: resolved
created: 2026-10-29
updated: 2026-10-29
verified_on: 2026-10-29
review_after: 2026-11-05
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-13
confidence: high
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
@@ -76,6 +76,21 @@ slot was not clean. Open `cron_incidents` rows went **2 → 4**.
4. Both radar incident rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) close only when a **clean** stage-1
slot runs — watch the next 07:00 weekday slot.
## Follow-up — the 2026-10-29 slot: the cap was not hit (recorded 2026-10-30)
**Resolved for this run.** The next stage-1 run (`dca8482e4f76`, **2026-10-29 07:00:59 → 10:16:12**,
3 h 15 m 13 s) closed **`completed`** *and* wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — the
first run of the split to do both. The handoff records **170 discovered / 33 duplicates avoided /
37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`,
`indeed blocked`, `wa_gov ok`. Stage 2 consumed it at **10:40:59**, delivered at **10:44:48** with
**20 Notion pages and 2 cover letters** (`missing_cover_letter_count 0`; board 455 → 475). Its two
ledger rows closed `resolved` at 10:16:12.
⚠ **Flaky, not solved.** A 3 h 15 m stage-1 run is still long enough for the cap to be reachable, and
the previous two slots in the split both died on it (2026-10-27 on the tool-call budget, 2026-10-28 on
the iteration cap). Treat a `completed` run with **no handoff file** as the same fault, whatever the
execution status says → [[incidents/2026-10-27-job-radar-budget-exhausted]].
## Related
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
@@ -0,0 +1,106 @@
---
title: Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers
type: incident
status: resolved
created: 2026-10-30
updated: 2026-10-30
verified_on: 2026-10-30
confidence: high
tags: [conduit, voice, tts, stt, config, session-ttl, incident]
sources:
- live-probe-2026-10-30
- live-config-2026-10-30
- live-log-2026-10-30
- session-20260908_031541_49c051
- plugin-tree-conduit_push
---
# Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers
**Window:** 2026-10-29 07:54 – 09:0x AWST · **Fixed:** 2026-10-29 08:02–08:4x AWST (dashboard restarted 08:03:26)
**Symptom:** Anthony asked *"why do you never answer when I call you on Conduit"* — every Conduit voice call connected and then died silently. No Mac connection, no Hermes turn, no error surfaced on the phone.
Three independent faults sat in series. Fixing any one alone would not have made a call work; each one masked the next.
## Evidence trail (why it was invisible)
- The Mac Mini speech server recorded **zero connections all day.** `last_request_at` on the plugin was `null` — meaning no call had *ever* reached the Hermes turn logic.
- `conduit-calls.json → history` was empty: no successful call had ever been recorded.
- The only traces were in the plugin log: a wrong-looking reason string per attempt, which is what made each fault look like the *next* fault.
## Fault A — Truncated speech-server URL (403)
`voice.speech_server.url` was set to the bare host path:
```
wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/ ⛔ (truncated)
wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime ✅
```
`plugin_api.py` (~line 2646) dials the configured value **verbatim** — it does not append `/realtime`. The bare `/v1/` returns HTTP **403**, the socket closes, and the plugin reports it as *"the speech server refused the key."* The key was fine; the path was wrong.
**Fix:** set `voice.speech_server.url` to `…:8443/v1/realtime`. Backup: `~/.hermes/config.yaml.bak.url-081525` (08:15:25).
## Fault B — Dashboard session TTL of 12 hours (WS 4401)
`dashboard.basic_auth.session_ttl_seconds` held the **12-hour default (43 200 s)**. The dashboard issues the phone its session, and when that session lapses the Conduit call socket answers **WS 4401 "Dashboard sign-in required"** — with no Hermes turn and no user-visible error.
Audit-log corroboration: the iPhone **re-authenticated at 08:00:59**, *minutes after* the silent call — i.e. it was walking around with an expired session and only logging back in when the dashboard happened to prompt.
**Fix:** raised to **30 days (2 592 000 s)** — matched to the refresh-token life so the phone's session no longer expires under it. Dashboard restarted **2026-10-29 08:03:26** (pid 4042003). Backup: `~/.hermes/config.yaml.bak.ttl-20261029080258`.
> ⚠ Two keys exist and only one was changed: `dashboard.basic_auth.session_ttl_seconds` = **2592000** (raised) and `dashboard.session_ttl_seconds` = **43200** (still 12 h, different key, different consumer). Check *both* when diagnosing a 4401 again.
## Fault C — Host STT and TTS providers both dead
Both halves of the voice loop were broken on the host, so even a correctly-signed call would have had no ears and no mouth.
| Setting | Was | Failed with | Now | Verified |
|---|---|---|---|---|
| `tts.provider` | `openai` | Mac `/v1/speech` → **502** | `ollama_local` | 6.94 s of real 64 kbps speech produced |
| `stt.provider` | `groq` | **403 / Cloudflare error 1010** | `local` | Hermes' own transcription path returned `success: True, provider: local` |
- Groq was **blocked, not mis-keyed** — it returns 403/1010 identically with a bogus key, so the failure was Cloudflare bot-blocking, not credentials.
- `tts.provider` had been flipped to `openai` by the **Conduit settings tool** earlier (that tool writes Hermes config). The app holds no `tts.provider` setting, so this fix will not be silently overwritten.
- `ollama_local` runs `~/.hermes/scripts/ollama-tts.sh` (model `phonon`); `local` STT runs `faster_whisper` (`base`).
- Backup: `~/.hermes/config.yaml.bak.stt-084536` (08:45:36); the TTS flip landed between the 08:15 and 08:45 backups.
## Verification (end-to-end, not just config)
Rather than trusting three config edits, the whole call path was driven for real: minted a valid HMAC dashboard session → got a WS ticket → let the plugin dial the Mac. Result:
- auth gate passed → `session.created` / `session.updated`
- speech in transcribed correctly ("Hello Conduit, can you hear me right now?")
- turn committed → audio streamed back → `response.done`
- **Mac counters 0 → 10 connections, 1 completed response, 2.98 s audio, 0 errors**
- plugin `connected: true`, handshake **41 ms**
- the iPhone minted its own WS tickets at **08:07** and **08:19** — i.e. against the new 30-day session
## App update to 2.2.0 changed the settings schema
The iPhone app self-updated to **2.2.0** (from 1.10.1) mid-window, and its settings schema changed with it: the phone now exposes **19 keys, none of them `voice.local.*`**. There is **no phone-side Local Live server address**, so the Local Live path cannot be configured from the host side at all — a prior assumption that it could is now obsolete.
Consequence: `voice.mode` was moved **`local` → `classic`** (request queued 08:47, approved in the app; `conduit-settings.json` now reads `classic`). Classic needs no phone-side server config because the plugin relays the whole call — which is precisely why Fault C (both host providers dead) had to be fixed for it to work at all.
## Call record
`conduit-calls.json → history` now holds **2 entries — the first real calls ever recorded**:
| Time (AWST) | Outcome |
|---|---|
| 2026-10-29 08:38:04 | No Mac connection, no Hermes turn. iPhone had to re-login at 08:39:05, a minute later. |
| 2026-10-29 08:49:36 | Placed *after* the Classic switch and the STT/TTS fixes. Session ended 08:49:35 — Anthony's verdict is unrecorded. |
## Open items
- The 08:49:36 Classic call has **no recorded outcome**. Nothing in the window proves a phone-originated call ran a real Hermes turn.
- `last_request_at` was still `null` at the end of the window: the call→Hermes-turn bridge remains **unproven** even though the transport is now healthy. Classic mode is the recommended path.
- If an app update or the settings sync ever writes the bare `/v1/` back into `voice.speech_server.url`, every call silently 403s again. Re-check it after any Conduit update.
- Watch for the Conduit settings tool re-flipping `tts.provider` should a `tts.provider` key ever appear in the app's schema.
## Related
- [Conduit](../systems/conduit.md) — service page (voice pipeline, call path, pairing)
- [Hermes Agent](../systems/hermes-agent.md) — TTS/STT provider block
- [Conduit call request refused (2026-10-24)](2026-10-24-conduit-call-request-refused.md) — the earlier, behavioural half of the "calls don't arrive" problem
- [Conduit memory-start timeout (2026-10-22)](2026-10-22-conduit-memory-start-timeout.md)