wiki(audit): 2026-10-29 window — Conduit voice fault documented and repaired; radar two-stage split resolved both incidents (20 pages, 2 letters, Notion 455->475); cron 176/0
This commit is contained in:
@@ -1,11 +1,11 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
|
||||
type: incident
|
||||
status: active
|
||||
status: resolved
|
||||
created: 2026-10-28
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
review_after: 2026-11-04
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
review_after: 2026-11-13
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
|
||||
@@ -101,6 +101,26 @@ The **first real two-stage cycle ran on 2026-10-28** and answered the open quest
|
||||
- **Open question 4 is answered: the row did not close.** `dca848_e93a538ade6b` stayed open because the 10-28
|
||||
slot was not clean, and a second row (`dca848_0a2f03b5952b`) opened — open incident rows **2 → 4**.
|
||||
|
||||
## Follow-up — the 2026-10-29 slot: the split held (recorded 2026-10-30)
|
||||
|
||||
**Resolved.** Stage 1 (`dca8482e4f76`) ran **2026-10-29 07:00:59 → 10:16:12** — 3 h 15 m 13 s — and
|
||||
closed **`completed`**. It wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` (24.8 KB):
|
||||
`run_date 2026-10-29T08:23:51+08:00`, **170 discovered / 33 duplicates avoided / 37 verified**,
|
||||
sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Both ledger rows this incident opened —
|
||||
`dca848_e93a538ade6b` and `dca848_0a2f03b5952b` — closed **`resolved` at 10:16:12**, and the run
|
||||
opened no new row. Open cron incident rows went **4 → 1**; the survivor is the Facebook monthly's
|
||||
`0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot.
|
||||
|
||||
Stage 2 (`5fc292ff0f3b`) fired at **10:13:46**, consumed the handoff at **10:40:59**, finished
|
||||
**10:44:48 `completed`** and **delivered** to topic 1030 in the same second (delivery `delivered`).
|
||||
Its `stage2_result`: **20 pages written / 20 `page_ids` / 2 cover letters / 2 same-day strong matches
|
||||
/ `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went
|
||||
**455 → 475**. The later slots no-oped as designed (11:30 → 10 m 46 s, 14:30 → 1 m 51 s, already-consumed
|
||||
handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns the report.
|
||||
|
||||
The questions this page left open are answered for this run: the handoff **was** written inside budget and
|
||||
consumed, and **no cover letter went missing on the first real run of the split**.
|
||||
|
||||
## Related
|
||||
|
||||
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
|
||||
type: incident
|
||||
status: active
|
||||
status: resolved
|
||||
created: 2026-10-29
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
review_after: 2026-11-05
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
review_after: 2026-11-13
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
|
||||
@@ -76,6 +76,21 @@ slot was not clean. Open `cron_incidents` rows went **2 → 4**.
|
||||
4. Both radar incident rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) close only when a **clean** stage-1
|
||||
slot runs — watch the next 07:00 weekday slot.
|
||||
|
||||
## Follow-up — the 2026-10-29 slot: the cap was not hit (recorded 2026-10-30)
|
||||
|
||||
**Resolved for this run.** The next stage-1 run (`dca8482e4f76`, **2026-10-29 07:00:59 → 10:16:12**,
|
||||
3 h 15 m 13 s) closed **`completed`** *and* wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — the
|
||||
first run of the split to do both. The handoff records **170 discovered / 33 duplicates avoided /
|
||||
37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`,
|
||||
`indeed blocked`, `wa_gov ok`. Stage 2 consumed it at **10:40:59**, delivered at **10:44:48** with
|
||||
**20 Notion pages and 2 cover letters** (`missing_cover_letter_count 0`; board 455 → 475). Its two
|
||||
ledger rows closed `resolved` at 10:16:12.
|
||||
|
||||
⚠ **Flaky, not solved.** A 3 h 15 m stage-1 run is still long enough for the cap to be reachable, and
|
||||
the previous two slots in the split both died on it (2026-10-27 on the tool-call budget, 2026-10-28 on
|
||||
the iteration cap). Treat a `completed` run with **no handoff file** as the same fault, whatever the
|
||||
execution status says → [[incidents/2026-10-27-job-radar-budget-exhausted]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
title: Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-30
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
confidence: high
|
||||
tags: [conduit, voice, tts, stt, config, session-ttl, incident]
|
||||
sources:
|
||||
- live-probe-2026-10-30
|
||||
- live-config-2026-10-30
|
||||
- live-log-2026-10-30
|
||||
- session-20260908_031541_49c051
|
||||
- plugin-tree-conduit_push
|
||||
---
|
||||
|
||||
# Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers
|
||||
|
||||
**Window:** 2026-10-29 07:54 – 09:0x AWST · **Fixed:** 2026-10-29 08:02–08:4x AWST (dashboard restarted 08:03:26)
|
||||
**Symptom:** Anthony asked *"why do you never answer when I call you on Conduit"* — every Conduit voice call connected and then died silently. No Mac connection, no Hermes turn, no error surfaced on the phone.
|
||||
|
||||
Three independent faults sat in series. Fixing any one alone would not have made a call work; each one masked the next.
|
||||
|
||||
## Evidence trail (why it was invisible)
|
||||
|
||||
- The Mac Mini speech server recorded **zero connections all day.** `last_request_at` on the plugin was `null` — meaning no call had *ever* reached the Hermes turn logic.
|
||||
- `conduit-calls.json → history` was empty: no successful call had ever been recorded.
|
||||
- The only traces were in the plugin log: a wrong-looking reason string per attempt, which is what made each fault look like the *next* fault.
|
||||
|
||||
## Fault A — Truncated speech-server URL (403)
|
||||
|
||||
`voice.speech_server.url` was set to the bare host path:
|
||||
|
||||
```
|
||||
wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/ ⛔ (truncated)
|
||||
wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime ✅
|
||||
```
|
||||
|
||||
`plugin_api.py` (~line 2646) dials the configured value **verbatim** — it does not append `/realtime`. The bare `/v1/` returns HTTP **403**, the socket closes, and the plugin reports it as *"the speech server refused the key."* The key was fine; the path was wrong.
|
||||
|
||||
**Fix:** set `voice.speech_server.url` to `…:8443/v1/realtime`. Backup: `~/.hermes/config.yaml.bak.url-081525` (08:15:25).
|
||||
|
||||
## Fault B — Dashboard session TTL of 12 hours (WS 4401)
|
||||
|
||||
`dashboard.basic_auth.session_ttl_seconds` held the **12-hour default (43 200 s)**. The dashboard issues the phone its session, and when that session lapses the Conduit call socket answers **WS 4401 "Dashboard sign-in required"** — with no Hermes turn and no user-visible error.
|
||||
|
||||
Audit-log corroboration: the iPhone **re-authenticated at 08:00:59**, *minutes after* the silent call — i.e. it was walking around with an expired session and only logging back in when the dashboard happened to prompt.
|
||||
|
||||
**Fix:** raised to **30 days (2 592 000 s)** — matched to the refresh-token life so the phone's session no longer expires under it. Dashboard restarted **2026-10-29 08:03:26** (pid 4042003). Backup: `~/.hermes/config.yaml.bak.ttl-20261029080258`.
|
||||
|
||||
> ⚠ Two keys exist and only one was changed: `dashboard.basic_auth.session_ttl_seconds` = **2592000** (raised) and `dashboard.session_ttl_seconds` = **43200** (still 12 h, different key, different consumer). Check *both* when diagnosing a 4401 again.
|
||||
|
||||
## Fault C — Host STT and TTS providers both dead
|
||||
|
||||
Both halves of the voice loop were broken on the host, so even a correctly-signed call would have had no ears and no mouth.
|
||||
|
||||
| Setting | Was | Failed with | Now | Verified |
|
||||
|---|---|---|---|---|
|
||||
| `tts.provider` | `openai` | Mac `/v1/speech` → **502** | `ollama_local` | 6.94 s of real 64 kbps speech produced |
|
||||
| `stt.provider` | `groq` | **403 / Cloudflare error 1010** | `local` | Hermes' own transcription path returned `success: True, provider: local` |
|
||||
|
||||
- Groq was **blocked, not mis-keyed** — it returns 403/1010 identically with a bogus key, so the failure was Cloudflare bot-blocking, not credentials.
|
||||
- `tts.provider` had been flipped to `openai` by the **Conduit settings tool** earlier (that tool writes Hermes config). The app holds no `tts.provider` setting, so this fix will not be silently overwritten.
|
||||
- `ollama_local` runs `~/.hermes/scripts/ollama-tts.sh` (model `phonon`); `local` STT runs `faster_whisper` (`base`).
|
||||
- Backup: `~/.hermes/config.yaml.bak.stt-084536` (08:45:36); the TTS flip landed between the 08:15 and 08:45 backups.
|
||||
|
||||
## Verification (end-to-end, not just config)
|
||||
|
||||
Rather than trusting three config edits, the whole call path was driven for real: minted a valid HMAC dashboard session → got a WS ticket → let the plugin dial the Mac. Result:
|
||||
|
||||
- auth gate passed → `session.created` / `session.updated`
|
||||
- speech in transcribed correctly ("Hello Conduit, can you hear me right now?")
|
||||
- turn committed → audio streamed back → `response.done`
|
||||
- **Mac counters 0 → 10 connections, 1 completed response, 2.98 s audio, 0 errors**
|
||||
- plugin `connected: true`, handshake **41 ms**
|
||||
- the iPhone minted its own WS tickets at **08:07** and **08:19** — i.e. against the new 30-day session
|
||||
|
||||
## App update to 2.2.0 changed the settings schema
|
||||
|
||||
The iPhone app self-updated to **2.2.0** (from 1.10.1) mid-window, and its settings schema changed with it: the phone now exposes **19 keys, none of them `voice.local.*`**. There is **no phone-side Local Live server address**, so the Local Live path cannot be configured from the host side at all — a prior assumption that it could is now obsolete.
|
||||
|
||||
Consequence: `voice.mode` was moved **`local` → `classic`** (request queued 08:47, approved in the app; `conduit-settings.json` now reads `classic`). Classic needs no phone-side server config because the plugin relays the whole call — which is precisely why Fault C (both host providers dead) had to be fixed for it to work at all.
|
||||
|
||||
## Call record
|
||||
|
||||
`conduit-calls.json → history` now holds **2 entries — the first real calls ever recorded**:
|
||||
|
||||
| Time (AWST) | Outcome |
|
||||
|---|---|
|
||||
| 2026-10-29 08:38:04 | No Mac connection, no Hermes turn. iPhone had to re-login at 08:39:05, a minute later. |
|
||||
| 2026-10-29 08:49:36 | Placed *after* the Classic switch and the STT/TTS fixes. Session ended 08:49:35 — Anthony's verdict is unrecorded. |
|
||||
|
||||
## Open items
|
||||
|
||||
- The 08:49:36 Classic call has **no recorded outcome**. Nothing in the window proves a phone-originated call ran a real Hermes turn.
|
||||
- `last_request_at` was still `null` at the end of the window: the call→Hermes-turn bridge remains **unproven** even though the transport is now healthy. Classic mode is the recommended path.
|
||||
- If an app update or the settings sync ever writes the bare `/v1/` back into `voice.speech_server.url`, every call silently 403s again. Re-check it after any Conduit update.
|
||||
- Watch for the Conduit settings tool re-flipping `tts.provider` should a `tts.provider` key ever appear in the app's schema.
|
||||
|
||||
## Related
|
||||
|
||||
- [Conduit](../systems/conduit.md) — service page (voice pipeline, call path, pairing)
|
||||
- [Hermes Agent](../systems/hermes-agent.md) — TTS/STT provider block
|
||||
- [Conduit call request refused (2026-10-24)](2026-10-24-conduit-call-request-refused.md) — the earlier, behavioural half of the "calls don't arrive" problem
|
||||
- [Conduit memory-start timeout (2026-10-22)](2026-10-22-conduit-memory-start-timeout.md)
|
||||
Reference in New Issue
Block a user