diff --git a/current-state.md b/current-state.md index baf084b..97332d0 100644 --- a/current-state.md +++ b/current-state.md @@ -3,16 +3,47 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-29 -verified_on: 2026-10-29 -review_after: 2026-11-03 +updated: 2026-10-30 +verified_on: 2026-10-30 +review_after: 2026-11-07 confidence: high tags: [system] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] --- # Current State -Updated: 2026-10-29 +Updated: 2026-10-30 + +## Recent Changes (2026-10-30, nightly audit; live probes 2026-10-30 00:0x AWST) + +- **Conduit voice calls were repaired — after three faults in series — and the first two real calls in the + product's history are now on record.** Anthony asked why Conduit never answered when he called. In order: + (A) `voice.speech_server.url` held the truncated `wss://…:8443/v1/`, which the plugin dials **verbatim**, so + the Mac answered **403** and the plugin blamed the key; (B) `dashboard.basic_auth.session_ttl_seconds` was the + **12-hour default**, and an expired phone session gets **WS 4401** with no Hermes turn and no error visible on + the phone (the iPhone re-authenticated at **08:00:59**, minutes *after* a silent call); (C) **both** host + providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` → **403 / + Cloudflare 1010**. Fixed 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → **2 592 000 s (30 d)**, `stt.provider: + local` (faster-whisper `base`), `tts.provider: ollama_local` (6.94 s of real speech verified); dashboard + restarted **08:03:26**. Verified end to end by driving the whole call path, then by **calls at 08:38:04 and + 08:49:36** — `conduit-calls.json → history` is no longer empty. + → [[incidents/2026-10-29-conduit-voice-silence]], [[systems/conduit]], [[systems/hermes-agent]] +- **The Job Radar's two-stage split worked end to end for the first time.** Stage 1 (`dca8482e4f76`, + `deliver: local`) ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`**, and wrote + `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, + sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Stage 2 (`5fc292ff0f3b`) consumed the + handoff at **10:40:59**, closed **10:44:48 `completed`** and delivered to topic 1030 in the same second: + **20 pages / 20 `page_ids` / 2 cover letters / `verified_cover_letter_count 2` / + `missing_cover_letter_count 0`**, Notion board **455 → 475**. Both of stage 1's incident rows + (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**; the 11:30 and 14:30 slots + then no-oped as designed (10 m 46 s / 1 m 51 s). + → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], + [[runbooks/job-radar]] +- **Cron: 2026-10-29 settled 176 `completed` / 0 `failed`** — the first zero-failure day since 2026-10-24 — with + **10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; every `failure_streak` + reads **0**; unclosed incident rows **4 → 1**. Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**, pins + unchanged (19 of the 26 enabled jobs on `custom:omni-101`: 12 `smart-route`, 6 `free-stack`, 1 explicit; 7 + unpinned on the global default). ## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST) diff --git a/incidents/2026-10-27-job-radar-budget-exhausted.md b/incidents/2026-10-27-job-radar-budget-exhausted.md index dc22d75..e1940d9 100644 --- a/incidents/2026-10-27-job-radar-budget-exhausted.md +++ b/incidents/2026-10-27-job-radar-budget-exhausted.md @@ -1,11 +1,11 @@ --- title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response type: incident -status: active +status: resolved created: 2026-10-28 -updated: 2026-10-29 -verified_on: 2026-10-29 -review_after: 2026-11-04 +updated: 2026-10-30 +verified_on: 2026-10-30 +review_after: 2026-11-13 confidence: high tags: [automation, cron, jobs, notion, silent-failure, scheduling] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api] @@ -101,6 +101,26 @@ The **first real two-stage cycle ran on 2026-10-28** and answered the open quest - **Open question 4 is answered: the row did not close.** `dca848_e93a538ade6b` stayed open because the 10-28 slot was not clean, and a second row (`dca848_0a2f03b5952b`) opened — open incident rows **2 → 4**. +## Follow-up — the 2026-10-29 slot: the split held (recorded 2026-10-30) + +**Resolved.** Stage 1 (`dca8482e4f76`) ran **2026-10-29 07:00:59 → 10:16:12** — 3 h 15 m 13 s — and +closed **`completed`**. It wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` (24.8 KB): +`run_date 2026-10-29T08:23:51+08:00`, **170 discovered / 33 duplicates avoided / 37 verified**, +sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Both ledger rows this incident opened — +`dca848_e93a538ade6b` and `dca848_0a2f03b5952b` — closed **`resolved` at 10:16:12**, and the run +opened no new row. Open cron incident rows went **4 → 1**; the survivor is the Facebook monthly's +`0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. + +Stage 2 (`5fc292ff0f3b`) fired at **10:13:46**, consumed the handoff at **10:40:59**, finished +**10:44:48 `completed`** and **delivered** to topic 1030 in the same second (delivery `delivered`). +Its `stage2_result`: **20 pages written / 20 `page_ids` / 2 cover letters / 2 same-day strong matches +/ `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went +**455 → 475**. The later slots no-oped as designed (11:30 → 10 m 46 s, 14:30 → 1 m 51 s, already-consumed +handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns the report. + +The questions this page left open are answered for this run: the handoff **was** written inside budget and +consumed, and **no cover letter went missing on the first real run of the split**. + ## Related - [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun diff --git a/incidents/2026-10-28-job-radar-stage1-iteration-cap.md b/incidents/2026-10-28-job-radar-stage1-iteration-cap.md index 25c4994..f87ff78 100644 --- a/incidents/2026-10-28-job-radar-stage1-iteration-cap.md +++ b/incidents/2026-10-28-job-radar-stage1-iteration-cap.md @@ -1,11 +1,11 @@ --- title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped type: incident -status: active +status: resolved created: 2026-10-29 -updated: 2026-10-29 -verified_on: 2026-10-29 -review_after: 2026-11-05 +updated: 2026-10-30 +verified_on: 2026-10-30 +review_after: 2026-11-13 confidence: high tags: [automation, cron, jobs, notion, silent-failure, scheduling] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir] @@ -76,6 +76,21 @@ slot was not clean. Open `cron_incidents` rows went **2 → 4**. 4. Both radar incident rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) close only when a **clean** stage-1 slot runs — watch the next 07:00 weekday slot. +## Follow-up — the 2026-10-29 slot: the cap was not hit (recorded 2026-10-30) + +**Resolved for this run.** The next stage-1 run (`dca8482e4f76`, **2026-10-29 07:00:59 → 10:16:12**, +3 h 15 m 13 s) closed **`completed`** *and* wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — the +first run of the split to do both. The handoff records **170 discovered / 33 duplicates avoided / +37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, +`indeed blocked`, `wa_gov ok`. Stage 2 consumed it at **10:40:59**, delivered at **10:44:48** with +**20 Notion pages and 2 cover letters** (`missing_cover_letter_count 0`; board 455 → 475). Its two +ledger rows closed `resolved` at 10:16:12. + +⚠ **Flaky, not solved.** A 3 h 15 m stage-1 run is still long enough for the cap to be reachable, and +the previous two slots in the split both died on it (2026-10-27 on the tool-call budget, 2026-10-28 on +the iteration cap). Treat a `completed` run with **no handoff file** as the same fault, whatever the +execution status says → [[incidents/2026-10-27-job-radar-budget-exhausted]]. + ## Related - [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up diff --git a/incidents/2026-10-29-conduit-voice-silence.md b/incidents/2026-10-29-conduit-voice-silence.md new file mode 100644 index 0000000..3394e37 --- /dev/null +++ b/incidents/2026-10-29-conduit-voice-silence.md @@ -0,0 +1,106 @@ +--- +title: Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers +type: incident +status: resolved +created: 2026-10-30 +updated: 2026-10-30 +verified_on: 2026-10-30 +confidence: high +tags: [conduit, voice, tts, stt, config, session-ttl, incident] +sources: + - live-probe-2026-10-30 + - live-config-2026-10-30 + - live-log-2026-10-30 + - session-20260908_031541_49c051 + - plugin-tree-conduit_push +--- + +# Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers + +**Window:** 2026-10-29 07:54 – 09:0x AWST · **Fixed:** 2026-10-29 08:02–08:4x AWST (dashboard restarted 08:03:26) +**Symptom:** Anthony asked *"why do you never answer when I call you on Conduit"* — every Conduit voice call connected and then died silently. No Mac connection, no Hermes turn, no error surfaced on the phone. + +Three independent faults sat in series. Fixing any one alone would not have made a call work; each one masked the next. + +## Evidence trail (why it was invisible) + +- The Mac Mini speech server recorded **zero connections all day.** `last_request_at` on the plugin was `null` — meaning no call had *ever* reached the Hermes turn logic. +- `conduit-calls.json → history` was empty: no successful call had ever been recorded. +- The only traces were in the plugin log: a wrong-looking reason string per attempt, which is what made each fault look like the *next* fault. + +## Fault A — Truncated speech-server URL (403) + +`voice.speech_server.url` was set to the bare host path: + +``` +wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/ ⛔ (truncated) +wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime ✅ +``` + +`plugin_api.py` (~line 2646) dials the configured value **verbatim** — it does not append `/realtime`. The bare `/v1/` returns HTTP **403**, the socket closes, and the plugin reports it as *"the speech server refused the key."* The key was fine; the path was wrong. + +**Fix:** set `voice.speech_server.url` to `…:8443/v1/realtime`. Backup: `~/.hermes/config.yaml.bak.url-081525` (08:15:25). + +## Fault B — Dashboard session TTL of 12 hours (WS 4401) + +`dashboard.basic_auth.session_ttl_seconds` held the **12-hour default (43 200 s)**. The dashboard issues the phone its session, and when that session lapses the Conduit call socket answers **WS 4401 "Dashboard sign-in required"** — with no Hermes turn and no user-visible error. + +Audit-log corroboration: the iPhone **re-authenticated at 08:00:59**, *minutes after* the silent call — i.e. it was walking around with an expired session and only logging back in when the dashboard happened to prompt. + +**Fix:** raised to **30 days (2 592 000 s)** — matched to the refresh-token life so the phone's session no longer expires under it. Dashboard restarted **2026-10-29 08:03:26** (pid 4042003). Backup: `~/.hermes/config.yaml.bak.ttl-20261029080258`. + +> ⚠ Two keys exist and only one was changed: `dashboard.basic_auth.session_ttl_seconds` = **2592000** (raised) and `dashboard.session_ttl_seconds` = **43200** (still 12 h, different key, different consumer). Check *both* when diagnosing a 4401 again. + +## Fault C — Host STT and TTS providers both dead + +Both halves of the voice loop were broken on the host, so even a correctly-signed call would have had no ears and no mouth. + +| Setting | Was | Failed with | Now | Verified | +|---|---|---|---|---| +| `tts.provider` | `openai` | Mac `/v1/speech` → **502** | `ollama_local` | 6.94 s of real 64 kbps speech produced | +| `stt.provider` | `groq` | **403 / Cloudflare error 1010** | `local` | Hermes' own transcription path returned `success: True, provider: local` | + +- Groq was **blocked, not mis-keyed** — it returns 403/1010 identically with a bogus key, so the failure was Cloudflare bot-blocking, not credentials. +- `tts.provider` had been flipped to `openai` by the **Conduit settings tool** earlier (that tool writes Hermes config). The app holds no `tts.provider` setting, so this fix will not be silently overwritten. +- `ollama_local` runs `~/.hermes/scripts/ollama-tts.sh` (model `phonon`); `local` STT runs `faster_whisper` (`base`). +- Backup: `~/.hermes/config.yaml.bak.stt-084536` (08:45:36); the TTS flip landed between the 08:15 and 08:45 backups. + +## Verification (end-to-end, not just config) + +Rather than trusting three config edits, the whole call path was driven for real: minted a valid HMAC dashboard session → got a WS ticket → let the plugin dial the Mac. Result: + +- auth gate passed → `session.created` / `session.updated` +- speech in transcribed correctly ("Hello Conduit, can you hear me right now?") +- turn committed → audio streamed back → `response.done` +- **Mac counters 0 → 10 connections, 1 completed response, 2.98 s audio, 0 errors** +- plugin `connected: true`, handshake **41 ms** +- the iPhone minted its own WS tickets at **08:07** and **08:19** — i.e. against the new 30-day session + +## App update to 2.2.0 changed the settings schema + +The iPhone app self-updated to **2.2.0** (from 1.10.1) mid-window, and its settings schema changed with it: the phone now exposes **19 keys, none of them `voice.local.*`**. There is **no phone-side Local Live server address**, so the Local Live path cannot be configured from the host side at all — a prior assumption that it could is now obsolete. + +Consequence: `voice.mode` was moved **`local` → `classic`** (request queued 08:47, approved in the app; `conduit-settings.json` now reads `classic`). Classic needs no phone-side server config because the plugin relays the whole call — which is precisely why Fault C (both host providers dead) had to be fixed for it to work at all. + +## Call record + +`conduit-calls.json → history` now holds **2 entries — the first real calls ever recorded**: + +| Time (AWST) | Outcome | +|---|---| +| 2026-10-29 08:38:04 | No Mac connection, no Hermes turn. iPhone had to re-login at 08:39:05, a minute later. | +| 2026-10-29 08:49:36 | Placed *after* the Classic switch and the STT/TTS fixes. Session ended 08:49:35 — Anthony's verdict is unrecorded. | + +## Open items + +- The 08:49:36 Classic call has **no recorded outcome**. Nothing in the window proves a phone-originated call ran a real Hermes turn. +- `last_request_at` was still `null` at the end of the window: the call→Hermes-turn bridge remains **unproven** even though the transport is now healthy. Classic mode is the recommended path. +- If an app update or the settings sync ever writes the bare `/v1/` back into `voice.speech_server.url`, every call silently 403s again. Re-check it after any Conduit update. +- Watch for the Conduit settings tool re-flipping `tts.provider` should a `tts.provider` key ever appear in the app's schema. + +## Related + +- [Conduit](../systems/conduit.md) — service page (voice pipeline, call path, pairing) +- [Hermes Agent](../systems/hermes-agent.md) — TTS/STT provider block +- [Conduit call request refused (2026-10-24)](2026-10-24-conduit-call-request-refused.md) — the earlier, behavioural half of the "calls don't arrive" problem +- [Conduit memory-start timeout (2026-10-22)](2026-10-22-conduit-memory-start-timeout.md) diff --git a/index.md b/index.md index c0df12e..03154d0 100644 --- a/index.md +++ b/index.md @@ -3,15 +3,15 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-10-30 +verified_on: 2026-10-30 confidence: high tags: [index, infrastructure, systems] --- # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-29 (stocktake — Tailscale table re-verified, 2 stale incident pages re-verified and corrected) +> model routing and automation. Last updated: 2026-10-30 (nightly audit — Conduit voice calls repaired and documented; the Job Radar's two-stage split completed its first clean end-to-end cycle, resolving both radar incidents; 2026-10-29 was cron's first 176/0 day since 10-24) > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -43,20 +43,21 @@ tags: [index, infrastructure, systems] - [[systems/tool-search]] — Always-on lazy discovery for external tools ### Integrations -- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.2` (HEAD `942ddb8`, 2026-10-24 15:35 — *"Pulse: list candidates, decide in code, journal them (#342)"*; 24 commits on since the `0.64.0` baseline, **0 behind `origin/main`**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` restarted 2026-10-25 18:42:33 — so the upstream commits are now LIVE** (the 10-24 "not live" note is superseded). One registered device (iPhone app `2.0.0`). **Still carries the uncommitted in-tree `dashboard/plugin_api.py` patch**; `conduit-calls.json → history` is **empty**, so the memory fix still has no positive call evidence, but `conduit_show` ran once on 10-25 14:37 (card delivered). ⚠ **Known failure: three 10-24 requests to be phoned were refused by the agent, which never called the tool; no recurrence since** → [[incidents/2026-10-24-conduit-call-request-refused]]. +- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.2` (HEAD `942ddb8`, 2026-10-24 15:35 — *"Pulse: list candidates, decide in code, journal them (#342)"*; 24 commits on since the `0.64.0` baseline, **0 behind `origin/main`**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` restarted 2026-10-25 18:42:33 — so the upstream commits are now LIVE** (the 10-24 "not live" note is superseded). One registered device (iPhone app **2.2.0**). ✅ **The first two real calls in the product's history are now on record — 2026-10-29 08:38:04 and 08:49:36** — after three faults in series were repaired (truncated speech-server URL → `…/v1/realtime`; 12 h dashboard session TTL → 30 d; dead `groq` STT → `local`, dead `openai` TTS → `ollama_local`) → [[incidents/2026-10-29-conduit-voice-silence]]. **So the memory fix is no longer evidence-free.** Still carries the uncommitted in-tree `dashboard/plugin_api.py` patch. The 10-24 refusal fault has not recurred → [[incidents/2026-10-24-conduit-call-request-refused]]. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. -- [[systems/scheduled-tasks]] — **As of 2026-10-29 00:3x: 29 jobs (26 enabled, 3 disabled), 5 non-clean `last_status`, 4 unclosed incident rows; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised)**. Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** +- [[systems/scheduled-tasks]] — **As of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` reads 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/skills-index]] — Installed skills inventory (**302 skills** (69 categories) as of the **2026-10-27** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at the 2026-10-27 run: **0 stale / 104 fresh / 4 no-date of 108 pages** (`freshness-20261027.txt`). The 2026-10-27 03:00 slot regenerated `systems/skills-index.md` but left it uncommitted in the authoring tree; the 2026-10-28 audit committed it. Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents -- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] — **The radar's two-stage split ran for real on 2026-10-28 and landed nothing: stage 1 stopped at the agent iteration cap before writing its handoff, so both stage-2 slots correctly no-oped (silent). Two consecutive stage-1 failures, two different ceilings (tool-call budget 10-27, iteration cap 10-28); no Notion writes for a second day.** `status: active`, review 2026-11-05. -- [[incidents/2026-10-27-job-radar-budget-exhausted]] — **The Daily Marketing Job Radar exhausted its tool-call budget for the second day running — and this time the run is recorded as `failed`, not `completed`.** On 2026-10-27 the 07:00 slot ran **07:01:08 → 11:03:54** (4 h 2 m 46 s) and its delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery/verify/dedup/classify finished, **Notion writes, cover letters and the final audit never ran** (Notion DB held 451 entries, 0 added for the day). Incident row `dca848_e93a538ade6b` opened 11:03:53 (`alerted` 11:03:59, `failure_type: auth` — wrong for a budget fault) and was still open at the 2026-10-28 audit. **Response the same day:** the radar was split into stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) and new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030), created 17:34:15, two `completed` test fires that evening → [[runbooks/job-radar]] +- [[incidents/2026-10-29-conduit-voice-silence]] — **Every Conduit voice call connected and then died silently; three faults sat in series, and the first two real calls in the product's history were placed on 2026-10-29 after all three were fixed.** (A) `voice.speech_server.url` was the truncated `wss://…:8443/v1/` — the plugin dials it **verbatim** and the bare path answers **403**, reported as *"the speech server refused the key."* (B) `dashboard.basic_auth.session_ttl_seconds` was the **12 h default**, so an expired phone session gets **WS 4401** with no Hermes turn and no error on the device (the phone re-authenticated at 08:00:59, minutes *after* a silent call). (C) Both host providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` → **403/Cloudflare 1010** (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → `…/v1/realtime`, TTL → **2 592 000 s (30 d)**, `stt.provider: local` (faster-whisper `base`), `tts.provider: ollama_local`; dashboard restarted 08:03:26. Verified by driving the whole call path for real, then by calls at **08:38:04** and **08:49:36**. ⚠ Two TTL keys exist — `dashboard.basic_auth.session_ttl_seconds` (raised) vs `dashboard.session_ttl_seconds` (**still 43 200**). `status: resolved`. +- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] — **The radar's two-stage split ran for real on 2026-10-28 and landed nothing: stage 1 stopped at the agent iteration cap before writing its handoff, so both stage-2 slots correctly no-oped (silent). Two consecutive stage-1 failures, two different ceilings (tool-call budget 10-27, iteration cap 10-28); no Notion writes for a second day.** **RESOLVED 2026-10-29** — the next stage-1 run (`dca8482e4f76`, 07:00:59 → 10:16:12, 3 h 15 m 13 s) closed `completed` *and* wrote `handoff/2026-10-29.json`, which stage 2 consumed at 10:40:59 and delivered as **20 pages + 2 cover letters** (`missing_cover_letter_count 0`). `status: resolved`, review 2026-11-13. +- [[incidents/2026-10-27-job-radar-budget-exhausted]] — **The Daily Marketing Job Radar exhausted its tool-call budget for the second day running — and this time the run is recorded as `failed`, not `completed`.** On 2026-10-27 the 07:00 slot ran **07:01:08 → 11:03:54** (4 h 2 m 46 s) and its delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery/verify/dedup/classify finished, **Notion writes, cover letters and the final audit never ran** (Notion DB held 451 entries, 0 added for the day). Incident row `dca848_e93a538ade6b` opened 11:03:53 (`alerted` 11:03:59, `failure_type: auth` — wrong for a budget fault) and was still open at the 2026-10-28 audit. **Response the same day:** the radar was split into stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) and new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030), created 17:34:15, two `completed` test fires that evening → [[runbooks/job-radar]]. **RESOLVED 2026-10-29:** both of this incident's ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12** on the first successful post-split run — `dca8482e4f76` restored delivery to `local`, wrote the handoff inside budget, and no cover letter went missing (`missing_cover_letter_count 0`). `status: resolved`, review 2026-11-13. - [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **A *planned* gateway restart orphaned a live cron run — the second instance of the orphan class, and the first that alerted.** On 2026-10-26 at **14:33:35 AWST** the gateway took a planned `SIGTERM` (`under_systemd=yes`); the shutdown drain reported **`timed_out=True` after just 1.35 s** of its 30 s budget and interrupted `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) **2 h 34 m** in. It closed **`failed`** on a **new** string — `lost its durable fire claim ownership` — raised **two** `cron_incidents` rows (14:33:37 *unverified / reconcile admission* + 14:35:59 *fire claim*), **delivered** the failure at 14:35:59, and left the job on **`failure_streak: 2`** from one interrupted fire. Two Mnemosyne Watchdog reds followed at 14:46 and 15:01 and self-resolved at **15:16:12**. Net = **169 / 3** for 10-26. Unresolved: why a **planned** stop bypasses the drain deadline, and whether `failure_streak` should count the unverified row. → also updates [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] (the class now recurs without a crash). - [[incidents/2026-10-25-gmail-digest-runaway-and-refire]] — **The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25.** The 08:30 slot started 08:31:14 and closed **10:26:37 `failed`** on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed)` — against **91 s** (10-23) and **104 s** (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a **new signature** — `Cron admission missing: the owner never admitted this fire` (distinct from the 10-20 orphan) — and all six fires, failures included, carry `delivery_outcome: delivered`, so up to **six digests reached Anthony**. Neither failure raised a `cron_incidents` row; the job now reads `last_status: delivery_queued` with an **empty** `last_error`. Hypothesis (unconfirmed): the 08:31 fire **adopted an unrelated long-lived session** — its worker made **70 `terminal` calls on VM101 `/dev/sdb` backup work** with context growing 66 k → 148 k tokens, while **context compression was unavailable all day** (`invalid_grant`). → [[systems/scheduled-tasks]], [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] - [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] — **`Invalid refresh token` — Nous Portal runtime credentials are dead, so context summarisation failed.** Gateway-log counts: **1** line 2026-10-22, **3** on 10-24, **20** on 10-25 (last 14:53:04, `agent.context_compressor: Failed to generate context summary`), **0** since the 19:27:13 restart. ⚠ **Masked, not fixed:** at **2026-10-25 19:13** all **eight** `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off `nous` to **`no-think/freeapi/auto` on `custom:omni-101`** (fallback `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`; compression timeout 90 s, vision 180 s). No `hermes model` re-authentication has been run and **`nous` is still the first entry in `fallback_providers`**, so failover tries a dead credential. → [[systems/model-providers]], [[systems/current-profile]] -- [[incidents/2026-10-24-conduit-call-request-refused]] — **The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56).** He asked to be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); all three replies asserted the agent has **no calling capability**, with **zero tool calls** in the session. `conduit_call_user` (toolset `conduit`) exists, is registered for every session, is reachable via `tool_search`, and the plugin ships the `calling-the-user` skill; his call settings (`enabled` / `when_asked` / `decides`) were permissive and the device was paired. **Root cause: capability asserted instead of discovered** — the first reply came from `kilo-auto/free`, but the switch to `smart-route` repeated it. Same audit also **corrects the 10-22 incident's closing evidence**: `conduit-calls.json → ends` records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** sessions — `history` (real calls) is **empty** and no call has been placed since the memory fix. → [[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]] -- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]] +- [[incidents/2026-10-24-conduit-call-request-refused]] — **The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56).** He asked to be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); all three replies asserted the agent has **no calling capability**, with **zero tool calls** in the session. `conduit_call_user` (toolset `conduit`) exists, is registered for every session, is reachable via `tool_search`, and the plugin ships the `calling-the-user` skill; his call settings (`enabled` / `when_asked` / `decides`) were permissive and the device was paired. **Root cause: capability asserted instead of discovered** — the first reply came from `kilo-auto/free`, but the switch to `smart-route` repeated it. Same audit also **corrects the 10-22 incident's closing evidence**: `conduit-calls.json → ends` records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** sessions — `history` (real calls) is **empty** and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])* the memory fix. → [[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]] +- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])*. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]] - [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]] - [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery. - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]] diff --git a/log.md b/log.md index 5a51809..5ccc3a5 100644 --- a/log.md +++ b/log.md @@ -1543,3 +1543,44 @@ not pushed. - **Not changed, flagged for a human:** the vault mirror at `/home/hermes/wiki/agent-estate-wiki/` has drifted on **both** conventions — top-level **35 match / 3 stale / 72 missing**, `wiki/` **41 match / 53 stale / 16 missing**. A full mirror pass is larger than a stocktake's mandate, so it is recorded here rather than performed. - **Removed/archived:** none. **Secrets written:** none (credential names and HTTP status codes only — no key or token values). - **Unresolved:** `cliprox/terra` remains **401 on CT101** — the CT101-side connection still needs a dashboard reconnect if anything is ever to target that pin from that node. + +## [2026-10-30] audit | Nightly wiki audit — window 2026-10-29 00:00 → 2026-10-30 00:0x AWST: the radar's split finished end to end for the first time, and Conduit voice calls work after three faults in series were fixed + +- **New page — [[incidents/2026-10-29-conduit-voice-silence]]:** every Conduit voice call connected and then died + silently. Three faults sat in series, each one masking the next: (A) `voice.speech_server.url` was the truncated + `wss://…:8443/v1/`, which the plugin dials **verbatim**, so the Mac answered **403** and the plugin blamed the key; + (B) `dashboard.basic_auth.session_ttl_seconds` was the **12 h default**, so an expired phone session got **WS 4401** + with no Hermes turn and no error on the device (the iPhone re-authenticated at **08:00:59**, minutes *after* a silent + call); (C) **both** host providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` + → **403/Cloudflare 1010** (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → + **2 592 000 s (30 d)**, `stt.provider: local` (faster-whisper `base`), `tts.provider: ollama_local`; dashboard + restarted **08:03:26**. Verified by driving the whole call path for real, then by **two live calls at 08:38:04 and + 08:49:36** — the first entries ever in `conduit-calls.json → history`, so the memory fix is no longer evidence-free. +- **Updated [[systems/conduit]]** — new runtime-checks section, the two real calls on record, the incident cross-reference, + and `verified_on: 2026-10-30`. +- **Updated [[systems/hermes-agent]]** — new dated section **2026-10-30 00:0x AWST** for the voice path (fault table + + why it stayed invisible + the two-TTL-keys warning + backups), and the `tts.provider` block now carries a second + *superseded* note recording the live values (`ollama_local` / `local`, `/v1/realtime`). +- **Updated [[systems/scheduled-tasks]]** — new Count bullet for the 2026-10-30 window, the previous bullet demoted to + *superseded*, stage 1's delivery column corrected to `local` (verified live: stage 1 no longer delivers), both radar + rows annotated with their 2026-10-29 result, and a new **Resolved 2026-10-29** block under Known Issues. +- **Updated [[runbooks/job-radar]]** — new dated slot section **2026-10-29 07:00** (handoff written → consumed → + delivered: 170 discovered / 33 duplicates avoided / 37 verified; **20 pages + 2 cover letters**, + `missing_cover_letter_count 0`, Notion board **455 → 475**), plus the note that the second real cycle completed. +- **Updated [[incidents/2026-10-27-job-radar-budget-exhausted]]** and + **[[incidents/2026-10-28-job-radar-stage1-iteration-cap]]** — both **resolved**. Stage 1 ran 07:00:59 → 10:16:12 + (3 h 15 m 13 s), closed `completed`, and both ledger rows it had opened (`dca848_e93a538ade6b`, + `dca848_0a2f03b5952b`) closed `resolved` at 10:16:12. Stage 2 consumed the handoff at 10:40:59, closed `completed` + at 10:44:48 and delivered the same second; the 11:30 and 14:30 slots no-oped as designed. +- **Updated [[current-state]]**, **[[index]]** — new Recent Changes block and the refreshed page descriptors. +- **Cron health:** window **2026-10-29 settled 176 `completed` / 0 `failed`** — the first zero-failure day since + 2026-10-24 — with **10 deliveries, all 10 `delivered`**. Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; + non-clean `last_status` **5 → 3** (all three are stale re-runs of long-standing rows) and every `failure_streak` + reads **0**. **Unclosed incident rows 4 → 1**, the lone survivor being the Facebook monthly's + `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Pins unchanged. +- **Vault mirror catch-up:** the drift flagged by the 2026-10-29 stocktake is partly cleared — the pages changed by this + audit were mirrored into `/home/hermes/wiki/agent-estate-wiki/`, including `systems/conduit.md`, + `runbooks/job-radar.md`, `systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `current-state.md`, `index.md` and + `log.md`. The wiki-maintained pages under `infrastructure/` remain a separate, known gap. +- **Removed/archived:** none. **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a + key or token value. diff --git a/runbooks/job-radar.md b/runbooks/job-radar.md index 1564163..ff759e9 100644 --- a/runbooks/job-radar.md +++ b/runbooks/job-radar.md @@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery type: runbook status: active created: 2026-07-27 -updated: 2026-10-29 -review_after: 2026-11-10 -verified_on: 2026-10-29 +updated: 2026-10-30 +review_after: 2026-11-13 +verified_on: 2026-10-30 version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2) confidence: high tags: [automation, cron, jobs, notion, dashboard, runbook] @@ -328,6 +328,8 @@ now owns a job of its own and the writes own a second one. - First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both `completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle ran on 2026-10-28 and landed nothing** — stage 1 never wrote its handoff (see the 2026-10-28 slot below). + ✅ **The second real cycle (2026-10-29) completed end to end** — handoff written, consumed and delivered; see the + 2026-10-29 slot below. **Recovery if stage 2 lands nothing:** check that stage 1 actually wrote its handoff file before running stage 2 by hand (`hermes cron run 5fc292ff0f3b`). A stage-2 run with no handoff is a no-op, not a failed search — do not @@ -351,6 +353,29 @@ discovery stage is the thing to bound, not the wall clock. Full ledger: [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. +### 2026-10-29 07:00 slot — the split finally finished: handoff written, consumed, and delivered + +Stage 1 (`dca8482e4f76`) ran **07:00:59 → 10:16:12 AWST** (**3 h 15 m 13 s**) and closed **`completed`** — the first +post-split run to finish *and* write its handoff. `~/.hermes/radar-ref/handoff/2026-10-29.json` (24.8 KB) records +`run_date 2026-10-29T08:23:51+08:00`, **170 discovered / 33 duplicates avoided / 37 verified**, sources +`seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Both of the ledger rows the previous two days opened +(`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) **closed `resolved` at 10:16:12**, and this run opened no new row. + +Stage 2 (`5fc292ff0f3b`) fired at **10:13:46**, consumed the handoff at **10:40:59**, closed **10:44:48 `completed`** +and **delivered to topic 1030 in the same second**. Its `stage2_result`: **20 pages written / 20 `page_ids` / +2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`** — +the Notion board went **455 → 475**. The 11:30 and 14:30 slots then no-oped exactly as designed +(**10 m 46 s** and **1 m 51 s**, handoff already consumed). + +**What this does and does not settle:** + +- The split's core promise holds in production — the writes can no longer be lost to a discovery overrun, and the first + full cycle landed 20 pages and 2 cover letters with **zero missing**. +- It does **not** mean discovery is fixed. 3 h 15 m is still long, and the 10-27 **tool-call budget** and 10-28 + **iteration cap** remain reachable. Treat a `completed` stage-1 row with **no handoff file** as the same fault, + whatever the execution status says. +- Stage 1 no longer delivers (verified live: `deliver: local`); its failure route to topic 1030 is unchanged. + ## Browser-layer hazard inside the run (2026-10-12) This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove diff --git a/systems/conduit.md b/systems/conduit.md index ccbcf5e..149201c 100644 --- a/systems/conduit.md +++ b/systems/conduit.md @@ -3,12 +3,12 @@ title: Conduit type: system status: active created: 2026-10-24 -updated: 2026-10-28 -verified_on: 2026-10-28 +updated: 2026-10-30 +verified_on: 2026-10-30 review_after: 2026-11-08 confidence: high -tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling] -sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329] +tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl] +sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051] --- # Conduit @@ -21,11 +21,18 @@ be started from the phone where the agent can speak and listen (`conduit-voice` Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[systems/hermes-ui]] (the browser desktop), [[systems/push-notifications]] (the older `hermes-push` bridge). -## Runtime checks (2026-10-26 00:1x AWST) -- **Working tree:** M dashboard/plugin_api.py — the uncommitted memory-timeout patch is **still present**. -- **`conduit-calls.json` → `history` entries:** **0** — the call-request path remains unused - → [[incidents/2026-10-24-conduit-call-request-refused]]. -- **Tool use in-window:** `conduit_show` ×1 — the plugin works for cards; the open incident concerns *placing calls*. +## Runtime checks (2026-10-30 00:0x AWST) +- **`conduit-calls.json` → `history` entries: 2** — the first real calls ever recorded + (2026-10-29 08:38:04 and 08:49:36). The path is no longer unused; the earlier + *"0 calls"* reading is superseded. See [[incidents/2026-10-29-conduit-voice-silence]]. +- **Calls still unproven end-to-end:** `last_request_at` remained `null` at the end of the + 2026-10-29 window — no call is yet known to have run a real Hermes turn. +- **`ends` vs `history`:** `ends` records *any turn-end that fires a watch* (including cron + sessions) — never read it as a call log. +- **Device:** iPhone app version now **2.2.0** (was 1.10.1 on 2026-10-28), **19 settings keys**, + **no `voice.local.*` keys** → Local Live cannot be configured host-side any more. +- **Voice providers were both dead and are now fixed** — `tts.provider: ollama_local`, + `stt.provider: local` (2026-10-29). Details below and in the 10-29 incident. ## Where it lives - **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push` @@ -46,36 +53,42 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste - **Data stores** (`~/.hermes/`): `conduit-health.sqlite3`, `conduit-location.sqlite3`, `conduit-phone.sqlite3`, plus JSON state — `conduit-push.json` (credential / gateway id / installation id / relay url / e2e — **no values recorded here**), `conduit-calls.json`, - `conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`. + `conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`, + `conduit-link-status.json`, `conduit-local-live.json` (**holds an api key — never copy its + contents into the wiki**). -## Current state (verified 2026-10-25 00:2x–00:5x AWST) -- **Dashboard gateway (re-verified 2026-10-26):** listening on `0.0.0.0:9119`; process started **2026-10-25 - 18:42:33** — **restarted since the 2026-10-24 entry, so the upstream commits are now live** (the recorded - *"not restarted since / not live"* note is superseded, not wrong at the time) — so it is still the process that picked up the memory-timeout patch, and - the `0.64.0` → `0.69.0` upstream commits are **not** live. -- **Device:** one registered — **iPhone** (`id 70aa63f3…`, `kid c24e8215…`), app version `2.0.0`, - features `chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards`; - tz `Australia/Perth`; sharing window 2026-10-16 → 2027-04-22. Paired **2026-10-22 12:54:39**. - `last_seen` / `last_sent` / `last_contact` all read **2026-10-23 19:10:30 AWST** (~29 h stale at - this audit) while health rows kept arriving until **2026-10-24 09:59:59** and window rollups were - recomputed at **13:56:35** — so those three fields track *phone-status* contact, not data flow. - (The 10-24 page recorded 19:02:16 from the same store; the epoch reads 19:10:35.) +## Current state (re-verified 2026-10-30 00:0x AWST) +- **Dashboard gateway:** listening on `0.0.0.0:9119`; **pid 4042003, started 2026-10-29 08:03:26** + — the restart that put the 30-day session TTL live (see below). The `0.64.0` → `0.69.0` upstream + commits remain **not** live (the process predates nothing relevant; the tree still only carries the + memory-timeout patch). +- **Device:** one registered — **iPhone**, app version **`2.2.0`** (self-updated from `1.10.1` on + 2026-10-29), `seen_at` **2026-10-29 18:56:44 AWST**, `mode: small`, **19 settings keys**. + `last_seen` / `last_sent` / `last_contact` are `null` in the current store — those three fields + track *phone-status* contact, not data flow (health rows and link pings keep arriving regardless: + `conduit-link-status.json` `app_seen_at` **2026-10-29 23:53:15**, `push_ok_at` **18:50:00**). + Paired **2026-10-22 12:54:39**; sharing window 2026-10-16 → 2027-04-22. +- **Dashboard session TTL (raised 2026-10-29):** `dashboard.basic_auth.session_ttl_seconds` = + **2592000 (30 days)**, up from the **43200 (12 h)** default. A lapsed session was making Conduit's + call socket answer **WS 4401 "Dashboard sign-in required"** — silently. ⚠ `dashboard.session_ttl_seconds` + is a **separate key and still 43200**; check both. - **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits `min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false. -- **Recent calls — none.** `conduit-calls.json → history` is **empty**: no call has been recorded - since the memory-timeout fix landed, so that fix still has **no positive evidence** behind it. The - “two call sessions closed cleanly 00:03:23 / 00:06:57” claim previously on this page was a **false - positive** — those `ends` entries are the two **cron** sessions of that night, and `ends` records - any turn-end that fires a watch, not calls → - [[incidents/2026-10-24-conduit-call-request-refused]]. -- **Data volume (2026-10-25 00:2x):** `conduit-health.sqlite3` **569.9 MB** (+8.2 MB in 25.7 h), - holding **1,520,229 `events`** (1,510,929 `sample` / 6,070 `sleep` / 3,230 `workout`), - **3,028 `buckets`** and **124 `windows`** for its single device; newest row **2026-10-24 09:59:59**, - newest `buckets.date` 2026-10-24; `conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB. +- **Recent calls — two, both 2026-10-29.** `conduit-calls.json → history` now reads **2** + (08:38:04, 08:49:36). The 08:38:04 call produced no Mac connection and no Hermes turn, and the + iPhone re-authenticated at 08:39:05 — the expired-session fault. The 08:49:36 call was placed after + the `classic` mode switch and the STT/TTS fixes, but the session ended one second earlier, so its + outcome is **unrecorded**. The memory-timeout patch therefore still has no *confirmed* call behind + it → [[incidents/2026-10-29-conduit-voice-silence]]. +- **Data volume (2026-10-30 00:0x):** `conduit-health.sqlite3` **930.0 MB** (+360 MB in 5 days; + +8.2 MB/25.7 h as measured on 10-25 — the rate has not held, so treat growth as **ingestion-paced, + unbounded**), holding **2,601,565 `events`** (2,592,189 `sample` / 6,146 `sleep` / 3,230 `workout`), + **4,105 `buckets`** and **411 `windows`**; newest health row **2026-10-29 23:12:01 AWST**. + `conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB (last measured 10-25). ⚠ The 10-24 page recorded **308 `buckets` / 62 `windows`**; a direct `COUNT(*)` reads - **3,028 / 124**, so the earlier pair was a narrower grouping — quote these. **No retention or - compaction has been observed** — the store grows from ingestion alone and is worth watching against - the CT460 disk budget. + **4,105 / 411**, so the earlier pair was a narrower grouping — quote these. **No retention or + compaction has been observed**, and the store is now closing on 1 GB — watch it against the CT460 + disk budget. ## Calls from Hermes (the `conduit_call_user` tool) The plugin registers a **`conduit_call_user`** tool (toolset `conduit`, 📞) for every session and @@ -88,20 +101,36 @@ the call rings when the asking turn ends and plays the reason first. Limits: `mi ⚠ **Known failure, 2026-10-24:** three requests to be phoned were refused by the agent, which asserted the capability did not exist **without calling the tool or loading the skill** → -[[incidents/2026-10-24-conduit-call-request-refused]]. +[[incidents/2026-10-24-conduit-call-request-refused]]. **Positive counter-example 2026-10-29:** in +that window the agent did look the tool up and did place calls (two entries in `history`), so the +failures were behavioural, not a missing capability. -## Voice pipeline (probed 2026-10-28) +## Voice pipeline (verified 2026-10-30) -Phone voice sessions ride the Mac mini speech server, which is also the Hermes TTS target: +Phone voice sessions ride the Mac mini speech server, which is also the Hermes TTS/STT target. +**All three settings below were wrong or dead as of 2026-10-28 and were fixed on 2026-10-29** — +see [[incidents/2026-10-29-conduit-voice-silence]]. -- `voice.speech_server.url` = `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/` — live-voice transport, - `wake_word.enabled: true`. -- `tts.provider: openai` → `https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech` (model `phonon`, - voice `alloy`) — see [[systems/hermes-agent]]. -- **Health check:** `GET https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/pool` returned - `{"size": 2, "in_use": 0}` with both units `idle` on 2026-10-28 00:2x — a **2-unit pool**. `/v1/models` is - **not implemented** (`{"detail":"Not Found"}`); probe `/v1/pool`, not `/v1/models`. +| Setting | Value (now) | Was | Failure when wrong | +|---|---|---|---| +| `voice.speech_server.url` | `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime` | `…:8443/v1/` | **403** — the plugin dials the value **verbatim**, it does not append `/realtime` | +| `tts.provider` | `ollama_local` (`~/.hermes/scripts/ollama-tts.sh`, model `phonon`) | `openai` → Mac `/v1/speech` | **502** | +| `stt.provider` | `local` (`faster_whisper`, model `base`) | `groq` | **403 / Cloudflare error 1010** — a *block*, not a bad key (fails identically with a bogus key) | + +- `voice.mode` = **`classic`** (switched from `local` 2026-10-29). Classic relays everything through + the plugin and needs **no phone-side server address** — which is why both host providers above had + to work. A Local Live path cannot be configured from the host at all: app **2.2.0**'s schema exposes + **19 keys, none of them `voice.local.*`**. - `voice.auto_tts: false` — assistant replies are not auto-spoken; audio appears only when requested. +- `wake_word.enabled: true`. +- **Health check:** `GET https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/pool` returned + `{"size": 2, "in_use": 0}` with both units `idle` on 2026-10-28 00:2x — a **2-unit pool**. `/v1/models` + is **not implemented** (`{"detail":"Not Found"}`); probe `/v1/pool`, not `/v1/models`. +- **End-to-end proof (2026-10-29):** driving the real call path produced `session.created` → + transcription of live speech → `response.done`, with the Mac counters moving **0 → 10 connections + / 1 completed response / 2.98 s audio / 0 errors** and the plugin handshaking in **41 ms**. +- ⚠ **Fragile URL:** the Conduit settings tool writes Hermes config (it had flipped `tts.provider` to + `openai`). If it or an app update ever writes the bare `/v1/` back, every call silently 403s again. ## Memory integration The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not @@ -118,15 +147,27 @@ See [[incidents/2026-10-22-conduit-memory-start-timeout]] and [[systems/mnemosyn git -C ~/.hermes/plugins/conduit_push log --oneline -1 && \ git -C ~/.hermes/plugins/conduit_push status --short -# is the gateway up, and since when? +# is the gateway up, and since when? (the restart that applies a session-TTL change) ss -tlnp | grep 9119 +# session TTLs — BOTH keys, they are different consumers +python3 -c "import yaml;c=yaml.safe_load(open('/home/hermes/.hermes/config.yaml'));\ +print('basic_auth',c['dashboard']['basic_auth'].get('session_ttl_seconds'),\ +'| dashboard',c['dashboard'].get('session_ttl_seconds'))" + +# voice transport + providers +python3 -c "import yaml;c=yaml.safe_load(open('/home/hermes/.hermes/config.yaml'));\ +print(c['voice']['speech_server']['url']);print(c['tts']['provider'],c['stt']['provider'])" + # did a voice call lose its memory context? grep -h "Conduit" ~/.hermes/logs/agent.log | grep -i "memory" | tail # did a real call happen? (history = calls; ends = turn-ends, NOT calls) python3 -c "import json,os;d=json.load(open(os.path.expanduser('~/.hermes/conduit-calls.json')));print(len(d['history']),'calls;',len(d['ends']),'turn-ends')" +# is the phone actually reaching the relay? +cat ~/.hermes/conduit-link-status.json + # state files (no secrets in the wiki) ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3 ``` @@ -134,16 +175,20 @@ ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3 ## Open items - Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it — it is now five releases behind HEAD. -- **Verify a call end-to-end.** No call has been placed since the fix, so `MEMORY_TIMEOUT_S = 60` is - still unexercised by a real call, and the 10-24 refusal means the chat route has never worked → - [[incidents/2026-10-24-conduit-call-request-refused]]. -- No retention policy on `conduit-health.sqlite3` (569.9 MB and growing). +- **Verify a call end-to-end from the phone.** The 2026-10-29 08:49:36 call has no recorded outcome, + so `MEMORY_TIMEOUT_S = 60` is still unexercised by a *confirmed* call, and `last_request_at` was + still `null` → [[incidents/2026-10-29-conduit-voice-silence]]. +- Re-check `voice.speech_server.url` after any Conduit/app update or settings sync — a bare `/v1/` + silently 403s every call. +- No retention policy on `conduit-health.sqlite3` (**930 MB** and growing unbounded). - The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* was raised. → [[systems/mnemosyne]] ## Related +- [[incidents/2026-10-29-conduit-voice-silence]] - [[incidents/2026-10-24-conduit-call-request-refused]] - [[incidents/2026-10-22-conduit-memory-start-timeout]] +- [[systems/hermes-agent]] - [[systems/mnemosyne]] - [[systems/scheduled-tasks]] - [[current-state]] diff --git a/systems/hermes-agent.md b/systems/hermes-agent.md index 2ddfa2c..687fe05 100644 --- a/systems/hermes-agent.md +++ b/systems/hermes-agent.md @@ -3,8 +3,8 @@ title: Hermes Agent type: system status: active created: 2026-07-22 -updated: 2026-10-28 -verified_on: 2026-10-28 +updated: 2026-10-30 +verified_on: 2026-10-30 confidence: high tags: [system, hermes, agent, llm] sources: [live-hermes-version, live-config, live-skill-inventory] @@ -166,6 +166,10 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi - **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS output appears only when explicitly requested. + - ⚠ **Superseded twice — live config checked 2026-10-30:** `tts.provider` is now **`ollama_local`** + (`~/.hermes/scripts/ollama-tts.sh`, model `phonon`) and `stt.provider` is now **`local`** (faster-whisper + `base`); `voice.speech_server.url` is now `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime` + (the bare `/v1/` answers **403**, and the plugin dials the value verbatim). See the 2026-10-30 section below. - ⚠ **Superseded — live config checked 2026-10-28** (change date not established): `tts.provider` is **`openai`** pointed at the **Mac mini speech server** — `tts.providers.openai.base_url` = `https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, `model: phonon`, `voice: alloy`. Live voice @@ -179,6 +183,34 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi cadence - [[systems/messaging-integrations]]. - **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]]. +## 2026-10-30 00:0x AWST — the Conduit voice path: a truncated URL, a 12-hour session TTL, and two dead providers + +Anthony asked why Conduit voice calls never got an answer. **Three faults sat in series** — each one masked the next — +and all three were fixed on 2026-10-29 between 08:02 and 08:4x AWST: + +| Fault | Was | Symptom | Now | +|---|---|---|---| +| `voice.speech_server.url` | `wss://…:8443/v1/` | dialled **verbatim** → HTTP **403**, reported as *"the speech server refused the key"* | `wss://…:8443/v1/realtime` | +| `dashboard.basic_auth.session_ttl_seconds` | `43200` (12 h default) | expired phone session → **WS 4401 "Dashboard sign-in required"**, no Hermes turn, no error on the device | `2592000` (30 d, matched to the refresh-token life) | +| `tts.provider` / `stt.provider` | `openai` / `groq` | Mac `/v1/speech` **502** / **403 Cloudflare 1010** | `ollama_local` / `local` | + +- **Why it stayed invisible:** the Mac recorded **zero connections**, `conduit-calls.json → history` was empty, and each + attempt logged a reason string pointing at the *next* fault — so one broken chain looked like three unrelated problems. + `last_request_at` was `null`: no call had ever reached Hermes' turn logic. +- **The 12 h TTL hid behind the others.** The dashboard had stopped issuing long-enough sessions, and the iPhone only + re-authenticated when the dashboard happened to prompt it — an audit-log entry shows a re-auth at **08:00:59**, minutes + *after* a silent call. +- **Two TTL keys exist.** `dashboard.basic_auth.session_ttl_seconds` was raised; the similarly named + `dashboard.session_ttl_seconds` **still reads 43200**. Check **both** on any future 4401. +- **Groq was blocked, not mis-keyed** — it answers 403/1010 with a bogus key too, so this was Cloudflare bot-blocking + rather than a credential problem. +- **Verified for real:** the whole path was driven (mint HMAC dashboard session → WS ticket → let the plugin dial the Mac), + then two live calls landed at **2026-10-29 08:38:04** and **08:49:36** — the first entries ever in + `conduit-calls.json → history`. `ollama_local` produced **6.94 s** of real 64 kbps speech; `local` STT returned + `success: True`. +- Backups: `config.yaml.bak.url-081525`, `config.yaml.bak.ttl-20261029080258`, `config.yaml.bak.stt-084536`; the dashboard + restarted **08:03:26** → [[incidents/2026-10-29-conduit-voice-silence]] · [[systems/conduit]]. + ## 2026-10-26 00:1x AWST — gateway restart record and the dead Nous credential - **Three gateway starts on 2026-10-25**: `00:32:20`, `18:30:04`, `19:27:13` AWST (after `2026-10-24 12:24:39`). diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index c94ada3..f38a7d1 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-29 -verified_on: 2026-10-29 -review_after: 2026-11-03 +updated: 2026-10-30 +verified_on: 2026-10-30 +review_after: 2026-11-06 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] @@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Current as of 2026-10-29 00:3x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real and the discovery stage still could not finish: stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.** +- **Current as of 2026-10-30 00:0x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**, and every `failure_streak` reads **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit). +- **Prior window — as of 2026-10-29 00:3x AWST *(superseded by the 2026-10-30 entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.** - **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)* - **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`). - **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`). @@ -51,8 +52,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) | | AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 | | Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | | -| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00 | -| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. See [[incidents/2026-10-27-job-radar-budget-exhausted]] | +| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | `local` (failures → topic 1030) | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00. ✅ **First full split run 2026-10-29:** 07:00:59 → 10:16:12 (3 h 15 m 13 s), `completed`, wrote `handoff/2026-10-29.json` (170 discovered / 33 duplicates avoided / 37 verified); both of its ledger rows closed `resolved` at 10:16:12. **Delivery is now `local`** — stage 2 owns the report | +| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. ✅ **First full split run 2026-10-29:** fired 10:13:46, consumed the handoff at 10:40:59, closed `completed` 10:44:48 and delivered the same second — **20 pages + 2 cover letters, `missing_cover_letter_count 0`**, Notion board 455 → 475; the 11:30 and 14:30 slots then no-oped correctly. See [[incidents/2026-10-27-job-radar-budget-exhausted]] | | DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant | | Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] | | Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent | @@ -97,6 +98,11 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] ## Known Issues +### Resolved 2026-10-29 + +- **The Job Radar's two-stage split finished end to end for the first time (`dca8482e4f76` + `5fc292ff0f3b`):** stage 1 ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`** and wrote `handoff/2026-10-29.json` (**170 discovered / 33 duplicates avoided / 37 verified**) — the first post-split run to write a handoff, after 10-27 died on the **tool-call budget** and 10-28 on the **iteration cap**. Stage 2 consumed it at **10:40:59**, closed `completed` at **10:44:48** and **delivered** to topic 1030 in the same second: **20 pages, 2 cover letters, `missing_cover_letter_count 0`**, Notion **455 → 475**. Both stage-1 ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**, and the run opened no new row → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], [[runbooks/job-radar]]. ⚠ The discovery stage is still **slow** (3 h 15 m) and both ceilings remain reachable — a `completed` stage-1 row with **no handoff file** is the same fault in new clothing. +- **Conduit voice calls — three faults in series, all fixed and verified by two real calls:** the trunk speech-server URL was truncated (`…/v1/` → **403**, blamed on the key), the dashboard session TTL was the **12 h default** (expired sessions answer **WS 4401** with no error on the phone), and **both** host providers were dead (`tts.provider: openai` → Mac `/v1/speech` **502**; `stt.provider: groq` → **403/Cloudflare 1010**). Repaired 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → **2 592 000 s**, `stt.provider: local`, `tts.provider: ollama_local`; dashboard restarted **08:03:26**. The path was driven end to end, then **calls at 08:38:04 and 08:49:36** — the first entries in `conduit-calls.json → history`. ⚠ Two TTL keys exist and only one was raised → [[incidents/2026-10-29-conduit-voice-silence]], [[systems/conduit]]. + ### Resolved 2026-10-12 - **Both `deepseek-v4-flash` incidents closed by a clean scheduled run (`fad5a86979d4`, 03:00):** the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents `fad5a8_a13bbc837e1c` (open since 2026-09-28, every fallback provider exhausted) and `fad5a8_c80b59a907f0` (since 2026-09-30, `deepseek-v4-flash` HTTP 400) **both auto-closed at 03:05:39**. Its stale pre-repin `last_status: error` cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one. - **`dca8482e4f76` timeout incident closed (`dca848_23fcd2b18aaa`):** the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm **auto-closed at 07:33:41**. Of the ten storm incidents, nine have now closed and **one remains**: `1b23c0_c35703f637b8` (`1b23c042e17d`, still `alerted`, next slot Fri 2026-10-16 08:00). See [[incidents/2026-10-09-cron-provider-timeout-storm]].