wiki(audit): 2026-10-29 window — Conduit voice fault documented and repaired; radar two-stage split resolved both incidents (20 pages, 2 letters, Notion 455->475); cron 176/0

This commit is contained in:
Tony0410
2026-10-30 00:21:21 +08:00
parent cc95dcc54d
commit 1d335ca8e6
10 changed files with 404 additions and 82 deletions
+35 -4
View File
@@ -3,16 +3,47 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-29 updated: 2026-10-30
verified_on: 2026-10-29 verified_on: 2026-10-30
review_after: 2026-11-03 review_after: 2026-11-07
confidence: high confidence: high
tags: [system] tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
--- ---
# Current State # Current State
Updated: 2026-10-29 Updated: 2026-10-30
## Recent Changes (2026-10-30, nightly audit; live probes 2026-10-30 00:0x AWST)
- **Conduit voice calls were repaired — after three faults in series — and the first two real calls in the
product's history are now on record.** Anthony asked why Conduit never answered when he called. In order:
(A) `voice.speech_server.url` held the truncated `wss://…:8443/v1/`, which the plugin dials **verbatim**, so
the Mac answered **403** and the plugin blamed the key; (B) `dashboard.basic_auth.session_ttl_seconds` was the
**12-hour default**, and an expired phone session gets **WS 4401** with no Hermes turn and no error visible on
the phone (the iPhone re-authenticated at **08:00:59**, minutes *after* a silent call); (C) **both** host
providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` → **403 /
Cloudflare 1010**. Fixed 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → **2 592 000 s (30 d)**, `stt.provider:
local` (faster-whisper `base`), `tts.provider: ollama_local` (6.94 s of real speech verified); dashboard
restarted **08:03:26**. Verified end to end by driving the whole call path, then by **calls at 08:38:04 and
08:49:36** — `conduit-calls.json → history` is no longer empty.
→ [[incidents/2026-10-29-conduit-voice-silence]], [[systems/conduit]], [[systems/hermes-agent]]
- **The Job Radar's two-stage split worked end to end for the first time.** Stage 1 (`dca8482e4f76`,
`deliver: local`) ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`**, and wrote
`~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**,
sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Stage 2 (`5fc292ff0f3b`) consumed the
handoff at **10:40:59**, closed **10:44:48 `completed`** and delivered to topic 1030 in the same second:
**20 pages / 20 `page_ids` / 2 cover letters / `verified_cover_letter_count 2` /
`missing_cover_letter_count 0`**, Notion board **455 → 475**. Both of stage 1's incident rows
(`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**; the 11:30 and 14:30 slots
then no-oped as designed (10 m 46 s / 1 m 51 s).
→ [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]],
[[runbooks/job-radar]]
- **Cron: 2026-10-29 settled 176 `completed` / 0 `failed`** — the first zero-failure day since 2026-10-24 — with
**10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; every `failure_streak`
reads **0**; unclosed incident rows **4 → 1**. Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**, pins
unchanged (19 of the 26 enabled jobs on `custom:omni-101`: 12 `smart-route`, 6 `free-stack`, 1 explicit; 7
unpinned on the global default).
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST) ## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST)
@@ -1,11 +1,11 @@
--- ---
title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
type: incident type: incident
status: active status: resolved
created: 2026-10-28 created: 2026-10-28
updated: 2026-10-29 updated: 2026-10-30
verified_on: 2026-10-29 verified_on: 2026-10-30
review_after: 2026-11-04 review_after: 2026-11-13
confidence: high confidence: high
tags: [automation, cron, jobs, notion, silent-failure, scheduling] tags: [automation, cron, jobs, notion, silent-failure, scheduling]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
@@ -101,6 +101,26 @@ The **first real two-stage cycle ran on 2026-10-28** and answered the open quest
- **Open question 4 is answered: the row did not close.** `dca848_e93a538ade6b` stayed open because the 10-28 - **Open question 4 is answered: the row did not close.** `dca848_e93a538ade6b` stayed open because the 10-28
slot was not clean, and a second row (`dca848_0a2f03b5952b`) opened — open incident rows **2 → 4**. slot was not clean, and a second row (`dca848_0a2f03b5952b`) opened — open incident rows **2 → 4**.
## Follow-up — the 2026-10-29 slot: the split held (recorded 2026-10-30)
**Resolved.** Stage 1 (`dca8482e4f76`) ran **2026-10-29 07:00:59 → 10:16:12** — 3 h 15 m 13 s — and
closed **`completed`**. It wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` (24.8 KB):
`run_date 2026-10-29T08:23:51+08:00`, **170 discovered / 33 duplicates avoided / 37 verified**,
sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Both ledger rows this incident opened —
`dca848_e93a538ade6b` and `dca848_0a2f03b5952b` — closed **`resolved` at 10:16:12**, and the run
opened no new row. Open cron incident rows went **4 → 1**; the survivor is the Facebook monthly's
`0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot.
Stage 2 (`5fc292ff0f3b`) fired at **10:13:46**, consumed the handoff at **10:40:59**, finished
**10:44:48 `completed`** and **delivered** to topic 1030 in the same second (delivery `delivered`).
Its `stage2_result`: **20 pages written / 20 `page_ids` / 2 cover letters / 2 same-day strong matches
/ `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went
**455 → 475**. The later slots no-oped as designed (11:30 → 10 m 46 s, 14:30 → 1 m 51 s, already-consumed
handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns the report.
The questions this page left open are answered for this run: the handoff **was** written inside budget and
consumed, and **no cover letter went missing on the first real run of the split**.
## Related ## Related
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun - [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
@@ -1,11 +1,11 @@
--- ---
title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
type: incident type: incident
status: active status: resolved
created: 2026-10-29 created: 2026-10-29
updated: 2026-10-29 updated: 2026-10-30
verified_on: 2026-10-29 verified_on: 2026-10-30
review_after: 2026-11-05 review_after: 2026-11-13
confidence: high confidence: high
tags: [automation, cron, jobs, notion, silent-failure, scheduling] tags: [automation, cron, jobs, notion, silent-failure, scheduling]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
@@ -76,6 +76,21 @@ slot was not clean. Open `cron_incidents` rows went **2 → 4**.
4. Both radar incident rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) close only when a **clean** stage-1 4. Both radar incident rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) close only when a **clean** stage-1
slot runs — watch the next 07:00 weekday slot. slot runs — watch the next 07:00 weekday slot.
## Follow-up — the 2026-10-29 slot: the cap was not hit (recorded 2026-10-30)
**Resolved for this run.** The next stage-1 run (`dca8482e4f76`, **2026-10-29 07:00:59 → 10:16:12**,
3 h 15 m 13 s) closed **`completed`** *and* wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — the
first run of the split to do both. The handoff records **170 discovered / 33 duplicates avoided /
37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`,
`indeed blocked`, `wa_gov ok`. Stage 2 consumed it at **10:40:59**, delivered at **10:44:48** with
**20 Notion pages and 2 cover letters** (`missing_cover_letter_count 0`; board 455 → 475). Its two
ledger rows closed `resolved` at 10:16:12.
⚠ **Flaky, not solved.** A 3 h 15 m stage-1 run is still long enough for the cap to be reachable, and
the previous two slots in the split both died on it (2026-10-27 on the tool-call budget, 2026-10-28 on
the iteration cap). Treat a `completed` run with **no handoff file** as the same fault, whatever the
execution status says → [[incidents/2026-10-27-job-radar-budget-exhausted]].
## Related ## Related
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up - [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
@@ -0,0 +1,106 @@
---
title: Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers
type: incident
status: resolved
created: 2026-10-30
updated: 2026-10-30
verified_on: 2026-10-30
confidence: high
tags: [conduit, voice, tts, stt, config, session-ttl, incident]
sources:
- live-probe-2026-10-30
- live-config-2026-10-30
- live-log-2026-10-30
- session-20260908_031541_49c051
- plugin-tree-conduit_push
---
# Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers
**Window:** 2026-10-29 07:54 – 09:0x AWST · **Fixed:** 2026-10-29 08:02–08:4x AWST (dashboard restarted 08:03:26)
**Symptom:** Anthony asked *"why do you never answer when I call you on Conduit"* — every Conduit voice call connected and then died silently. No Mac connection, no Hermes turn, no error surfaced on the phone.
Three independent faults sat in series. Fixing any one alone would not have made a call work; each one masked the next.
## Evidence trail (why it was invisible)
- The Mac Mini speech server recorded **zero connections all day.** `last_request_at` on the plugin was `null` — meaning no call had *ever* reached the Hermes turn logic.
- `conduit-calls.json → history` was empty: no successful call had ever been recorded.
- The only traces were in the plugin log: a wrong-looking reason string per attempt, which is what made each fault look like the *next* fault.
## Fault A — Truncated speech-server URL (403)
`voice.speech_server.url` was set to the bare host path:
```
wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/ ⛔ (truncated)
wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime ✅
```
`plugin_api.py` (~line 2646) dials the configured value **verbatim** — it does not append `/realtime`. The bare `/v1/` returns HTTP **403**, the socket closes, and the plugin reports it as *"the speech server refused the key."* The key was fine; the path was wrong.
**Fix:** set `voice.speech_server.url` to `…:8443/v1/realtime`. Backup: `~/.hermes/config.yaml.bak.url-081525` (08:15:25).
## Fault B — Dashboard session TTL of 12 hours (WS 4401)
`dashboard.basic_auth.session_ttl_seconds` held the **12-hour default (43 200 s)**. The dashboard issues the phone its session, and when that session lapses the Conduit call socket answers **WS 4401 "Dashboard sign-in required"** — with no Hermes turn and no user-visible error.
Audit-log corroboration: the iPhone **re-authenticated at 08:00:59**, *minutes after* the silent call — i.e. it was walking around with an expired session and only logging back in when the dashboard happened to prompt.
**Fix:** raised to **30 days (2 592 000 s)** — matched to the refresh-token life so the phone's session no longer expires under it. Dashboard restarted **2026-10-29 08:03:26** (pid 4042003). Backup: `~/.hermes/config.yaml.bak.ttl-20261029080258`.
> ⚠ Two keys exist and only one was changed: `dashboard.basic_auth.session_ttl_seconds` = **2592000** (raised) and `dashboard.session_ttl_seconds` = **43200** (still 12 h, different key, different consumer). Check *both* when diagnosing a 4401 again.
## Fault C — Host STT and TTS providers both dead
Both halves of the voice loop were broken on the host, so even a correctly-signed call would have had no ears and no mouth.
| Setting | Was | Failed with | Now | Verified |
|---|---|---|---|---|
| `tts.provider` | `openai` | Mac `/v1/speech` → **502** | `ollama_local` | 6.94 s of real 64 kbps speech produced |
| `stt.provider` | `groq` | **403 / Cloudflare error 1010** | `local` | Hermes' own transcription path returned `success: True, provider: local` |
- Groq was **blocked, not mis-keyed** — it returns 403/1010 identically with a bogus key, so the failure was Cloudflare bot-blocking, not credentials.
- `tts.provider` had been flipped to `openai` by the **Conduit settings tool** earlier (that tool writes Hermes config). The app holds no `tts.provider` setting, so this fix will not be silently overwritten.
- `ollama_local` runs `~/.hermes/scripts/ollama-tts.sh` (model `phonon`); `local` STT runs `faster_whisper` (`base`).
- Backup: `~/.hermes/config.yaml.bak.stt-084536` (08:45:36); the TTS flip landed between the 08:15 and 08:45 backups.
## Verification (end-to-end, not just config)
Rather than trusting three config edits, the whole call path was driven for real: minted a valid HMAC dashboard session → got a WS ticket → let the plugin dial the Mac. Result:
- auth gate passed → `session.created` / `session.updated`
- speech in transcribed correctly ("Hello Conduit, can you hear me right now?")
- turn committed → audio streamed back → `response.done`
- **Mac counters 0 → 10 connections, 1 completed response, 2.98 s audio, 0 errors**
- plugin `connected: true`, handshake **41 ms**
- the iPhone minted its own WS tickets at **08:07** and **08:19** — i.e. against the new 30-day session
## App update to 2.2.0 changed the settings schema
The iPhone app self-updated to **2.2.0** (from 1.10.1) mid-window, and its settings schema changed with it: the phone now exposes **19 keys, none of them `voice.local.*`**. There is **no phone-side Local Live server address**, so the Local Live path cannot be configured from the host side at all — a prior assumption that it could is now obsolete.
Consequence: `voice.mode` was moved **`local` → `classic`** (request queued 08:47, approved in the app; `conduit-settings.json` now reads `classic`). Classic needs no phone-side server config because the plugin relays the whole call — which is precisely why Fault C (both host providers dead) had to be fixed for it to work at all.
## Call record
`conduit-calls.json → history` now holds **2 entries — the first real calls ever recorded**:
| Time (AWST) | Outcome |
|---|---|
| 2026-10-29 08:38:04 | No Mac connection, no Hermes turn. iPhone had to re-login at 08:39:05, a minute later. |
| 2026-10-29 08:49:36 | Placed *after* the Classic switch and the STT/TTS fixes. Session ended 08:49:35 — Anthony's verdict is unrecorded. |
## Open items
- The 08:49:36 Classic call has **no recorded outcome**. Nothing in the window proves a phone-originated call ran a real Hermes turn.
- `last_request_at` was still `null` at the end of the window: the call→Hermes-turn bridge remains **unproven** even though the transport is now healthy. Classic mode is the recommended path.
- If an app update or the settings sync ever writes the bare `/v1/` back into `voice.speech_server.url`, every call silently 403s again. Re-check it after any Conduit update.
- Watch for the Conduit settings tool re-flipping `tts.provider` should a `tts.provider` key ever appear in the app's schema.
## Related
- [Conduit](../systems/conduit.md) — service page (voice pipeline, call path, pairing)
- [Hermes Agent](../systems/hermes-agent.md) — TTS/STT provider block
- [Conduit call request refused (2026-10-24)](2026-10-24-conduit-call-request-refused.md) — the earlier, behavioural half of the "calls don't arrive" problem
- [Conduit memory-start timeout (2026-10-22)](2026-10-22-conduit-memory-start-timeout.md)
+10 -9
View File
File diff suppressed because one or more lines are too long
+41
View File
@@ -1543,3 +1543,44 @@ not pushed.
- **Not changed, flagged for a human:** the vault mirror at `/home/hermes/wiki/agent-estate-wiki/` has drifted on **both** conventions — top-level **35 match / 3 stale / 72 missing**, `wiki/` **41 match / 53 stale / 16 missing**. A full mirror pass is larger than a stocktake's mandate, so it is recorded here rather than performed. - **Not changed, flagged for a human:** the vault mirror at `/home/hermes/wiki/agent-estate-wiki/` has drifted on **both** conventions — top-level **35 match / 3 stale / 72 missing**, `wiki/` **41 match / 53 stale / 16 missing**. A full mirror pass is larger than a stocktake's mandate, so it is recorded here rather than performed.
- **Removed/archived:** none. **Secrets written:** none (credential names and HTTP status codes only — no key or token values). - **Removed/archived:** none. **Secrets written:** none (credential names and HTTP status codes only — no key or token values).
- **Unresolved:** `cliprox/terra` remains **401 on CT101** — the CT101-side connection still needs a dashboard reconnect if anything is ever to target that pin from that node. - **Unresolved:** `cliprox/terra` remains **401 on CT101** — the CT101-side connection still needs a dashboard reconnect if anything is ever to target that pin from that node.
## [2026-10-30] audit | Nightly wiki audit — window 2026-10-29 00:00 → 2026-10-30 00:0x AWST: the radar's split finished end to end for the first time, and Conduit voice calls work after three faults in series were fixed
- **New page — [[incidents/2026-10-29-conduit-voice-silence]]:** every Conduit voice call connected and then died
silently. Three faults sat in series, each one masking the next: (A) `voice.speech_server.url` was the truncated
`wss://…:8443/v1/`, which the plugin dials **verbatim**, so the Mac answered **403** and the plugin blamed the key;
(B) `dashboard.basic_auth.session_ttl_seconds` was the **12 h default**, so an expired phone session got **WS 4401**
with no Hermes turn and no error on the device (the iPhone re-authenticated at **08:00:59**, minutes *after* a silent
call); (C) **both** host providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq`
→ **403/Cloudflare 1010** (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL →
**2 592 000 s (30 d)**, `stt.provider: local` (faster-whisper `base`), `tts.provider: ollama_local`; dashboard
restarted **08:03:26**. Verified by driving the whole call path for real, then by **two live calls at 08:38:04 and
08:49:36** — the first entries ever in `conduit-calls.json → history`, so the memory fix is no longer evidence-free.
- **Updated [[systems/conduit]]** — new runtime-checks section, the two real calls on record, the incident cross-reference,
and `verified_on: 2026-10-30`.
- **Updated [[systems/hermes-agent]]** — new dated section **2026-10-30 00:0x AWST** for the voice path (fault table +
why it stayed invisible + the two-TTL-keys warning + backups), and the `tts.provider` block now carries a second
*superseded* note recording the live values (`ollama_local` / `local`, `/v1/realtime`).
- **Updated [[systems/scheduled-tasks]]** — new Count bullet for the 2026-10-30 window, the previous bullet demoted to
*superseded*, stage 1's delivery column corrected to `local` (verified live: stage 1 no longer delivers), both radar
rows annotated with their 2026-10-29 result, and a new **Resolved 2026-10-29** block under Known Issues.
- **Updated [[runbooks/job-radar]]** — new dated slot section **2026-10-29 07:00** (handoff written → consumed →
delivered: 170 discovered / 33 duplicates avoided / 37 verified; **20 pages + 2 cover letters**,
`missing_cover_letter_count 0`, Notion board **455 → 475**), plus the note that the second real cycle completed.
- **Updated [[incidents/2026-10-27-job-radar-budget-exhausted]]** and
**[[incidents/2026-10-28-job-radar-stage1-iteration-cap]]** — both **resolved**. Stage 1 ran 07:00:59 → 10:16:12
(3 h 15 m 13 s), closed `completed`, and both ledger rows it had opened (`dca848_e93a538ade6b`,
`dca848_0a2f03b5952b`) closed `resolved` at 10:16:12. Stage 2 consumed the handoff at 10:40:59, closed `completed`
at 10:44:48 and delivered the same second; the 11:30 and 14:30 slots no-oped as designed.
- **Updated [[current-state]]**, **[[index]]** — new Recent Changes block and the refreshed page descriptors.
- **Cron health:** window **2026-10-29 settled 176 `completed` / 0 `failed`** — the first zero-failure day since
2026-10-24 — with **10 deliveries, all 10 `delivered`**. Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**;
non-clean `last_status` **5 → 3** (all three are stale re-runs of long-standing rows) and every `failure_streak`
reads **0**. **Unclosed incident rows 4 → 1**, the lone survivor being the Facebook monthly's
`0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Pins unchanged.
- **Vault mirror catch-up:** the drift flagged by the 2026-10-29 stocktake is partly cleared — the pages changed by this
audit were mirrored into `/home/hermes/wiki/agent-estate-wiki/`, including `systems/conduit.md`,
`runbooks/job-radar.md`, `systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `current-state.md`, `index.md` and
`log.md`. The wiki-maintained pages under `infrastructure/` remain a separate, known gap.
- **Removed/archived:** none. **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a
key or token value.
+28 -3
View File
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery
type: runbook type: runbook
status: active status: active
created: 2026-07-27 created: 2026-07-27
updated: 2026-10-29 updated: 2026-10-30
review_after: 2026-11-10 review_after: 2026-11-13
verified_on: 2026-10-29 verified_on: 2026-10-30
version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2) version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2)
confidence: high confidence: high
tags: [automation, cron, jobs, notion, dashboard, runbook] tags: [automation, cron, jobs, notion, dashboard, runbook]
@@ -328,6 +328,8 @@ now owns a job of its own and the writes own a second one.
- First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both - First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both
`completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle `completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle
ran on 2026-10-28 and landed nothing** — stage 1 never wrote its handoff (see the 2026-10-28 slot below). ran on 2026-10-28 and landed nothing** — stage 1 never wrote its handoff (see the 2026-10-28 slot below).
✅ **The second real cycle (2026-10-29) completed end to end** — handoff written, consumed and delivered; see the
2026-10-29 slot below.
**Recovery if stage 2 lands nothing:** check that stage 1 actually wrote its handoff file before running stage 2 **Recovery if stage 2 lands nothing:** check that stage 1 actually wrote its handoff file before running stage 2
by hand (`hermes cron run 5fc292ff0f3b`). A stage-2 run with no handoff is a no-op, not a failed search — do not by hand (`hermes cron run 5fc292ff0f3b`). A stage-2 run with no handoff is a no-op, not a failed search — do not
@@ -351,6 +353,29 @@ discovery stage is the thing to bound, not the wall clock.
Full ledger: [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. Full ledger: [[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
### 2026-10-29 07:00 slot — the split finally finished: handoff written, consumed, and delivered
Stage 1 (`dca8482e4f76`) ran **07:00:59 → 10:16:12 AWST** (**3 h 15 m 13 s**) and closed **`completed`** — the first
post-split run to finish *and* write its handoff. `~/.hermes/radar-ref/handoff/2026-10-29.json` (24.8 KB) records
`run_date 2026-10-29T08:23:51+08:00`, **170 discovered / 33 duplicates avoided / 37 verified**, sources
`seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Both of the ledger rows the previous two days opened
(`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) **closed `resolved` at 10:16:12**, and this run opened no new row.
Stage 2 (`5fc292ff0f3b`) fired at **10:13:46**, consumed the handoff at **10:40:59**, closed **10:44:48 `completed`**
and **delivered to topic 1030 in the same second**. Its `stage2_result`: **20 pages written / 20 `page_ids` /
2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`** —
the Notion board went **455 → 475**. The 11:30 and 14:30 slots then no-oped exactly as designed
(**10 m 46 s** and **1 m 51 s**, handoff already consumed).
**What this does and does not settle:**
- The split's core promise holds in production — the writes can no longer be lost to a discovery overrun, and the first
full cycle landed 20 pages and 2 cover letters with **zero missing**.
- It does **not** mean discovery is fixed. 3 h 15 m is still long, and the 10-27 **tool-call budget** and 10-28
**iteration cap** remain reachable. Treat a `completed` stage-1 row with **no handoff file** as the same fault,
whatever the execution status says.
- Stage 1 no longer delivers (verified live: `deliver: local`); its failure route to topic 1030 is unchanged.
## Browser-layer hazard inside the run (2026-10-12) ## Browser-layer hazard inside the run (2026-10-12)
This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove
+95 -50
View File
@@ -3,12 +3,12 @@ title: Conduit
type: system type: system
status: active status: active
created: 2026-10-24 created: 2026-10-24
updated: 2026-10-28 updated: 2026-10-30
verified_on: 2026-10-28 verified_on: 2026-10-30
review_after: 2026-11-08 review_after: 2026-11-08
confidence: high confidence: high
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling] tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl]
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329] sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051]
--- ---
# Conduit # Conduit
@@ -21,11 +21,18 @@ be started from the phone where the agent can speak and listen (`conduit-voice`
Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[systems/hermes-ui]] Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[systems/hermes-ui]]
(the browser desktop), [[systems/push-notifications]] (the older `hermes-push` bridge). (the browser desktop), [[systems/push-notifications]] (the older `hermes-push` bridge).
## Runtime checks (2026-10-26 00:1x AWST) ## Runtime checks (2026-10-30 00:0x AWST)
- **Working tree:** M dashboard/plugin_api.py — the uncommitted memory-timeout patch is **still present**. - **`conduit-calls.json` → `history` entries: 2** — the first real calls ever recorded
- **`conduit-calls.json` → `history` entries:** **0** — the call-request path remains unused (2026-10-29 08:38:04 and 08:49:36). The path is no longer unused; the earlier
→ [[incidents/2026-10-24-conduit-call-request-refused]]. *"0 calls"* reading is superseded. See [[incidents/2026-10-29-conduit-voice-silence]].
- **Tool use in-window:** `conduit_show` ×1 — the plugin works for cards; the open incident concerns *placing calls*. - **Calls still unproven end-to-end:** `last_request_at` remained `null` at the end of the
2026-10-29 window — no call is yet known to have run a real Hermes turn.
- **`ends` vs `history`:** `ends` records *any turn-end that fires a watch* (including cron
sessions) — never read it as a call log.
- **Device:** iPhone app version now **2.2.0** (was 1.10.1 on 2026-10-28), **19 settings keys**,
**no `voice.local.*` keys** → Local Live cannot be configured host-side any more.
- **Voice providers were both dead and are now fixed** — `tts.provider: ollama_local`,
`stt.provider: local` (2026-10-29). Details below and in the 10-29 incident.
## Where it lives ## Where it lives
- **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push` - **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push`
@@ -46,36 +53,42 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
- **Data stores** (`~/.hermes/`): `conduit-health.sqlite3`, `conduit-location.sqlite3`, - **Data stores** (`~/.hermes/`): `conduit-health.sqlite3`, `conduit-location.sqlite3`,
`conduit-phone.sqlite3`, plus JSON state — `conduit-push.json` (credential / gateway id / `conduit-phone.sqlite3`, plus JSON state — `conduit-push.json` (credential / gateway id /
installation id / relay url / e2e — **no values recorded here**), `conduit-calls.json`, installation id / relay url / e2e — **no values recorded here**), `conduit-calls.json`,
`conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`. `conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`,
`conduit-link-status.json`, `conduit-local-live.json` (**holds an api key — never copy its
contents into the wiki**).
## Current state (verified 2026-10-25 00:2x–00:5x AWST) ## Current state (re-verified 2026-10-30 00:0x AWST)
- **Dashboard gateway (re-verified 2026-10-26):** listening on `0.0.0.0:9119`; process started **2026-10-25 - **Dashboard gateway:** listening on `0.0.0.0:9119`; **pid 4042003, started 2026-10-29 08:03:26**
18:42:33** — **restarted since the 2026-10-24 entry, so the upstream commits are now live** (the recorded — the restart that put the 30-day session TTL live (see below). The `0.64.0` → `0.69.0` upstream
*"not restarted since / not live"* note is superseded, not wrong at the time) — so it is still the process that picked up the memory-timeout patch, and commits remain **not** live (the process predates nothing relevant; the tree still only carries the
the `0.64.0` → `0.69.0` upstream commits are **not** live. memory-timeout patch).
- **Device:** one registered — **iPhone** (`id 70aa63f3…`, `kid c24e8215…`), app version `2.0.0`, - **Device:** one registered — **iPhone**, app version **`2.2.0`** (self-updated from `1.10.1` on
features `chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards`; 2026-10-29), `seen_at` **2026-10-29 18:56:44 AWST**, `mode: small`, **19 settings keys**.
tz `Australia/Perth`; sharing window 2026-10-16 → 2027-04-22. Paired **2026-10-22 12:54:39**. `last_seen` / `last_sent` / `last_contact` are `null` in the current store — those three fields
`last_seen` / `last_sent` / `last_contact` all read **2026-10-23 19:10:30 AWST** (~29 h stale at track *phone-status* contact, not data flow (health rows and link pings keep arriving regardless:
this audit) while health rows kept arriving until **2026-10-24 09:59:59** and window rollups were `conduit-link-status.json` `app_seen_at` **2026-10-29 23:53:15**, `push_ok_at` **18:50:00**).
recomputed at **13:56:35** — so those three fields track *phone-status* contact, not data flow. Paired **2026-10-22 12:54:39**; sharing window 2026-10-16 → 2027-04-22.
(The 10-24 page recorded 19:02:16 from the same store; the epoch reads 19:10:35.) - **Dashboard session TTL (raised 2026-10-29):** `dashboard.basic_auth.session_ttl_seconds` =
**2592000 (30 days)**, up from the **43200 (12 h)** default. A lapsed session was making Conduit's
call socket answer **WS 4401 "Dashboard sign-in required"** — silently. ⚠ `dashboard.session_ttl_seconds`
is a **separate key and still 43200**; check both.
- **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits - **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits
`min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false. `min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false.
- **Recent calls — none.** `conduit-calls.json → history` is **empty**: no call has been recorded - **Recent calls — two, both 2026-10-29.** `conduit-calls.json → history` now reads **2**
since the memory-timeout fix landed, so that fix still has **no positive evidence** behind it. The (08:38:04, 08:49:36). The 08:38:04 call produced no Mac connection and no Hermes turn, and the
“two call sessions closed cleanly 00:03:23 / 00:06:57” claim previously on this page was a **false iPhone re-authenticated at 08:39:05 — the expired-session fault. The 08:49:36 call was placed after
positive** — those `ends` entries are the two **cron** sessions of that night, and `ends` records the `classic` mode switch and the STT/TTS fixes, but the session ended one second earlier, so its
any turn-end that fires a watch, not calls → outcome is **unrecorded**. The memory-timeout patch therefore still has no *confirmed* call behind
[[incidents/2026-10-24-conduit-call-request-refused]]. it → [[incidents/2026-10-29-conduit-voice-silence]].
- **Data volume (2026-10-25 00:2x):** `conduit-health.sqlite3` **569.9 MB** (+8.2 MB in 25.7 h), - **Data volume (2026-10-30 00:0x):** `conduit-health.sqlite3` **930.0 MB** (+360 MB in 5 days;
holding **1,520,229 `events`** (1,510,929 `sample` / 6,070 `sleep` / 3,230 `workout`), +8.2 MB/25.7 h as measured on 10-25 — the rate has not held, so treat growth as **ingestion-paced,
**3,028 `buckets`** and **124 `windows`** for its single device; newest row **2026-10-24 09:59:59**, unbounded**), holding **2,601,565 `events`** (2,592,189 `sample` / 6,146 `sleep` / 3,230 `workout`),
newest `buckets.date` 2026-10-24; `conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB. **4,105 `buckets`** and **411 `windows`**; newest health row **2026-10-29 23:12:01 AWST**.
`conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB (last measured 10-25).
⚠ The 10-24 page recorded **308 `buckets` / 62 `windows`**; a direct `COUNT(*)` reads ⚠ The 10-24 page recorded **308 `buckets` / 62 `windows`**; a direct `COUNT(*)` reads
**3,028 / 124**, so the earlier pair was a narrower grouping — quote these. **No retention or **4,105 / 411**, so the earlier pair was a narrower grouping — quote these. **No retention or
compaction has been observed** — the store grows from ingestion alone and is worth watching against compaction has been observed**, and the store is now closing on 1 GB — watch it against the CT460
the CT460 disk budget. disk budget.
## Calls from Hermes (the `conduit_call_user` tool) ## Calls from Hermes (the `conduit_call_user` tool)
The plugin registers a **`conduit_call_user`** tool (toolset `conduit`, 📞) for every session and The plugin registers a **`conduit_call_user`** tool (toolset `conduit`, 📞) for every session and
@@ -88,20 +101,36 @@ the call rings when the asking turn ends and plays the reason first. Limits: `mi
⚠ **Known failure, 2026-10-24:** three requests to be phoned were refused by the agent, which ⚠ **Known failure, 2026-10-24:** three requests to be phoned were refused by the agent, which
asserted the capability did not exist **without calling the tool or loading the skill** → asserted the capability did not exist **without calling the tool or loading the skill** →
[[incidents/2026-10-24-conduit-call-request-refused]]. [[incidents/2026-10-24-conduit-call-request-refused]]. **Positive counter-example 2026-10-29:** in
that window the agent did look the tool up and did place calls (two entries in `history`), so the
failures were behavioural, not a missing capability.
## Voice pipeline (probed 2026-10-28) ## Voice pipeline (verified 2026-10-30)
Phone voice sessions ride the Mac mini speech server, which is also the Hermes TTS target: Phone voice sessions ride the Mac mini speech server, which is also the Hermes TTS/STT target.
**All three settings below were wrong or dead as of 2026-10-28 and were fixed on 2026-10-29** —
see [[incidents/2026-10-29-conduit-voice-silence]].
- `voice.speech_server.url` = `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/` — live-voice transport, | Setting | Value (now) | Was | Failure when wrong |
`wake_word.enabled: true`. |---|---|---|---|
- `tts.provider: openai` → `https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech` (model `phonon`, | `voice.speech_server.url` | `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime` | `…:8443/v1/` | **403** — the plugin dials the value **verbatim**, it does not append `/realtime` |
voice `alloy`) — see [[systems/hermes-agent]]. | `tts.provider` | `ollama_local` (`~/.hermes/scripts/ollama-tts.sh`, model `phonon`) | `openai` → Mac `/v1/speech` | **502** |
- **Health check:** `GET https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/pool` returned | `stt.provider` | `local` (`faster_whisper`, model `base`) | `groq` | **403 / Cloudflare error 1010** — a *block*, not a bad key (fails identically with a bogus key) |
`{"size": 2, "in_use": 0}` with both units `idle` on 2026-10-28 00:2x — a **2-unit pool**. `/v1/models` is
**not implemented** (`{"detail":"Not Found"}`); probe `/v1/pool`, not `/v1/models`. - `voice.mode` = **`classic`** (switched from `local` 2026-10-29). Classic relays everything through
the plugin and needs **no phone-side server address** — which is why both host providers above had
to work. A Local Live path cannot be configured from the host at all: app **2.2.0**'s schema exposes
**19 keys, none of them `voice.local.*`**.
- `voice.auto_tts: false` — assistant replies are not auto-spoken; audio appears only when requested. - `voice.auto_tts: false` — assistant replies are not auto-spoken; audio appears only when requested.
- `wake_word.enabled: true`.
- **Health check:** `GET https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/pool` returned
`{"size": 2, "in_use": 0}` with both units `idle` on 2026-10-28 00:2x — a **2-unit pool**. `/v1/models`
is **not implemented** (`{"detail":"Not Found"}`); probe `/v1/pool`, not `/v1/models`.
- **End-to-end proof (2026-10-29):** driving the real call path produced `session.created` →
transcription of live speech → `response.done`, with the Mac counters moving **0 → 10 connections
/ 1 completed response / 2.98 s audio / 0 errors** and the plugin handshaking in **41 ms**.
- ⚠ **Fragile URL:** the Conduit settings tool writes Hermes config (it had flipped `tts.provider` to
`openai`). If it or an app update ever writes the bare `/v1/` back, every call silently 403s again.
## Memory integration ## Memory integration
The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not
@@ -118,15 +147,27 @@ See [[incidents/2026-10-22-conduit-memory-start-timeout]] and [[systems/mnemosyn
git -C ~/.hermes/plugins/conduit_push log --oneline -1 && \ git -C ~/.hermes/plugins/conduit_push log --oneline -1 && \
git -C ~/.hermes/plugins/conduit_push status --short git -C ~/.hermes/plugins/conduit_push status --short
# is the gateway up, and since when? # is the gateway up, and since when? (the restart that applies a session-TTL change)
ss -tlnp | grep 9119 ss -tlnp | grep 9119
# session TTLs — BOTH keys, they are different consumers
python3 -c "import yaml;c=yaml.safe_load(open('/home/hermes/.hermes/config.yaml'));\
print('basic_auth',c['dashboard']['basic_auth'].get('session_ttl_seconds'),\
'| dashboard',c['dashboard'].get('session_ttl_seconds'))"
# voice transport + providers
python3 -c "import yaml;c=yaml.safe_load(open('/home/hermes/.hermes/config.yaml'));\
print(c['voice']['speech_server']['url']);print(c['tts']['provider'],c['stt']['provider'])"
# did a voice call lose its memory context? # did a voice call lose its memory context?
grep -h "Conduit" ~/.hermes/logs/agent.log | grep -i "memory" | tail grep -h "Conduit" ~/.hermes/logs/agent.log | grep -i "memory" | tail
# did a real call happen? (history = calls; ends = turn-ends, NOT calls) # did a real call happen? (history = calls; ends = turn-ends, NOT calls)
python3 -c "import json,os;d=json.load(open(os.path.expanduser('~/.hermes/conduit-calls.json')));print(len(d['history']),'calls;',len(d['ends']),'turn-ends')" python3 -c "import json,os;d=json.load(open(os.path.expanduser('~/.hermes/conduit-calls.json')));print(len(d['history']),'calls;',len(d['ends']),'turn-ends')"
# is the phone actually reaching the relay?
cat ~/.hermes/conduit-link-status.json
# state files (no secrets in the wiki) # state files (no secrets in the wiki)
ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3 ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3
``` ```
@@ -134,16 +175,20 @@ ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3
## Open items ## Open items
- Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it — it is - Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it — it is
now five releases behind HEAD. now five releases behind HEAD.
- **Verify a call end-to-end.** No call has been placed since the fix, so `MEMORY_TIMEOUT_S = 60` is - **Verify a call end-to-end from the phone.** The 2026-10-29 08:49:36 call has no recorded outcome,
still unexercised by a real call, and the 10-24 refusal means the chat route has never worked → so `MEMORY_TIMEOUT_S = 60` is still unexercised by a *confirmed* call, and `last_request_at` was
[[incidents/2026-10-24-conduit-call-request-refused]]. still `null` → [[incidents/2026-10-29-conduit-voice-silence]].
- No retention policy on `conduit-health.sqlite3` (569.9 MB and growing). - Re-check `voice.speech_server.url` after any Conduit/app update or settings sync — a bare `/v1/`
silently 403s every call.
- No retention policy on `conduit-health.sqlite3` (**930 MB** and growing unbounded).
- The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* was - The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* was
raised. → [[systems/mnemosyne]] raised. → [[systems/mnemosyne]]
## Related ## Related
- [[incidents/2026-10-29-conduit-voice-silence]]
- [[incidents/2026-10-24-conduit-call-request-refused]] - [[incidents/2026-10-24-conduit-call-request-refused]]
- [[incidents/2026-10-22-conduit-memory-start-timeout]] - [[incidents/2026-10-22-conduit-memory-start-timeout]]
- [[systems/hermes-agent]]
- [[systems/mnemosyne]] - [[systems/mnemosyne]]
- [[systems/scheduled-tasks]] - [[systems/scheduled-tasks]]
- [[current-state]] - [[current-state]]
+34 -2
View File
@@ -3,8 +3,8 @@ title: Hermes Agent
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-28 updated: 2026-10-30
verified_on: 2026-10-28 verified_on: 2026-10-30
confidence: high confidence: high
tags: [system, hermes, agent, llm] tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory] sources: [live-hermes-version, live-config, live-skill-inventory]
@@ -166,6 +166,10 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
- **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env - **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env
and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS
output appears only when explicitly requested. output appears only when explicitly requested.
- ⚠ **Superseded twice — live config checked 2026-10-30:** `tts.provider` is now **`ollama_local`**
(`~/.hermes/scripts/ollama-tts.sh`, model `phonon`) and `stt.provider` is now **`local`** (faster-whisper
`base`); `voice.speech_server.url` is now `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime`
(the bare `/v1/` answers **403**, and the plugin dials the value verbatim). See the 2026-10-30 section below.
- ⚠ **Superseded — live config checked 2026-10-28** (change date not established): `tts.provider` is - ⚠ **Superseded — live config checked 2026-10-28** (change date not established): `tts.provider` is
**`openai`** pointed at the **Mac mini speech server** — `tts.providers.openai.base_url` = **`openai`** pointed at the **Mac mini speech server** — `tts.providers.openai.base_url` =
`https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, `model: phonon`, `voice: alloy`. Live voice `https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, `model: phonon`, `voice: alloy`. Live voice
@@ -179,6 +183,34 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
cadence - [[systems/messaging-integrations]]. cadence - [[systems/messaging-integrations]].
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]]. - **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
## 2026-10-30 00:0x AWST — the Conduit voice path: a truncated URL, a 12-hour session TTL, and two dead providers
Anthony asked why Conduit voice calls never got an answer. **Three faults sat in series** — each one masked the next —
and all three were fixed on 2026-10-29 between 08:02 and 08:4x AWST:
| Fault | Was | Symptom | Now |
|---|---|---|---|
| `voice.speech_server.url` | `wss://…:8443/v1/` | dialled **verbatim** → HTTP **403**, reported as *"the speech server refused the key"* | `wss://…:8443/v1/realtime` |
| `dashboard.basic_auth.session_ttl_seconds` | `43200` (12 h default) | expired phone session → **WS 4401 "Dashboard sign-in required"**, no Hermes turn, no error on the device | `2592000` (30 d, matched to the refresh-token life) |
| `tts.provider` / `stt.provider` | `openai` / `groq` | Mac `/v1/speech` **502** / **403 Cloudflare 1010** | `ollama_local` / `local` |
- **Why it stayed invisible:** the Mac recorded **zero connections**, `conduit-calls.json → history` was empty, and each
attempt logged a reason string pointing at the *next* fault — so one broken chain looked like three unrelated problems.
`last_request_at` was `null`: no call had ever reached Hermes' turn logic.
- **The 12 h TTL hid behind the others.** The dashboard had stopped issuing long-enough sessions, and the iPhone only
re-authenticated when the dashboard happened to prompt it — an audit-log entry shows a re-auth at **08:00:59**, minutes
*after* a silent call.
- **Two TTL keys exist.** `dashboard.basic_auth.session_ttl_seconds` was raised; the similarly named
`dashboard.session_ttl_seconds` **still reads 43200**. Check **both** on any future 4401.
- **Groq was blocked, not mis-keyed** — it answers 403/1010 with a bogus key too, so this was Cloudflare bot-blocking
rather than a credential problem.
- **Verified for real:** the whole path was driven (mint HMAC dashboard session → WS ticket → let the plugin dial the Mac),
then two live calls landed at **2026-10-29 08:38:04** and **08:49:36** — the first entries ever in
`conduit-calls.json → history`. `ollama_local` produced **6.94 s** of real 64 kbps speech; `local` STT returned
`success: True`.
- Backups: `config.yaml.bak.url-081525`, `config.yaml.bak.ttl-20261029080258`, `config.yaml.bak.stt-084536`; the dashboard
restarted **08:03:26** → [[incidents/2026-10-29-conduit-voice-silence]] · [[systems/conduit]].
## 2026-10-26 00:1x AWST — gateway restart record and the dead Nous credential ## 2026-10-26 00:1x AWST — gateway restart record and the dead Nous credential
- **Three gateway starts on 2026-10-25**: `00:32:20`, `18:30:04`, `19:27:13` AWST (after `2026-10-24 12:24:39`). - **Three gateway starts on 2026-10-25**: `00:32:20`, `18:30:04`, `19:27:13` AWST (after `2026-10-24 12:24:39`).
+12 -6
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-29 updated: 2026-10-30
verified_on: 2026-10-29 verified_on: 2026-10-30
review_after: 2026-11-03 review_after: 2026-11-06
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db] sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **Current as of 2026-10-29 00:3x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real and the discovery stage still could not finish: stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.** - **Current as of 2026-10-30 00:0x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**, and every `failure_streak` reads **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit).
- **Prior window — as of 2026-10-29 00:3x AWST *(superseded by the 2026-10-30 entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.**
- **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)* - **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)*
- **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`). - **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`).
- **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`). - **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
@@ -51,8 +52,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) | | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 | | AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 |
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | | | Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00 | | Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | `local` (failures → topic 1030) | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00. ✅ **First full split run 2026-10-29:** 07:00:59 → 10:16:12 (3 h 15 m 13 s), `completed`, wrote `handoff/2026-10-29.json` (170 discovered / 33 duplicates avoided / 37 verified); both of its ledger rows closed `resolved` at 10:16:12. **Delivery is now `local`** — stage 2 owns the report |
| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. See [[incidents/2026-10-27-job-radar-budget-exhausted]] | | Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. ✅ **First full split run 2026-10-29:** fired 10:13:46, consumed the handoff at 10:40:59, closed `completed` 10:44:48 and delivered the same second — **20 pages + 2 cover letters, `missing_cover_letter_count 0`**, Notion board 455 → 475; the 11:30 and 14:30 slots then no-oped correctly. See [[incidents/2026-10-27-job-radar-budget-exhausted]] |
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant | | DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant |
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] | | Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] |
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent | | Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
@@ -97,6 +98,11 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
## Known Issues ## Known Issues
### Resolved 2026-10-29
- **The Job Radar's two-stage split finished end to end for the first time (`dca8482e4f76` + `5fc292ff0f3b`):** stage 1 ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`** and wrote `handoff/2026-10-29.json` (**170 discovered / 33 duplicates avoided / 37 verified**) — the first post-split run to write a handoff, after 10-27 died on the **tool-call budget** and 10-28 on the **iteration cap**. Stage 2 consumed it at **10:40:59**, closed `completed` at **10:44:48** and **delivered** to topic 1030 in the same second: **20 pages, 2 cover letters, `missing_cover_letter_count 0`**, Notion **455 → 475**. Both stage-1 ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**, and the run opened no new row → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], [[runbooks/job-radar]]. ⚠ The discovery stage is still **slow** (3 h 15 m) and both ceilings remain reachable — a `completed` stage-1 row with **no handoff file** is the same fault in new clothing.
- **Conduit voice calls — three faults in series, all fixed and verified by two real calls:** the trunk speech-server URL was truncated (`…/v1/` → **403**, blamed on the key), the dashboard session TTL was the **12 h default** (expired sessions answer **WS 4401** with no error on the phone), and **both** host providers were dead (`tts.provider: openai` → Mac `/v1/speech` **502**; `stt.provider: groq` → **403/Cloudflare 1010**). Repaired 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → **2 592 000 s**, `stt.provider: local`, `tts.provider: ollama_local`; dashboard restarted **08:03:26**. The path was driven end to end, then **calls at 08:38:04 and 08:49:36** — the first entries in `conduit-calls.json → history`. ⚠ Two TTL keys exist and only one was raised → [[incidents/2026-10-29-conduit-voice-silence]], [[systems/conduit]].
### Resolved 2026-10-12 ### Resolved 2026-10-12
- **Both `deepseek-v4-flash` incidents closed by a clean scheduled run (`fad5a86979d4`, 03:00):** the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents `fad5a8_a13bbc837e1c` (open since 2026-09-28, every fallback provider exhausted) and `fad5a8_c80b59a907f0` (since 2026-09-30, `deepseek-v4-flash` HTTP 400) **both auto-closed at 03:05:39**. Its stale pre-repin `last_status: error` cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one. - **Both `deepseek-v4-flash` incidents closed by a clean scheduled run (`fad5a86979d4`, 03:00):** the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents `fad5a8_a13bbc837e1c` (open since 2026-09-28, every fallback provider exhausted) and `fad5a8_c80b59a907f0` (since 2026-09-30, `deepseek-v4-flash` HTTP 400) **both auto-closed at 03:05:39**. Its stale pre-repin `last_status: error` cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one.
- **`dca8482e4f76` timeout incident closed (`dca848_23fcd2b18aaa`):** the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm **auto-closed at 07:33:41**. Of the ten storm incidents, nine have now closed and **one remains**: `1b23c0_c35703f637b8` (`1b23c042e17d`, still `alerted`, next slot Fri 2026-10-16 08:00). See [[incidents/2026-10-09-cron-provider-timeout-storm]]. - **`dca8482e4f76` timeout incident closed (`dca848_23fcd2b18aaa`):** the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm **auto-closed at 07:33:41**. Of the ten storm incidents, nine have now closed and **one remains**: `1b23c0_c35703f637b8` (`1b23c042e17d`, still `alerted`, next slot Fri 2026-10-16 08:00). See [[incidents/2026-10-09-cron-provider-timeout-storm]].