Files
agent-estate-wiki/systems/scheduled-tasks.md
T

163 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Scheduled Tasks system active 2026-07-22 2026-11-08 2026-11-08 2026-11-09 high
cron
scheduled
automation
live-cron-list
live-jobs-json
cron-executions-db
live-executions-db
live-deliveries-db
live-sessions-db

Scheduled Tasks

Count

  • Current as of 2026-11-07 (window 2026-11-06; live jobs.json + executions.db + cron_incidents + deliveries.db + sessions, read 2026-11-07 00:3x-00:5x AWST): 29 jobs (26 enabled / 3 disabled - Important-mail monitor, Rhino AgentMail Inbox Watcher, AgentMail Rhino Inbox Poll; count and split unchanged across five consecutive audits, fleet updated_at 2026-11-07T00:39:46+08:00). 11-06 settled: 172 rows, 16 distinct jobs - 170 completed / 1 failed / 1 running (this audit, 23:59:31). The single failed row is a new, non-restart failure signature: the Job Radar's 11:30 Stage-2 slot (95347fa5…) was discarded as stale at 13:26:44 on Fire claim ownership lost; stale result was discarded. because a direct fire of the same job, started 08:27:09, was still holding the claim - that fire completed 13:26:22 and delivered, and the ledger records no gateway restart in the window -> incidents/2026-11-06-job-radar-concurrent-fire-claim-discard. Deliveries 10 rows / 10 delivered / 0 for_failure; the radar still landed the day in full at 14:30 (MISSING_COVER_LETTER_COUNT 0, 4 same-day strong rows, 4 cover letters, 20 non-empty artifacts, Run status: COMPLETE). Fleet non-clean last_status unchanged at 1 (066bb5191280, delivery_failed, streak 0). Unclosed incident rows 0 -> 0 - no ledger row has last_seen_at on or after 2026-11-06 (newest activity remains the 2026-11-05T12:36:17 closures), so the discarded run raised no incident row and no alert; incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts remains the one open page. The stocktake's overrun streak fully unwound: 1a8e90ea2271 ran 03:00:32 -> 04:11:35 (1 h 11 m 03 s) against 7 h 40 m -> 5 h 41 m -> 3 h 02 m on 11-03/11-04/11-05. Nothing added, retired, disabled or repinned.
  • Current as of 2026-11-06 (window 2026-11-05; live jobs.json + executions.db + cron_incidents + deliveries.db + sessions, read 2026-11-06 00:1x-00:4x AWST): 29 jobs (26 enabled / 3 disabled) - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. 11-05 settled: 175 rows, 18 distinct jobs - 173 completed / 2 running / 0 failed - the first zero-failed ledger day since 2026-11-02. Both running rows are explained and neither is lost work: ea79a2107e36 (Mnemosyne Health Monitor, 12:00:45 slot) was still in flight 12 h+ with no delivery at audit time, and cbfe434a3b5e is this audit (23:59:31). Deliveries 10 rows / 10 delivered / 0 for_failure. Fleet non-clean last_status 3 -> 1 (only 066bb5191280, Uber Eats Monthly Spend Report, delivery_failed, streak 0). Unclosed incident rows 3 -> 0 - all three closed in-window: 85edf6_c52a11a62044 (Tech & AI Newsletter Digest, unknown) at 07:07:42 and both Job Radar rows (dca848_dfd4bea84f2a timeout + dca848_3b1bb1e9b003 unknown) at 12:36:17. That empties the ledger for the first time since 2026-11-02 and closes out incidents/2026-11-04-upstream-stream-failure-0700-cohort and incidents/2026-11-03-job-radar-stream-active-timeout in full. New this window: a live-secret exposure -> incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.
  • Current as of 2026-11-05 (window 2026-11-04; live jobs.json + executions.db + cron_incidents + deliveries.db, read 2026-11-05 00:5x AWST): 29 jobs (26 enabled / 3 disabled). 11-04 settled: 170 rows - 167 completed / 2 failed / 1 running; deliveries 8 rows / 8 delivered; non-clean last_status 2 -> 3; unclosed incident rows 1 -> 3 (all state = alerted). The two failures are one signature: dca8482e4f76 (Daily Marketing Job Radar - Discovery, streak 2) and 85edf6772bca (Tech & AI Newsletter Digest (Analytical), streak 1) both died on RuntimeError: Upstream stream failed before completion. 36 s apart (09:41:44 / 09:42:20) after 2 h 41 m each, both on custom:omni-101 / smart-route. The third non-clean row is 066bb5191280 (Uber Eats Monthly Spend Report) delivery_failed, streak 0. Also in-window: 1a8e90ea2271 (wiki-stocktake-maintenance) overran 5 h 41 m 46 s and reported incomplete (third consecutive day), and CT299 was taken offline by a second wedged vzdump -> incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage; the 07:00 pair -> incidents/2026-11-04-upstream-stream-failure-0700-cohort. Corrected 2026-11-05 01:3x AWST (live): this bullet's first draft read the fleet as 158 jobs / 144 enabled / 14 disabled and 11-04 as ending 1 item_suppressed. Three independent reads (jobs.json, hermes cron list, and the in-flight job count) all give 29 jobs / 26 enabled / 3 disabled, and no item_suppressed row exists on 11-04 - the 170th row is this audit, still running. Both errors are corrected across scheduled-tasks, current-state, log and index. 11-03 has since settled to 171 completed / 1 failed (172 rows; its running row completed), making 11-03 and 11-04 the only two consecutive days with a failed row since 10-31. ⚠ deepseek-paid-smart cannot currently dispatch on the omniroute (CT600) endpoint: keyed probe 2026-11-05 01:33:12 AWST - POST http://omniroute:20128/v1/chat/completions -> HTTP 503 ALL_TARGETS_SKIPPED (poolSize 5, attempted 0), while POST http://100.78.63.20:20128/v1/chat/completions (omniroute-1) -> HTTP 200, resolving the alias to deepseek/deepseek-v4-flash. The fleet's only deepseek-paid-smart pin is this audit job (cbfe434a3b5e, provider custom:omni-101 = the omniroute-1 node) - which is why it still completes. An unkeyed /v1/models probe returning 401 on both hosts is auth-gating, not health. ⚠ **Re-verified 2026-11-06 (stocktake): on omniroute-1 (100.78.63.20) the raw model id deepseek/deepseek-v4-flash returns HTTP 401 No active credentials for provider: deepseek while the fleet's deepseek-paid-smart alias returns HTTP 200 and resolves to deepseek/deepseek-v4-flash. That 401 is an alias-routing artefact, not an outage — always probe the alias the fleet actually pins, never the raw model id. → incidents/2026-10-14-omniroute-node-connection-refused-storm -> runbooks/job-radar CL_MODEL caveat.
  • Current as of 2026-11-04 (window 2026-11-03; live jobs.json + executions.db + cron_incidents + deliveries.db) — the ledger produced its own red row. 2026-11-03 settled 170 completed / 1 failed / 1 running / 0 unknown (172 rows) against 11-02's 177 / 2 — the first failed row since 10-31, and the failure is the Job Radar Stage 1 (dca8482e4f76) dying on RuntimeError: stream active timeout after 6 h 28 m 18 s → incidents/2026-11-03-job-radar-stream-active-timeout. Fleet 29 jobs — 26 enabled / 3 disabled (unchanged). Non-clean last_status 1 → 2: dca8482e4f76 (error, streak 1) joins 066bb5191280 (delivery_failed, unchanged). Unclosed incident rows 0 → 1 (dca848_dfd4bea84f2a). Deliveries: 7 rows, 7/7 delivered (07:13:49, 07:30:35, 08:39:25, 12:37:09, 13:28:51 for_failure, 13:49:11, 18:45:39). ⚠ The 03:00 stocktake ran long for the second day: 1a8e90ea2271 03:00:30.495724 → 10:41:13.851030 = 7 h 40 m 43 s, completed / suppressed, delivered nothing and again left its artefacts uncommitted (systems/skills-index.md modified, raw/stocktake/freshness-20261103.txt untracked — committed by this audit). Sessions in the window: 16 (14 cron, 1 ios, 1 desktop).
  • Prior window — as of 2026-11-03 00:1x AWST (live jobs.json + executions.db + cron_incidents + deliveries.db) — the first fully clean ledger day in five, but two long runs ended in a silent cap and one of them raised no incident at all. Fleet 29 jobs — 26 enabled / 3 disabled (unchanged). Window 2026-11-02 settled 177 completed / 2 running / 0 unknown / 0 failed (179 rows; 96 of the completions are 96eb95ce12f2's 15-minute watchdog slots) against 167 / 1 / 2 on 11-01, 160 / 4 on 10-31 and 138 / 0 on 10-30. Both 11-01 unknown rows resolved on 11-02: the Sunday 18:00 Weekly Review reconciled at 01:27:22 (be6ec11a8e68428084af9ab18fb148db, completed / delivered, 7 h 26 m 43 s after it started) and its incident 79f9bb_9c443b602eec auto-closed at 01:27:21.606691 → incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure (now resolved; the "lost review" was never lost). ecc1fc13655a's 11-01 ghost error also cleared — its 11-02 12:00:33 slot ran completed / delivered at 12:29:26. Unclosed incident rows 1 → 0 — the first zero in the period. Non-clean last_status 2 → 1: only 066bb5191280 (Uber Eats Monthly Spend Report, delivery_failed, failure_streak: 0, last_run 2026-11-01T07:18:57) — the 11-01 DM-thread miss, which did not recur; it has no executions row on 11-02 at all. Delivery health: 9 rows on 11-02, 9/9 delivered (01:27:21, 07:30:35, 07:42:21, 08:10:18, 08:41:14, 09:07:20, 10:23:15, 12:29:25, 18:45:37). ⚠ Two long runs finished completed with error: NULL and delivery_outcome: suppressed after ~9 h — and the first raised no incident: (a) dca8482e4f76 (Job Radar Stage 1) 07:00:33 → 16:01:52 = 9 h 01 m 19 s, hit the agent iteration cap before the handoff, so both Stage 2 slots (92665337… 11:30, 05a3f00b… 14:30 safety net) exited [SILENT] and nothing reached Notion — the same class that raised incidents on 10-27 and 10-28 raised none this time because the terminal state flipped to completed → incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence; (b) 1a8e90ea2271 (daily stocktake) 03:00:31 → 12:44:00 = 9 h 43 m 29 s, suppressed — it delivered no report (unlike 11-01's 03:00:37 delivery) and left its output uncommitted (systems/skills-index.md modified, raw/stocktake/freshness-20261102.txt untracked; the 2026-11-03 audit committed both on its behalf). See Known Issues → Open below.
  • Prior window (2026-11-02 00:4x AWST, live jobs.json + executions.db + cron_incidents + deliveries.db + gateway log) — the fleet ran clean all day, then failed in three different ways and missed a delivery. Fleet 29 jobs — 26 enabled / 3 disabled (unchanged). Window 2026-11-01 settled 167 completed / 1 running / 2 unknown (+ this audit running, started 23:59:28) against 161 / 4 on 10-31, 138 / 0 on 10-30 and 177 / 0 on 10-29 — the first non-clean day in four. The two unknown rows are the whole story of the day: the Sunday 18:00 Weekly Review never ran and never delivered ("Gateway connection failed; no local fallback", incident 79f9bb_9c443b602eec still alerted at audit time, job reads error / failure_streak: 1), and one 96eb95ce12f2 watchdog slot. Three other jobs were SIGTERMed (-15) mid-adoption without writing a durable terminal state (85edf6772bca @07:30:46, 96eb95ce12f2 and 0e753e9720fd @09:02:18) — with no gateway restart or reload anywhere in the window (PID 175269 up since 10-31 19:24:40; the relay counter ran unbroken through both signals) → incidents/2026-11-01-cron-worker-sigterm-orphaned-claims and incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure. Delivery health: 8 delivery rows on 11-01, 7 delivered / 1 failed — the failure is 066bb5191280's thread_id 34795 DM-thread miss (the thread-less fallback still reaches the DM; the run is recorded failed anyway). **Non-clean last_status 4 → 2** (ea79a2107e36, 79f9bb41ecf5). ⚠ **ecc1fc13655areadserrorwithfailure_streak: 1andlast_run 2026-11-01T12:01:00while owning NOexecutionsrow for that slot and NOcron_incidentsrow** — a status field with no artefact behind it. **Unclosed incident rows 1 → 1, but a different row:** all three rows the 10-31 audit left open closed on 11-01 (02:39:33 ×2 forea79a2, **10:44:12** for 0e753e_e7f8194a7d06— the Facebook-repin row open since **2026-10-01**, 31 days). ⚠ **Telegram's transport flapped all day** (dual-stackapi.telegram.org ConnectTimeouts, IPv4-literal fallbacks, a disconnect at 20:51:53, polling restarts at 22:33:30 / 23:12:23) and the **relay adapter has been absent ~29 h** (Reconnecting relay` attempt 336 at 11-02 00:10:47) → systems/messaging-integrations.
  • Prior window (2026-11-01 00:2x AWST, live jobs.json + executions.db + cron_incidents + gateway log) — the last five hours of 10-31 ran clean, but an 18:06 gateway reload force-drained at its full 1 800 s cap and killed a live cron run. Fleet 29 jobs — 26 enabled / 3 disabled (jobs.json.updated_at 2026-11-01T00:05:42+08:00). Window 2026-10-31 settled 160 completed / 4 failed (+ this audit running, started 23:59:17) against 138 / 0 on 10-30 and 177 / 0 on 10-29. All four failures are one fault: ea79a2107e36 (Mnemosyne Health Monitor) closed failed at 18:39:15 after 6 h 38 m 09 s on RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership, plus three derivative 96eb95ce12f2 (log-error watchdog) reds at 18:39:23 / 18:45:23 / 19:00:22 (watchdog cleared on its 19:15:21 slot) → incidents/2026-10-31-gateway-reload-force-drain-orphaned-run. Mechanism (new for this class): a SIGUSR1 reload at 18:06:41 deferred its stop with "cap=1800s", waited the entire cap on one managed work unit (pid 397) — the cron unit itself was logged restart_safe: True and excluded from the wait — then force-drained at 18:37:38 ("drain timed out with interrupted agents"). A second, clean restart followed at 19:23:39 → 19:24:40 (drain took 0.00s, timed_out=False; "Previous gateway exited cleanly"), gateway PID 175269 up since 19:23:56, and nothing failed after 19:15:21. Non-clean last_status 3 → 4, failing (run failed outright) 0 → 1 — ea79a2107e36 reads error with failure_streak: 2, the fleet's only non-zero streak; its next slot (2026-11-01 00:00:41) was already running at audit time. Unclosed incident rows 1 → 3: ea79a2_a4661554a3c2 (first seen 2026-10-26 14:35:59) was re-used/refreshed (last_seen 18:39:15) and new row ea79a2_9197df32320b opened 18:06:44 (alerted 18:06:48) — two rows, one event, same job. Delivery health recovered decisively: 8 deliveries on 10-31, 8/8 delivered (five normal + three failure notices) against zero rows on 10-30 — the notice path that failed silently on 10-30 worked. ⚠ The stocktake (1a8e90ea2271) ran 03:00:58 → 06:44:31 (3 h 43 m 33 s) — its longest run since 10-22 — closed completed with delivery suppressed and committed nothing, leaving 7 modified files + 1 untracked (raw/stocktake/freshness-20261031.txt); this audit committed them. ⚠ Two Telegram adapter rebuilds (telegram_network_error, "updater.stop() did not finish … likely CLOSE-WAIT socket") at 05:03:42 and 09:12:11, both self-recovered. ⚠ Asana is unreachable on both paths from CT460 — the asana MCP server has never authenticated (~/.hermes/mcp-tokens/ holds only asana.client.json) and the hosted connector rejects opt_fields as a non-list (422); 7 728 OAuth-failure log lines in the retained window, 1 156 of them on 10-31 → incidents/2026-10-31-asana-access-failures. Sessions in window: 17 (14 cron, 3 desktop — two Conduit Pulse sessions and a voice-settings background investigation, the latter two still running at audit time).
  • ⚠ Superseded by the 2026-11-01 entry above — Current as of 2026-10-31 00:3x AWST (live jobs.json + executions.db) — the fleet is whole, but it is whole because a 5 h 30 m HOST OUTAGE ended, not because anything was repaired. Fleet 29 jobs — 26 enabled / 3 disabled (jobs.json.updated_at 2026-10-31T00:29:59+08:00). Window 2026-10-30: 138 executions — 136 completed, 0 failed, 2 running (the 16:37 catch-up + this audit), against 177 / 177 / 0 on 10-29 → ≈39 slots lost. Zero execution rows exist between 10:58:20 and 16:37:21: PVE host pve rebooted (boot 2026-10-30 16:24:00, ssh … uptime -s), CT460 came back 16:28:00, the previous gateway life (pid=3913841, started 2026-10-28T17:21:58Z) was logged as exited UNCLEANLY — "no exit path ran — SIGKILL / OOM / VM death" at 16:35:50, and the scheduler resumed with an 8-run burst at 16:37:21–16:38:22. Root cause not established (no OOM evidence; that ledger string is a candidate list, not a diagnosis) → incidents/2026-10-30-ct460-host-outage-scheduler-gap. ea79a2107e36 (Mnemosyne Health Monitor) is ENABLED AGAIN and clean: its 00:01:08 slot did not die at 01:04:22 — it closed completed at 10:02:48 after 10 h 01 m 40 s and delivered, auto-closing incident ea79a2_d93c3acdc4f2 resolved at 2026-10-30T10:02:48.662+08:00; live last_status: ok, failure_streak: 0. Its 12:01 slot was lost to the outage and is the still-running 16:37:21 catch-up. Non-clean last_status 4 → 3, failing (run failed outright) 1 → 0, unclosed incident rows 2 → 1 (survivor: 0e753e_e7f8194a7d06, due 11-01). deliveries.db finished 2026-10-30 with 8 rows, all delivered (latest 18:46:04). 2026-10-31 opened clean (5 runs, all completed).
  • ⚠ Superseded by the 2026-10-31 entry above — Cron as of 2026-10-30 03:3x AWST (live jobs.json + executions.db) — the 10-29 clean day held, but the fleet changed shape overnight: the Mnemosyne Health Monitor was auto-paused and its failure notice could not be delivered. Fleet now 29 jobs (25 enabled / 4 disabled) — ea79a2107e36 (Mnemosyne Health Monitor) was auto-paused at 2026-10-30 01:04:22 after its 00:01:08 slot closed unknown on "Cron execution unverified; reconcile ~/.hermes/cron/admissions/ad897cbc9f…json: Gateway connection failed; no local fallback", which opened incident ea79a2_d93c3acdc4f2 (failure_type: unknown, state detected). Non-clean last_status is now 4 — the three long-standing survivors (0e753e9720fd error since 2026-10-01, fad5a86979d4 and 066bb5191280 delivery_failed) plus ea79a2107e36 (error, failure_streak: 1) — so the standing "every failure_streak reads 0" claim is now false. Window 2026-10-29 settled 177 completed / 0 failed (the page previously said 176 — the 23:59 Wiki Audit row cbfe434a3b5e had still been running at audit time and has since settled completed); 2026-10-28 stands at 169 / 2 (also +1, same cause). Unclosed incident rows 1 → 2: dca848_e93a538ade6b and dca848_0a2f03b5952b both closed resolved at 10:16:12, leaving the Facebook monthly's 0e753e_e7f8194a7d06 (detected, cannot clear before its 2026-11-01 slot) plus the new ea79a2_d93c3acdc4f2. Delivery health: 10 deliveries on 10-29, 10/10 delivered, 0 failures (one to topic 1030 — the radar report); zero delivery rows were created on 2026-10-30, because the Mnemosyne Health Monitor's own failure notice failed to send (send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed) — a failure notice that was itself never delivered. The radar's two-stage split held its ground: dca8482e4f76 (stage 1) and 5fc292ff0f3b (stage 2) both read ok with streak 0, and every previously-red row (1b23c042e17d, ecc1fc13655a, 08eb86f02d37, 85edf6772bca, 0e519874d280, c0a9fa314542) has since cleared to ok on its own scheduled slot — the page's older per-row notes still claimed error for several of them. Pins: 18 of the 25 enabled jobs pin custom:omni-101 (11 smart-route, 6 free-stack, 1 deepseek-paid-smart); the other 7 enabled rows are no_agent jobs with no pin.
  • Prior window — as of 2026-10-30 00:0x AWST (superseded by the 2026-10-30 03:3x entry above) (live jobs.json + executions.db) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week. Fleet unchanged at 29 jobs (26 enabled / 3 disabled); non-clean last_status 5 → 3, and every failure_streak read 0 — the three survivors are all stale re-runs of long-standing rows (0e753e9720fd error since 2026-10-01, fad5a86979d4 and 066bb5191280 delivery_failed). Window 2026-10-29 settled 176 completed / 0 failed (+ this audit running) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at 168 / 2. Unclosed incident rows 4 → 1: dca848_e93a538ade6b and dca848_0a2f03b5952b both closed resolved at 10:16:12, leaving only the Facebook monthly's 0e753e_e7f8194a7d06, which cannot clear before its 2026-11-01 slot. Delivery health: 10 deliveries on 10-29, 10/10 delivered, 0 failures (one to topic 1030 — the radar report). dca8482e4f76 (stage 1) ran 07:00:59 → 10:16:12 (3 h 15 m 13 s), closed completed, and wrote ~/.hermes/radar-ref/handoff/2026-10-29.json — 170 discovered / 33 duplicates avoided / 37 verified, run_date 2026-10-29T08:23:51+08:00, sources seek ok, linkedin ok, indeed blocked, wa_gov ok — opening no new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). 5fc292ff0f3b (stage 2) fired 10:13:46, consumed the handoff 10:40:59, closed 10:44:48 completed and delivered the same second: 20 pages / 20 page_ids / 2 cover letters / 2 same-day strong matches / verified_cover_letter_count 2 / missing_cover_letter_count 0; the Notion board went 455 → 475. Both later slots no-oped as designed (11:30 → 10 m 46 s, 14:30 → 1 m 51 s). Stage 1 correctly no longer delivers. See incidents/2026-10-27-job-radar-budget-exhausted and incidents/2026-10-28-job-radar-stage1-iteration-cap, now both resolved. Pins unchanged — 19 of the 26 enabled jobs pin custom:omni-101 (12 smart-route, 6 free-stack, 1 explicit).
  • Prior window — as of 2026-10-29 00:3x AWST (superseded by the 2026-10-30 entry above) (live jobs.json + executions.db) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped. Fleet unchanged at 29 jobs (26 enabled / 3 disabled); non-clean last_status 5 → 5; unclosed incident rows 2 → 4. Window 2026-10-28 settled 168 / 2, and both failures are in-window. dca8482e4f76 (Job Radar stage 1) ran 07:00:32 → 10:33:18 (3 h 32 m 46 s), closed failed, failure_streak: 2, on "hit the agent iteration cap before the handoff could be written"; ~/.hermes/radar-ref/handoff/ holds only 2026-10-27.json, and stage 2 (5fc292ff0f3b) fired 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43, both completed / suppressed — the no-op path, confirmed in production → incidents/2026-10-28-job-radar-stage1-iteration-cap. The other failure is the 03:00 stocktake (1a8e90ea2271): 03:00:28 → 03:37:58 (37 m 30 s), failed, streak 1, delivery suppressed, on RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203 — the provider could not deserialize the model's own 512 KB final response. It filed incident 1a8e90_f57ab4cafab8 in state detected and never alerted, and left systems/skills-index.md modified and raw/stocktake/freshness-20261028.txt untracked; the 2026-10-29 audit committed both. Steel left the stack in-window — and was then fully removed: browser.cloud_provider/camera_provider are back to camofox (config mtime 2026-10-29 00:15:13), the browser-steel plugin was deleted (backup config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z written 00:13:24), and by the end of 2026-10-29 the Steel deployments on CT201 (containers, images, network, project dir, and the Serve routes /steel and /steel-api) and CT333 (container + image) were removed entirely — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. STEEL_BASE_URL is no longer in ~/.hermes/.env; it survived only in the inherited environment of processes already running at the cleanup, and the gateway was restarted on 2026-10-29 01:21 AWST to clear it → systems/browser-backend, decisions/rollback-steel-to-camofox, comparisons/steel-vs-camoufox. 2026-10-27 settled 173 / 1 (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). Pins: 12 smart-route / 6 free-stack / 7 unpinned / 1 deepseek-paid-smart.
  • Prior window — as of 2026-10-28 00:2x AWST (live jobs.json + executions.db) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: 172 completed / 1 failed (+ this audit running). The single failure is dca8482e4f76 (Daily Marketing Job Radar - Discovery), which ran 07:01:08 → 11:03:54 (4 h 2 m 46 s), closed failed, and delivered a report that opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery/verify/dedup/classify finished, Notion ingestion, cover-letter generation and the final audit did not run. Incident row dca848_e93a538ade6b opened 11:03:53, alerted 11:03:59, still open (failure_type: auth, which does not describe a budget fault) → incidents/2026-10-27-job-radar-budget-exhausted. Response, same day: the radar was split into a two-stage pipeline — stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5, deliver: local, failures → topic 1030) and new stage 2 5fc292ff0f3b (Daily Marketing Job Radar - Write and Letters, 30 11,14 * * 1-5, Telegram topic 1030, created 2026-10-27T17:34:15; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both completed, delivery suppressed, last_status: ok) → runbooks/job-radar. Fleet 28 → 29 jobs (26 enabled / 3 disabled). Unclosed incident rows 3 → 2: the two rows the 2026-10-26 planned restart raised (ea79a2_a4661554a3c2, ea79a2_c3ab2d784ba3) auto-closed 00:38:32, and the radar row opened. Non-clean last_status 4 → 5 (churn): ea79a2107e36 cleared to ok; dca8482e4f76 joined as error (streak 1) and 08eb86f02d37 re-joined as delivery_queued — still non-clean: 0e753e9720fd (error), fad5a86979d4 (delivery_failed), 066bb5191280 (delivery_failed). Pins unchanged. The 2026-10-26 orphan class did not recur — every 10-27 run except the radar's completed. (superseded by the 2026-10-29 entry above)
  • Prior window — as of 2026-10-27 00:1x AWST (superseded by the 2026-10-28 entry above) (live jobs.json + executions.db) — a planned restart interrupted a live job, and the fleet got a loud failure instead of a silent one. Window 2026-10-26 settled 169 completed / 3 failed, and all three failures are a single fault: ea79a2107e36 (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → incidents/2026-10-26-planned-restart-orphaned-cron-run. 2026-10-25 settled 172 / 2. Fleet unchanged: 28 jobs — 25 enabled / 3 disabled (d40085da631c Important-mail monitor, f77b7e46bd09 Rhino AgentMail Inbox Watcher, 0a7f933d06e2 AgentMail Rhino Inbox Poll). Non-clean last_status 5 → 4: c0a9fa314542 (GitHub Trending Radar) and 08eb86f02d37 (Gmail Action Digest) both cleared to ok, while ea79a2107e36 joins as error, streak 2; 0e753e9720fd (error) and fad5a86979d4 + 066bb5191280 (delivery_failed) unchanged. Unclosed incident rows 2 → 3: both ea79a2_* rows — 14:33:37 "Cron execution unverified; reconcile admissions/724a69108b…json" and 14:35:59 "lost its durable fire claim ownership" — are still alerted, alongside 0e753e_e7f8194a7d06 (detected since 2026-10-01). One row closed: c0a9fa_764558b5ae83 auto-closed 09:13:00 when the radar's 09:00 slot ran clean — every row of the 2026-10-16 503 resource-pressure incident is now closed → incidents/2026-10-16-omniroute-503-resource-pressure. Pins unchanged: 11 custom:omni-101 / smart-route, 6 custom:omni-101 / free-stack, 2 custom:omni / smart-route, 1 custom:omni-101 / deepseek-paid-smart, 8 unpinned (global default still custom:omni-101 / smart-route). One gateway start on 2026-10-26 (14:35:18, after a planned SIGTERM at 14:33:35; teardown 54.06 s, .clean_shutdown marker skipped) → systems/hermes-agent. ⚠ fad5a86979d4's 03:08 slot completed but delivered delivery_failed — Thread 34795 not found, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ The radar's 07:00 slot closed completed at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted” — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model deepseek-paid-smart → runbooks/job-radar. ⚠ Off-cron, unchanged: Nous Portal credentials still invalid → incidents/2026-10-22-nous-portal-refresh-token-invalid. Sessions in window: 17 (15 cron, 1 ios, 1 telegram).
  • Current as of 2026-10-26 00:1x AWST (superseded by the 2026-10-27 entry above) (live jobs.json + executions.db) — fleet is clean, but one job misbehaved badly and the global model default moved back. Window 2026-10-25 settled 172 completed / 2 failed; both failures are 08eb86f02d37 (Gmail Action Digest), which also fired six times that day → incidents/2026-10-25-gmail-digest-runaway-and-refire. 2026-10-24 settled 168 / 0 (the 2026-10-25 entry read 167 — its day was still open). Fleet unchanged: 28 jobs — 25 enabled / 3 disabled; all 28 IDs still on this page. Unclosed incident rows still 2 (0e753e_e7f8194a7d06 detected since 2026-10-01; c0a9fa_764558b5ae83 alerted since 2026-10-19) — nothing opened or closed in-window, so neither 10-25 failure raised an incident row. Non-clean last_status 4 → 5: 08eb86f02d37 joins as delivery_queued with an empty last_error (never resolved to ok), alongside 0e753e9720fd (error), fad5a86979d4 + 066bb5191280 (delivery_failed) and c0a9fa314542 (error, streak 1). Pins unchanged: 11 custom:omni-101 / smart-route, 6 custom:omni-101 / free-stack, 2 custom:omni / smart-route, 1 deepseek-paid-smart, 8 unpinned. ⚠ Global default reverted to OmniRoute on 2026-10-24 19:06 (custom:omni-101 / smart-route), so the 8 unpinned jobs are back on the same upstream as the pinned ones — the fleet is a single path again → systems/current-profile. ⚠ Three gateway starts on 2026-10-25 (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → systems/hermes-agent. ⚠ Nous Portal credential still invalid (Invalid refresh token) — context summaries unavailable on 10-25 (20 log lines); all 8 auxiliary slots were moved to no-think/freeapi/auto on custom:omni-101 at 19:13 → incidents/2026-10-22-nous-portal-refresh-token-invalid. Sessions in window: 21 (18 cron — 15 smart-route, 2 free-stack, 1 deepseek-paid-smart; 1 desktop; 1 oneshot; 1 telegram).
  • Current as of 2026-10-25 00:1x AWST (jobs.json.updated_at 2026-10-25T00:12:26+08:00) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's. Window 2026-10-24: 167 completed / 0 failed (+ this audit running); 2026-10-23 settled at 170 / 0 (169 at the 10-24 audit). That makes 10-22 → 10-24 three clean days in a row. Fleet unchanged: 28 jobs — 25 enabled / 3 disabled. Unclosed incident rows still 2 and nothing opened or closed in-window: c0a9fa_764558b5ae83 (alerted since 10-19, next slot Mon 10-26 09:00) and 0e753e_e7f8194a7d06 (detected since 10-01, next slot 11-01). Non-clean last_status still 4: c0a9fa314542 (error, streak 1 — Mon 10-26), 0e753e9720fd (error, 0 — 11-01), fad5a86979d4 and 066bb5191280 (delivery_failed — Mon 10-26 / 11-01). Pins unchanged: 11 custom:omni-101 / smart-route, 8 unpinned (global default), 6 custom:omni-101 / free-stack, 2 custom:omni / smart-route, 1 custom:omni-101 / deepseek-paid-smart (this audit). 14 sessions in the window: 11 cron + 3 desktop — two of the desktop ones are the 12:53–12:56 call requests that the agent refused → incidents/2026-10-24-conduit-call-request-refused. ⚠ New off-cron fault, open: the agent told Anthony it cannot make phone calls, three times, with zero tool calls, while conduit_call_user was one tool_search away → same incident page. (Superseded by the 2026-10-26 entry above.)
  • Current as of 2026-10-24 00:2x AWST (superseded by the 2026-10-25 entry above) (jobs.json.updated_at 2026-10-24T00:17:36+08:00) — a clean day, and the 2026-10-09 timeout storm is finally closed. Window 2026-10-23: 169 completed / 0 failed (+ this audit running) — the third zero-failure day in four; 2026-10-22 settled at 173 / 0. Fleet unchanged: 28 jobs — 25 enabled / 3 disabled. 1b23c042e17d (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its last two incident rows (1b23c0_c35703f637b8 + 1b23c0_103cbe04a79b) — all 12 rows of the 10-09 storm are closed and that job reads ok again → incidents/2026-10-09-cron-provider-timeout-storm. Unclosed incident rows 4 → 2: c0a9fa_764558b5ae83 (alerted since 10-19, next slot Mon 10-26 09:00) and 0e753e_e7f8194a7d06 (detected since 10-01, next slot 11-01). Non-clean last_status 5 → 4: c0a9fa314542 (error, streak 1 — Mon 10-26), 0e753e9720fd (error, 0 — 11-01), fad5a86979d4 and 066bb5191280 (delivery_failed — Mon 10-26 / 11-01). Pins unchanged: 11 custom:omni-101 / smart-route, 8 unpinned (global default), 6 custom:omni-101 / free-stack, 2 custom:omni / smart-route, 1 custom:omni-101 / deepseek-paid-smart (this audit). 14 sessions in the window: 13 cron + 1 desktop (18:26:12 — the Conduit phone-call session) → systems/conduit. The stocktake's 10-22 overrun did not repeat: its 10-23 03:07 slot ran and committed (250c735: freshness 2 stale / 94 fresh, skills-index 306). → systems/skills-index
  • Current as of 2026-10-23 00:1x AWST (superseded by the 2026-10-24 entry above) (jobs.json.updated_at 2026-10-23T00:07:25+08:00) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing. Window 2026-10-22: 172 completed / 0 failed (+ this audit running) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. A DB recount by started_at and by claimed_at puts 2026-10-21 at 168 / 1, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. Fleet unchanged: 28 jobs — 25 enabled / 3 disabled. Five incidents closed in-window and none opened — 0e5198_be2291809e56 + 0e5198_ec2fac98a91d (0e519874d280, closed 08:01:40), 06a0a1_9e0ff4eedc49 (10:14:34), c8d529_e8c70250f9be (18:17:59), dca848_014affeb6954 (07:59:22) — so unclosed rows fall 9 → 4 (3 alerted / 1 detected: 1b23c0_c35703f637b8, 1b23c0_103cbe04a79b, c0a9fa_764558b5ae83, 0e753e_e7f8194a7d06). Non-clean last_status falls 9 → 5: 1b23c042e17d (error, streak 2 — next slot Fri 2026-10-23 08:00), c0a9fa314542 (error, 1 — Mon 10-26), 0e753e9720fd (error, 0 — 11-01), fad5a86979d4 (delivery_failed — Mon 10-26 03:00), 066bb5191280 (delivery_failed — 11-01). All four cleared rows had been red since 2026-10-15 (the omni-101 503 burst) and each ran clean on its first slot of 10-22. Pins: 11 custom:omni-101 / smart-route, 8 unpinned (provider/model null → global default), 6 custom:omni-101 / free-stack, 2 custom:omni / smart-route, 1 custom:omni-101 / deepseek-paid-smart (this audit). ⚠ Stocktake anomaly: 1a8e90ea2271 ran 03:00:28 → 04:38:19 (97 m 58 s) against 16 m 01 s on 10-21, finished completed with delivery suppressed — and committed nothing. It left its regenerated systems/skills-index.md uncommitted (mtime 10-22 03:00, with a ## .Archive (41) section reading 347) and its diagnosed fixes unapplied → incidents/2026-08-15-skills-index-drift. 27 sessions in the window: 15 cron, 3 subagent (the AI-Newsletter research fan-out), 6 desktop (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models mimo-v2.5, or-inclusionai_ling-2_6-1t, minimax/minimax-m3.1-flash), 2 acp (0 messages), 1 oneshot memory-capture probe. → systems/skills-index, incidents/2026-08-15-skills-index-drift
  • Current as of 2026-10-22 00:1x AWST (superseded by the 2026-10-23 entry above) (jobs.json.updated_at 2026-10-22T00:11:31+08:00) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen. Window 2026-10-21: 171 completed / 1 failed (+ this audit running). The single failure is dca8482e4f76 (Daily Marketing Job Radar, 07:00 weekday, custom:omni-101 / smart-route), which died at 07:40:48 after 40 m 36 s on RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s — not the 503 resource-pressure string, which appears in no 2026-10-21 row → incidents/2026-10-21-omniroute-pre-dispatch-filter-skip. Fleet unchanged: 28 jobs — 25 enabled / 3 disabled. Non-clean last_status stays at 9 (7 error / 2 delivery_failed) but the composition moved: 08eb86f02d37 cleared (its 08:30 slot ran clean; incident 08eb86_472c7b609c80 resolved 2026-10-21 08:37:21) while dca8482e4f76 re-entered (streak 1). Survivors, with next run: dca8482e4f76 (1, 10-22 07:00), 0e519874d280 (2, 10-22 08:00), 06a0a1c8704f (1, 10-22 10:00), c8d52904e1d6 (1, 10-22 18:00), 1b23c042e17d (2, Fri 10-23 08:00), c0a9fa314542 (1, Mon 10-26 09:00), fad5a86979d4 (delivery_failed, Mon 10-26 03:00), 0e753e9720fd (error, 0, 11-01 09:00), 066bb5191280 (delivery_failed, 11-01 02:00). Incident ledger: 1 opened / 1 closed, unclosed stays 9 (8 alerted / 1 detected). Pins: 17 of the 28 jobs pin custom:omni-101 (10 smart-route, 6 free-stack, 1 deepseek-paid-smart), 7 carry no pin at all (provider/model null → global default), 1 pins custom:omni / smart-route. 2026-10-20 settled at 169 completed / 1 unknown — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. 12 sessions in the window, all cron — no interactive sessions. ⚠ Infrastructure changed in-window: Hermes Agent was updated to 0.22.3 at 22:47 → 22:57 with its pre-update snapshot skipped → systems/hermes-agent, infrastructure/backups.
  • ⚠ Current as of 2026-10-21 00:0x AWST (superseded by the 2026-10-22 entry above) (jobs.json.updated_at 2026-10-21T00:07:08+08:00) — a clean day, one orphaned run, and one long-running silent failure. Window 2026-10-20: 168 completed / 0 failed (plus 1 unknown and this audit) — the first zero-failure day since 2026-10-12, ending a seven-day run of failures, and the omni-101 503 fault did not recur (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). Zero incident rows opened and 4 closed (dca848_867d9caa96a8, 85edf6_256c5e150bc4, ecc1fc_c9976ea24553, ecc1fc_21a72aa11cc4), so unclosed rows fall 13 → 9 (8 alerted / 1 detected). ⚠ The exception is a failure this ledger does not record: 08eb86f02d37 (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed unknown when the gateway exited uncleanly — “Scheduler restarted after this execution's owner exited before a durable terminal state”; no delivery row exists for the slot and no incident row was raised → incidents/2026-10-20-gateway-unclean-exit-orphaned-run. Non-clean last_status 12 → 9 (7 error / 2 delivery_failed); every failure_streak ≤ 2. Fleet unchanged: 28 jobs — 25 enabled / 3 disabled. ⚠ Discovery — green row, dead job: e27da6c34bbb (Config Backup, 11:00) reads ok while its Gitea push has failed every night since 2026-08-08 (75 unpushed commits) → incidents/2026-08-08-gitea-config-backup-push-dead. Model pins unchanged — 20 of 28 jobs still pin OmniRoute — but the 8 unpinned jobs moved upstream to nous direct with the global default change, so the fleet now has two upstream paths. 2 interactive sessions in the window (talaria-ios tax work; a desktop turn that received the canned non-answer). → incidents/2026-10-16-omniroute-503-resource-pressure
  • Current as of 2026-10-20 00:09 AWST (superseded by the 2026-10-21 entry above) — the omni-101 503 fault is BACK, and it is now the fleet's defining risk. Window 2026-10-19: 159 completed / 13 failed, against 10-18's 168 / 1 and 10-17's 163 / 5. 9 genuine model-call failures across 8 jobs all carry the identical RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly. on custom:omni-101, in two clusters — 06:01 (one job) and 07:01 → 12:01 (seven more runs across seven jobs). The other 4 rows are derivative Mnemosyne Watchdog exits (96eb95ce12f2 at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the second burst in four days on the same node with the same blast radius, while /api/health read 200 {"status":"ok"} throughout — so incidents/2026-10-16-omniroute-503-resource-pressure is back to status: active (it had been downgraded to monitoring 44 h earlier). Fleet unchanged at 28 jobs — 25 enabled / 3 disabled. Non-clean last_status 7 → 12 (10 error + 2 delivery_failed); unclosed incident rows 13 (12 alerted / 1 detected) — 4 opened and 2 closed on 10-19. Every failure_streak still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. 15 sessions in the window, all cron — no interactive sessions. → incidents/2026-10-16-omniroute-503-resource-pressure
  • Current as of 2026-10-19 00:20 AWST (superseded by the 2026-10-20 entry above) (jobs.json.updated_at 2026-10-19T00:17:06+08:00) — one failure in 24 h, and it was a failure mode this wiki had not recorded before. Fleet 28 jobs — 25 enabled / 3 disabled (unchanged). Window 2026-10-18: 168 completed, 1 failed — the failure is ecc1fc13655a (FreshRSS Midday, 12:01:10 → 12:08:25), which wrote and delivered a canned “Your request was not processed…” sentence to Telegram (delivery_outcome: delivered at 12:08:11) before dying on RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response. → incidents/2026-10-18-degenerate-stream-canned-substitute. Non-clean last_status: 8 → 7 — 1a8e90ea2271 (10-18 03:01) and cbfe434a3b5e (this audit, 10-18 00:42) cleared on schedule, leaving 1b23c042e17d (error, streak 2, next Fri 10-23 08:00), 0e519874d280 (Mon 10-19 08:00 — next to fire), c8d52904e1d6 + 06a0a1c8704f (Thu 10-22), ecc1fc13655a (10-19 12:00), and the monthlies 0e753e9720fd + 066bb5191280 (both 11-01). Open incidents 8 → 7 (6 alerted / 1 detected); 1 opened, 2 resolved in-window. Tallies: 10-17 = 163 / 5, 10-18 = 168 / 1, 10-19 so far 5 / 0 (+ this audit running); 11 sessions in the window, all cron — no interactive sessions. → incidents/2026-10-18-degenerate-stream-canned-substitute
  • Current as of 2026-10-18 00:31 AWST (jobs.json.updated_at 2026-10-18T00:31:17+08:00) — the omni-101 503 burst is over and the fleet is unwinding normally. Fleet 28 jobs — 25 enabled / 3 disabled. 8 carry a non-clean last_status (down from 12 at the 10-16 storm peak): 1b23c042e17d (error, streak 2), cbfe434a3b5e (1), 1a8e90ea2271 (1), 0e519874d280 (1), 06a0a1c8704f (1), c8d52904e1d6 (1), 0e753e9720fd (error, streak 0), 066bb5191280 (delivery_failed). Four rows cleared as their slots came round: b63ed5817212 (10-17 06:01), 08eb86f02d37 (10-17 08:31), ecc1fc13655a and ea79a2107e36 (both 10-17 12:02). Open incidents 12 → 8 (5 alerted / 3 detected); 9 incidents opened and 6 closed inside this window. Daily tallies: 10-16 = 165 completed / 8 failed, 10-17 = 162 / 5 (all five before 03:01; DB re-read 2026-10-20: 163 / 5 — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. on custom:omni-101, 10-16 08:00 → 10-17 03:00. → incidents/2026-10-16-omniroute-503-resource-pressure
  • ⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery. 12 non-clean last_status (was 3 at the 10-13 audit): 9 new rows from the omni-101 outage — 08eb86f02d37 (streak 2), ecc1fc13655a (2), dca8482e4f76, c8d52904e1d6, 06a0a1c8704f, 85edf6772bca, 0e519874d280, 1a8e90ea2271, cbfe434a3b5e (all streak 1) — plus the three long-standing rows 1b23c042e17d (since 10-09), 0e753e9720fd (10-01) and 066bb5191280 (delivery_failed, 10-01). Three jobs already cleared inside this window: b63ed5817212 (10-16 00:02), 1b31187397a5 (10-15 22:01), ea79a2107e36 (10-16 00:09). Structural finding: all 18 enabled model-calling jobs are pinned to custom:omni-101 — the cron fleet has no model-call failover. → incidents/2026-10-14-omniroute-node-connection-refused-storm
  • Total jobs: 29 — 26 enabled, 3 disabled (re-verified from live ~/.hermes/cron/jobs.json at 2026-10-31 00:3x AWST). ⚠ Superseded reading: the 2026-10-30 03:3x audit counted 25 enabled / 4 disabled; the 4th disabled job was ea79a2107e36 (Mnemosyne Health Monitor), and it is enabled again as of 2026-10-30 10:02:48 (last_status: ok, streak 0) — so every "4th disabled job" statement below is superseded. The job count itself is unchanged at 29. The count fell 30 → 29 on 2026-10-10 (redundant DHL monitor 62619aa6bcb3 retired) and 29 → 28 when Mosquito Tracker 1b31187397a5 was removed — its last execution was 2026-10-16 15:00 (completed, delivered) and it is absent from jobs.json as of this audit. See Removed Jobs. The 4th disabled job is ea79a2107e36 (Mnemosyne Health Monitor), auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing d40085da631c (Important-mail monitor), f77b7e46bd09 (Rhino AgentMail Inbox Watcher) and 0a7f933d06e2 (AgentMail Rhino Inbox Poll).
  • Non-clean last_status: 4 as of 2026-11-01 00:2x AWST — 2 error + 2 delivery_failed (ea79a2107e36 error / failure_streak: 2 — the 18:06 reload orphan; 0e753e9720fd error / 0; fad5a86979d4 and 066bb5191280 delivery_failed / 0). ⚠ Superseded: the 2026-10-31 00:3x reading was 3 — 1 error + 2 delivery_failed; before that, the 2026-10-30 03:3x reading was 4 (2 error + 2 delivery_failed); its first entry, ea79a2107e36 (Mnemosyne Health Monitor, error, failure_streak: 1), has cleared to ok / 0 — its 00:01:08 slot was still running, not failed, and closed completed at 10:02:48. The job list that follows is the 03:3x snapshot. Live jobs.json (authoritative for the per-job counters): ea79a2107e36 (Mnemosyne Health Monitor — error, failure_streak: 1, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident ea79a2_d93c3acdc4f2 open detected), 0e753e9720fd (Facebook Birthday Monthly Scrape — error, streak 0, next slot 11-01 09:00), fad5a86979d4 (Facebook Birthday Weekly Registry Sync — delivery_failed, streak 0, most recent failure 2026-10-26 03:08:15) and 066bb5191280 (Uber Eats Monthly Spend Report — delivery_failed, streak 0, next slot 11-01 02:00). History: 3 (10-12 audit) → 12 (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → 4 now, the increase being the Mnemosyne Health Monitor's overnight gateway fault.
  • Failing (last run failed outright): 1 as of 2026-11-01 00:2x AWST — ea79a2107e36 (Mnemosyne Health Monitor), failure_streak: 2, last run 2026-10-31 12:01:06 → 18:39:15 (orphaned by the 18:06:41 gateway reload). ⚠ Superseded: the 2026-10-31 00:3x reading was 0; before that, the 2026-10-30 03:3x reading was 1 — ea79a2107e36 (Mnemosyne Health Monitor), failure_streak: 1 — and that verdict was the reconciliation path, not the run: the slot closed completed at 10:02:48 after 10 h 01 m 40 s and the counters reset. Only delivery-only failures remain (2). Its 2026-10-30 00:01:08 slot closed unknown at 01:04:22 on "Cron execution unverified; reconcile ~/.hermes/cron/admissions/ad897cbc9f…json: Gateway connection failed; no local fallback", and the job was auto-paused at that moment — it is now the 4th disabled job. ⚠ Its failure notice was never delivered: hermes cron list reports "The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed", and deliveries.db holds zero rows for 2026-10-30. A failure notice that fails to send is the same green-row/dead-job trap as incidents/2026-08-08-gitea-config-backup-push-dead — the incident row ea79a2_d93c3acdc4f2 is the only trace. The other previously-red jobs (1b23c042e17d, ecc1fc13655a, 08eb86f02d37, 85edf6772bca, 0e519874d280, c0a9fa314542) have all cleared to ok with streak 0 on their own scheduled slots.
    • Historical (2026-10-01 → 2026-10-05), now all remediated: dca8482e4f76 403 MODEL_NOT_IN_PLAN ×7; 1a8e90ea2271 HTTP 410 ×11 (retired qwen3.5:397b); fad5a86979d4 HTTP 400 ×3; 0e753e9720fd HTTP 400 ×1; 06a0a1c8704f, 1b23c042e17d and 79f9bb41ecf5 HTTP 401 (the cliprox/terra connection auth expiry, settled 3/3 on 2026-10-04).
  • Date-bucketing note (added 2026-10-25): executions.db stores started_at with a +08:00 offset, and SQLite's date() normalises to UTC — so date(started_at) moves every row that ran 00:00–07:59 AWST onto the previous day. Daily tallies on this page are bucketed with substr(started_at,1,10), which preserves AWST calendar days. A recount via date() reads low by roughly a day's early-morning runs; this is the same class of drift that put 10-21 at 168 instead of 171.
  • Failing (delivery only): 1 — fad5a86979d4 (Facebook Birthday Weekly Registry Sync — the same dead-thread cause, most recently 2026-10-26 03:08:15). It carries failure_streak: 0 because the run succeeded; only the delivery failed. ✅ 066bb5191280 (Uber Eats Monthly Spend Report) was remediated 2026-11-01 — retargeted from the bare telegram target to telegram:1793951355; the 2026-10 report generated and was delivered on its 2026-11-01 02:00 slot.
  • Repin applied 2026-10-05: free-stack ×6 (06a0a1c8704f, 1b23c042e17d, 79f9bb41ecf5, 1a8e90ea2271, fad5a86979d4, 0e753e9720fd) and smart-route ×1 (dca8482e4f76), all on custom:omni-101. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. free-stack is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs.
  • Verified live, not assumed: 79f9bb41ecf5 succeeded on a manual run 2026-10-05 23:39 and 06a0a1c8704f at 23:45 (both incidents auto-resolved); dca8482e4f76 completed 23:56 and delivered to telegram:-1004321904721:1030 (incident resolved 2026-10-06 00:05:07); and 1a8e90ea2271 completed automatically on 2026-10-06 03:00. Four of the seven are therefore verified by live run. The remaining three (1b23c042e17d, fad5a86979d4, 0e753e9720fd) are repinned but not yet re-run — they are due on their normal schedules. 1a8e90ea2271 has since run twice more on schedule (2026-10-06 03:00 → 33f4e95; 2026-10-07 03:00 → 8c63ef2), so the stocktake repin is verified across consecutive automatic runs.
  • Streak counting note: the N failures in a row figure in hermes cron list comes from the failure_streak counter in jobs.json. executions.db retains only a rolling window of rows — the live floor is now 2026-10-24 09:22:36 (1,001-row cap), so DB-based recounts undercount older streaks and cannot reproduce tallies for days before that floor. Quote the jobs.json counter as authoritative and state the floor whenever a DB recount is cited. Live counters as of 2026-10-30 03:3x AWST: ea79a2107e36 = 1; every other job = 0.
  • Re-verified 2026-10-06 AWST: job count unchanged at 30 (27 enabled / 3 disabled). 066bb5191280 and 0e753e9720fd still carry the bare-telegram target warning (configured thread_id 34795 for telegram:1793951355 was not found).
  • Re-verified 2026-10-07 AWST: job count unchanged at 30 (27 enabled / 3 disabled); 0 failing — all 30 failure_streak counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 completed, 0 failed, 1 running (the audit itself); 14 sessions, all cron — no interactive (non-cron) sessions. The stocktake job 1a8e90ea2271 ran on schedule at 03:00 and regenerated systems/skills-index (344 → 346 skills) plus raw/stocktake/freshness-20261006.txt, committing to Gitea as 33f4e95 — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin last_status: error (1b23c042e17d, fad5a86979d4, 0e753e9720fd).
  • A stale notice, not a live outage, drove this review. The interactive session on 2026-10-05 23:16 began because Anthony replied to a 85edf6772bca failure notice that had been sitting in topic 346 since the 2026-09-29 07:00 run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already resolved on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check executions.started_at (and the cron_incidents state) before acting on a notice.
  • Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00): job count unchanged at 30 (27 enabled / 3 disabled); 0 failing — every failure_streak reads 0. Window tally (executions.db): 176 completed, 1 failed, 1 running (the audit itself). The single failure is 62619aa6bcb3 (DHL Tracking, 08:00) — a Camofox HTTP read timeout (ensure_browser() → TimeoutError: timed out), notified 08:00:58 and auto-resolved 11:01:13 when the next 3-hourly run succeeded (incident 62619a_f485ccfd4a0d). 13 sessions started in the window, all source = cron — no interactive sessions; 12 deliveries, 0 delivery failures. The same three stale pre-repin last_status: error jobs remain (1b23c042e17d, fad5a86979d4, 0e753e9720fd); the stocktake job 1a8e90ea2271 ran on schedule at 03:00 for the second consecutive day.
  • Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, 0 failing (2026-10-06) → 30, 0 failing (2026-10-07) → 30, 0 failing (2026-10-08) → 29, 0 failing (2026-10-10) → 29, 0 failing (2026-10-11) → 29, 0 failing (2026-10-12)
  • Re-verified 2026-10-12 AWST (window 2026-10-11 00:00 → 2026-10-12 00:00): 29 jobs (26/3), 0 failing, 5 non-clean last_status; window tally 172 completed, 0 failed (the 2026-10-11 day finalised one run higher than the previous audit's snapshot, which caught it mid-flight at 171 + 1 running). 18 sessions — 17 cron + 1 interactive.
  • Re-verified 2026-10-13 AWST (window 2026-10-12 00:00 → 2026-10-13 00:00): 29 jobs (26/3), 0 failing, non-clean last_status down to 3. Window tally 174 completed, 0 failed — third consecutive clean day (10-10 175/0, 10-11 172/0, 10-12 174/0). 5 open incidents → 2: three closed on 2026-10-12 (see #Resolved 2026-10-12). 18 sessions in the window, all cron, no interactive sessions; 0 new incident pages required. Two findings led the window: the GitHub Trending Radar delivery token is dead (getMe → 401, live-verified) and the Daily Marketing Job Radar run hit Camofox memory pressure (2.5 GB peak, heavy swap, self-recovered by restart).

Job Table

Name ID Schedule Delivery Skills Script Notes
AgentMail Rhino Inbox Poll 0a7f933d06e2 * * * * * origin terminal agentmail-poll.py disabled since 2026-07-27 (legacy)
AI Newsletter Curation for Patrick (Weekly) 06a0a1c8704f 0 10 * * 4 telegram:-1004321904721:346 — — ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was custom:Omniroute:20128 / cliprox/terra, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live last_status: ok, streak 0
Daily Birthday Checker a432898b55a4 30 7 * * * telegram:1793951355 — birthday_check.py
Daily Marketing Job Radar — Discovery (stage 1) dca8482e4f76 0 7 * * 1-5 local (failures → topic 1030) job-search-automation — ✅ Repinned 2026-10-05 → custom:omni-101 / smart-route (was command-code / gpt-5.6-terra, 403 MODEL_NOT_IN_PLAN ×7). ✅ First scheduled success on the repin 2026-10-12 07:00 (finished 07:33:50, delivered) — its 2026-10-09 timeout incident dca848_23fcd2b18aaa auto-closed 07:33:41. ⚠ The 2026-10-12 run hit Camofox memory pressure — the service reached its containment ceiling (2.5 GB peak, MemoryHigh=2G, swap 1 GB fully consumed, available: 0B), so /tabs calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ The 2026-10-19 07:01 slot failed on the omni-101 503 (new incident dca848_867d9caa96a8, alerted) — the second time this job has been caught by that node's faults. Live last_status: error, failure_streak: 1, next slot 2026-10-20 07:00. ✅ First full split run 2026-10-29: 07:00:59 → 10:16:12 (3 h 15 m 13 s), completed, wrote handoff/2026-10-29.json (170 discovered / 33 duplicates avoided / 37 verified); both of its ledger rows closed resolved at 10:16:12. Delivery is now local — stage 2 owns the report ✅ Live status re-verified 2026-10-30 03:3x: last_status: ok, failure_streak: 0 — the "Live last_status: error, failure_streak: 1" note above is superseded; the job cleared on its own 10-20 07:00 slot and has run clean since, including the successful 2026-10-29 split run.
Daily Marketing Job Radar — Write and Letters (stage 2) 5fc292ff0f3b 30 11,14 * * 1-5 telegram:-1004321904721:1030 job-search-automation — ✅ Created 2026-10-27 17:34 — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). custom:omni-101 / smart-route; first fires 2026-10-27 17:35 and 17:38 both completed. ✅ First full split run 2026-10-29: fired 10:13:46, consumed the handoff at 10:40:59, closed completed 10:44:48 and delivered the same second — 20 pages + 2 cover letters, missing_cover_letter_count 0, Notion board 455 → 475; the 11:30 and 14:30 slots then no-oped correctly. See incidents/2026-10-27-job-radar-budget-exhausted
Enhanced Weekend Adventure Planner 1b23c042e17d 0 8 * * 5 telegram:-1004321904721:102 — — 2026-09-30 delivery-target fix confirmed working (delivered 2026-10-02, flag cleared). ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the failed cliprox/terra pin, HTTP 401 ×1). ⚠ Last run 2026-10-09 08:00 failed — RuntimeError: Request timed out. (incident 1b23c0_c35703f637b8, still alerted, opened 2026-10-09 08:17) — so it still carries last_status: error, streak 1 → now streak 2. ⚠ The Fri 2026-10-16 08:00 slot failed too — RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. (incident 1b23c0_103cbe04a79b, alerted). It cannot clear before Fri 2026-10-23 08:00, so a single missed slot now reads as a two-week-old failure. → incidents/2026-10-16-omniroute-503-resource-pressure ✅ Live status re-verified 2026-10-30 03:3x: last_status: ok, failure_streak: 0 — the "still carries last_status: error … now streak 2" note above is superseded; the job cleared on its Fri 2026-10-23 08:00 slot and has run clean since.
Evening Stand-Down Briefing e530c89f2be0 45 18 * * 1-5 telegram:-1004321904721 — evening_standdown.py no-agent
Facebook Birthday Monthly Scrape 0e753e9720fd 0 9 1 * * telegram — — ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the catalog-missing nvidia/deepseek-ai/deepseek-v4-flash pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident 0e753e_e7f8194a7d06 (opened 2026-10-01 09:00:55) is still open in state detected, and last_status still reads error; next slot 2026-11-01 09:00, which is when the repin finally gets its live test. ⚠ Bare telegram still targets the home DM — retarget to a group topic
Facebook Birthday Weekly Registry Sync fad5a86979d4 0 3 * * 1 telegram — birthday_check.py ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the catalog-missing nvidia/deepseek-ai/deepseek-v4-flash pin, HTTP 400 ×3). ✅ Verified live 2026-10-12 03:00 — completed and delivered (448 birthdays); the stale last_status: error cleared after two weeks and both deepseek-v4-flash incidents auto-closed at 03:05:39 (fad5a8_a13bbc837e1c, fad5a8_c80b59a907f0). ⚠ Live status is no longer ok: the 2026-10-19 03:04:47 run completed but recorded delivery_failed — configured thread_id 34795 for telegram:1793951355:34795 was not found (the CLI renders it as "ran, but the result was not delivered", then "delivered without thread_id"). The run is fine; only the send is. No incident row was created for this. ⚠ Bare telegram target should move to a group topic
FreshRSS Midday News Briefing ecc1fc13655a 0 12 * * * telegram:-1004321904721:346 — freshrss_intelligence_v2.py ⚠ The 2026-10-16 12:00 slot failed on the omni-101 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed that incident. ⚠ Then 2026-10-18 12:00 failed differently — a model repetition loop aborted the run at 12:08:25 after it had delivered a canned "Your request was not processed…" line to the topic (12:08:11) → incidents/2026-10-18-degenerate-stream-canned-substitute. ⚠ Then the 2026-10-19 12:01 slot failed again, on the omni-101 503 — ecc1fc_c9976ea24553 (a 10-16 row, refreshed 10-19 12:02:58, alerted) plus the separate ecc1fc_21a72aa11cc4 degenerate-stream row from 10-18, so this one job holds two open incidents. Live last_status: error, failure_streak: 2, next slot 2026-10-20 12:00 ✅ Live status re-verified 2026-10-30 03:3x: last_status: ok, failure_streak: 0 — the "Live last_status: error, failure_streak: 2" note above is superseded; the job cleared on its own daily 12:00 slot after 2026-10-19.
GitHub Trending Radar — Weekly Roundup c0a9fa314542 0 9 * * 1 telegram:1793951355 — — ⚠ Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13). A getMe call against the telegram.bot_token in /home/hermes/workspace/github-trending-radar/config.json returns HTTP 401 Unauthorized. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at digest.py's send(). The agent recovered the content with --dry-run and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new @BotFather token replaces the dead one. This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong. ⚠ The status reading changed again on 2026-10-19: its scheduled Monday slot (0 9 * * 1) at 09:01 failed on the omni-101 503 (new incident c0a9fa_764558b5ae83, alerted), so this week's roundup did not run at all and last_status now reads error (streak 1). The token question is therefore untested for a second consecutive Monday — the job has not reached digest.py on a Monday since 2026-10-12. Next real test: Mon 2026-10-26 09:00. ✅ Live status re-verified 2026-10-30 03:3x: last_status: ok, failure_streak: 0 — the "reads error (streak 1)" note above is superseded; the job ran clean on its Mon 2026-10-26 09:00 slot, so the 10-19 outage has cleared. The revoked-bot-token finding above stands as a separate, still-unverified delivery-path question — that Monday run reached the model without raising a 503, which does not by itself prove the token works.
Gmail Action Digest 08eb86f02d37 30 8 * * * telegram:-1004321904721:notifications — gmail_action_digest_v3.py ⚠ The 2026-10-16 08:30 slot failed on the omni-101 503 (streak 2 at the storm peak); and its own 10-17 08:31 run restored a clean last_status. ⚠ But the incident never closed: 08eb86_472c7b609c80 was refreshed on 2026-10-19 08:33 by the same 503, and is still alerted. Live last_status: error, streak 1 ✅ Live status re-verified 2026-10-30 03:3x: last_status: ok, failure_streak: 0 — the "Live last_status: error, streak 1" note above is superseded; the job cleared on its own daily 08:30 slot after 2026-10-19.
Important-mail monitor d40085da631c every 30m telegram:1793951355 — _cron_silent_monitor.py disabled since 2026-08-29 (Gmail read-state incident)
Job Tracker Dashboard Refresh 2fd50c87b0d0 0 */2 * * * telegram:-1004321904721:1030 — dashboard_watchdog.py no-agent
Mnemosyne Auto-Consolidate 0c771463c3cd every 30m local — consolidate_mnemosyne.sh no-agent
Mnemosyne DB Backup e5ed215fc91c 0 3 * * 0 telegram:1793951355 — mnemosyne-backup.sh no-agent
Mnemosyne Embedding Coverage Monitor b63ed5817212 0 */6 * * * telegram:-1004321904721:102 — check_embedding_coverage.py Alerts if coverage < 95%. ⚠ Both 503-era slots failed (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own 10-17 06:01 run. ⚠ Both 2026-10-19 slots failed the same way (06:01:05 and 12:01:16), taking its streak to 2 and re-triggering the watchdog; its 18:01 slot succeeded, clearing the streak and auto-closing b63ed5_bbe44c89aac1 at 18:01:56. Live last_status: ok. With 0 */6 * * * this job is the fleet's best 503 sentinel — four probes a day
Mnemosyne Health Monitor ea79a2107e36 0 */12 * * * telegram:-1004321904721:102 mnemosyne-operations — Reports only on exception. Live status re-verified 2026-10-31 00:3x: ENABLED, last_status: ok, failure_streak: 0; incident ea79a2_d93c3acdc4f2 closed resolved 2026-10-30T10:02:48 (⚠ the 2026-10-30 03:3x DISABLED/unknown note later in this row is superseded by that; its 12:01 slot was lost to the CT460 host outage and fired late at 16:37:21). ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then again at 2026-10-19 12:01 — incident ea79a2_747b27556fd0 was first opened 10-16 12:03, refreshed 10-19 12:02:55, and auto-closed 2026-10-20 00:04:24 when the 00:01 slot succeeded. Live last_status: ok, streak 0 ⚠ Live status re-verified 2026-10-30 03:3x — the job is now DISABLED and failing: its 2026-10-30 00:01:08 slot closed unknown at 01:04:22 on "Cron execution unverified; reconcile ~/.hermes/cron/admissions/ad897cbc9f…json: Gateway connection failed; no local fallback", opening incident ea79a2_d93c3acdc4f2 (failure_type: unknown, state detected), and the scheduler auto-paused the job at that moment. Live last_status: error, failure_streak: 1, enabled: false. The "Live last_status: ok, streak 0" note above is superseded. ⚠ The failure notice was never delivered — hermes cron list reports "live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed" and deliveries.db has zero rows for 2026-10-30. See incidents/2026-10-30-mnemosyne-health-monitor-paused.
Mnemosyne Watchdog 96eb95ce12f2 */15 * * * * telegram:-1004321904721:102 — mnemosyne-watchdog.py no-agent, silent ≤10 conflicts; also runs hermes-runtime-deps-guard.sh every 15 min (added 2026-09-30). ⚠ Its reds are derivative — it double-counts other jobs' failures. During the 503 burst it exited 1 on 4 slots (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: 🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…. One sentinel watching another job, not four independent faults. The pattern repeated exactly in burst 2 — four more reds on 2026-10-19 (06:16, 06:31, 12:16, 12:31), all quoting the same coverage-monitor error, all clearing at 12:46:11 (96eb95_565ec1deb1c7, a 10-16 row refreshed through 10-19 12:31). 8 of the 26 failure rows across the two bursts were this one sentinel
Perth New Restaurant & Cafe Openings (Weekly) c8d52904e1d6 0 18 * * 4 telegram:-1004321904721:102 — — delivery target corrected 2026-09-30; verified delivered, status ok
Political News Digest (Mon/Thu) 0e519874d280 0 8 * * 1,4 telegram:-1004321904721:346 — ⚠ Two node faults, no code fault: 2026-10-15 08:06 RuntimeError: Connection error. (0e5198_be2291809e56) and 2026-10-19 08:01 HTTP 503 … resource pressure (new incident 0e5198_ec2fac98a91d, alerted). Both are open; failure_streak: 2. Next slot Thu 2026-10-22 08:00 ✅ Live status re-verified 2026-10-30 03:3x: last_status: ok, failure_streak: 0 — the "failure_streak: 2" note above is superseded; the job cleared on its Mon/Thu 08:00 slots after 2026-10-19.
Rhino AgentMail Inbox Watcher f77b7e46bd09 every 5m origin — agentmail-rhino-watch.sh disabled since 2026-07-27 (legacy)
Tech & AI Newsletter Digest (Analytical) 85edf6772bca 0 7 * * * telegram:-1004321904721:346 — ⚠ 2026-10-19 07:01 slot failed on the omni-101 503 (new incident 85edf6_256c5e150bc4, alerted) — its first failure since its 10-15 connection-refused red cleared on 10-16. Live last_status: error, streak 1, next slot 2026-10-20 07:00 ✅ Live status re-verified 2026-10-30 03:3x: last_status: ok, failure_streak: 0 — the "Live last_status: error, streak 1" note above is superseded; the job cleared on its own daily 07:00 slot after 2026-10-19.
Uber Eats Monthly Spend Report 066bb5191280 0 2 1 * * telegram:1793951355 camofox-browser-automation, browser-backend-verification — ✅ Fixed 2026-11-01: delivery retargeted to telegram:1793951355 (was a bare telegram target that inherited the dead thread_id 34795 → "configured thread_id 34795 … was not found; delivered without thread_id"). Also repaired the cron prompt, which pointed at port 9093 (nonexistent) instead of 9377, and wired in the tested pipeline scrape-uber-eats-orders.py → build-uber-eats-report.py → generate-uber-eats-report.py. The Oct 2026 report was generated and verified against the live Uber DOM → reports/uber-eats/index.
Weekly Review 79f9bb41ecf5 0 18 * * 0 origin weekly-review — ⚠ Failed 2026-10-04 18:00 on the cliprox/terra pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ Repinned 2026-10-05 23:39 → custom:omni-101 / free-stack and verified by a live manual run; the incident auto-resolved. Now last_status: ok, streak 0
weekly-save-restricted-bot-restart a688ed7b7ab0 0 3 * * 0 local — restart-save-restricted-bot.sh no-agent; script restored 2026-09-30
Wiki Audit — Nightly cbfe434a3b5e 59 23 * * * local task-approach — Reviews past 24h for wiki changes. ⚠ The 2026-10-16 23:59 slot failed on the omni-101 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident cbfe43_9b89947a046a is detected and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). Not a stale session lock — the execution row carries the explicit 503 error text
wiki-stocktake-maintenance 1a8e90ea2271 0 3 * * * local — wiki-maintain.sh ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the retired qwen3.5:397b, HTTP 410 ×11). ✅ Verified by live run 2026-10-06 03:00 — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as 33f4e95. Live last_status: ok, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ The Quartz rebuild step is a no-op in practice — no quartz.config.ts exists and npx is unavailable, so the static site in public/ is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit 8c63ef2) — a second consecutive automatic success, verified. The script itself was never broken. ⚠ The 2026-10-17 03:00 slot failed on the same omni-101 503 — the first miss since the 2026-10-15 storm slot (which died with RuntimeError: Connection error.; the 10-16 slot succeeded). raw/stocktake/freshness-20261017.txt was not written and systems/skills-index.md stays stamped 2026-10-16 (the 10-16 slot succeeded, so freshness-20261016.txt exists); the 2026-10-18 03:00 slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident 1a8e90_3c9c1d573cde detected. Note the ledger tags this row failure_type: unknown even though the execution carries an explicit 503 message
🦝 Hermes Config Backup to Gitea (Daily) ✅ RESOLVED 2026-11-07 — push live again, 92-commit backlog drained and pushed, script hardened to exit non-zero on failure so the job row can no longer read ok while the push is dead. Live run exit 0; origin/main == local HEAD (a50a5170c1c0); rev-list --count origin/main..main = 0. → incidents/2026-08-08-gitea-config-backup-push-dead e27da6c34bbb 0 11 * * * origin — hermes-config-backup.sh no-agent

Removed Jobs

Name Removed Reason
Workday start reminder 2026-08-26 No longer present in live cron list
CT 221 Miner Watchdog 2026-08-26 Job d5516d0a5147 removed; CT221 status unknown/inspected
DHL Tracking - Parcel 8052838382 2026-10-10 62619aa6bcb3 — parcel delivered, monitor redundant
Mosquito Tracker 2026-10-16 (after the 15:00 run) 1b31187397a5 — absent from jobs.json at the 2026-10-18 audit; last execution 2026-10-16 15:00 was completed / delivered
Review Agent Estate Phase One 2026-08-26 One-time job; completed 2026-08-08, now cleaned up

Disabled Jobs (not removed) — 3 as of 2026-10-31

Name ID Disabled since Reason
Important-mail monitor d40085da631c 2026-08-29 Paused during Gmail read-state incident. See incidents/2026-08-29-gmail-read-state.
Rhino AgentMail Inbox Watcher f77b7e46bd09 2026-07-27 Legacy AgentMail polling; superseded by newer patterns.
AgentMail Rhino Inbox Poll 0a7f933d06e2 2026-07-27 Legacy AgentMail polling; superseded by newer patterns.
Mnemosyne Health Monitor (re-enabled 2026-10-30) ea79a2107e36 2026-10-30 01:04:22 → re-enabled same day ⚠ SUPERSEDED as of 2026-10-31 — the job is ENABLED (enabled: true, last_status: ok, failure_streak: 0) and incident ea79a2_d93c3acdc4f2 closed resolved 2026-10-30T10:02:48.662+08:00. The following describes the 03:3x snapshot only. Auto-paused by the scheduler after its 00:01:08 slot closed unknown on a gateway-connection failure ("Cron execution unverified; reconcile ~/.hermes/cron/admissions/ad897cbc9f…json: Gateway connection failed; no local fallback"). Live last_status: error, failure_streak: 1, incident ea79a2_d93c3acdc4f2 open detected. Its failure notice was never delivered — deliveries.db has zero rows for 2026-10-30. See incidents/2026-10-30-mnemosyne-health-monitor-paused.

Known Issues

Resolved 2026-11-02

  • The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot. Execution be6ec11a8e68428084af9ab18fb148db (job 79f9bb41ecf5, scheduled_instant 2026-11-01T10:00:00+00:00) started 2026-11-01T18:00:39.125776+08:00 and finally closed 2026-11-02T01:27:22.000881+08:00 with status: completed, error: NULL, delivery_outcome: delivered; the delivery row is timestamped 01:27:21.802246 and incident 79f9bb_9c443b602eec auto-closed at 01:27:21.606691. Same execution row, not a re-run — the unknown terminal state recorded at 18:05:13 was replaced when the run wrote its real outcome. → incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure is now resolved. Method lesson for this audit: the 2026-11-02 00:4x reading of status: unknown + closed_at: NULL was accurate at the time of reading and had inverted 43 minutes later. A nightly audit that runs ~2 h after midnight must not treat a null terminal state on a long-running job as final — the reconciler's window is shorter than these runs, so unknown on a long job is a timing artefact, not a verdict. The same false-negative logic applies to ecc1fc13655a, whose 11-01 ghost error cleared when its 11-02 12:00:33 slot closed completed / delivered at 12:29:26.

Resolved 2026-10-30

  • The Mnemosyne Health Monitor was never really down — and it is enabled again (ea79a2107e36). Its 00:01:08 slot was still running at the 03:3x reading; the unknown/error verdict came from the admission reconciliation, not the run. It closed completed at 10:02:48 after 10 h 01 m 40 s and delivered, auto-closing incident ea79a2_d93c3acdc4f2 resolved at 2026-10-30T10:02:48.662+08:00. Live: enabled: true, last_status: ok, failure_streak: 0. Unclosed incident rows 2 → 1, non-clean last_status 4 → 3, failing 1 → 0. ⚠ The pause was real to record, but the "job is disabled / notice never delivered" state described in incidents/2026-10-30-mnemosyne-health-monitor-paused lasted only until 10:02:48 — deliveries.db finished 2026-10-30 with 8 rows, all delivered (latest 18:46:04), so the zero-row/never-delivered reading was a 03:3x artefact of the day having just begun.
  • CT460's 5 h 30 m host outage is closed only because the reboot ended it — nothing was repaired. PVE pve booted 2026-10-30 16:24:00, CT460 16:28:00, scheduler resumed 16:37:21 with 8 catch-up runs → incidents/2026-10-30-ct460-host-outage-scheduler-gap. ≈39 slots lost, 0 failures. No monitor alerted — the two jobs that could have (cbfe434a3b5e NFS mount monitor, 1a8e90ea2271 m93p LXC boot monitor) are disabled, and this remains the open item.

Resolved 2026-10-29

  • The Job Radar's two-stage split finished end to end for the first time (dca8482e4f76 + 5fc292ff0f3b): stage 1 ran 07:00:59 → 10:16:12 (3 h 15 m 13 s), closed completed and wrote handoff/2026-10-29.json (170 discovered / 33 duplicates avoided / 37 verified) — the first post-split run to write a handoff, after 10-27 died on the tool-call budget and 10-28 on the iteration cap. Stage 2 consumed it at 10:40:59, closed completed at 10:44:48 and delivered to topic 1030 in the same second: 20 pages, 2 cover letters, missing_cover_letter_count 0, Notion 455 → 475. Both stage-1 ledger rows (dca848_e93a538ade6b, dca848_0a2f03b5952b) closed resolved at 10:16:12, and the run opened no new row → incidents/2026-10-27-job-radar-budget-exhausted, incidents/2026-10-28-job-radar-stage1-iteration-cap, runbooks/job-radar. ⚠ The discovery stage is still slow (3 h 15 m) and both ceilings remain reachable — a completed stage-1 row with no handoff file is the same fault in new clothing.
  • Conduit voice calls — three faults in series, all fixed and verified by two real calls: the trunk speech-server URL was truncated (…/v1/ → 403, blamed on the key), the dashboard session TTL was the 12 h default (expired sessions answer WS 4401 with no error on the phone), and both host providers were dead (tts.provider: openai → Mac /v1/speech 502; stt.provider: groq → 403/Cloudflare 1010). Repaired 08:02–08:4x: URL → …:8443/v1/realtime, TTL → 2 592 000 s, stt.provider: local, tts.provider: ollama_local; dashboard restarted 08:03:26. The path was driven end to end, then calls at 08:38:04 and 08:49:36 — the first entries in conduit-calls.json → history. ⚠ Two TTL keys exist and only one was raised → incidents/2026-10-29-conduit-voice-silence, systems/conduit.

Resolved 2026-10-12

  • Both deepseek-v4-flash incidents closed by a clean scheduled run (fad5a86979d4, 03:00): the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents fad5a8_a13bbc837e1c (open since 2026-09-28, every fallback provider exhausted) and fad5a8_c80b59a907f0 (since 2026-09-30, deepseek-v4-flash HTTP 400) both auto-closed at 03:05:39. Its stale pre-repin last_status: error cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one.
  • dca8482e4f76 timeout incident closed (dca848_23fcd2b18aaa): the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm auto-closed at 07:33:41. Of the ten storm incidents, nine have now closed and one remains: 1b23c0_c35703f637b8 (1b23c042e17d, still alerted, next slot Fri 2026-10-16 08:00). See incidents/2026-10-09-cron-provider-timeout-storm.
  • Net effect: open cron incidents 5 → 2. The two survivors are 0e753e_e7f8194a7d06 (detected since 2026-10-01, untouched because its job does not run until 2026-11-01) and 1b23c0_c35703f637b8 (alerted, cleared on the next Friday slot).

Resolved 2026-10-07

  • DHL Tracking 62619aa6bcb3 — single Camofox timeout, self-recovered (2026-10-07 08:00 → 11:01): the 08:00 run died in ensure_browser() with TimeoutError: timed out on a browser HTTP call (dhl_tracking_monitor.py:233), i.e. Camofox was momentarily unresponsive. The failure notice was sent at 08:00:58 and the incident (62619a_f485ccfd4a0d, failure_type: timeout) auto-resolved at 11:01:13 when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. This is the first detect → notify → auto-resolve incident cycle since the 2026-09-30 managed-runtime event — the incident machinery itself worked as designed. No action needed; a repeat would justify a Camofox health probe inside the script.
  • The tracked parcel arrived. The 14:00 run on 2026-10-07 reported waybill 8052838382 Delivered (Kensington → Perth). The monitor has therefore achieved its purpose and runs 5×/day for no remaining benefit — a candidate for removal, flagged rather than removed (the job is Anthony's to retire).

Resolved 2026-10-05

  • Dead model pins across 7 jobs — repinned to routing aliases (resolved 2026-10-05 23:16 → 2026-10-06 00:05): Seven jobs were each pinned to a raw model name that had died upstream — cliprox/terra (HTTP 401, connection auth expired, 3 jobs), ollama-cloud/qwen3.5:397b (HTTP 410, retired 2026-09-25), nvidia/deepseek-ai/deepseek-v4-flash (HTTP 400, dropped from the live catalog, 2 jobs) and command-code/gpt-5.6-terra (HTTP 403 MODEL_NOT_IN_PLAN) — across 2026-10-01 → 2026-10-05. All seven were repinned to provider custom:omni-101 using routing aliases — free-stack ×6, smart-route ×1 — deliberately, because the fault was raw-pin brittleness and re-pinning to another raw name would have re-created it. failure_streak is now 0 on all 30 jobs, down from 7 failing. Three were verified by live run: 79f9bb41ecf5 (23:39), 06a0a1c8704f (23:45) and dca8482e4f76 (23:56, delivered to topic 1030 — its first success since 2026-09-25). The remaining four are repinned but not yet re-run; the stale last_status: error field clears on each one's next scheduled run. See incidents/2026-10-05-dead-model-pin-repin.
    • The cliprox/terra incident is closed for cron purposes: no enabled job references that pin any more. The underlying proxy connection was never re-authenticated as part of that remediation — the jobs were moved off it instead. ⚠ Corrected 2026-10-29: a live probe shows the pin now returns HTTP 200 on CT600 (custom:Omniroute:20128) — the connection has since been re-authenticated there — while custom:omni-101 (CT101) still returns HTTP 401. See incidents/2026-10-01-cliproxy-connection-auth-expired.
    • wiki-stocktake-maintenance resumes its 03:00 schedule, so wiki-maintain.sh no longer needs a daily manual run.
  • GitHub Trending Radar c0a9fa314542 — token-revocation claim refuted (2026-10-05): SUPERSEDED 2026-10-13. The Mon 09:00 run completed on 2026-10-05 and 2026-10-12, which was read as proof the getMe → 401 diagnosis was wrong. It was not: a cron run can be green while the script's own delivery hop is dead, and the 2026-10-13 live getMe against the token in /home/hermes/workspace/github-trending-radar/config.json returns 401. Corrected above and in Open items. Lesson: a completed run does not clear a credential fault unless the run exercised the same call that failed.
  • A stale failure notice, not a live outage, opened the 2026-10-05 review: the interactive session began because Anthony acted on a 85edf6772bca failure notice sitting in topic 346 since the 2026-09-29 07:00 run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already resolved on 2026-09-30 17:55:13. The notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. Check executions.started_at and the cron_incidents state before acting on a notice.

Resolved 2026-09-30

  • Telegram thread delivery failures — root cause found and fixed: jobs fad5a86979d4, 1b23c042e17d, c8d52904e1d6 reported configured thread_id 34795 for telegram:1793951355 was not found / Message thread not found. The target was wrong, not the topic: a bare deliver: "telegram" resolves to the home DM (1793951355), where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong). 1b23c042e17d and c8d52904e1d6 were retargeted to telegram:-1004321904721:102 and verified end-to-end. See systems/messaging-integrations.
  • Managed-runtime dependency failure (2026-09-28 → 30): Jobs failed with Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1). Real cause was ModuleNotFoundError: No module named 'dotenv' / 'mnemosyne' in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by hermes-runtime-deps-guard.sh v2, run every 15 min by job 96eb95ce12f2. All managed venvs verified. See incidents/2026-09-28-hermes-managed-runtime-deps.
  • weekly-save-restricted-bot-restart missing script (2026-08-30): Previously failed with "Script not found: restart-save-restricted-bot.sh". The script existed at /home/hermes/scripts/ but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job now ok, streak 0.
  • Residual pre-fix dispatch errors cleared: cbfe434a3b5e returned to ok on its next run as predicted. The 1a8e90ea2271 error of the same era also cleared — its current failure is a different, upstream problem (below).
  • Stale delivery_failed flag on 1b23c042e17d cleared (confirmed 2026-10-02): the 2026-09-30 retarget held — the job's next run delivered to topic 102.

Open

  • The Job Radar's Stage 1 lost a second consecutive day, and the fault changed (NEW 2026-11-04): dca8482e4f76 ran 6 h 28 m 18 s on 2026-11-03 and died on RuntimeError: stream active timeout (not the iteration cap), going silent for 65 m 21 s first. The ledger went red this time (failed + incident dca848_dfd4bea84f2a + a delivered notice) → incidents/2026-11-03-job-radar-stream-active-timeout. 2026-11-02's and 2026-11-03's candidate sets are both unrecovered. ✅ CLEARED in substance 2026-11-07: the 11-03 ledger row closed resolved at 2026-11-05T12:36:17, and dca8482e4f76 has run clean twice since - 5 h 35 m 35 s on 11-05 and 1 h 28 m 37 s on 11-06, both completed / suppressed. The two unrecovered candidate sets stay lost; nothing in the ledger is red -> systems/scheduled-tasks.

  • Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):

    • dca8482e4f76 (Job Radar Stage 1) ran 07:00:33 → 16:01:52 = 9 h 01 m 19 s and closed status: completed / error: NULL / delivery_outcome: suppressed after hitting the agent iteration cap before the handoff was written. Both Stage 2 slots declined (92665337… 11:30 and the 05a3f00b… 14:30 safety net both exited [SILENT]), so nothing reached Notion. This is the same class that raised dca848_e93a538ade6b (10-27) and dca848_0a2f03b5952b (10-28) — whose error columns are literally the runs' INCOMPLETE reports — yet 2026-11-02 produced no cron_incidents row at all, because the terminal state flipped from non-clean to completed. Three caps in seven days (10-27, 10-28, 11-02) is a pattern, not bad luck. → incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence. Detection rule: for a job whose deliverable is a file, executions.status is not evidence — check the artefact.
    • 1a8e90ea2271 (wiki stocktake) ran 03:00:31 → 12:44:00 = 9 h 43 m 29 s, completed / suppressed, and delivered no report (11-01 delivered at 03:00:37; 11-02 has no delivery row near 03:00). It left its output uncommitted — systems/skills-index.md modified and raw/stocktake/freshness-20261102.txt untracked — which the 2026-11-03 audit committed on its behalf. The artefacts themselves are sound (freshness report: 4 stale / 113 fresh at a >21-day threshold; skills-index regenerated).
  • GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13): live getMe on the token in /home/hermes/workspace/github-trending-radar/config.json returns HTTP 401, so digest.py's Telegram send fails every Monday while the cron run itself stays green (last_status: ok). A completed run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with --dry-run and its own response was delivered. Fix: issue a fresh token from @BotFather, replace it in config.json, and re-run getMe to confirm; do not re-read a green cron status as proof of delivery. This corrects the 2026-10-05 "claim refuted" entry, which was wrong. See systems/scheduled-tasks job c0a9fa314542.

  • Camofox memory pressure during long browser runs (NEW 2026-10-13): the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf: MemoryHigh=2G, MemoryMax=3G, swap max 1G). Live probe mid-run read Memory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G) with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the /tabs endpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See systems/browser-backend.

  • Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07): wiki-maintain.sh documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no quartz.config.ts is present and npx is unavailable, so public/ has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description.

  • skills-index under-counts by one skill (NEW 2026-10-07): live find ~/.hermes/skills -name SKILL.md -type f returns 347, but the generated page says 346 (70 categories). The generator de-dups on skill basename within a category, so the nested duplicate productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md collapses into its parent and is silently dropped. The other duplicated basename (hermes-api-vault, at the root and under devops/) survives because the two land in different categories. Remove the nested duplicate directory, or key the generator on path rather than basename.

  • A delivered notice is not a successful job: cliprox/terra failures recorded delivery_outcome: delivered because the failure notice is itself a deliverable. Read the execution status column, not the delivery outcome, when judging whether a job produced its output.

  • Stale provider key custom:omni (1 enabled + 1 disabled): the provider was removed from custom_providers (live set on 2026-10-01: litellm, guacano, Omniroute:20128, command-code, omni-101). Re-verified 2026-10-06: e530c89f2be0 (enabled) and d40085da631c (disabled) still name it; both run smart-route, and the default custom provider resolves to the same base URL (http://100.78.63.20:20128), so no failure has been observed — but the references should be normalised to custom:omni-101. The 2026-10-05 repin moved seven jobs to an explicit custom:omni-101, leaving these two as the only remaining custom:omni references.

  • Bare telegram delivery targets (3 jobs) — demonstrably breaking sends: 0e753e9720fd, fad5a86979d4, 066bb5191280 deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical: 066bb5191280 (02:11) and 0e753e9720fd (09:01) both recorded delivery_outcome: failed with configured thread_id 34795 for telegram:1793951355 was not found. Work generated correctly; the delivery did not. Retarget all three to telegram:-1004321904721:<topic>. See systems/messaging-integrations.

  • Two wiki trees, drifted in both directions (NEW 2026-10-02): /home/hermes/workspace/agent-estate-wiki is the canonical source (Gitea remote Anthony/agent-estate-wiki, Quartz build, wiki-maintain.sh target). /home/hermes/wiki/agent-estate-wiki is the Obsidian/Ignis vault copy the iPad reads — pages nested under wiki/, and no git remote at all. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged backwards (systems/ignis.md, systems/nextcloud.md, three incidents/ pages exist only in the vault; raw/stocktake/*, agents/, incidents/2026-09-02-sftpgo-outage.md only in canonical) — a naive one-way sync deletes pages. See systems/hermes-agent.

  • Work mailbox unreachable (non-cron; affects on-demand lookups): the work Outlook account is configured with password auth while the tenant advertises only XOAuth2, so himalaya -a work fails with login not supported. No cron job polls it. See incidents/2026-08-26-work-outlook-basic-auth-disabled.

  • The omni-101 503 burst left no node-side evidence (NEW 2026-10-18): 9 model calls across 7 jobs died on RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly. between 2026-10-16 08:00 and 2026-10-17 03:00 — while the node answered /api/health → 200 {"status":"ok"} and /v1/models → 401 before, during and after (re-probed 2026-10-18 00:33). Nothing on the Hermes side records the node's CPU/RAM/queue depth at failure time, so the "resource pressure" cause is taken on the router's word; if it recurs, capture host metrics during the burst, not afterwards. The incident ledger also tags seven of the nine rows failure_type: unknown despite the explicit message (only the two script-exit watchdog incidents get a type), so failure_type cannot be used to triage 503s — filter on the error text. Root cause unknown; not a repeat of the 10-14 TCP-refusal storm (which was reachability, this was serving).

  • It recurred on 2026-10-19 with the same 9-call blast radius (NEW 2026-10-20): the identical HTTP 503 … resource pressure string took 9 model calls across 8 jobs again (06:01, then 07:01 → 12:01), and again the node answered /api/health → 200 {"status":"ok"} and /v1/models → 401 before, during and after (re-probed 2026-10-20 00:03). Two bursts, four days apart, same node, same shape, no failover — and nothing was instrumented during either one. Action for the next occurrence: while jobs are still failing, run a real completion against custom:omni-101 and capture the node's host CPU / RAM / queue depth — not afterwards. The ledger again tagged every new row failure_type: unknown, so find these by the error text, never by failure_type. Also verified 2026-10-20: all eight affected jobs read provider: custom:omni-101, model: smart-route in jobs.json — one alias, one node, no diversity. → incidents/2026-10-16-omniroute-503-resource-pressure → incidents/2026-10-16-omniroute-503-resource-pressure

Historical

  • Mnemosyne Watchdog incident (2026-08-19): Job 96eb95ce12f2 cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See systems/mnemosyne.
  • Session storage failures (2026-08-10): Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See current-state.
  • Model repinning (2026-08-08): 13 jobs repinned from dead/stale provider combos to smart-route or gpt-5.6-terra.
  • Wiki Audit model drift (2026-08-07): Global inference config change caused RuntimeError: Skipped to prevent unintended spend on an unpinned job.

Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09)

Fleet: 30 jobs · 27 enabled · 3 disabled · 10 carrying a non-clean last_status at 2026-10-10 00:22 (jobs.json.updated_at 2026-10-10T00:22:17+08:00). Executions by AWST day: 10-07 {'completed': 177, 'failed': 1} · 10-08 {'completed': 178, 'failed': 2} · 10-09 {'completed': 162, 'failed': 15, 'running': 1}.

The 2026-10-09 timeout storm. 16 failed executions across 11 distinct jobs, every one RuntimeError: Request timed out. on the routing aliases behind custom:omni-101, 00:00 → 15:00 AWST; every run from 18:00 onward completed. Full timeline, job list and incident table: incidents/2026-10-09-cron-provider-timeout-storm.

  • Four incidents auto-closed on the next clean run (96eb95 07:01, b63ed5 18:01, 1b3118 22:05, ea79a2 2026-10-10 00:03); eight remain open — 1a8e90_789224ad589c, cbfe43_fa81052baaf7, 1a8e90_dc6b2d40ec44, dca848_23fcd2b18aaa, 85edf6_18bedbeec5cd, 1b23c0_c35703f637b8, 08eb86_634f1a296f78, ecc1fc_58b6c38296f3.
  • 1a8e90ea2271 is the only job still at failure_streak: 2 (10-08 HTTP 403 + 10-09 timeout); its 03:00 run today decides whether the streak clears.
  • 96eb95ce12f2 (watchdog) failed twice as a cascade — it exits 1 when it finds ERROR lines in agent.log, so it reported the storm rather than suffering it.

The 2026-10-08 window has no audit entry. The nightly audit's own 23:59 run on 2026-10-08 failed (RuntimeError: Request timed out., incident cbfe43_fa81052baaf7, first seen 2026-10-09T00:32:09) and committed nothing; the last committed audit covers the 2026-10-07 window (commit 659356a, 2026-10-08 00:12). The 2026-10-09 run covers both days.

Stocktake: two-day gap, artifacts recovered. The 2026-10-08 03:01 run left systems/skills-index.md modified and raw/stocktake/freshness-20261008.txt untracked; both were stranded (the job's own commit step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator ran, so no freshness-20261009.txt exists and the skills index is still the 2026-10-08 output.

DHL monitor — retirement instructed, still enabled. Anthony replied "Delete this job" to the 20:26 DHL delivery notice on 2026-10-09; the session replied that 62619aa6bcb3 was "stopped and deleted". Live jobs.json at 2026-10-10T00:22 shows 62619aa6bcb3 still enabled — paused_at: null, last_status: ok, schedule 0 8,11,14,17,20 * * * — so it runs again at 08:00. The parcel (8052838382) was reported Delivered on 2026-10-07, making the monitor redundant. This audit did not mutate cron state; retiring the job is open item #1 below.

Memory recall relevance. "Are you up to date" returned a stale Uber Eats episode on 2026-10-09; capture and vector health were verified good (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is not in live config. See incidents/2026-10-09-memory-recall-relevance.

Open items (2026-10-09)

  1. Retire 62619aa6bcb3 (DHL Tracking — Parcel 8052838382) — instructed 2026-10-09 21:24, not executed. DONE 2026-10-10 — verified retired by ID lookup; fleet dropped to 29 jobs (26 enabled). See log.md and the 2026-10-10 audit block below.
  2. Decide and persist the recall-ranking change in memory.mnemosyne; re-run the trigger question as a regression test.
  3. Close the eight open cron incidents once their next clean runs land; watch 1a8e90ea2271 (streak 2). DONE — all eight cleared: five on 2026-10-10 and dca848_23fcd2b18aaa on 2026-10-12 07:33:41. (A different pair, 1b23c0_c35703f637b8 + 0e753e_e7f8194a7d06, remains open for unrelated reasons — see ### Open above.)
  4. Stale delivery targets — 0e753e9720fd, fad5a86979d4 (last_status: error) and 066bb5191280 (delivery_failed) unchanged since the 2026-10-07 audit.
  5. Quartz site rebuild is skipped by wiki-maintain.sh step 3 (no quartz.config.ts, no npx) — carried from 2026-10-07, still open.
  6. Skills index off by one (347 SKILL.md on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.

Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST)

Fleet: 29 jobs - 26 enabled - 3 disabled (jobs.json.updated_at 2026-10-11T00:19:44+08:00). The count fell from 30 to 29 because 62619aa6bcb3 (DHL Tracking - Parcel 8052838382) is gone: open item #1 of the 2026-10-09 audit is CLOSED (below). Disabled: d40085da631c (Important-mail monitor), f77b7e46bd09 (Rhino AgentMail Inbox Watcher), 0a7f933d06e2 (AgentMail Rhino Inbox Poll).

Executions by AWST day (executions.db): 10-07 177 completed / 1 failed - 10-08 178 / 2 - 10-09 163 / 15 - 10-10 174 completed / 0 failed (+1 left running). The 2026-10-10 fleet was wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm.

Non-clean last_status: 5 jobs, down from 10.

Job ID last_status streak
Facebook Birthday Monthly Scrape 0e753e9720fd error 0
Facebook Birthday Weekly Registry Sync fad5a86979d4 error 0
Daily Marketing Job Radar — Discovery (stage 1) dca8482e4f76 error 1
Enhanced Weekend Adventure Planner 1b23c042e17d error 1
Uber Eats Monthly Spend Report 066bb5191280 delivery_failed 0

The three streak 0 rows are stale last_status fields, not live failure streaks. 1a8e90ea2271 (stocktake) is back to last_status: ok.

Incidents: 7 closed during 2026-10-10, 5 open. The 10-09 storm unwound as each job next ran clean (ea79a2 00:03, cbfe43 00:27, 1a8e90ea2271 x2 03:06, 85edf6 07:04, 08eb86 08:32, ecc1fc 12:05). Still open: 1b23c042e17d + dca8482e4f76 (timeout, alerted, both first seen 2026-10-09) and 0e753e9720fd + fad5a86979d4 (agent; open since 2026-10-01 / 2026-09-30).

Open item #1 CLOSED - the DHL monitor is retired. Anthony instructed removal again on the evening of 2026-10-10 and the job is absent from live jobs.json, verified 2026-10-11 00:19 by direct ID lookup (no job id contains 62619; no job name matches dhl/parcel). This is the third "deleted" claim for this job - the 2026-10-07 and 2026-10-09 reports were both false because the check was the absence of job output rather than a positive lookup. Removals must be verified by ID lookup.

Still open: the Quartz rebuild step in wiki-maintain.sh (no quartz.config.ts) - skills-index off-by-one (347 SKILL.md vs 346 reported) - the 5 non-clean last_status rows above - Mattermost platform unreachable (systems/messaging-integrations) - the gateway runtime env is outside the 15-minute dependency guard (incidents/2026-09-28-hermes-managed-runtime-deps).


Re-verified 2026-10-12 (audit window 2026-10-11 00:00 → 2026-10-12 00:2x AWST; live probes 00:0x-00:2x)

Fleet: 29 jobs — 26 enabled / 3 disabled (jobs.json updated_at 2026-10-12T00:10:42+08:00). Disabled set unchanged: d40085da631c (Important-mail monitor), f77b7e46bd09 (Rhino AgentMail Inbox Watcher), 0a7f933d06e2 (AgentMail Rhino Inbox Poll — legacy).

Executions by AWST day (executions.db): 10-07 177 completed / 1 failed — 10-08 178 / 2 — 10-09 163 / 15 — 10-10 175 completed / 0 failed (finalised; the 10-11 audit saw 174 + 1 still running) — 10-11 171 completed / 0 failed + 1 running (this audit). Two consecutive zero-failure days after the 2026-10-09 provider-timeout storm. Heaviest jobs on 10-11: runtime-deps guard 96eb95ce12f2 96 runs (15-min cadence), 0c771463c3cd 46, 2fd50c87b0d0 12, coverage monitor b63ed5817212 3, health monitor ea79a2107e36 2, stocktake 1a8e90ea2271 1.

Non-clean last_status: 5 jobs — unchanged from 2026-10-10. 0e753e9720fd (error, streak 0), fad5a86979d4 (error, streak 0), dca8482e4f76 (error, streak 1), 1b23c042e17d (error, streak 1), 066bb5191280 (delivery_failed, streak 0). The two streak-1 jobs are the Mon-Fri and Friday-only schedulers: their last_error is RuntimeError: Request timed out. from Friday 2026-10-09 (the storm) and neither has had a scheduled slot since — dca8482e4f76 next runs 07:00 Mon, 1b23c042e17d Fri 08:00, so both should clear themselves. Everything else reads last_status: ok (this build reports ok, not success).

Incidents: 0 opened / 0 closed in the window — 5 unresolved, unchanged. alerted 3: fad5a8_c80b59a907f0 (weekly registry sync, scope agent, first seen 2026-09-30, deepseek-v4-flash HTTP 400), dca848_23fcd2b18aaa + 1b23c0_c35703f637b8 (timeout, both first seen 2026-10-09 — the last two left from the storm). detected 2: fad5a8_a13bbc837e1c (2026-09-28, every provider in the fallback chain exhausted), 0e753e_e7f8194a7d06 (2026-10-01, same deepseek-v4-flash 400). Table totals: alerted 3 / detected 2 / closed 8 / resolved 63. Last closures remain the seven on 2026-10-10.

Sessions in the window: 18 — 17 cron + 1 interactive (talaria-ios, 2026-10-11 19:13 → 19:51, 41 messages, "best VNC clients" research). The 03:00 stocktake 1a8e90ea2271 ran clean on schedule (completed 03:00:40 → 03:11:29) and committed d981e30.

False alarm — the 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION was a timezone mislabel, not a capture stall. The run compared a naive-UTC working_memory.created_at (2026-10-10 13:24:53) against AWST now and reported "10.7 h of no captures"; the true gap was 2.7 h (the row landed 21:24:53 AWST, inside the session that produced it). No pipeline fault. The same job's 12:05 and 2026-10-12 00:02 runs returned [SILENT] (green). → systems/mnemosyne

Still open (unchanged): Quartz rebuild step inert (wiki-maintain.sh step 3 — no quartz.config.ts); skills-index off-by-one (347 SKILL.md on disk vs 346 reported); the 5 non-clean last_status rows; Mattermost platform unreachable (systems/messaging-integrations); gateway runtime env outside the 15-minute dependency guard (incidents/2026-09-28-hermes-managed-runtime-deps).

Re-verified 2026-10-13 (audit window 2026-10-12 00:00 → 2026-10-12 23:59 AWST; live probes 2026-10-13 00:05–00:45)

Headline: a clean run hid a dead credential. The cron table was green — 174 completed / 0 failed — and two of the three non-clean last_status rows cleared overnight, yet one job that reported success on paper had its delivery hop dead the whole time. Detail below.

Cron tally: 2026-10-11 172 completed / 0 failed (the one row still running at the last audit finished clean); 2026-10-12 174 completed / 0 failed / 1 running — and the running row is this audit itself (cbfe434a3b5e, started 23:59:41). Third consecutive clean day. The execution-log retention floor has moved up to 2026-10-07 08:15, so 10-07 rows are now pruned (their earlier tally is no longer reproducible from this DB).

Incidents: open 5 → 2. Three closed on 2026-10-12 — fad5a8_a13bbc837e1c and fad5a8_c80b59a907f0 at 03:05:39 (both deepseek-v4-flash, closed by the first clean scheduled run of fad5a86979d4); dca848_23fcd2b18aaa at 07:33:41 (closed by dca8482e4f76's own successful 07:00 run). Table totals now alerted 1 / detected 1 / closed 8 / resolved 66 (76 rows). Survivors: 0e753e_e7f8194a7d06 (detected since 2026-10-01 — cannot clear until its job next runs on 2026-11-01) and 1b23c0_c35703f637b8 (alerted since 2026-10-09, next slot Fri 2026-10-16 08:00).

Job-by-job (verified against the executions log, delivery_outcome included):

  • fad5a86979d4 — Facebook Birthday Weekly Registry Sync 03:00:45 → 03:05:47, completed, delivered. Output: Birthday Dashboard, 448 unique birthdays, 1 contact match. This is the repin's live proof.
  • dca8482e4f76 — Daily Marketing Job Radar 07:00:47 → 07:33:50, completed, delivered to topic 1030 (32-minute run; the 2026-10-09 timeout incident closed 9 seconds before it finished). Run was messy — see the Camofox finding — but it produced and delivered.
  • c0a9fa314542 — GitHub Trending Radar 09:00:41 → 09:08:33, completed, delivered — and yet its own send failed (finding 1). A completed row plus delivery_outcome: delivered is not proof the job's payload went out; here the delivered artifact was the agent's recovered digest, not the script's.
  • 1b23c042e17d — Enhanced Weekend Adventure Planner 2026-10-09 08:00:38 → 08:17:46, failed (RuntimeError: Request timed out., notice delivered). No run since; next slot Fri 2026-10-16 08:00.
  • 0e753e9720fd — Facebook Birthday Monthly Scrape: no runs at all inside the retained window (monthly, next 2026-11-01), so its error/streak-0 row is still untested rather than fixed.

Sessions in the window: 18, all cron, no interactive session — a quiet day, in contrast to the previous window's talaria-ios research session. Busiest by message count: dca8482e4f76 (140), 1a8e90ea2271 stocktake (97), Wiki Audit (28), c0a9fa314542 (25). The 03:00 stocktake ran clean and pushed 0eee5f1 to origin/master: skills-index regenerated (346 skills / 70 categories, 347 files on disk — the known off-by-one), freshness audit 54 stale / 32 fresh / 7 no-date of 93 pages, Quartz step skipped on the documented path.

Mnemosyne (12:02 health run, green): working 760/786 embedded = 96.7%, episodic 1050/1056 = 99.4% — both above the 95% floor, so no reindex needed. Orphan embeddings 0. FTS5 aligned exactly (fts_working 786/786, fts_episodes 1056/1056, fts_facts 876/876). 0 unconsolidated working rows; 1 unresolved conflict (threshold 10); last consolidation 0.3 h earlier. 2252 memories total (786 working / 1056 episodic / 1 triple), 395 consolidated facts. Disk 15 GB free (81% used). No exceptions → nothing to act on, and nothing to change on systems/mnemosyne beyond the coverage figures.

Finding 1 — GitHub Trending Radar's Telegram token is revoked (live-verified). getMe against the telegram.bot_token in /home/hermes/workspace/github-trending-radar/config.json returns HTTP 401 Unauthorized. The token is a plausible length (46 chars), so it is revoked/invalid rather than malformed, and there is no valid replacement anywhere on the box. The fetch path is healthy — GitHub trending pulled 100 repos and wrote 26 cache files in 24 h — so only digest.py's send() fails. Consequence: the Monday digest reached the DM only because the agent noticed the failure and re-ran with --dry-run and its own response was delivered. The 2026-10-05 "token-revocation claim refuted" entry was wrong and has been corrected: two green runs were read as proof the token worked, when a green cron row only means the agent finished. Fix: new token from @BotFather → config.json → re-run getMe. Credential present, value not recorded here.

Finding 2 — Camofox hit its memory ceiling during the Job Radar run. Mid-run probe of camofox-browser.service: Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M) with 383 tasks and the service up 9 days. Swap fully consumed, zero reclaimable headroom → /tabs and page evals hung. The agent restarted the service (back to ~2.1 GB used of 14 GB on CT460) and completed the extraction one tab at a time with immediate close. Same containment drop-in as documented (resource-containment.conf); the limits held, the workload is what needs bounding. → systems/browser-backend

Nothing added/removed: no new system, no retired job, so index.md is unchanged this cycle. No page archived: the DHL monitor stays where the 2026-10-10 audit put it (redundant, in Removed Jobs).

Still open after this cycle: GitHub Trending Radar token (new 2026-10-13); Camofox containment vs. multi-tab workloads (new 2026-10-13); Quartz rebuild step inert; skills-index off-by-one; 3 non-clean last_status rows (0e753e9720fd, 1b23c042e17d, 066bb5191280); Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.

Re-verified 2026-10-14

Window: 2026-10-13 00:00 → 23:59 AWST; live probes 2026-10-14 00:00–00:30 AWST. Fleet 29 jobs (26 enabled / 3 disabled) — unchanged this cycle. Open incidents 2 → 3.

Cron tally — the clean streak broke. 2026-10-13 = 167 completed / 1 failed (plus this audit's own row in flight at probe time, which finalises the day at 168). Tail: 10-10 175 / 0, 10-11 172 / 0, 10-12 175 / 0 → the first failed cron run since 2026-10-09 (a day that recorded 15).

Finding 1 — wiki-stocktake-maintenance (1a8e90ea2271) died on a dead opencode credential. 03:00:43 → 03:01:57, RuntimeError: HTTP 404: No active credentials for provider: opencode. Pinned custom:omni-101:free-stack. New incident 1a8e90_d11698bd7275 (state: detected, failure_type: agent, alerted_at: NULL — failure streak 1, so under the 3-strike alert rule). This is the job's third failure in six days, all three on the opencode path: 10-08 HTTP 403: OpenCode's free tier can only be used from within OpenCode (1a8e90_789224ad589c), 10-09 timeout (1a8e90_dc6b2d40ec44), 10-13 404 no credentials (1a8e90_d11698bd7275). The first two resolved on 2026-10-10; this one clears only when the 2026-10-14 03:00 run succeeds — i.e. each incident clears only on the next successful run of its own job. → incidents/2026-10-05-dead-model-pin-repin

Finding 2 — the failure stranded its own generated artifacts; this audit committed them. The script's step 1 (skills-index regeneration) and step 2 (freshness report) both completed and wrote to disk, then the agent/LLM turn that follows died → nothing committed, nothing pushed. Left in the working tree: systems/skills-index.md (modified — date-only change: updated/verified_on 10-12 → 10-13; 346 skills / 70 categories unchanged) and raw/stocktake/freshness-20261013.txt (untracked — 48 stale / 38 fresh / 7 no-date of 93 pages, down from 54 stale / 32 fresh on 10-12). Both are in this audit's commit. Structural point: the agent row is what commits, so a perfectly healthy script run still publishes nothing when the LLM turn dies. There is no green-row alternative — the row went red.

Retention floor moved. executions.db now holds 1,001 rows (1,000 terminal + 1 in flight); MAX_TERMINAL_EXECUTIONS = 1000 in cron/executions.py. Oldest retained terminal row: 2026-10-08 07:11. Days above the floor are complete; older ones undercount — 10-08 now reads 126 completed / 1 failed where its finalised tally was higher. Run-level evidence older than the floor is pruned, not recoverable.

Delivery. 10-13: 159 suppressed / 9 delivered / 1 no-record. Delivered = Daily Marketing Job Radar (dca8482e4f76), Mosquito Tracker (1b31187397a5 ×3), Tech & AI Newsletter Digest (85edf6772bca), Daily Birthday Checker (a432898b55a4), Gmail Action Digest (08eb86f02d37), FreshRSS Midday News Briefing (ecc1fc13655a), Evening Stand-Down Briefing (e530c89f2be0). No unexpected delivery; the two dead-thread delivery faults (0e753e9720fd, 066bb5191280, both Telegram thread_id 34795) are unchanged since 10-01.

Mnemosyne. All six monitor runs returned the silent token — Health Monitor 00:02 and 12:02, Embedding Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, i.e. both coverages at/above the 95 % floor and no reindex. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %). Wrinkle: the 12:02 health run emitted the token markdown-bolded rather than bare; the delivery layer still suppressed it (delivery_outcome: suppressed), so the guard held — but the matcher is tolerant of wrapping rather than literal. → systems/mnemosyne

Non-clean last_status: 3 → 4. Cleared: fad5a86979d4 (success 10-12 03:05), dca8482e4f76 (success 10-13 07:02). Added: 1a8e90ea2271. Remaining: 0e753e9720fd (10-01), 1b23c042e17d (10-09), 066bb5191280 (10-01), 1a8e90ea2271 (10-13).

Sessions. 15 in the window, all cron, zero interactive — third consecutive window with no interactive session.

Pages touched: systems/scheduled-tasks.md (this block, frontmatter, non-clean list, retention note), systems/skills-index.md (committed stocktake artifact), current-state.md, index.md, log.md. Nothing added/removed: no new system, no retired job, no page archived → index.md structure unchanged.

Still open after this cycle: wiki-stocktake-maintenance opencode credential (new 2026-10-13 — watch the 10-14 03:00 run: streak 3 triggers an alert, and the durable fix is to repin the job off free-stack-opencode or repair OmniRoute's opencode provider credentials); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert; skills-index off-by-one; 4 non-clean last_status rows (above); Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.

Re-verified 2026-10-15 / 2026-10-16 — the omni-101 connection-refused storm

Window: 2026-10-14 00:00 → 2026-10-16 00:00 AWST (two days — the 2026-10-14 23:59 audit died inside the same outage, so the 10-14 window was never covered). Live probes 2026-10-16 00:0x–00:2x. Fleet 29 jobs (26 enabled / 3 disabled) — unchanged. Open incidents 3 → 12 (9 alerted / 3 detected).

The tally broke hard. 2026-10-14 = 160 completed / 12 failed; 2026-10-15 = 151 completed / 24 failed / 1 running. Against a four-day clean run (10-10 175/0, 10-11 172/0, 10-12 175/0, 10-13 167/1). 36 failure rows across 13 jobs — but only 24 are genuine model-call failures on 12 jobs; the other 12 rows are the Mnemosyne Watchdog reddening on a log sentinel (symptom, not fault). Day-level, 10-15 is the worst cron day since 2026-10-09 (15).

Root cause. httpcore.ConnectError: [Errno 111] Connection refused against http://100.78.63.20:20128 — the omniroute-1 node, provider custom:omni-101 (config.yaml: custom provider name: omni-101). Surfaced as openai.APIConnectionError: Connection error. → RuntimeError: Connection error. 328 Errno 111 lines in the live errors.log, 504 across rotations. Window 10-14 08:30:27 → 10-15 18:07, recovered by 10-15 22:01. → incidents/2026-10-14-omniroute-node-connection-refused-storm

Structural finding — no model-call failover anywhere in the fleet. Verified from jobs.json: all 18 enabled agent-mode jobs are pinned to custom:omni-101 (smart-route ×10, free-stack ×6, deepseek-paid-smart ×2). The only other model-capable cron pin, custom:omni (CT600, omniroute:20128), sits on e530c89f2be0 — which is no_agent: true, a script-only job, so its provider field is vestigial. Completing runs inside the window by provider: 210 no_agent / no provider + 1 no_agent on custom:omni — that is the entire green of the fleet, and zero model calls completed in 34 hours. ⚠ A first pass at this window read Evening Stand-Down's clean 18:45 runs as proof CT600 stayed up. It is not: Stand-Down never calls a model. Never read a no_agent job's green row as provider-health evidence.

Worst-affected jobs: b63ed5817212 Mnemosyne Embedding Coverage ×6 (every slot for 30 h), 1b31187397a5 Mosquito Tracker ×4, ea79a2107e36 Mnemosyne Health Monitor ×3, 08eb86f02d37 Gmail Action Digest ×2, ecc1fc13655a FreshRSS Midday ×2, then one apiece for 85edf6772bca, dca8482e4f76, 0e519874d280, 06a0a1c8704f, c8d52904e1d6, 1a8e90ea2271, cbfe434a3b5e. Full table in the incident page. User-visible cost: Digests that never went out — Tech & AI, Daily Marketing Job Radar, Political News, AI Newsletter Curation for Patrick, Gmail Action Digest ×2, FreshRSS Midday ×2, and the weekly Perth New Restaurant (missed outright; next chance 10-22). The missed audit also meant the 10-14 window went uncovered.

Recovery verified by schedule, not by probe alone. 1b31187397a5 completed 10-15 22:01:56 (streak 4 → 0); b63ed5817212 10-16 00:02:29; 96eb95ce12f2 10-16 00:01:36; ea79a2107e36 10-16 00:09:48. Live 10-16 00:1x: GET /v1/models on 100.78.63.20:20128 → HTTP 401 in 10 ms (up, auth required) and the same on omniroute:20128 / 100.88.81.19.

Non-clean last_status: 3 → 12. Cleared during the window: b63ed5817212, 1b31187397a5, ea79a2107e36 (all on successful post-storm runs). Added: the nine rows listed under Count above. Unchanged long-standing: 1b23c042e17d (10-09 error), 0e753e9720fd (10-01 error), 066bb5191280 (10-01 delivery_failed, dead Telegram thread_id 34795).

Mnemosyne. Both monitors sat on the failing node, so no coverage figures were refreshed in the window — last explicit numbers remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %); the 10-16 00:09 Health Monitor run is the first clean read since 10-12. → systems/mnemosyne

Stocktake gap (self-healing). 1a8e90ea2271 failed 10-15 03:00 → raw/stocktake/freshness-20261015.txt was never written and systems/skills-index.md still carries the 10-14 stamp. Its next slot is 10-16 03:00, so the gap closes on its own — not regenerated by hand here, to avoid racing the scheduled run.

Sessions. 36 sessions in the two-day window, all cron, zero interactive — fourth consecutive window with no interactive session.

Pages touched: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (new), systems/omniroute.md, systems/mnemosyne.md, systems/scheduled-tasks.md (this block, frontmatter, Count), current-state.md, index.md, log.md. Nothing removed or archived: no retired job, no obsolete system → index.md structure gains one page.

Still open after this cycle: the no-failover topology itself (all 18 model-calling jobs on one OmniRoute node — the durable fix is a second provider pin or a router-side failover, and it needs Anthony's call); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab workloads; Quartz rebuild step inert; skills-index off-by-one; 12 non-clean last_status rows; Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.

Re-verified 2026-10-24 (audit window 2026-10-23 00:00 → 2026-10-24 00:3x AWST; live probes 00:1x–00:3x)

Count. As of 2026-10-24 00:2x AWST (jobs.json.updated_at 2026-10-24T00:17:36+08:00): 28 jobs — 25 enabled / 3 disabled, unchanged for a sixth consecutive audit. Window 2026-10-23: 169 completed / 0 failed (+ this audit running), the third zero-failure day in four; 2026-10-22 settled at 173 / 0.

The 2026-10-09 timeout storm closed. 1b23c042e17d (Enhanced Weekend Adventure Planner, weekly Fri 08:00) ran clean on its 10-23 slot, closing the storm's last two rows (1b23c0_c35703f637b8, 1b23c0_103cbe04a79b) at 08:04. The storm now holds zero unclosed rows and the job reads ok again → incidents/2026-10-09-cron-provider-timeout-storm.

Incident ledger: unclosed 4 → 2 (no rows opened, two closed). Remaining: c0a9fa_764558b5ae83 (alerted since 10-19, next slot Mon 10-26 09:00 — the GitHub Trending Radar revoked token) and 0e753e_e7f8194a7d06 (detected since 10-01, next slot 11-01 — Facebook Birthday monthly scrape).

Non-clean last_status: 5 → 4 — c0a9fa314542 (error, streak 1, Mon 10-26), 0e753e9720fd (error, 0, 11-01), fad5a86979d4 (delivery_failed, Mon 10-26), 066bb5191280 (delivery_failed, 11-01). No new failure signature appeared in the window.

Pins unchanged — 11 custom:omni-101 / smart-route, 8 unpinned (global default), 6 custom:omni-101 / free-stack, 2 custom:omni / smart-route, 1 custom:omni-101 / deepseek-paid-smart (this audit).

Sessions: 14 in the window — 13 cron + 1 desktop (2026-10-23 18:26:12, 13 messages, the Conduit phone-call session that exposed the memory-timeout fault below).

New fault found and fixed in-window (not a cron job): the Conduit voice plugin gave the Mnemosyne provider start a hard-coded 10 s budget, so three calls (10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) were abandoned by the memory layer and ran without recall. Patched the same evening to a 60 s default (CONDUIT_MEMORY_TIMEOUT_S override) plus a TokenError guard that degrades instead of aborting; dashboard restarted 18:54:43 and no abort line exists after it. The patch is uncommitted in the plugin's working tree → incidents/2026-10-22-conduit-memory-start-timeout, systems/conduit.

Stocktake: the 10-22 sixfold overrun did not repeat — the 10-23 03:07 slot ran and committed (250c735: freshness 2 stale / 94 fresh, skills-index 306).

Pages touched: systems/conduit.md (new), incidents/2026-10-22-conduit-memory-start-timeout.md (new), incidents/2026-10-09-cron-provider-timeout-storm.md (status → resolved), systems/mnemosyne.md, current-state.md, index.md, log.md, plus frontmatter added to two legacy incident pages (see log). Nothing removed or archived: no job retired, no system obsolete in the window.

Still open: the single-node no-failover topology behind every model-calling job; the GitHub Trending Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 stopped; the Conduit patch being uncommitted; the 561.7 MB Conduit health store with no retention.

Re-verified 2026-10-25 (audit window 2026-10-24 00:00 → 2026-10-25 00:5x AWST; live probes 00:1x–00:5x)

A third consecutive clean day — 167 / 0 on 2026-10-24, with 2026-10-23 settling at 170 / 0. Fleet unchanged at 28 jobs (25 enabled / 3 disabled), 4 non-clean last_status, 2 unclosed incident rows, pins unchanged. Nothing opened and nothing closed in the incident ledger in-window. 14 sessions: 11 cron + 3 desktop.

The window's fault is off-cron. On 2026-10-24 12:53–12:56 Anthony asked three times to be phoned to test Conduit calls (Call me now to test calls. → Yes. You do → CAL ME); every reply asserted the agent has no calling capability, with zero tool calls in the session — no tool_search, no skill_view, no conduit_call_user. The tool exists, is registered for every session, is reachable through lazy discovery, and the plugin ships the calling-the-user skill; his call settings (enabled / when_asked / decides / alerts) were permissive and the device was paired. The first reply came from kilo-auto/free, but the mid-session switch to smart-route repeated it — capability asserted instead of discovered, not a broken tool → incidents/2026-10-24-conduit-call-request-refused.

Conduit re-verified. Plugin 0.69.0 / HEAD f000bf0 (22 commits on from the 0.64.0 / eaa14be recorded on 10-24) while the gateway process still dates from 2026-10-23 18:54:43, so the upstream commits are not live and the memory-timeout patch is uncommitted and now five releases behind HEAD. Device phone-status contact last 10-23 19:10:30; health ingestion ran to 10-24 09:59:59; store 569.9 MB / 1,520,229 events / 3,028 buckets / 124 windows — the 10-24 page's 308 buckets / 62 windows was a narrower grouping, not a collapse. ⚠ Evidence correction: conduit-calls.json → history is empty, and the 10-22 incident's “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne cron turn-ends of that night (ends records any turn-end that fires a watch, MAX_RECENT_ENDS = 64) — so the memory fix still has no positive call evidence → incidents/2026-10-22-conduit-memory-start-timeout, systems/conduit.

Method note: daily tallies are bucketed with substr(started_at,1,10) — date(started_at) normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day (see the Count section).

Pages touched: incidents/2026-10-24-conduit-call-request-refused.md (new), systems/conduit.md, incidents/2026-10-22-conduit-memory-start-timeout.md (correction), systems/scheduled-tasks.md, current-state.md, index.md, log.md. Nothing removed or archived: no job retired, no system obsolete, no decision reversed in the window.

Still open: the single-node no-failover topology behind every model-calling job; the GitHub Trending Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 stopped; the Conduit patch uncommitted and five releases behind HEAD; the 569.9 MB Conduit health store with no retention; and a call that has never been placed since the memory fix.

Re-verified 2026-11-02 (audit window 2026-11-01 00:00 → 2026-11-02 00:4x AWST; live probes 00:0x–00:5x)

Settled: 167 completed / 1 running / 2 unknown — the first non-clean day after three consecutive clean-or-nearly-clean days (10-31 was 161 / 4). Fleet unchanged at 29 jobs, 26 enabled, 3 disabled. Nothing was added, removed, disabled or repinned in the window.

The day's three failure modes, none of which is a repeat of 10-31

# Fault Jobs Durable record Page
1 SIGTERM (-15) after adoption, no terminal state — "whether side effects ran is unknown" 85edf6772bca 07:30:46, 96eb95ce12f2 + 0e753e9720fd 09:02:18 3 cron_incidents rows, all resolved; one execution left unknown incidents/2026-11-01-cron-worker-sigterm-orphaned-claims
2 Gateway handoff failure — "Cron execution unverified … Gateway connection failed; no local fallback" b63ed5817212 (self-healed 68 s) and 79f9bb41ecf5 2 rows; 79f9bb_9c443b602eec still alerted incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure
3 last_status: error with no execution row and no incident row ecc1fc13655a, last_run 2026-11-01T12:01:00 none same page, "Second silent gap"

Fault 1 is a fourth variant of the orphan class (after 10-20 crash, 10-26 drained-SIGTERM, 10-31 cap force-drain) and the first with no gateway restart anywhere near it — so it cannot be explained by reload/drain behaviour. Fault 3 is the inverse of the usual silent failure: instead of a green row over a dead job, there is a red status with no row at all.

The 09:00 pair — cold-start cost meets a 30 s ack window

At 09:00:49.708 the spawner logged Cron external worker for job '96eb95ce12f2' did not acknowledge within 30s; leaving the durable execution claim untouched; the worker then spent 09:00:49 → 09:01:02 doing a full cold boot (Plugin discovery complete: 83 found, 69 enabled) before starting its job at 09:01:13 and being signalled at ~09:02:13. The 30 s window is smaller than this host's cold-start time. Not yet formalised as an Open item because the signal source is unidentified — see the incident page's open questions.

Delivery health: 8 rows, 7 delivered

deliveries.db for 11-01: 02:39:33 ✔, 03:00:37 ✔, 07:18:50 ✘ (066bb5191280, configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id), 07:34:49 ✔, 08:36:51 ✔, 10:44:12 ✔, 11:01:12 ✔ — and nothing after 11:01:12, which is how the missing Weekly Review was detected. The 066bb5191280 case is the second variant documented in systems/messaging-integrations (thread-less fallback send, run still marked failed).

Incident ledger movement

  • Closed on 11-01: ea79a2_a4661554a3c2 + ea79a2_9197df32320b (02:39:33 — the Mnemosyne Health Monitor pair from 10-26 / 10-31) and 0e753e_e7f8194a7d06 (10:44:12) — the Facebook birthday repin row that had been open since 2026-10-01 09:00:55, the oldest row on the board. That job's delivery_outcome: delivered on the same run.
  • Opened on 11-01: 5 rows; 4 resolved, 1 open.
  • Now unclosed: exactly one — 79f9bb_9c443b602eec (state: alerted, acked_at: None).

Deliberately not done

  • No full vault mirror pass (drift counts recorded in log.md; a complete pass is bigger than an audit's mandate).
  • No edits to ### Open beyond the 066bb5191280 note above: the 10-13 GitHub-token and Camofox containment items were not re-probed this window, so their lines are left as last verified.
  • Nothing removed or archived — no job retired, no system obsoleted, no decision reversed.

Still open (carried): the single-node no-failover topology behind every model-calling job; the GitHub Trending Radar token; Camofox containment under multi-tab load; the Conduit health store with no retention; plus new: the unreconciled admission files ~/.hermes/cron/admissions/141448f9…json and 69b5588f…json (both 11-01) and the unidentified -15 signal source.

Re-verified 2026-11-03 (audit window 2026-11-02 00:00 → 2026-11-03 00:2x AWST; live probes 00:0x–00:2x)

The first fully clean ledger day in five — and it contains two failures the ledger does not show.

Settled: 177 completed / 2 running / 0 unknown / 0 failed (179 rows) against 167 / 1 / 2 on 11-01, 160 / 4 on 10-31 and 138 / 0 on 10-30. Fleet unchanged at 29 jobs, 26 enabled, 3 disabled. Nothing was added, removed, disabled or repinned in the window. 96 of the completions are 96eb95ce12f2's 15-minute watchdog slots.

Both 11-01 unknown rows resolved in-window (see §Resolved 2026-11-02). Unclosed incident rows 1 → 0 — the first zero in the period. Non-clean last_status 2 → 1. Deliveries 9 rows, 9/9 delivered.

Both faults of the day are invisible in the ledger

# Fault Job What the ledger says Page
1 Iteration cap reached before the handoff — 9 h 01 m 19 s, Stage 2 never triggered, nothing reached Notion, and no cron_incidents row was raised dca8482e4f76 07:00:33 → 16:01:52 completed / error: NULL / delivery_outcome: suppressed incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence
2 9 h 43 m run, no report delivered, output left uncommitted 1a8e90ea2271 03:00:31 → 12:44:00 completed / error: NULL / delivery_outcome: suppressed this page → Known Issues → Open

Fault 1 is the third cap in seven days (10-27 tool-call budget, 10-28 iteration cap, 11-02 iteration cap) and the first that raised no incident at all — the two earlier rows carry their runs' INCOMPLETE reports verbatim in cron_incidents.error. That is the sharpest form of the §9 anti-pattern yet: the ledger did not merely stay green over a dead job, it lost the red row it used to produce. Silence from cron_incidents is not success.

Delivery health: 9 rows, 9 delivered

deliveries.db for 11-02: 01:27:21 ✔ (the Weekly Review recovery), 07:30:35 ✔, 07:42:21 ✔, 08:10:18 ✔, 08:41:14 ✔, 09:07:20 ✔, 10:23:15 ✔, 12:29:25 ✔, 18:45:37 ✔. The 11-01 066bb5191280 thread-miss did not recur. That job has no executions row on 11-02 at all (0 2 1 * *, next slot 2026-12-01), so its 11-01 delivery_failed flag will sit on the fleet for the rest of November — the fleet's only non-clean last_status.

Stocktake artefacts committed on the stocktake's behalf

1a8e90ea2271 finished with systems/skills-index.md modified and raw/stocktake/freshness-20261102.txt untracked in the working tree. Both are valid artefacts — freshness 4 stale / 113 fresh at a >21-day threshold; skills-index regenerated with Devops 108 → 109 (pulse-turn-handling added) — so this audit committed both rather than leave them to be overwritten by the next slot.

Method note (correction carried forward)

Two rules now bind this audit's tallies: group by substr(started_at,1,10), never date(...) (which normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day); and do not read a running/unknown row as a verdict until the job's expected runtime has elapsed — the Weekly Review was still working at the 00:4x reading and delivered at 01:27, 43 minutes later. An audit that runs two hours after midnight will read every long job mid-flight.

Pages touched

incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md (new), incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (status → resolved, Resolution section + answers to its open questions 1–3 added), systems/scheduled-tasks.md, systems/browser-backend.md, systems/skills-index.md (stocktake output, committed), raw/stocktake/freshness-20261102.txt (stocktake output, committed), index.md, log.md. Nothing removed or archived: no job retired, no system obsolete, no decision reversed in the window.

Still open (carried + new)

The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery token; Camofox containment under multi-tab load; the Conduit health store with no retention; the unreconciled admission files ~/.hermes/cron/admissions/141448f9…json and 69b5588f…json; the unidentified -15 signal source; and the 30 s worker-ack window vs this host's cold-start time. Plus new: whether 2026-11-02's 164 deduped Job Radar candidates are recoverable, whether a cap-hit should keep being recorded as completed, and why the 03:00 stocktake now takes ~9 h 45 m.

Re-verified 2026-11-04 (window 2026-11-03, AWST)

Summary. One genuine red row, raised by the harness itself. 2026-11-03 settled 170 completed / 1 failed / 1 running / 0 unknown (172 rows). The single failed row is the Job Radar Stage 1 (dca8482e4f76) — 07:00:33.819190 → 13:28:52.305995, RuntimeError: stream active timeout, a new terminal signature for a job whose last three failures were all the agent iteration cap. Its run session compacted five times (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) and went silent for 65 m 21 s after its last tool result at 12:23:28 before the harness wrote "This turn did not complete." at 13:28:49 → incidents/2026-11-03-job-radar-stream-active-timeout.

Count (live, 2026-11-04)

Measure 2026-11-03 window Change vs 11-02 window
Fleet 29 jobs — 26 enabled / 3 disabled unchanged
Ledger day 170 completed / 1 failed / 1 running / 0 unknown (172 rows) 11-02: 177 / 2 / 0 / 0
Non-clean last_status 2 (dca8482e4f76 error streak 1; 066bb5191280 delivery_failed) 1 → 2
Unclosed incident rows 1 (dca848_dfd4bea84f2a, open, alerted 13:28:55.141618) 0 → 1
Deliveries 7 rows, 7/7 delivered 11-02: 9 / 9
Gateway PID 175269 (started 2026-10-31 19:23:56) — no restart unchanged
Dashboard PID 176771 (started 2026-10-31 19:35:02) unchanged
Sessions in window 16 (14 cron, 1 ios 20260908_031541_49c051, 1 desktop 20261031_190813_95fbde) 14 started, all cron

What changed in the window

  • Job Radar Stage 1 died on a provider stream timeout — see the new incident page; it also answered the 11-02 page's open question 3: output is suppressed when the terminal state is completed, and delivered when it is failed. The cap-produced green row is the anomaly, not the red one.
  • Stage 2 declined twice, correctly (5fc292ff0f3b 11:30:36 → 11:32:16; 14:30:32 → 14:33:11) — no handoff existed. ~/.hermes/radar-ref/handoff/ still holds only 10-27, 10-29, 10-30.
  • New system page: systems/hermes-hq-push — plugin v0.9.27, tree mtime 2026-11-03 11:06, hermes plugins list → enabled (user), config.yaml plugins.enabled line 7556. Its manifest states it replaces dispatch-push, and the replacement is clean: no dispatch-push reference survives in config.yaml and its directory is gone. Distinct from the Cadu/Vory bridge on systems/push-notifications, which is untouched and still degraded.
  • The 18:00 Weekly Review's 7 h 26 m recovery held — no new unknown rows, and ecc1fc13655a's ghost error did not reappear in this window.

Files touched by this audit

  • New: incidents/2026-11-03-job-radar-stream-active-timeout.md, systems/hermes-hq-push.md.
  • Updated: this page (+frontmatter, +Count bullet, +this section), index.md, incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md (11-03 row + 2026-11-04 update), log.md.
  • Committed on the stocktake's behalf: systems/skills-index.md (modified) and raw/stocktake/freshness-20261103.txt (untracked) — the 03:00 job left both in the working tree again.
  • Nothing removed or archived: no job added, retired, disabled or repinned; no system obsoleted; no decision reversed; systems/push-notifications.md left as-is (not re-verified this window).

Still open (2026-11-04 — carries its predecessor forward)

The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery token; Camofox containment under multi-tab load; the Conduit health store with no retention; the unreconciled admission files ~/.hermes/cron/admissions/141448f9…json and 69b5588f…json; the unidentified -15 signal source; and the 30 s worker-ack window vs this host's cold-start time. New: whether the 65 m 21 s stream silence is the stream-active watchdog threshold or a silently dead provider socket; whether the five-compaction churn contributes; whether a stream timeout on a file-deliverable job should retry automatically; radar-ref/handoff/ retention (newest file 10-30); why Stage 2 still has no handoff to wait on; and why the 03:00 stocktake takes 7 h 40 m and commits nothing.

Re-verified 2026-11-06 (window 2026-11-05, AWST; live reads 2026-11-06 00:1x-00:4x)

The day was a recovery day, not a repeat

  • Both 11-04 casualties ran clean. 85edf6772bca (Tech & AI Newsletter Digest (Analytical), streak 1) completed its 07:00 run in 7 m 01 s (07:00:42 -> 07:07:43); its incident row closed at 07:07:42. dca8482e4f76 (Daily Marketing Job Radar - Discovery, streak 2, dead twice on 11-04) also completed its 07:00 run - but took 5 h 35 m 35 s (07:00:42 -> 12:36:18), and that duration is what held its two incident rows open until 12:36:17. Neither job re-raised RuntimeError: Upstream stream failed before completion.
  • No unknown, timeout or error row exists anywhere in the 11-05 ledger - the first such day since 11-02.
  • Deliveries: 10 rows, all delivered, none flagged for_failure - 07:02:11, 07:07:42, 07:30:39, 08:06:00, 08:35:30, 10:41:20, 12:15:25, 18:45:49, 19:25:00, 23:27:09 AWST.

Long runs (completed, not failures)

Job Slot Duration Against its norm
ea79a2107e36 Mnemosyne Health Monitor 00:00 7 h 01 m 28 s (00:00:43 -> 07:02:11) normal 17-31 m
1a8e90ea2271 wiki-stocktake-maintenance 03:00 3 h 01 m 58 s (03:00:32 -> 06:02:30) 11-02 7 h 40 m, 11-03 5 h 41 m
dca8482e4f76 Job Radar - Discovery 07:00 5 h 35 m 35 s recovered from 2 failures
5fc292ff0f3b Job Radar - Stage 2 14:30 8 h 56 m 17 s (14:30:53 -> 23:27:10) delivered 23:27:09
5fc292ff0f3b Job Radar - Stage 2 11:30 10 m 33 s normal
85edf6772bca Newsletter Digest 07:00 7 m 01 s normal

The 03:00 stocktake overran a third day - and committed this time

1a8e90ea2271 ran 03:00:32 -> 06:02:30 (3 h 01 m 58 s), so the streak now reads 7 h 40 m -> 5 h 41 m -> 3 h 02 m (improving, still ~6x norm), and it did commit its artefacts (2f3fc99, 05:55) - against the 11-04 state where it "reported incomplete and committed nothing". The vault repo /home/hermes/wiki picked that up at 05:58 (ee00b5c). The nightly audit did not have to commit the skills-index on the stocktake's behalf this pass.

Sessions in the window

26 sessions had activity touching 2026-11-05 - 21 cron, 4 subagent, 1 desktop. The only long-form one is the morning Mnemosyne Health Monitor run (local-b433c51e..., cron, 17 messages, 07:02-07:50): it requested tool approvals and received denials. No configuration change came out of it, and it is consistent with that job's 7 h 01 m ballooning above.

Still open (2026-11-06 - carries its predecessor forward)

Everything on the 2026-11-04 list stands unchanged and not re-probed this window (single-node no-failover topology; the GitHub Trending Radar delivery token; Camofox containment under multi-tab load; the Conduit health store with no retention; the unreconciled admission files; the unidentified -15 signal source; the 30 s worker-ack window vs cold-start; whether a stream-silence timeout should retry a file-deliverable job; radar-ref/handoff/ retention). New this window:

  • Live secrets sit verbatim in Mnemosyne rows and in two cron prompts. Three .env values (TELEGRAM_BOT_TOKEN, MATTERMOST_TOKEN, HERMES_CUSTOM_OMNIROUTE_20128_API_KEY) appear in memories, episodic_memory, working_memory, gists and memoria_facts.context_snippet plus the FTS mirrors - 17 rows, 4 + 6 + 7 per key - and each of the 7 uncompressed rotation snapshots reproduces the whole DB; all four values also sit verbatim in the 1.7 GB session store state.db, the likely capture origin. The Camofox key is embedded in the prompts of 0e753e9720fd and fad5a86979d4 (both enabled Facebook Birthday jobs) and therefore in cron/deliveries.db - the only config-backup/ DB carrying any tested value. Found, not remediated -> incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.
  • ea79a2107e36 ballooned twice out of two runs (7 h 01 m from the 00:00 slot, then still running 12 h after its 12:00:45 slot) - against a 17-31 m normal. Cause unexplained; the 12:00 run had delivered nothing at audit time.
  • 066bb5191280 (Uber Eats Monthly Spend Report) is now the fleet's only non-clean last_status (delivery_failed, streak 0) - carried, not re-diagnosed this window.
  • deepseek-paid-smart on omniroute (CT600) - carried, not re-probed this window. 11-05 read: HTTP 503 ALL_TARGETS_SKIPPED on omniroute:20128 vs HTTP 200 on omniroute-1 (100.78.63.20:20128 -> deepseek/deepseek-v4-flash). This audit remains the fleet's only deepseek-paid-smart pin (provider custom:omni-101), which is why it still completes.

Freshness

raw/stocktake/freshness-20261106.txt (re-run after this audit's writes): 0 stale (>=90d), 124 fresh, 3 no-date of 127 pages - the new incident page included. The three no-date pages are unchanged (SCHEMA.md, experiments/active/example.md, log.md).

Files touched by this audit

  • New: incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md, raw/stocktake/freshness-20261106.txt.
  • Updated: this page (+frontmatter, +Count bullet, +this section), current-state.md, index.md, log.md, incidents/2026-11-03-job-radar-stream-active-timeout.md, incidents/2026-11-04-upstream-stream-failure-0700-cohort.md (both -> status: resolved with closure timestamps), and the vault mirror (/home/hermes/wiki, both conventions).
  • Nothing removed or archived: no job added, retired, disabled or repinned; no system obsoleted; no decision reversed.

Re-verified 2026-11-07 (window 2026-11-06, AWST; live reads 2026-11-07 00:3x-00:5x)

The day

2026-11-06 closed 172 execution rows across 16 distinct jobs: 170 completed / 1 failed / 1 running (this audit, 23:59:31 - so the running row is not lost work). Deliveries 10 rows / 10 delivered / 0 for_failure (00:14:38, 07:06:34, 07:30:36, 08:29:16, 08:35:53, 12:20:23, 13:26:22, 14:42:58, 18:45:37, 19:48:39). Fleet 29 jobs (26 enabled / 3 disabled) - unchanged for a fifth consecutive audit; non-clean last_status still 1 (066bb5191280, delivery_failed, streak 0). Unclosed incident rows 0 -> 0: no cron_incidents row carries last_seen_at >= 2026-11-06 and none is outside closed/resolved, so the window's one failed row raised no row in the ledger and no alert (see below).

The day ends a one-day zero-failed streak: 11-05 was the first clean ledger day since 11-02, and 11-06 breaks it again - but by a new mechanism with no restart and no upstream provider involvement.

The window's only failure: a concurrent fire discarded the Job Radar's 11:30 slot

Full page: incidents/2026-11-06-job-radar-concurrent-fire-claim-discard. In short:

  • Stage 1 (dca8482e4f76) ran 07:00:44 -> 08:29:21 (1 h 28 m 37 s), completed, suppressed - its second consecutive clean run.
  • A direct fire of 5fc292ff0f3b (Write and Letters) started 08:27:09 (source: direct, not the scheduler's builtin) and ran 4 h 59 m 13 s, completing and delivering at 13:26:22.
  • The scheduler's own 11:30:32 slot had started and was running concurrently; at 13:26:38 it logged "fire claim ownership lost; interrupting stale run" and closed 13:26:44 failed with Fire claim ownership lost; stale result was discarded.
  • The 14:30:32 slot then ran a normal cycle (12 m 27 s, delivered 14:42:59) and the day landed in full: SAME_DAY_STRONG_COUNT 4, VERIFIED_COVER_LETTER_COUNT 4, MISSING_COVER_LETTER_COUNT 0, cache total 494, 10 today-dated rows, 20 artifacts present and non-empty, Run status: COMPLETE.
  • New signature meaning: every earlier lost its durable fire claim case came from a restart/reload orphaning a live run (incidents/2026-10-26-planned-restart-orphaned-cron-run, incidents/2026-10-31-gateway-reload-force-drain-orphaned-run). This one had no restart - the claim was lost to another fire of the same job. executions.db holds exactly one row whose error contains fire claim: this one.

Long runs in-window (>90 m; normal cycles are 12 m - 1 h)

Job Slot Duration Outcome
5fc292ff0f3b (direct fire) 08:27:09 -> 13:26:22 4 h 59 m 13 s completed, delivered 13:26:22
5fc292ff0f3b (11:30 slot) 11:30:32 -> 13:26:44 1 h 56 m 12 s failed - discarded as stale
ea79a2107e36 (Mnemosyne Health Monitor) 12:00:43 -> 19:48:39 7 h 47 m 56 s completed, delivered 19:48:39
dca8482e4f76 (Stage 1) 07:00:44 -> 08:29:21 1 h 28 m 37 s completed, suppressed
1a8e90ea2271 (stocktake) 03:00:32 -> 04:11:35 1 h 11 m 03 s completed, suppressed
5fc292ff0f3b (14:30 slot) 14:30:32 -> 14:42:59 12 m 27 s completed, delivered

The stocktake's overrun is over. 1a8e90ea2271 ran 1 h 11 m 03 s - the four-day streak reads 7 h 40 m -> 5 h 41 m -> 3 h 02 m -> 1 h 11 m for 11-03 -> 11-06, and it committed normally.

The previous audit's open question is closed: the Mnemosyne Health Monitor's 11-05 12:00:45 slot, which was still in flight 12 h+ at the last audit with no delivery, completed at 2026-11-06T00:14:39 after 12 h 13 m 54 s and delivered (00:14:38). Its next 12:00 slot (11-06) then took 7 h 47 m 56 s and also delivered. Ballooned but never lost, and no incident row was ever raised for either.

Sessions

15 cron sessions started in the 11-06 window (495 messages / 273 tool calls), with 16 sessions carrying activity (one desktop session, 146 messages). Busiest by message count: wiki-stocktake-maintenance 157, Daily Marketing Job Radar - Discovery 92, Enhanced Weekend Adventure Planner 82, Daily Marketing Job Radar - Write and Letters (14:30) 28, Mnemosyne Health Monitor 27.

Still open

Freshness

wiki-maintain.sh (full run: skills-index regenerated -> freshness audit -> Quartz rebuild + serve restart) completed at 2026-11-07T00:45:32+08:00, run after this audit's writes:

  • Skills index: systems/skills-index.md regenerated - 302 skills / 68 categories (unchanged; the wrapper de-dup fix holds).
  • Freshness: 0 stale (>=21d), 125 fresh, 3 no-date of 128 pages -> raw/stocktake/freshness-20261107.txt. Nothing crossed the 21-day threshold. The page count is +3 on the 2026-11-05 stocktake's 125 (122 fresh / 3 no-date), consistent with the new pages the last two audits and this one added - this audit contributes incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md.
  • Site: Quartz parsed 128 Markdown files and emitted 769 files (1 m 02 s); deployed to public/ and hermes-wiki-serve.service restarted. public/ is generated output - excluded from the freshness count by design.

Files touched by this audit

  • New: incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md, raw/stocktake/freshness-20261107.txt.
  • Updated: this page (+frontmatter, +Count bullet, +this section), current-state.md, index.md, log.md, runbooks/job-radar.md, and the vault mirror (/home/hermes/wiki, both conventions).
  • Nothing removed or archived: no job added, retired, disabled or repinned; no system obsoleted; no decision reversed.

Re-verified 2026-11-08 (window 2026-11-07, AWST; live reads 2026-11-08 00:0x-00:5x)

The day

  • 2026-11-07 settled 163 ledger rows — 159 completed / 3 failed / 1 running (the running row is this audit, cbfe434a3b5e, fired 23:59:58). Same fleet as the previous five audits: 29 jobs, 26 enabled.
  • Deliveries: 7 rows / 7 delivered / 0 for_failure (10 on each of 11-05 and 11-06 — no delivery fault to explain).

The window's failure: a restart drain orphaned two in-flight runs

  • Two of the three failed rows are one event. ea79a2107e36 (Mnemosyne Health Monitor, 00:00:42.918 → 05:02:22.870) and 1a8e90ea2271 (wiki-stocktake-maintenance, 03:00:31.515 → 05:02:22.174) both closed with RuntimeError: Cron job '…' lost its durable fire claim ownership. The drain happened at 05:00:43/47; systemd restarted the gateway at 05:01:00 (Scheduled restart job, restart counter is at 3.) after a requested restart sat in after-turn mode for the full 1800 s (#77184).
  • The restart was requested at ~04:29:52 — 2 s before both restart-safe cron workers exited status 1 ("after adopting this execution, without writing a durable terminal state; whether side effects ran is unknown", logged 04:29:54) — i.e. inside the conduit_push self-update window (0.122.2 → 0.134.4, 04:27:55 → 04:32:32, its own ledger recording restarting_gateway = failed). Working hypothesis, not proven: the plugin asked for the restart and could not perform it. The 0.22.5 → 0.23.0 updater is excluded (its receipt starts 05:05:13). → incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain
  • The third failed row is the watchdog working: 96eb95ce12f2 (05:02:41.105 → 05:02:59.640) exited 1 having caught the health monitor's RuntimeError traceback in agent.log — it reported the event ~19 s after it was written.

Still open

  • ⚠ Unclosed incident rows 0 → 2, and neither alerted. Both belong to the stocktake job and sit in state detected with alerted_at: null: 1a8e90_2f370ef764f5 (first seen 2026-11-07T05:02:21.573) and 1a8e90_628d914f1f97 (2026-11-07T04:29:54.815). The health monitor's two rows were closed 15:43:17 and resolved 17:08:52 the same day. This is the red row, no alert shape the 11-06 audit recorded for Job Radar, now reproduced on a different job.
  • ⚠ Non-clean last_status 1 → 2: 1a8e90ea2271 is now error with failure_streak: 2 — the 03:00 stocktake slot has failed on consecutive days (11-06 and 11-07) and again left its stocktake edits uncommitted; the 11-07 in-day entry folded those edits into its own commit so the tree was not left dirty. The other non-clean row is unchanged: 066bb5191280 (Uber Eats Monthly Spend Report), delivery_failed, streak 0, "configured thread_id 34795 … was not found".
  • incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts stays OPEN (untouched in-window).
  • Job Radar ran clean on 11-07 — no slot failed, no claim conflict; the 11-06 concurrency page is unchanged.

Long runs in-window (>90 m; normal cycles are 12 m - 1 h)

  • Mnemosyne Health Monitor ea79a2107e36 — 5 h 01 m before the drain killed it (its two previous slots ran 7 h 47 m 56 s and 12 h 13 m 54 s without losing the claim).
  • wiki-stocktake-maintenance 1a8e90ea2271 — 1 h 59 m 51 s before the drain killed it.

Skill inventory movement (in-window — 2026-11-07 07:43 AWST)

  • The stocktake's skills-index regeneration now reads 299 skills / 67 categories, down from 302 / 68, because four skills were moved into ~/.hermes/skills/.archive/ at 07:43: cron-model-repair, fleet-wide-failure-triage, hermes-instance-update, mail-lookup. The parked archive grew 46 → 50, and the Cron Model Repair category vanished with its only member.
  • ⚠ No active skill references any of the four — a grep of every SKILL.md outside .archive/ returns no absorbing skill, so if this was a curator consolidation the "absorbed into X" pointer is missing. Their content survives only in the parked copies.
  • One skill appeared: microsoft-office (Productivity 25 → 26), part of a 10-file skill refresh at 05:36 that lines up with the 0.22.5 → 0.23.0 update's own skill materialisation (the others were computer-use, claude-design, popular-web-designs, document-to-action-items, meeting-action-items, product-price-monitor, weekly-review-planning, grounded-citations, dogfood).
  • Filesystem vs. page: 300 SKILL.md on disk vs. 299 listed — the generator de-dups one nested duplicate, so 299 is the unique figure (behaviour unchanged).

Freshness / files touched