Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76 (Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and cover letters never ran (Notion DB held 451 entries, 0 added). Incident row dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified). Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5, deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5, topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3). Also: the two incident rows the 2026-10-26 planned restart raised auto-closed 00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of 108) — committed here and index.md corrected; tailscale.md peer table rebuilt from tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained the voice pipeline section. New: incidents/2026-10-27-job-radar-budget-exhausted.md Nothing removed or archived. No secrets.
94 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Scheduled Tasks | system | active | 2026-07-22 | 2026-10-28 | 2026-10-28 | 2026-11-03 | high |
|
|
Scheduled Tasks
Count
- Current as of 2026-10-28 00:2x AWST (live
jobs.json+executions.db) — the radar's budget overrun stopped being silent, and the radar is now two jobs. Window 2026-10-27: 172completed/ 1failed(+ this auditrunning). The single failure isdca8482e4f76(Daily Marketing Job Radar - Discovery), which ran 07:01:08 → 11:03:54 (4 h 2 m 46 s), closedfailed, and delivered a report that opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery/verify/dedup/classify finished, Notion ingestion, cover-letter generation and the final audit did not run. Incident rowdca848_e93a538ade6bopened 11:03:53,alerted11:03:59, still open (failure_type: auth, which does not describe a budget fault) → incidents/2026-10-27-job-radar-budget-exhausted. Response, same day: the radar was split into a two-stage pipeline — stage 1dca8482e4f76(Discovery,0 7 * * 1-5,deliver: local, failures → topic 1030) and new stage 25fc292ff0f3b(Daily Marketing Job Radar - Write and Letters,30 11,14 * * 1-5, Telegram topic 1030, created 2026-10-27T17:34:15; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, bothcompleted, deliverysuppressed,last_status: ok) → runbooks/job-radar. Fleet 28 → 29 jobs (26 enabled / 3 disabled). Unclosed incident rows 3 → 2: the two rows the 2026-10-26 planned restart raised (ea79a2_a4661554a3c2,ea79a2_c3ab2d784ba3) auto-closed 00:38:32, and the radar row opened. Non-cleanlast_status4 → 5 (churn):ea79a2107e36cleared took;dca8482e4f76joined aserror(streak 1) and08eb86f02d37re-joined asdelivery_queued— still non-clean:0e753e9720fd(error),fad5a86979d4(delivery_failed),066bb5191280(delivery_failed). Pins unchanged. The 2026-10-26 orphan class did not recur — every 10-27 run except the radar's completed. - Prior window — as of 2026-10-27 00:1x AWST (superseded by the 2026-10-28 entry above) (live
jobs.json+executions.db) — a planned restart interrupted a live job, and the fleet got a loud failure instead of a silent one. Window 2026-10-26 settled 169completed/ 3failed, and all three failures are a single fault:ea79a2107e36(Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → incidents/2026-10-26-planned-restart-orphaned-cron-run. 2026-10-25 settled 172 / 2. Fleet unchanged: 28 jobs — 25 enabled / 3 disabled (d40085da631cImportant-mail monitor,f77b7e46bd09Rhino AgentMail Inbox Watcher,0a7f933d06e2AgentMail Rhino Inbox Poll). Non-cleanlast_status5 → 4:c0a9fa314542(GitHub Trending Radar) and08eb86f02d37(Gmail Action Digest) both cleared took, whileea79a2107e36joins aserror, streak 2;0e753e9720fd(error) andfad5a86979d4+066bb5191280(delivery_failed) unchanged. Unclosed incident rows 2 → 3: bothea79a2_*rows — 14:33:37 "Cron execution unverified; reconcileadmissions/724a69108b…json" and 14:35:59 "lost its durable fire claim ownership" — are stillalerted, alongside0e753e_e7f8194a7d06(detectedsince 2026-10-01). One row closed:c0a9fa_764558b5ae83auto-closed 09:13:00 when the radar's 09:00 slot ran clean — every row of the 2026-10-16 503 resource-pressure incident is now closed → incidents/2026-10-16-omniroute-503-resource-pressure. Pins unchanged: 11custom:omni-101/smart-route, 6custom:omni-101/free-stack, 2custom:omni/smart-route, 1custom:omni-101/deepseek-paid-smart, 8 unpinned (global default stillcustom:omni-101/smart-route). One gateway start on 2026-10-26 (14:35:18, after a plannedSIGTERMat 14:33:35; teardown 54.06 s,.clean_shutdownmarker skipped) → systems/hermes-agent. ⚠fad5a86979d4's 03:08 slot completed but delivereddelivery_failed— Thread 34795 not found, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ The radar's 07:00 slot closedcompletedat 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted” — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with modeldeepseek-paid-smart→ runbooks/job-radar. ⚠ Off-cron, unchanged: Nous Portal credentials still invalid → incidents/2026-10-22-nous-portal-refresh-token-invalid. Sessions in window: 17 (15cron, 1ios, 1telegram). - Current as of 2026-10-26 00:1x AWST (superseded by the 2026-10-27 entry above) (live
jobs.json+executions.db) — fleet is clean, but one job misbehaved badly and the global model default moved back. Window 2026-10-25 settled 172completed/ 2failed; both failures are08eb86f02d37(Gmail Action Digest), which also fired six times that day → incidents/2026-10-25-gmail-digest-runaway-and-refire. 2026-10-24 settled 168 / 0 (the 2026-10-25 entry read 167 — its day was still open). Fleet unchanged: 28 jobs — 25 enabled / 3 disabled; all 28 IDs still on this page. Unclosed incident rows still 2 (0e753e_e7f8194a7d06detectedsince 2026-10-01;c0a9fa_764558b5ae83alertedsince 2026-10-19) — nothing opened or closed in-window, so neither 10-25 failure raised an incident row. Non-cleanlast_status4 → 5:08eb86f02d37joins asdelivery_queuedwith an emptylast_error(never resolved took), alongside0e753e9720fd(error),fad5a86979d4+066bb5191280(delivery_failed) andc0a9fa314542(error, streak 1). Pins unchanged: 11custom:omni-101/smart-route, 6custom:omni-101/free-stack, 2custom:omni/smart-route, 1deepseek-paid-smart, 8 unpinned. ⚠ Global default reverted to OmniRoute on 2026-10-24 19:06 (custom:omni-101/smart-route), so the 8 unpinned jobs are back on the same upstream as the pinned ones — the fleet is a single path again → systems/current-profile. ⚠ Three gateway starts on 2026-10-25 (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → systems/hermes-agent. ⚠ Nous Portal credential still invalid (Invalid refresh token) — context summaries unavailable on 10-25 (20 log lines); all 8auxiliaryslots were moved tono-think/freeapi/autooncustom:omni-101at 19:13 → incidents/2026-10-22-nous-portal-refresh-token-invalid. Sessions in window: 21 (18cron— 15smart-route, 2free-stack, 1deepseek-paid-smart; 1desktop; 1oneshot; 1telegram). - Current as of 2026-10-25 00:1x AWST (
jobs.json.updated_at 2026-10-25T00:12:26+08:00) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's. Window 2026-10-24: 167completed/ 0failed(+ this auditrunning); 2026-10-23 settled at 170 / 0 (169 at the 10-24 audit). That makes 10-22 → 10-24 three clean days in a row. Fleet unchanged: 28 jobs — 25 enabled / 3 disabled. Unclosed incident rows still 2 and nothing opened or closed in-window:c0a9fa_764558b5ae83(alertedsince 10-19, next slot Mon 10-26 09:00) and0e753e_e7f8194a7d06(detectedsince 10-01, next slot 11-01). Non-cleanlast_statusstill 4:c0a9fa314542(error, streak 1 — Mon 10-26),0e753e9720fd(error, 0 — 11-01),fad5a86979d4and066bb5191280(delivery_failed— Mon 10-26 / 11-01). Pins unchanged: 11custom:omni-101/smart-route, 8 unpinned (global default), 6custom:omni-101/free-stack, 2custom:omni/smart-route, 1custom:omni-101/deepseek-paid-smart(this audit). 14 sessions in the window: 11cron+ 3desktop— two of thedesktopones are the 12:53–12:56 call requests that the agent refused → incidents/2026-10-24-conduit-call-request-refused. ⚠ New off-cron fault, open: the agent told Anthony it cannot make phone calls, three times, with zero tool calls, whileconduit_call_userwas onetool_searchaway → same incident page. (Superseded by the 2026-10-26 entry above.) - Current as of 2026-10-24 00:2x AWST (superseded by the 2026-10-25 entry above) (
jobs.json.updated_at 2026-10-24T00:17:36+08:00) — a clean day, and the 2026-10-09 timeout storm is finally closed. Window 2026-10-23: 169completed/ 0failed(+ this auditrunning) — the third zero-failure day in four; 2026-10-22 settled at 173 / 0. Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.1b23c042e17d(Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its last two incident rows (1b23c0_c35703f637b8+1b23c0_103cbe04a79b) — all 12 rows of the 10-09 storm are closed and that job readsokagain → incidents/2026-10-09-cron-provider-timeout-storm. Unclosed incident rows 4 → 2:c0a9fa_764558b5ae83(alertedsince 10-19, next slot Mon 10-26 09:00) and0e753e_e7f8194a7d06(detectedsince 10-01, next slot 11-01). Non-cleanlast_status5 → 4:c0a9fa314542(error, streak 1 — Mon 10-26),0e753e9720fd(error, 0 — 11-01),fad5a86979d4and066bb5191280(delivery_failed— Mon 10-26 / 11-01). Pins unchanged: 11custom:omni-101/smart-route, 8 unpinned (global default), 6custom:omni-101/free-stack, 2custom:omni/smart-route, 1custom:omni-101/deepseek-paid-smart(this audit). 14 sessions in the window: 13cron+ 1desktop(18:26:12 — the Conduit phone-call session) → systems/conduit. The stocktake's 10-22 overrun did not repeat: its 10-23 03:07 slot ran and committed (250c735: freshness 2 stale / 94 fresh, skills-index 306). → systems/skills-index - Current as of 2026-10-23 00:1x AWST (superseded by the 2026-10-24 entry above) (
jobs.json.updated_at 2026-10-23T00:07:25+08:00) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing. Window 2026-10-22: 172completed/ 0failed(+ this auditrunning) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. A DB recount bystarted_atand byclaimed_atputs 2026-10-21 at 168 / 1, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. Fleet unchanged: 28 jobs — 25 enabled / 3 disabled. Five incidents closed in-window and none opened —0e5198_be2291809e56+0e5198_ec2fac98a91d(0e519874d280, closed 08:01:40),06a0a1_9e0ff4eedc49(10:14:34),c8d529_e8c70250f9be(18:17:59),dca848_014affeb6954(07:59:22) — so unclosed rows fall 9 → 4 (3alerted/ 1detected:1b23c0_c35703f637b8,1b23c0_103cbe04a79b,c0a9fa_764558b5ae83,0e753e_e7f8194a7d06). Non-cleanlast_statusfalls 9 → 5:1b23c042e17d(error, streak 2 — next slot Fri 2026-10-23 08:00),c0a9fa314542(error, 1 — Mon 10-26),0e753e9720fd(error, 0 — 11-01),fad5a86979d4(delivery_failed— Mon 10-26 03:00),066bb5191280(delivery_failed— 11-01). All four cleared rows had been red since 2026-10-15 (theomni-101503 burst) and each ran clean on its first slot of 10-22. Pins: 11custom:omni-101/smart-route, 8 unpinned (provider/modelnull → global default), 6custom:omni-101/free-stack, 2custom:omni/smart-route, 1custom:omni-101/deepseek-paid-smart(this audit). ⚠ Stocktake anomaly:1a8e90ea2271ran 03:00:28 → 04:38:19 (97 m 58 s) against 16 m 01 s on 10-21, finishedcompletedwith deliverysuppressed— and committed nothing. It left its regeneratedsystems/skills-index.mduncommitted (mtime 10-22 03:00, with a## .Archive (41)section reading 347) and its diagnosed fixes unapplied → incidents/2026-08-15-skills-index-drift. 27 sessions in the window: 15cron, 3subagent(the AI-Newsletter research fan-out), 6desktop(voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, modelsmimo-v2.5,or-inclusionai_ling-2_6-1t,minimax/minimax-m3.1-flash), 2acp(0 messages), 1oneshotmemory-capture probe. → systems/skills-index, incidents/2026-08-15-skills-index-drift - Current as of 2026-10-22 00:1x AWST (superseded by the 2026-10-23 entry above) (
jobs.json.updated_at 2026-10-22T00:11:31+08:00) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen. Window 2026-10-21: 171completed/ 1failed(+ this auditrunning). The single failure isdca8482e4f76(Daily Marketing Job Radar, 07:00 weekday,custom:omni-101/smart-route), which died at 07:40:48 after 40 m 36 s onRuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s— not the 503 resource-pressure string, which appears in no 2026-10-21 row → incidents/2026-10-21-omniroute-pre-dispatch-filter-skip. Fleet unchanged: 28 jobs — 25 enabled / 3 disabled. Non-cleanlast_statusstays at 9 (7error/ 2delivery_failed) but the composition moved:08eb86f02d37cleared (its 08:30 slot ran clean; incident08eb86_472c7b609c80resolved 2026-10-21 08:37:21) whiledca8482e4f76re-entered (streak 1). Survivors, with next run:dca8482e4f76(1, 10-22 07:00),0e519874d280(2, 10-22 08:00),06a0a1c8704f(1, 10-22 10:00),c8d52904e1d6(1, 10-22 18:00),1b23c042e17d(2, Fri 10-23 08:00),c0a9fa314542(1, Mon 10-26 09:00),fad5a86979d4(delivery_failed, Mon 10-26 03:00),0e753e9720fd(error, 0, 11-01 09:00),066bb5191280(delivery_failed, 11-01 02:00). Incident ledger: 1 opened / 1 closed, unclosed stays 9 (8alerted/ 1detected). Pins: 17 of the 28 jobs pincustom:omni-101(10smart-route, 6free-stack, 1deepseek-paid-smart), 7 carry no pin at all (provider/modelnull → global default), 1 pinscustom:omni/smart-route. 2026-10-20 settled at 169completed/ 1unknown— the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. 12 sessions in the window, allcron— no interactive sessions. ⚠ Infrastructure changed in-window: Hermes Agent was updated to0.22.3at 22:47 → 22:57 with its pre-update snapshotskipped→ systems/hermes-agent, infrastructure/backups. - ⚠ Current as of 2026-10-21 00:0x AWST (superseded by the 2026-10-22 entry above) (
jobs.json.updated_at 2026-10-21T00:07:08+08:00) — a clean day, one orphaned run, and one long-running silent failure. Window 2026-10-20: 168completed/ 0failed(plus 1unknownand this audit) — the first zero-failure day since 2026-10-12, ending a seven-day run of failures, and theomni-101503 fault did not recur (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). Zero incident rows opened and 4 closed (dca848_867d9caa96a8,85edf6_256c5e150bc4,ecc1fc_c9976ea24553,ecc1fc_21a72aa11cc4), so unclosed rows fall 13 → 9 (8alerted/ 1detected). ⚠ The exception is a failure this ledger does not record:08eb86f02d37(Gmail Action Digest, 08:30) was claimed 08:31:03 and closedunknownwhen the gateway exited uncleanly — “Scheduler restarted after this execution's owner exited before a durable terminal state”; no delivery row exists for the slot and no incident row was raised → incidents/2026-10-20-gateway-unclean-exit-orphaned-run. Non-cleanlast_status12 → 9 (7error/ 2delivery_failed); everyfailure_streak≤ 2. Fleet unchanged: 28 jobs — 25 enabled / 3 disabled. ⚠ Discovery — green row, dead job:e27da6c34bbb(Config Backup, 11:00) readsokwhile its Gitea push has failed every night since 2026-08-08 (75 unpushed commits) → incidents/2026-08-08-gitea-config-backup-push-dead. Model pins unchanged — 20 of 28 jobs still pin OmniRoute — but the 8 unpinned jobs moved upstream tonousdirect with the global default change, so the fleet now has two upstream paths. 2 interactive sessions in the window (talaria-iostax work; adesktopturn that received the canned non-answer). → incidents/2026-10-16-omniroute-503-resource-pressure - Current as of 2026-10-20 00:09 AWST (superseded by the 2026-10-21 entry above) — the
omni-101503 fault is BACK, and it is now the fleet's defining risk. Window 2026-10-19: 159completed/ 13failed, against 10-18's 168 / 1 and 10-17's 163 / 5. 9 genuine model-call failures across 8 jobs all carry the identicalRuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.oncustom:omni-101, in two clusters — 06:01 (one job) and 07:01 → 12:01 (seven more runs across seven jobs). The other 4 rows are derivative Mnemosyne Watchdog exits (96eb95ce12f2at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the second burst in four days on the same node with the same blast radius, while/api/healthread200 {"status":"ok"}throughout — so incidents/2026-10-16-omniroute-503-resource-pressure is back tostatus: active(it had been downgraded tomonitoring44 h earlier). Fleet unchanged at 28 jobs — 25 enabled / 3 disabled. Non-cleanlast_status7 → 12 (10error+ 2delivery_failed); unclosed incident rows 13 (12alerted/ 1detected) — 4 opened and 2 closed on 10-19. Everyfailure_streakstill reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. 15 sessions in the window, allcron— no interactive sessions. → incidents/2026-10-16-omniroute-503-resource-pressure - Current as of 2026-10-19 00:20 AWST (superseded by the 2026-10-20 entry above) (
jobs.json.updated_at 2026-10-19T00:17:06+08:00) — one failure in 24 h, and it was a failure mode this wiki had not recorded before. Fleet 28 jobs — 25 enabled / 3 disabled (unchanged). Window 2026-10-18: 168completed, 1failed— the failure isecc1fc13655a(FreshRSS Midday, 12:01:10 → 12:08:25), which wrote and delivered a canned “Your request was not processed…” sentence to Telegram (delivery_outcome: deliveredat 12:08:11) before dying onRuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.→ incidents/2026-10-18-degenerate-stream-canned-substitute. Non-cleanlast_status: 8 → 7 —1a8e90ea2271(10-18 03:01) andcbfe434a3b5e(this audit, 10-18 00:42) cleared on schedule, leaving1b23c042e17d(error, streak 2, next Fri 10-23 08:00),0e519874d280(Mon 10-19 08:00 — next to fire),c8d52904e1d6+06a0a1c8704f(Thu 10-22),ecc1fc13655a(10-19 12:00), and the monthlies0e753e9720fd+066bb5191280(both 11-01). Open incidents 8 → 7 (6alerted/ 1detected); 1 opened, 2 resolved in-window. Tallies: 10-17 = 163 / 5, 10-18 = 168 / 1, 10-19 so far 5 / 0 (+ this auditrunning); 11 sessions in the window, allcron— no interactive sessions. → incidents/2026-10-18-degenerate-stream-canned-substitute - Current as of 2026-10-18 00:31 AWST (
jobs.json.updated_at 2026-10-18T00:31:17+08:00) — theomni-101503 burst is over and the fleet is unwinding normally. Fleet 28 jobs — 25 enabled / 3 disabled. 8 carry a non-cleanlast_status(down from 12 at the 10-16 storm peak):1b23c042e17d(error, streak 2),cbfe434a3b5e(1),1a8e90ea2271(1),0e519874d280(1),06a0a1c8704f(1),c8d52904e1d6(1),0e753e9720fd(error, streak 0),066bb5191280(delivery_failed). Four rows cleared as their slots came round:b63ed5817212(10-17 06:01),08eb86f02d37(10-17 08:31),ecc1fc13655aandea79a2107e36(both 10-17 12:02). Open incidents 12 → 8 (5alerted/ 3detected); 9 incidents opened and 6 closed inside this window. Daily tallies: 10-16 = 165 completed / 8 failed, 10-17 = 162 / 5 (all five before 03:01; DB re-read 2026-10-20: 163 / 5 — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology:RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.oncustom:omni-101, 10-16 08:00 → 10-17 03:00. → incidents/2026-10-16-omniroute-503-resource-pressure - ⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery. 12 non-clean
last_status(was 3 at the 10-13 audit): 9 new rows from theomni-101outage —08eb86f02d37(streak 2),ecc1fc13655a(2),dca8482e4f76,c8d52904e1d6,06a0a1c8704f,85edf6772bca,0e519874d280,1a8e90ea2271,cbfe434a3b5e(all streak 1) — plus the three long-standing rows1b23c042e17d(since 10-09),0e753e9720fd(10-01) and066bb5191280(delivery_failed, 10-01). Three jobs already cleared inside this window:b63ed5817212(10-16 00:02),1b31187397a5(10-15 22:01),ea79a2107e36(10-16 00:09). Structural finding: all 18 enabled model-calling jobs are pinned tocustom:omni-101— the cron fleet has no model-call failover. → incidents/2026-10-14-omniroute-node-connection-refused-storm - Total jobs: 28 — 25 enabled, 3 disabled (verified from live
~/.hermes/cron/jobs.jsonat 2026-10-19 00:20 AWST). The count fell 30 → 29 on 2026-10-10 (redundant DHL monitor62619aa6bcb3retired) and 29 → 28 when Mosquito Tracker1b31187397a5was removed — its last execution was 2026-10-16 15:00 (completed, delivered) and it is absent fromjobs.jsonas of this audit. See Removed Jobs. - Non-clean
last_status: 12 as of 2026-10-20 00:09 — 10error+ 2delivery_failed. The burst re-dirtied four rows that were clean at the 10-19 audit (dca8482e4f76,85edf6772bca,08eb86f02d37,c0a9fa314542) and deepened two of the existing seven (0e519874d280andecc1fc13655aboth tofailure_streak: 2);fad5a86979d4newly readsdelivery_failed(bare-telegramtarget).b63ed5817212(dirtied twice) and96eb95ce12f2(four derivative reds) cleared the same day (18:01:56 / 12:46:11), andea79a2107e36cleared on its 10-20 00:01 slot. Survivors, with next run:ecc1fc13655a(streak 2, 10-20 12:00),0e519874d280(2, Thu 10-22 08:00),1b23c042e17d(2, Fri 10-23 08:00),dca8482e4f76(1, 10-20 07:00),85edf6772bca(1, 10-20 07:00),08eb86f02d37(1, 10-20 08:30),c0a9fa314542(1, Mon 10-26 09:00),06a0a1c8704f(1, Thu 10-22 10:00),c8d52904e1d6(1, Thu 10-22 18:00),fad5a86979d4(delivery_failed, Mon 10-26 03:00),0e753e9720fd(error, 0, 11-01 09:00),066bb5191280(delivery_failed, 0, 11-01 02:00). History: 3 (10-12 audit) → 12 (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 now. Three of these fire within eight hours of this audit, so the count will move again by morning. - Failing (last run failed outright): 0 — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider
custom:omni-101using routing aliases, and everyfailure_streakinjobs.jsonnow reads 0. Four of the seven jobs had still carried a pre-repinlast_status: error(1b23c042e17d,fad5a86979d4,0e753e9720fd,1a8e90ea2271) — a stale field, not a live failure; it clears on each job's next scheduled run.1a8e90ea2271cleared on 2026-10-06 03:00, running on schedule for the first time since 2026-09-25; the remaining three (1b23c042e17d,fad5a86979d4,0e753e9720fd) are due Fri 08:00 / Mon 03:00 / 1st 09:00.fad5a86979d4cleared on 2026-10-12 03:00, leaving two (1b23c042e17d,0e753e9720fd). See incidents/2026-10-05-dead-model-pin-repin.- Historical (2026-10-01 → 2026-10-05), now all remediated:
dca8482e4f76403MODEL_NOT_IN_PLAN×7;1a8e90ea2271HTTP 410 ×11 (retiredqwen3.5:397b);fad5a86979d4HTTP 400 ×3;0e753e9720fdHTTP 400 ×1;06a0a1c8704f,1b23c042e17dand79f9bb41ecf5HTTP 401 (thecliprox/terraconnection auth expiry, settled 3/3 on 2026-10-04).
- Historical (2026-10-01 → 2026-10-05), now all remediated:
- Date-bucketing note (added 2026-10-25):
executions.dbstoresstarted_atwith a+08:00offset, and SQLite'sdate()normalises to UTC — sodate(started_at)moves every row that ran 00:00–07:59 AWST onto the previous day. Daily tallies on this page are bucketed withsubstr(started_at,1,10), which preserves AWST calendar days. A recount viadate()reads low by roughly a day's early-morning runs; this is the same class of drift that put 10-21 at 168 instead of 171. - Failing (delivery only): 1 —
066bb5191280: the report itself generated; the send failed against the DM thread (baretelegramtarget, 2026-10-01). Unchanged this window. - Repin applied 2026-10-05:
free-stack×6 (06a0a1c8704f,1b23c042e17d,79f9bb41ecf5,1a8e90ea2271,fad5a86979d4,0e753e9720fd) andsmart-route×1 (dca8482e4f76), all oncustom:omni-101. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it.free-stackis in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs. - Verified live, not assumed:
79f9bb41ecf5succeeded on a manual run 2026-10-05 23:39 and06a0a1c8704fat 23:45 (both incidents auto-resolved);dca8482e4f76completed 23:56 and delivered totelegram:-1004321904721:1030(incident resolved 2026-10-06 00:05:07); and1a8e90ea2271completed automatically on 2026-10-06 03:00. Four of the seven are therefore verified by live run. The remaining three (1b23c042e17d,fad5a86979d4,0e753e9720fd) are repinned but not yet re-run — they are due on their normal schedules.1a8e90ea2271has since run twice more on schedule (2026-10-06 03:00 →33f4e95; 2026-10-07 03:00 →8c63ef2), so the stocktake repin is verified across consecutive automatic runs. - Streak counting note: the
N failures in a rowfigure inhermes cron listcomes from thefailure_streakcounter injobs.json.executions.dbretains only rows from 2026-09-30 onward, so DB-based recounts undercount older streaks. Live counters as of 2026-10-06: all 0. - Re-verified 2026-10-06 AWST: job count unchanged at 30 (27 enabled / 3 disabled).
066bb5191280and0e753e9720fdstill carry the bare-telegramtarget warning (configured thread_id 34795 for telegram:1793951355 was not found). - Re-verified 2026-10-07 AWST: job count unchanged at 30 (27 enabled / 3 disabled); 0 failing — all 30
failure_streakcounters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176completed, 0failed, 1running(the audit itself); 14 sessions, allcron— no interactive (non-cron) sessions. The stocktake job1a8e90ea2271ran on schedule at 03:00 and regenerated systems/skills-index (344 → 346 skills) plusraw/stocktake/freshness-20261006.txt, committing to Gitea as33f4e95— the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repinlast_status: error(1b23c042e17d,fad5a86979d4,0e753e9720fd). - A stale notice, not a live outage, drove this review. The interactive session on 2026-10-05 23:16 began because Anthony replied to a
85edf6772bcafailure notice that had been sitting in topic346since the 2026-09-29 07:00 run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were alreadyresolvedon 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Checkexecutions.started_at(and thecron_incidentsstate) before acting on a notice. - Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00): job count unchanged at 30 (27 enabled / 3 disabled); 0 failing — every
failure_streakreads 0. Window tally (executions.db): 176completed, 1failed, 1running(the audit itself). The single failure is62619aa6bcb3(DHL Tracking, 08:00) — a Camofox HTTP read timeout (ensure_browser()→TimeoutError: timed out), notified 08:00:58 and auto-resolved 11:01:13 when the next 3-hourly run succeeded (incident62619a_f485ccfd4a0d). 13 sessions started in the window, allsource = cron— no interactive sessions; 12 deliveries, 0 delivery failures. The same three stale pre-repinlast_status: errorjobs remain (1b23c042e17d,fad5a86979d4,0e753e9720fd); the stocktake job1a8e90ea2271ran on schedule at 03:00 for the second consecutive day. - Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, 0 failing (2026-10-06) → 30, 0 failing (2026-10-07) → 30, 0 failing (2026-10-08) → 29, 0 failing (2026-10-10) → 29, 0 failing (2026-10-11) → 29, 0 failing (2026-10-12)
- Re-verified 2026-10-12 AWST (window 2026-10-11 00:00 → 2026-10-12 00:00): 29 jobs (26/3), 0 failing, 5 non-clean
last_status; window tally 172completed, 0failed(the 2026-10-11 day finalised one run higher than the previous audit's snapshot, which caught it mid-flight at 171 + 1running). 18 sessions — 17cron+ 1 interactive. - Re-verified 2026-10-13 AWST (window 2026-10-12 00:00 → 2026-10-13 00:00): 29 jobs (26/3), 0 failing, non-clean
last_statusdown to 3. Window tally 174completed, 0failed— third consecutive clean day (10-10175/0, 10-11172/0, 10-12174/0). 5 open incidents → 2: three closed on 2026-10-12 (see #Resolved 2026-10-12). 18 sessions in the window, allcron, no interactive sessions; 0 new incident pages required. Two findings led the window: the GitHub Trending Radar delivery token is dead (getMe→ 401, live-verified) and the Daily Marketing Job Radar run hit Camofox memory pressure (2.5 GB peak, heavy swap, self-recovered by restart).
Job Table
| Name | ID | Schedule | Delivery | Skills | Script | Notes |
|---|---|---|---|---|---|---|
| AgentMail Rhino Inbox Poll | 0a7f933d06e2 |
* * * * * |
origin | terminal | agentmail-poll.py |
disabled since 2026-07-27 (legacy) |
| AI Newsletter Curation for Patrick (Weekly) | 06a0a1c8704f |
0 10 * * 4 |
telegram:-1004321904721:346 | — | — | ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was custom:Omniroute:20128 / cliprox/terra, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live last_status: ok, streak 0 |
| Daily Birthday Checker | a432898b55a4 |
30 7 * * * |
telegram:1793951355 | — | birthday_check.py |
|
| Daily Marketing Job Radar — Discovery (stage 1) | dca8482e4f76 |
0 7 * * 1-5 |
telegram:-1004321904721:1030 | job-search-automation | — | ✅ Repinned 2026-10-05 → custom:omni-101 / smart-route (was command-code / gpt-5.6-terra, 403 MODEL_NOT_IN_PLAN ×7). ✅ First scheduled success on the repin 2026-10-12 07:00 (finished 07:33:50, delivered) — its 2026-10-09 timeout incident dca848_23fcd2b18aaa auto-closed 07:33:41. ⚠ The 2026-10-12 run hit Camofox memory pressure — the service reached its containment ceiling (2.5 GB peak, MemoryHigh=2G, swap 1 GB fully consumed, available: 0B), so /tabs calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ The 2026-10-19 07:01 slot failed on the omni-101 503 (new incident dca848_867d9caa96a8, alerted) — the second time this job has been caught by that node's faults. Live last_status: error, failure_streak: 1, next slot 2026-10-20 07:00 |
| Daily Marketing Job Radar — Write and Letters (stage 2) | 5fc292ff0f3b |
30 11,14 * * 1-5 |
telegram:-1004321904721:1030 | job-search-automation | — | ✅ Created 2026-10-27 17:34 — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). custom:omni-101 / smart-route; first fires 2026-10-27 17:35 and 17:38 both completed. See incidents/2026-10-27-job-radar-budget-exhausted |
| DHL Tracking - Parcel 8052838382 | 62619aa6bcb3 |
0 8,11,14,17,20 * * * |
telegram:1793951355 | — | dhl_tracking_monitor.py |
no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (ensure_browser() → TimeoutError: timed out) — incident 62619a_f485ccfd4a0d, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 Delivered — the parcel has arrived, so the monitor is now redundant |
| Enhanced Weekend Adventure Planner | 1b23c042e17d |
0 8 * * 5 |
telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix confirmed working (delivered 2026-10-02, flag cleared). ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the failed cliprox/terra pin, HTTP 401 ×1). ⚠ Last run 2026-10-09 08:00 failed — RuntimeError: Request timed out. (incident 1b23c0_c35703f637b8, still alerted, opened 2026-10-09 08:17) — so it still carries last_status: error, streak 1 → now streak 2. ⚠ The Fri 2026-10-16 08:00 slot failed too — RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. (incident 1b23c0_103cbe04a79b, alerted). It cannot clear before Fri 2026-10-23 08:00, so a single missed slot now reads as a two-week-old failure. → incidents/2026-10-16-omniroute-503-resource-pressure |
| Evening Stand-Down Briefing | e530c89f2be0 |
45 18 * * 1-5 |
telegram:-1004321904721 | — | evening_standdown.py |
no-agent |
| Facebook Birthday Monthly Scrape | 0e753e9720fd |
0 9 1 * * |
telegram | — | — | ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the catalog-missing nvidia/deepseek-ai/deepseek-v4-flash pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident 0e753e_e7f8194a7d06 (opened 2026-10-01 09:00:55) is still open in state detected, and last_status still reads error; next slot 2026-11-01 09:00, which is when the repin finally gets its live test. ⚠ Bare telegram still targets the home DM — retarget to a group topic |
| Facebook Birthday Weekly Registry Sync | fad5a86979d4 |
0 3 * * 1 |
telegram | — | birthday_check.py |
✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the catalog-missing nvidia/deepseek-ai/deepseek-v4-flash pin, HTTP 400 ×3). ✅ Verified live 2026-10-12 03:00 — completed and delivered (448 birthdays); the stale last_status: error cleared after two weeks and both deepseek-v4-flash incidents auto-closed at 03:05:39 (fad5a8_a13bbc837e1c, fad5a8_c80b59a907f0). ⚠ Live status is no longer ok: the 2026-10-19 03:04:47 run completed but recorded delivery_failed — configured thread_id 34795 for telegram:1793951355:34795 was not found (the CLI renders it as "ran, but the result was not delivered", then "delivered without thread_id"). The run is fine; only the send is. No incident row was created for this. ⚠ Bare telegram target should move to a group topic |
| FreshRSS Midday News Briefing | ecc1fc13655a |
0 12 * * * |
telegram:-1004321904721:346 | — | freshrss_intelligence_v2.py |
⚠ The 2026-10-16 12:00 slot failed on the omni-101 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed that incident. ⚠ Then 2026-10-18 12:00 failed differently — a model repetition loop aborted the run at 12:08:25 after it had delivered a canned "Your request was not processed…" line to the topic (12:08:11) → incidents/2026-10-18-degenerate-stream-canned-substitute. ⚠ Then the 2026-10-19 12:01 slot failed again, on the omni-101 503 — ecc1fc_c9976ea24553 (a 10-16 row, refreshed 10-19 12:02:58, alerted) plus the separate ecc1fc_21a72aa11cc4 degenerate-stream row from 10-18, so this one job holds two open incidents. Live last_status: error, failure_streak: 2, next slot 2026-10-20 12:00 |
| GitHub Trending Radar — Weekly Roundup | c0a9fa314542 |
0 9 * * 1 |
telegram:1793951355 | — | — | ⚠ Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13). A getMe call against the telegram.bot_token in /home/hermes/workspace/github-trending-radar/config.json returns HTTP 401 Unauthorized. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at digest.py's send(). The agent recovered the content with --dry-run and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new @BotFather token replaces the dead one. This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong. ⚠ The status reading changed again on 2026-10-19: its scheduled Monday slot (0 9 * * 1) at 09:01 failed on the omni-101 503 (new incident c0a9fa_764558b5ae83, alerted), so this week's roundup did not run at all and last_status now reads error (streak 1). The token question is therefore untested for a second consecutive Monday — the job has not reached digest.py on a Monday since 2026-10-12. Next real test: Mon 2026-10-26 09:00. |
| Gmail Action Digest | 08eb86f02d37 |
30 8 * * * |
telegram:-1004321904721:notifications | — | gmail_action_digest_v3.py |
⚠ The 2026-10-16 08:30 slot failed on the omni-101 503 (streak 2 at the storm peak); and its own 10-17 08:31 run restored a clean last_status. ⚠ But the incident never closed: 08eb86_472c7b609c80 was refreshed on 2026-10-19 08:33 by the same 503, and is still alerted. Live last_status: error, streak 1 |
| Important-mail monitor | d40085da631c |
every 30m |
telegram:1793951355 | — | _cron_silent_monitor.py |
disabled since 2026-08-29 (Gmail read-state incident) |
| Job Tracker Dashboard Refresh | 2fd50c87b0d0 |
0 */2 * * * |
telegram:-1004321904721:1030 | — | dashboard_watchdog.py |
no-agent |
| Mnemosyne Auto-Consolidate | 0c771463c3cd |
every 30m |
local | — | consolidate_mnemosyne.sh |
no-agent |
| Mnemosyne DB Backup | e5ed215fc91c |
0 3 * * 0 |
telegram:1793951355 | — | mnemosyne-backup.sh |
no-agent |
| Mnemosyne Embedding Coverage Monitor | b63ed5817212 |
0 */6 * * * |
telegram:-1004321904721:102 | — | check_embedding_coverage.py |
Alerts if coverage < 95%. ⚠ Both 503-era slots failed (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own 10-17 06:01 run. ⚠ Both 2026-10-19 slots failed the same way (06:01:05 and 12:01:16), taking its streak to 2 and re-triggering the watchdog; its 18:01 slot succeeded, clearing the streak and auto-closing b63ed5_bbe44c89aac1 at 18:01:56. Live last_status: ok. With 0 */6 * * * this job is the fleet's best 503 sentinel — four probes a day |
| Mnemosyne Health Monitor | ea79a2107e36 |
0 */12 * * * |
telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then again at 2026-10-19 12:01 — incident ea79a2_747b27556fd0 was first opened 10-16 12:03, refreshed 10-19 12:02:55, and auto-closed 2026-10-20 00:04:24 when the 00:01 slot succeeded. Live last_status: ok, streak 0 |
| Mnemosyne Watchdog | 96eb95ce12f2 |
*/15 * * * * |
telegram:-1004321904721:102 | — | mnemosyne-watchdog.py |
no-agent, silent ≤10 conflicts; also runs hermes-runtime-deps-guard.sh every 15 min (added 2026-09-30). ⚠ Its reds are derivative — it double-counts other jobs' failures. During the 503 burst it exited 1 on 4 slots (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: 🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…. One sentinel watching another job, not four independent faults. The pattern repeated exactly in burst 2 — four more reds on 2026-10-19 (06:16, 06:31, 12:16, 12:31), all quoting the same coverage-monitor error, all clearing at 12:46:11 (96eb95_565ec1deb1c7, a 10-16 row refreshed through 10-19 12:31). 8 of the 26 failure rows across the two bursts were this one sentinel |
| Perth New Restaurant & Cafe Openings (Weekly) | c8d52904e1d6 |
0 18 * * 4 |
telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified delivered, status ok |
| Political News Digest (Mon/Thu) | 0e519874d280 |
0 8 * * 1,4 |
telegram:-1004321904721:346 | — | ⚠ Two node faults, no code fault: 2026-10-15 08:06 RuntimeError: Connection error. (0e5198_be2291809e56) and 2026-10-19 08:01 HTTP 503 … resource pressure (new incident 0e5198_ec2fac98a91d, alerted). Both are open; failure_streak: 2. Next slot Thu 2026-10-22 08:00 |
|
| Rhino AgentMail Inbox Watcher | f77b7e46bd09 |
every 5m |
origin | — | agentmail-rhino-watch.sh |
disabled since 2026-07-27 (legacy) |
| Tech & AI Newsletter Digest (Analytical) | 85edf6772bca |
0 7 * * * |
telegram:-1004321904721:346 | — | ⚠ 2026-10-19 07:01 slot failed on the omni-101 503 (new incident 85edf6_256c5e150bc4, alerted) — its first failure since its 10-15 connection-refused red cleared on 10-16. Live last_status: error, streak 1, next slot 2026-10-20 07:00 |
|
| Uber Eats Monthly Spend Report | 066bb5191280 |
0 2 1 * * |
telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (configured thread_id 34795 for telegram:1793951355 was not found) — the 2026-09 report generated fine, only the send failed. Bare telegram → home DM; retarget to a group topic |
| Weekly Review | 79f9bb41ecf5 |
0 18 * * 0 |
origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the cliprox/terra pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ Repinned 2026-10-05 23:39 → custom:omni-101 / free-stack and verified by a live manual run; the incident auto-resolved. Now last_status: ok, streak 0 |
| weekly-save-restricted-bot-restart | a688ed7b7ab0 |
0 3 * * 0 |
local | — | restart-save-restricted-bot.sh |
no-agent; script restored 2026-09-30 |
| Wiki Audit — Nightly | cbfe434a3b5e |
59 23 * * * |
local | task-approach | — | Reviews past 24h for wiki changes. ⚠ The 2026-10-16 23:59 slot failed on the omni-101 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident cbfe43_9b89947a046a is detected and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). Not a stale session lock — the execution row carries the explicit 503 error text |
| wiki-stocktake-maintenance | 1a8e90ea2271 |
0 3 * * * |
local | — | wiki-maintain.sh |
✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the retired qwen3.5:397b, HTTP 410 ×11). ✅ Verified by live run 2026-10-06 03:00 — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as 33f4e95. Live last_status: ok, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ The Quartz rebuild step is a no-op in practice — no quartz.config.ts exists and npx is unavailable, so the static site in public/ is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit 8c63ef2) — a second consecutive automatic success, verified. The script itself was never broken. ⚠ The 2026-10-17 03:00 slot failed on the same omni-101 503 — the first miss since the 2026-10-15 storm slot (which died with RuntimeError: Connection error.; the 10-16 slot succeeded). raw/stocktake/freshness-20261017.txt was not written and systems/skills-index.md stays stamped 2026-10-16 (the 10-16 slot succeeded, so freshness-20261016.txt exists); the 2026-10-18 03:00 slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident 1a8e90_3c9c1d573cde detected. Note the ledger tags this row failure_type: unknown even though the execution carries an explicit 503 message |
🦝 Hermes Config Backup to Gitea (Daily) ⚠ push has failed every night since 2026-08-08 — 75 local commits unpushed, job still reads ok → incidents/2026-08-08-gitea-config-backup-push-dead |
e27da6c34bbb |
0 11 * * * |
origin | — | hermes-config-backup.sh |
no-agent |
Removed Jobs
| Name | Removed | Reason |
|---|---|---|
| Workday start reminder | 2026-08-26 | No longer present in live cron list |
| CT 221 Miner Watchdog | 2026-08-26 | Job d5516d0a5147 removed; CT221 status unknown/inspected |
| DHL Tracking - Parcel 8052838382 | 2026-10-10 | 62619aa6bcb3 — parcel delivered, monitor redundant |
| Mosquito Tracker | 2026-10-16 (after the 15:00 run) | 1b31187397a5 — absent from jobs.json at the 2026-10-18 audit; last execution 2026-10-16 15:00 was completed / delivered |
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
Disabled Jobs (not removed)
| Name | ID | Disabled since | Reason |
|---|---|---|---|
| Important-mail monitor | d40085da631c |
2026-08-29 | Paused during Gmail read-state incident. See incidents/2026-08-29-gmail-read-state. |
| Rhino AgentMail Inbox Watcher | f77b7e46bd09 |
2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| AgentMail Rhino Inbox Poll | 0a7f933d06e2 |
2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
Known Issues
Resolved 2026-10-12
- Both
deepseek-v4-flashincidents closed by a clean scheduled run (fad5a86979d4, 03:00): the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidentsfad5a8_a13bbc837e1c(open since 2026-09-28, every fallback provider exhausted) andfad5a8_c80b59a907f0(since 2026-09-30,deepseek-v4-flashHTTP 400) both auto-closed at 03:05:39. Its stale pre-repinlast_status: errorcleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one. dca8482e4f76timeout incident closed (dca848_23fcd2b18aaa): the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm auto-closed at 07:33:41. Of the ten storm incidents, nine have now closed and one remains:1b23c0_c35703f637b8(1b23c042e17d, stillalerted, next slot Fri 2026-10-16 08:00). See incidents/2026-10-09-cron-provider-timeout-storm.- Net effect: open cron incidents 5 → 2. The two survivors are
0e753e_e7f8194a7d06(detectedsince 2026-10-01, untouched because its job does not run until 2026-11-01) and1b23c0_c35703f637b8(alerted, cleared on the next Friday slot).
Resolved 2026-10-07
- DHL Tracking
62619aa6bcb3— single Camofox timeout, self-recovered (2026-10-07 08:00 → 11:01): the 08:00 run died inensure_browser()withTimeoutError: timed outon a browser HTTP call (dhl_tracking_monitor.py:233), i.e. Camofox was momentarily unresponsive. The failure notice was sent at 08:00:58 and the incident (62619a_f485ccfd4a0d,failure_type: timeout) auto-resolved at 11:01:13 when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. This is the first detect → notify → auto-resolve incident cycle since the 2026-09-30 managed-runtime event — the incident machinery itself worked as designed. No action needed; a repeat would justify a Camofox health probe inside the script. - The tracked parcel arrived. The 14:00 run on 2026-10-07 reported waybill 8052838382
Delivered(Kensington → Perth). The monitor has therefore achieved its purpose and runs 5×/day for no remaining benefit — a candidate for removal, flagged rather than removed (the job is Anthony's to retire).
Resolved 2026-10-05
- Dead model pins across 7 jobs — repinned to routing aliases (resolved 2026-10-05 23:16 → 2026-10-06 00:05): Seven jobs were each pinned to a raw model name that had died upstream —
cliprox/terra(HTTP 401, connection auth expired, 3 jobs),ollama-cloud/qwen3.5:397b(HTTP 410, retired 2026-09-25),nvidia/deepseek-ai/deepseek-v4-flash(HTTP 400, dropped from the live catalog, 2 jobs) andcommand-code/gpt-5.6-terra(HTTP 403MODEL_NOT_IN_PLAN) — across 2026-10-01 → 2026-10-05. All seven were repinned to providercustom:omni-101using routing aliases —free-stack×6,smart-route×1 — deliberately, because the fault was raw-pin brittleness and re-pinning to another raw name would have re-created it.failure_streakis now 0 on all 30 jobs, down from 7 failing. Three were verified by live run:79f9bb41ecf5(23:39),06a0a1c8704f(23:45) anddca8482e4f76(23:56, delivered to topic 1030 — its first success since 2026-09-25). The remaining four are repinned but not yet re-run; the stalelast_status: errorfield clears on each one's next scheduled run. See incidents/2026-10-05-dead-model-pin-repin.- The
cliprox/terraincident is closed for cron purposes: no enabled job references that pin any more. The underlying proxy connection was never re-authenticated — the jobs were moved off it instead. See incidents/2026-10-01-cliproxy-connection-auth-expired. wiki-stocktake-maintenanceresumes its 03:00 schedule, sowiki-maintain.shno longer needs a daily manual run.
- The
- GitHub Trending Radar
c0a9fa314542— token-revocation claim refuted (2026-10-05): SUPERSEDED 2026-10-13. The Mon 09:00 run completed on 2026-10-05 and 2026-10-12, which was read as proof thegetMe→ 401 diagnosis was wrong. It was not: a cron run can be green while the script's own delivery hop is dead, and the 2026-10-13 livegetMeagainst the token in/home/hermes/workspace/github-trending-radar/config.jsonreturns 401. Corrected above and in Open items. Lesson: a completed run does not clear a credential fault unless the run exercised the same call that failed. - A stale failure notice, not a live outage, opened the 2026-10-05 review: the interactive session began because Anthony acted on a
85edf6772bcafailure notice sitting in topic346since the 2026-09-29 07:00 run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were alreadyresolvedon 2026-09-30 17:55:13. The notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. Checkexecutions.started_atand thecron_incidentsstate before acting on a notice.
Resolved 2026-09-30
- Telegram thread delivery failures — root cause found and fixed: jobs
fad5a86979d4,1b23c042e17d,c8d52904e1d6reportedconfigured thread_id 34795 for telegram:1793951355 was not found/Message thread not found. The target was wrong, not the topic: a baredeliver: "telegram"resolves to the home DM (1793951355), where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong).1b23c042e17dandc8d52904e1d6were retargeted totelegram:-1004321904721:102and verified end-to-end. See systems/messaging-integrations. - Managed-runtime dependency failure (2026-09-28 → 30): Jobs failed with
Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1). Real cause wasModuleNotFoundError: No module named 'dotenv'/'mnemosyne'in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed byhermes-runtime-deps-guard.shv2, run every 15 min by job96eb95ce12f2. All managed venvs verified. See incidents/2026-09-28-hermes-managed-runtime-deps. weekly-save-restricted-bot-restartmissing script (2026-08-30): Previously failed with "Script not found:restart-save-restricted-bot.sh". The script existed at/home/hermes/scripts/but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job nowok, streak 0.- Residual pre-fix dispatch errors cleared:
cbfe434a3b5ereturned tookon its next run as predicted. The1a8e90ea2271error of the same era also cleared — its current failure is a different, upstream problem (below). - Stale
delivery_failedflag on1b23c042e17dcleared (confirmed 2026-10-02): the 2026-09-30 retarget held — the job's next run delivered to topic102.
Open
-
GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13): live
getMeon the token in/home/hermes/workspace/github-trending-radar/config.jsonreturns HTTP 401, sodigest.py's Telegram send fails every Monday while the cron run itself stays green (last_status: ok). A completed run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with--dry-runand its own response was delivered. Fix: issue a fresh token from @BotFather, replace it inconfig.json, and re-rungetMeto confirm; do not re-read a green cron status as proof of delivery. This corrects the 2026-10-05 "claim refuted" entry, which was wrong. See systems/scheduled-tasks jobc0a9fa314542. -
Camofox memory pressure during long browser runs (NEW 2026-10-13): the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (
~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf:MemoryHigh=2G,MemoryMax=3G, swap max 1G). Live probe mid-run readMemory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G)with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the/tabsendpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See systems/browser-backend. -
Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07):
wiki-maintain.shdocuments a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: noquartz.config.tsis present andnpxis unavailable, sopublic/has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description. -
skills-indexunder-counts by one skill (NEW 2026-10-07): livefind ~/.hermes/skills -name SKILL.md -type freturns 347, but the generated page says 346 (70 categories). The generator de-dups on skill basename within a category, so the nested duplicateproductivity/obsidian-git-sync/obsidian-git-sync/SKILL.mdcollapses into its parent and is silently dropped. The other duplicated basename (hermes-api-vault, at the root and underdevops/) survives because the two land in different categories. Remove the nested duplicate directory, or key the generator on path rather than basename. -
A delivered notice is not a successful job:
cliprox/terrafailures recordeddelivery_outcome: deliveredbecause the failure notice is itself a deliverable. Read the executionstatuscolumn, not the delivery outcome, when judging whether a job produced its output. -
Stale provider key
custom:omni(1 enabled + 1 disabled): the provider was removed fromcustom_providers(live set on 2026-10-01:litellm,guacano,Omniroute:20128,command-code,omni-101). Re-verified 2026-10-06:e530c89f2be0(enabled) andd40085da631c(disabled) still name it; both runsmart-route, and the defaultcustomprovider resolves to the same base URL (http://100.78.63.20:20128), so no failure has been observed — but the references should be normalised tocustom:omni-101. The 2026-10-05 repin moved seven jobs to an explicitcustom:omni-101, leaving these two as the only remainingcustom:omnireferences. -
Bare
telegramdelivery targets (3 jobs) — demonstrably breaking sends:0e753e9720fd,fad5a86979d4,066bb5191280deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical:066bb5191280(02:11) and0e753e9720fd(09:01) both recordeddelivery_outcome: failedwithconfigured thread_id 34795 for telegram:1793951355 was not found. Work generated correctly; the delivery did not. Retarget all three totelegram:-1004321904721:<topic>. See systems/messaging-integrations. -
Two wiki trees, drifted in both directions (NEW 2026-10-02):
/home/hermes/workspace/agent-estate-wikiis the canonical source (Gitea remoteAnthony/agent-estate-wiki, Quartz build,wiki-maintain.shtarget)./home/hermes/wiki/agent-estate-wikiis the Obsidian/Ignis vault copy the iPad reads — pages nested underwiki/, and no git remote at all. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged backwards (systems/ignis.md,systems/nextcloud.md, threeincidents/pages exist only in the vault;raw/stocktake/*,agents/,incidents/2026-09-02-sftpgo-outage.mdonly in canonical) — a naive one-way sync deletes pages. See systems/hermes-agent. -
Work mailbox unreachable (non-cron; affects on-demand lookups): the
workOutlook account is configured with password auth while the tenant advertises onlyXOAuth2, sohimalaya -a workfails withlogin not supported. No cron job polls it. See incidents/2026-08-26-work-outlook-basic-auth-disabled. -
The
omni-101503 burst left no node-side evidence (NEW 2026-10-18): 9 model calls across 7 jobs died onRuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.between 2026-10-16 08:00 and 2026-10-17 03:00 — while the node answered/api/health→200 {"status":"ok"}and/v1/models→401before, during and after (re-probed 2026-10-18 00:33). Nothing on the Hermes side records the node's CPU/RAM/queue depth at failure time, so the "resource pressure" cause is taken on the router's word; if it recurs, capture host metrics during the burst, not afterwards. The incident ledger also tags seven of the nine rowsfailure_type: unknowndespite the explicit message (only the two script-exit watchdog incidents get a type), sofailure_typecannot be used to triage 503s — filter on the error text. Root cause unknown; not a repeat of the 10-14 TCP-refusal storm (which was reachability, this was serving). -
It recurred on 2026-10-19 with the same 9-call blast radius (NEW 2026-10-20): the identical
HTTP 503 … resource pressurestring took 9 model calls across 8 jobs again (06:01, then 07:01 → 12:01), and again the node answered/api/health→200 {"status":"ok"}and/v1/models→401before, during and after (re-probed 2026-10-20 00:03). Two bursts, four days apart, same node, same shape, no failover — and nothing was instrumented during either one. Action for the next occurrence: while jobs are still failing, run a real completion againstcustom:omni-101and capture the node's host CPU / RAM / queue depth — not afterwards. The ledger again tagged every new rowfailure_type: unknown, so find these by the error text, never byfailure_type. Also verified 2026-10-20: all eight affected jobs readprovider: custom:omni-101,model: smart-routeinjobs.json— one alias, one node, no diversity. → incidents/2026-10-16-omniroute-503-resource-pressure → incidents/2026-10-16-omniroute-503-resource-pressure
Historical
- Mnemosyne Watchdog incident (2026-08-19): Job
96eb95ce12f2cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See systems/mnemosyne. - Session storage failures (2026-08-10): Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See current-state.
- Model repinning (2026-08-08): 13 jobs repinned from dead/stale provider combos to
smart-routeorgpt-5.6-terra. - Wiki Audit model drift (2026-08-07): Global inference config change caused
RuntimeError: Skipped to prevent unintended spendon an unpinned job.
Related
- systems/mnemosyne
- runbooks/job-radar
- systems/hermes-agent
- systems/messaging-integrations
- incidents/2026-09-28-hermes-managed-runtime-deps
- incidents/2026-10-01-cliproxy-connection-auth-expired
- incidents/2026-08-26-work-outlook-basic-auth-disabled
Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09)
Fleet: 30 jobs · 27 enabled · 3 disabled · 10 carrying a non-clean last_status at
2026-10-10 00:22 (jobs.json.updated_at 2026-10-10T00:22:17+08:00). Executions by AWST day:
10-07 {'completed': 177, 'failed': 1} · 10-08 {'completed': 178, 'failed': 2} · 10-09 {'completed': 162, 'failed': 15, 'running': 1}.
The 2026-10-09 timeout storm. 16 failed executions across 11 distinct jobs, every one
RuntimeError: Request timed out. on the routing aliases behind custom:omni-101, 00:00 → 15:00 AWST;
every run from 18:00 onward completed. Full timeline, job list and incident table:
incidents/2026-10-09-cron-provider-timeout-storm.
- Four incidents auto-closed on the next clean run (
96eb9507:01,b63ed518:01,1b311822:05,ea79a22026-10-10 00:03); eight remain open —1a8e90_789224ad589c,cbfe43_fa81052baaf7,1a8e90_dc6b2d40ec44,dca848_23fcd2b18aaa,85edf6_18bedbeec5cd,1b23c0_c35703f637b8,08eb86_634f1a296f78,ecc1fc_58b6c38296f3. 1a8e90ea2271is the only job still atfailure_streak: 2(10-08 HTTP 403 + 10-09 timeout); its 03:00 run today decides whether the streak clears.96eb95ce12f2(watchdog) failed twice as a cascade — it exits 1 when it finds ERROR lines inagent.log, so it reported the storm rather than suffering it.
The 2026-10-08 window has no audit entry. The nightly audit's own 23:59 run on 2026-10-08 failed
(RuntimeError: Request timed out., incident cbfe43_fa81052baaf7, first seen 2026-10-09T00:32:09) and
committed nothing; the last committed audit covers the 2026-10-07 window (commit 659356a, 2026-10-08
00:12). The 2026-10-09 run covers both days.
Stocktake: two-day gap, artifacts recovered. The 2026-10-08 03:01 run left systems/skills-index.md
modified and raw/stocktake/freshness-20261008.txt untracked; both were stranded (the job's own commit
step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator
ran, so no freshness-20261009.txt exists and the skills index is still the 2026-10-08 output.
DHL monitor — retirement instructed, still enabled. Anthony replied "Delete this job" to the 20:26
DHL delivery notice on 2026-10-09; the session replied that 62619aa6bcb3 was "stopped and deleted".
Live jobs.json at 2026-10-10T00:22 shows 62619aa6bcb3 still enabled — paused_at: null,
last_status: ok, schedule 0 8,11,14,17,20 * * * — so it runs again at 08:00. The parcel
(8052838382) was reported Delivered on 2026-10-07, making the monitor redundant. This audit did not
mutate cron state; retiring the job is open item #1 below.
Memory recall relevance. "Are you up to date" returned a stale Uber Eats episode on 2026-10-09; capture and vector health were verified good (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is not in live config. See incidents/2026-10-09-memory-recall-relevance.
Open items (2026-10-09)
RetireDONE 2026-10-10 — verified retired by ID lookup; fleet dropped to 29 jobs (26 enabled). See log.md and the 2026-10-10 audit block below.62619aa6bcb3(DHL Tracking — Parcel8052838382) — instructed 2026-10-09 21:24, not executed.- Decide and persist the recall-ranking change in
memory.mnemosyne; re-run the trigger question as a regression test. Close the eight open cron incidents once their next clean runs land; watchDONE — all eight cleared: five on 2026-10-10 and1a8e90ea2271(streak 2).dca848_23fcd2b18aaaon 2026-10-12 07:33:41. (A different pair,1b23c0_c35703f637b8+0e753e_e7f8194a7d06, remains open for unrelated reasons — see### Openabove.)- Stale delivery targets —
0e753e9720fd,fad5a86979d4(last_status: error) and066bb5191280(delivery_failed) unchanged since the 2026-10-07 audit. - Quartz site rebuild is skipped by
wiki-maintain.shstep 3 (noquartz.config.ts, nonpx) — carried from 2026-10-07, still open. - Skills index off by one (347
SKILL.mdon disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.
Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST)
Fleet: 29 jobs - 26 enabled - 3 disabled (jobs.json.updated_at 2026-10-11T00:19:44+08:00).
The count fell from 30 to 29 because 62619aa6bcb3 (DHL Tracking - Parcel 8052838382) is gone:
open item #1 of the 2026-10-09 audit is CLOSED (below). Disabled: d40085da631c (Important-mail
monitor), f77b7e46bd09 (Rhino AgentMail Inbox Watcher), 0a7f933d06e2 (AgentMail Rhino Inbox Poll).
Executions by AWST day (executions.db): 10-07 177 completed / 1 failed - 10-08 178 / 2 -
10-09 163 / 15 - 10-10 174 completed / 0 failed (+1 left running). The 2026-10-10 fleet was
wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm.
Non-clean last_status: 5 jobs, down from 10.
| Job | ID | last_status |
streak |
|---|---|---|---|
| Facebook Birthday Monthly Scrape | 0e753e9720fd |
error |
0 |
| Facebook Birthday Weekly Registry Sync | fad5a86979d4 |
error |
0 |
| Daily Marketing Job Radar — Discovery (stage 1) | dca8482e4f76 |
error |
1 |
| Enhanced Weekend Adventure Planner | 1b23c042e17d |
error |
1 |
| Uber Eats Monthly Spend Report | 066bb5191280 |
delivery_failed |
0 |
The three streak 0 rows are stale last_status fields, not live failure streaks. 1a8e90ea2271
(stocktake) is back to last_status: ok.
Incidents: 7 closed during 2026-10-10, 5 open. The 10-09 storm unwound as each job next ran clean
(ea79a2 00:03, cbfe43 00:27, 1a8e90ea2271 x2 03:06, 85edf6 07:04, 08eb86 08:32, ecc1fc
12:05). Still open: 1b23c042e17d + dca8482e4f76 (timeout, alerted, both first seen
2026-10-09) and 0e753e9720fd + fad5a86979d4 (agent; open since 2026-10-01 / 2026-09-30).
Open item #1 CLOSED - the DHL monitor is retired. Anthony instructed removal again on the evening
of 2026-10-10 and the job is absent from live jobs.json, verified 2026-10-11 00:19 by direct ID
lookup (no job id contains 62619; no job name matches dhl/parcel). This is the third "deleted"
claim for this job - the 2026-10-07 and 2026-10-09 reports were both false because the check was
the absence of job output rather than a positive lookup. Removals must be verified by ID lookup.
Still open: the Quartz rebuild step in wiki-maintain.sh (no quartz.config.ts) - skills-index
off-by-one (347 SKILL.md vs 346 reported) - the 5 non-clean last_status rows above - Mattermost
platform unreachable (systems/messaging-integrations) - the gateway runtime env is outside the
15-minute dependency guard (incidents/2026-09-28-hermes-managed-runtime-deps).
Re-verified 2026-10-12 (audit window 2026-10-11 00:00 → 2026-10-12 00:2x AWST; live probes 00:0x-00:2x)
Fleet: 29 jobs — 26 enabled / 3 disabled (jobs.json updated_at 2026-10-12T00:10:42+08:00). Disabled
set unchanged: d40085da631c (Important-mail monitor), f77b7e46bd09 (Rhino AgentMail Inbox Watcher),
0a7f933d06e2 (AgentMail Rhino Inbox Poll — legacy).
Executions by AWST day (executions.db): 10-07 177 completed / 1 failed — 10-08 178 / 2 —
10-09 163 / 15 — 10-10 175 completed / 0 failed (finalised; the 10-11 audit saw 174 + 1 still
running) — 10-11 171 completed / 0 failed + 1 running (this audit). Two consecutive
zero-failure days after the 2026-10-09 provider-timeout storm. Heaviest jobs on 10-11: runtime-deps guard
96eb95ce12f2 96 runs (15-min cadence), 0c771463c3cd 46, 2fd50c87b0d0 12, coverage monitor
b63ed5817212 3, health monitor ea79a2107e36 2, stocktake 1a8e90ea2271 1.
Non-clean last_status: 5 jobs — unchanged from 2026-10-10. 0e753e9720fd (error, streak 0),
fad5a86979d4 (error, streak 0), dca8482e4f76 (error, streak 1), 1b23c042e17d (error, streak 1),
066bb5191280 (delivery_failed, streak 0). The two streak-1 jobs are the Mon-Fri and Friday-only schedulers:
their last_error is RuntimeError: Request timed out. from Friday 2026-10-09 (the storm) and neither has
had a scheduled slot since — dca8482e4f76 next runs 07:00 Mon, 1b23c042e17d Fri 08:00, so both should
clear themselves. Everything else reads last_status: ok (this build reports ok, not success).
Incidents: 0 opened / 0 closed in the window — 5 unresolved, unchanged. alerted 3:
fad5a8_c80b59a907f0 (weekly registry sync, scope agent, first seen 2026-09-30, deepseek-v4-flash HTTP 400),
dca848_23fcd2b18aaa + 1b23c0_c35703f637b8 (timeout, both first seen 2026-10-09 — the last two left from
the storm). detected 2: fad5a8_a13bbc837e1c (2026-09-28, every provider in the fallback chain exhausted),
0e753e_e7f8194a7d06 (2026-10-01, same deepseek-v4-flash 400). Table totals: alerted 3 / detected 2 /
closed 8 / resolved 63. Last closures remain the seven on 2026-10-10.
Sessions in the window: 18 — 17 cron + 1 interactive (talaria-ios, 2026-10-11 19:13 → 19:51,
41 messages, "best VNC clients" research). The 03:00 stocktake 1a8e90ea2271 ran clean on schedule
(completed 03:00:40 → 03:11:29) and committed d981e30.
False alarm — the 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION was a timezone mislabel, not a
capture stall. The run compared a naive-UTC working_memory.created_at (2026-10-10 13:24:53) against AWST
now and reported "10.7 h of no captures"; the true gap was 2.7 h (the row landed 21:24:53 AWST, inside the
session that produced it). No pipeline fault. The same job's 12:05 and 2026-10-12 00:02 runs returned
[SILENT] (green). → systems/mnemosyne
Still open (unchanged): Quartz rebuild step inert (wiki-maintain.sh step 3 — no quartz.config.ts);
skills-index off-by-one (347 SKILL.md on disk vs 346 reported); the 5 non-clean last_status rows;
Mattermost platform unreachable (systems/messaging-integrations); gateway runtime env outside the
15-minute dependency guard (incidents/2026-09-28-hermes-managed-runtime-deps).
Re-verified 2026-10-13 (audit window 2026-10-12 00:00 → 2026-10-12 23:59 AWST; live probes 2026-10-13 00:05–00:45)
Headline: a clean run hid a dead credential. The cron table was green — 174 completed / 0 failed —
and two of the three non-clean last_status rows cleared overnight, yet one job that reported success on
paper had its delivery hop dead the whole time. Detail below.
Cron tally: 2026-10-11 172 completed / 0 failed (the one row still running at the last audit
finished clean); 2026-10-12 174 completed / 0 failed / 1 running — and the running row is this audit
itself (cbfe434a3b5e, started 23:59:41). Third consecutive clean day. The execution-log retention
floor has moved up to 2026-10-07 08:15, so 10-07 rows are now pruned (their earlier tally is no longer
reproducible from this DB).
Incidents: open 5 → 2. Three closed on 2026-10-12 — fad5a8_a13bbc837e1c and fad5a8_c80b59a907f0 at
03:05:39 (both deepseek-v4-flash, closed by the first clean scheduled run of fad5a86979d4);
dca848_23fcd2b18aaa at 07:33:41 (closed by dca8482e4f76's own successful 07:00 run). Table totals
now alerted 1 / detected 1 / closed 8 / resolved 66 (76 rows). Survivors: 0e753e_e7f8194a7d06
(detected since 2026-10-01 — cannot clear until its job next runs on 2026-11-01) and
1b23c0_c35703f637b8 (alerted since 2026-10-09, next slot Fri 2026-10-16 08:00).
Job-by-job (verified against the executions log, delivery_outcome included):
fad5a86979d4— Facebook Birthday Weekly Registry Sync 03:00:45 → 03:05:47, completed, delivered. Output: Birthday Dashboard, 448 unique birthdays, 1 contact match. This is the repin's live proof.dca8482e4f76— Daily Marketing Job Radar 07:00:47 → 07:33:50, completed, delivered to topic 1030 (32-minute run; the 2026-10-09 timeout incident closed 9 seconds before it finished). Run was messy — see the Camofox finding — but it produced and delivered.c0a9fa314542— GitHub Trending Radar 09:00:41 → 09:08:33, completed,delivered— and yet its own send failed (finding 1). Acompletedrow plusdelivery_outcome: deliveredis not proof the job's payload went out; here the delivered artifact was the agent's recovered digest, not the script's.1b23c042e17d— Enhanced Weekend Adventure Planner 2026-10-09 08:00:38 → 08:17:46, failed (RuntimeError: Request timed out., notice delivered). No run since; next slot Fri 2026-10-16 08:00.0e753e9720fd— Facebook Birthday Monthly Scrape: no runs at all inside the retained window (monthly, next 2026-11-01), so itserror/streak-0 row is still untested rather than fixed.
Sessions in the window: 18, all cron, no interactive session — a quiet day, in contrast to the
previous window's talaria-ios research session. Busiest by message count: dca8482e4f76 (140),
1a8e90ea2271 stocktake (97), Wiki Audit (28), c0a9fa314542 (25). The 03:00 stocktake ran clean and
pushed 0eee5f1 to origin/master: skills-index regenerated (346 skills / 70 categories, 347 files on
disk — the known off-by-one), freshness audit 54 stale / 32 fresh / 7 no-date of 93 pages, Quartz step
skipped on the documented path.
Mnemosyne (12:02 health run, green): working 760/786 embedded = 96.7%, episodic 1050/1056 =
99.4% — both above the 95% floor, so no reindex needed. Orphan embeddings 0. FTS5 aligned exactly
(fts_working 786/786, fts_episodes 1056/1056, fts_facts 876/876). 0 unconsolidated working rows;
1 unresolved conflict (threshold 10); last consolidation 0.3 h earlier. 2252 memories total (786 working /
1056 episodic / 1 triple), 395 consolidated facts. Disk 15 GB free (81% used). No exceptions → nothing to
act on, and nothing to change on systems/mnemosyne beyond the coverage figures.
Finding 1 — GitHub Trending Radar's Telegram token is revoked (live-verified). getMe against the
telegram.bot_token in /home/hermes/workspace/github-trending-radar/config.json returns HTTP 401
Unauthorized. The token is a plausible length (46 chars), so it is revoked/invalid rather than malformed,
and there is no valid replacement anywhere on the box. The fetch path is healthy — GitHub trending
pulled 100 repos and wrote 26 cache files in 24 h — so only digest.py's send() fails. Consequence: the
Monday digest reached the DM only because the agent noticed the failure and re-ran with --dry-run and
its own response was delivered. The 2026-10-05 "token-revocation claim refuted" entry was wrong and has
been corrected: two green runs were read as proof the token worked, when a green cron row only means the
agent finished. Fix: new token from @BotFather → config.json → re-run getMe. Credential present, value
not recorded here.
Finding 2 — Camofox hit its memory ceiling during the Job Radar run. Mid-run probe of
camofox-browser.service: Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M) with 383 tasks and the service up 9 days. Swap fully consumed, zero reclaimable headroom →
/tabs and page evals hung. The agent restarted the service (back to ~2.1 GB used of 14 GB on CT460) and
completed the extraction one tab at a time with immediate close. Same containment drop-in as documented
(resource-containment.conf); the limits held, the workload is what needs bounding. → systems/browser-backend
Nothing added/removed: no new system, no retired job, so index.md is unchanged this cycle. No page
archived: the DHL monitor stays where the 2026-10-10 audit put it (redundant, in Removed Jobs).
Still open after this cycle: GitHub Trending Radar token (new 2026-10-13); Camofox containment vs.
multi-tab workloads (new 2026-10-13); Quartz rebuild step inert; skills-index off-by-one; 3 non-clean
last_status rows (0e753e9720fd, 1b23c042e17d, 066bb5191280); Mattermost platform unreachable;
gateway runtime env outside the 15-minute dependency guard.
Re-verified 2026-10-14
Window: 2026-10-13 00:00 → 23:59 AWST; live probes 2026-10-14 00:00–00:30 AWST. Fleet 29 jobs (26 enabled / 3 disabled) — unchanged this cycle. Open incidents 2 → 3.
Cron tally — the clean streak broke. 2026-10-13 = 167 completed / 1 failed (plus this audit's own row
in flight at probe time, which finalises the day at 168). Tail: 10-10 175 / 0, 10-11 172 / 0, 10-12 175 / 0
→ the first failed cron run since 2026-10-09 (a day that recorded 15).
Finding 1 — wiki-stocktake-maintenance (1a8e90ea2271) died on a dead opencode credential.
03:00:43 → 03:01:57, RuntimeError: HTTP 404: No active credentials for provider: opencode.
Pinned custom:omni-101:free-stack. New incident 1a8e90_d11698bd7275 (state: detected,
failure_type: agent, alerted_at: NULL — failure streak 1, so under the 3-strike alert rule).
This is the job's third failure in six days, all three on the opencode path: 10-08
HTTP 403: OpenCode's free tier can only be used from within OpenCode (1a8e90_789224ad589c), 10-09 timeout
(1a8e90_dc6b2d40ec44), 10-13 404 no credentials (1a8e90_d11698bd7275). The first two resolved on
2026-10-10; this one clears only when the 2026-10-14 03:00 run succeeds — i.e. each incident clears only on
the next successful run of its own job. → incidents/2026-10-05-dead-model-pin-repin
Finding 2 — the failure stranded its own generated artifacts; this audit committed them.
The script's step 1 (skills-index regeneration) and step 2 (freshness report) both completed and wrote to disk,
then the agent/LLM turn that follows died → nothing committed, nothing pushed. Left in the working tree:
systems/skills-index.md (modified — date-only change: updated/verified_on 10-12 → 10-13; 346 skills /
70 categories unchanged) and raw/stocktake/freshness-20261013.txt (untracked — 48 stale / 38 fresh /
7 no-date of 93 pages, down from 54 stale / 32 fresh on 10-12). Both are in this audit's commit.
Structural point: the agent row is what commits, so a perfectly healthy script run still publishes nothing
when the LLM turn dies. There is no green-row alternative — the row went red.
Retention floor moved. executions.db now holds 1,001 rows (1,000 terminal + 1 in flight);
MAX_TERMINAL_EXECUTIONS = 1000 in cron/executions.py. Oldest retained terminal row: 2026-10-08 07:11.
Days above the floor are complete; older ones undercount — 10-08 now reads 126 completed / 1 failed where its
finalised tally was higher. Run-level evidence older than the floor is pruned, not recoverable.
Delivery. 10-13: 159 suppressed / 9 delivered / 1 no-record. Delivered = Daily Marketing Job Radar
(dca8482e4f76), Mosquito Tracker (1b31187397a5 ×3), Tech & AI Newsletter Digest (85edf6772bca),
Daily Birthday Checker (a432898b55a4), Gmail Action Digest (08eb86f02d37), FreshRSS Midday News Briefing
(ecc1fc13655a), Evening Stand-Down Briefing (e530c89f2be0). No unexpected delivery; the two dead-thread
delivery faults (0e753e9720fd, 066bb5191280, both Telegram thread_id 34795) are unchanged since 10-01.
Mnemosyne. All six monitor runs returned the silent token — Health Monitor 00:02 and 12:02, Embedding
Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, i.e. both coverages at/above the 95 % floor and
no reindex. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic
1050/1056 = 99.4 %). Wrinkle: the 12:02 health run emitted the token markdown-bolded rather than bare;
the delivery layer still suppressed it (delivery_outcome: suppressed), so the guard held — but the matcher is
tolerant of wrapping rather than literal. → systems/mnemosyne
Non-clean last_status: 3 → 4. Cleared: fad5a86979d4 (success 10-12 03:05), dca8482e4f76 (success
10-13 07:02). Added: 1a8e90ea2271. Remaining: 0e753e9720fd (10-01), 1b23c042e17d (10-09),
066bb5191280 (10-01), 1a8e90ea2271 (10-13).
Sessions. 15 in the window, all cron, zero interactive — third consecutive window with no interactive
session.
Pages touched: systems/scheduled-tasks.md (this block, frontmatter, non-clean list, retention note),
systems/skills-index.md (committed stocktake artifact), current-state.md, index.md, log.md.
Nothing added/removed: no new system, no retired job, no page archived → index.md structure unchanged.
Still open after this cycle: wiki-stocktake-maintenance opencode credential (new 2026-10-13 — watch the
10-14 03:00 run: streak 3 triggers an alert, and the durable fix is to repin the job off free-stack-opencode
or repair OmniRoute's opencode provider credentials); GitHub Trending Radar token (2026-10-13, next run
Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert;
skills-index off-by-one; 4 non-clean last_status rows (above); Mattermost platform unreachable;
gateway runtime env outside the 15-minute dependency guard.
Re-verified 2026-10-15 / 2026-10-16 — the omni-101 connection-refused storm
Window: 2026-10-14 00:00 → 2026-10-16 00:00 AWST (two days — the 2026-10-14 23:59 audit died inside the
same outage, so the 10-14 window was never covered). Live probes 2026-10-16 00:0x–00:2x.
Fleet 29 jobs (26 enabled / 3 disabled) — unchanged. Open incidents 3 → 12 (9 alerted / 3 detected).
The tally broke hard. 2026-10-14 = 160 completed / 12 failed; 2026-10-15 = 151 completed /
24 failed / 1 running. Against a four-day clean run (10-10 175/0, 10-11 172/0, 10-12 175/0,
10-13 167/1). 36 failure rows across 13 jobs — but only 24 are genuine model-call failures on
12 jobs; the other 12 rows are the Mnemosyne Watchdog reddening on a log sentinel (symptom, not
fault). Day-level, 10-15 is the worst cron day since 2026-10-09 (15).
Root cause. httpcore.ConnectError: [Errno 111] Connection refused against
http://100.78.63.20:20128 — the omniroute-1 node, provider custom:omni-101
(config.yaml: custom provider name: omni-101). Surfaced as openai.APIConnectionError: Connection error.
→ RuntimeError: Connection error. 328 Errno 111 lines in the live errors.log, 504 across rotations.
Window 10-14 08:30:27 → 10-15 18:07, recovered by 10-15 22:01. → incidents/2026-10-14-omniroute-node-connection-refused-storm
Structural finding — no model-call failover anywhere in the fleet. Verified from jobs.json: all 18
enabled agent-mode jobs are pinned to custom:omni-101 (smart-route ×10, free-stack ×6,
deepseek-paid-smart ×2). The only other model-capable cron pin, custom:omni (CT600, omniroute:20128),
sits on e530c89f2be0 — which is no_agent: true, a script-only job, so its provider field is
vestigial. Completing runs inside the window by provider: 210 no_agent / no provider + 1 no_agent on
custom:omni — that is the entire green of the fleet, and zero model calls completed in 34 hours.
⚠ A first pass at this window read Evening Stand-Down's clean 18:45 runs as proof CT600 stayed up. It is
not: Stand-Down never calls a model. Never read a no_agent job's green row as provider-health evidence.
Worst-affected jobs: b63ed5817212 Mnemosyne Embedding Coverage ×6 (every slot for 30 h),
1b31187397a5 Mosquito Tracker ×4, ea79a2107e36 Mnemosyne Health Monitor ×3, 08eb86f02d37 Gmail Action
Digest ×2, ecc1fc13655a FreshRSS Midday ×2, then one apiece for 85edf6772bca, dca8482e4f76,
0e519874d280, 06a0a1c8704f, c8d52904e1d6, 1a8e90ea2271, cbfe434a3b5e. Full table in the incident
page. User-visible cost: Digests that never went out — Tech & AI, Daily Marketing Job Radar, Political
News, AI Newsletter Curation for Patrick, Gmail Action Digest ×2, FreshRSS Midday ×2, and the weekly Perth
New Restaurant (missed outright; next chance 10-22). The missed audit also meant the 10-14 window went
uncovered.
Recovery verified by schedule, not by probe alone. 1b31187397a5 completed 10-15 22:01:56 (streak
4 → 0); b63ed5817212 10-16 00:02:29; 96eb95ce12f2 10-16 00:01:36; ea79a2107e36 10-16
00:09:48. Live 10-16 00:1x: GET /v1/models on 100.78.63.20:20128 → HTTP 401 in 10 ms (up, auth
required) and the same on omniroute:20128 / 100.88.81.19.
Non-clean last_status: 3 → 12. Cleared during the window: b63ed5817212, 1b31187397a5,
ea79a2107e36 (all on successful post-storm runs). Added: the nine rows listed under Count above.
Unchanged long-standing: 1b23c042e17d (10-09 error), 0e753e9720fd (10-01 error), 066bb5191280
(10-01 delivery_failed, dead Telegram thread_id 34795).
Mnemosyne. Both monitors sat on the failing node, so no coverage figures were refreshed in the window — last explicit numbers remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %); the 10-16 00:09 Health Monitor run is the first clean read since 10-12. → systems/mnemosyne
Stocktake gap (self-healing). 1a8e90ea2271 failed 10-15 03:00 → raw/stocktake/freshness-20261015.txt
was never written and systems/skills-index.md still carries the 10-14 stamp. Its next slot is 10-16 03:00,
so the gap closes on its own — not regenerated by hand here, to avoid racing the scheduled run.
Sessions. 36 sessions in the two-day window, all cron, zero interactive — fourth consecutive
window with no interactive session.
Pages touched: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (new),
systems/omniroute.md, systems/mnemosyne.md, systems/scheduled-tasks.md (this block, frontmatter,
Count), current-state.md, index.md, log.md.
Nothing removed or archived: no retired job, no obsolete system → index.md structure gains one page.
Still open after this cycle: the no-failover topology itself (all 18 model-calling jobs on one
OmniRoute node — the durable fix is a second provider pin or a router-side failover, and it needs Anthony's
call); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab
workloads; Quartz rebuild step inert; skills-index off-by-one; 12 non-clean last_status rows;
Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.
Re-verified 2026-10-24 (audit window 2026-10-23 00:00 → 2026-10-24 00:3x AWST; live probes 00:1x–00:3x)
Count. As of 2026-10-24 00:2x AWST (jobs.json.updated_at 2026-10-24T00:17:36+08:00): 28 jobs —
25 enabled / 3 disabled, unchanged for a sixth consecutive audit. Window 2026-10-23: 169 completed /
0 failed (+ this audit running), the third zero-failure day in four; 2026-10-22 settled at 173 / 0.
The 2026-10-09 timeout storm closed. 1b23c042e17d (Enhanced Weekend Adventure Planner, weekly Fri 08:00)
ran clean on its 10-23 slot, closing the storm's last two rows (1b23c0_c35703f637b8,
1b23c0_103cbe04a79b) at 08:04. The storm now holds zero unclosed rows and the job reads ok
again → incidents/2026-10-09-cron-provider-timeout-storm.
Incident ledger: unclosed 4 → 2 (no rows opened, two closed). Remaining: c0a9fa_764558b5ae83
(alerted since 10-19, next slot Mon 10-26 09:00 — the GitHub Trending Radar revoked token) and
0e753e_e7f8194a7d06 (detected since 10-01, next slot 11-01 — Facebook Birthday monthly scrape).
Non-clean last_status: 5 → 4 — c0a9fa314542 (error, streak 1, Mon 10-26), 0e753e9720fd
(error, 0, 11-01), fad5a86979d4 (delivery_failed, Mon 10-26), 066bb5191280 (delivery_failed, 11-01).
No new failure signature appeared in the window.
Pins unchanged — 11 custom:omni-101 / smart-route, 8 unpinned (global default), 6
custom:omni-101 / free-stack, 2 custom:omni / smart-route, 1 custom:omni-101 /
deepseek-paid-smart (this audit).
Sessions: 14 in the window — 13 cron + 1 desktop (2026-10-23 18:26:12, 13 messages, the Conduit
phone-call session that exposed the memory-timeout fault below).
New fault found and fixed in-window (not a cron job): the Conduit voice plugin gave the Mnemosyne
provider start a hard-coded 10 s budget, so three calls (10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16)
were abandoned by the memory layer and ran without recall. Patched the same evening to a 60 s default
(CONDUIT_MEMORY_TIMEOUT_S override) plus a TokenError guard that degrades instead of aborting; dashboard
restarted 18:54:43 and no abort line exists after it. The patch is uncommitted in the plugin's
working tree → incidents/2026-10-22-conduit-memory-start-timeout, systems/conduit.
Stocktake: the 10-22 sixfold overrun did not repeat — the 10-23 03:07 slot ran and committed
(250c735: freshness 2 stale / 94 fresh, skills-index 306).
Pages touched: systems/conduit.md (new), incidents/2026-10-22-conduit-memory-start-timeout.md (new),
incidents/2026-10-09-cron-provider-timeout-storm.md (status → resolved), systems/mnemosyne.md,
current-state.md, index.md, log.md, plus frontmatter added to two legacy incident pages (see log).
Nothing removed or archived: no job retired, no system obsolete in the window.
Still open: the single-node no-failover topology behind every model-calling job; the GitHub Trending Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 stopped; the Conduit patch being uncommitted; the 561.7 MB Conduit health store with no retention.
Re-verified 2026-10-25 (audit window 2026-10-24 00:00 → 2026-10-25 00:5x AWST; live probes 00:1x–00:5x)
A third consecutive clean day — 167 / 0 on 2026-10-24, with 2026-10-23 settling at 170 / 0. Fleet
unchanged at 28 jobs (25 enabled / 3 disabled), 4 non-clean last_status, 2 unclosed incident
rows, pins unchanged. Nothing opened and nothing closed in the incident ledger in-window. 14
sessions: 11 cron + 3 desktop.
The window's fault is off-cron. On 2026-10-24 12:53–12:56 Anthony asked three times to be phoned to
test Conduit calls (Call me now to test calls. → Yes. You do → CAL ME); every reply asserted the
agent has no calling capability, with zero tool calls in the session — no tool_search, no
skill_view, no conduit_call_user. The tool exists, is registered for every session, is reachable
through lazy discovery, and the plugin ships the calling-the-user skill; his call settings
(enabled / when_asked / decides / alerts) were permissive and the device was paired. The first
reply came from kilo-auto/free, but the mid-session switch to smart-route repeated it — capability
asserted instead of discovered, not a broken tool →
incidents/2026-10-24-conduit-call-request-refused.
Conduit re-verified. Plugin 0.69.0 / HEAD f000bf0 (22 commits on from the 0.64.0 /
eaa14be recorded on 10-24) while the gateway process still dates from 2026-10-23 18:54:43, so the
upstream commits are not live and the memory-timeout patch is uncommitted and now five releases
behind HEAD. Device phone-status contact last 10-23 19:10:30; health ingestion ran to
10-24 09:59:59; store 569.9 MB / 1,520,229 events / 3,028 buckets / 124 windows — the 10-24
page's 308 buckets / 62 windows was a narrower grouping, not a collapse. ⚠ Evidence
correction: conduit-calls.json → history is empty, and the 10-22 incident's “two calls closed
cleanly 00:03:23 / 00:06:57” were the two Mnemosyne cron turn-ends of that night (ends records
any turn-end that fires a watch, MAX_RECENT_ENDS = 64) — so the memory fix still has no positive
call evidence → incidents/2026-10-22-conduit-memory-start-timeout, systems/conduit.
Method note: daily tallies are bucketed with substr(started_at,1,10) — date(started_at)
normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day (see the Count section).
Pages touched: incidents/2026-10-24-conduit-call-request-refused.md (new), systems/conduit.md,
incidents/2026-10-22-conduit-memory-start-timeout.md (correction), systems/scheduled-tasks.md,
current-state.md, index.md, log.md. Nothing removed or archived: no job retired, no system
obsolete, no decision reversed in the window.
Still open: the single-node no-failover topology behind every model-calling job; the GitHub Trending Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 stopped; the Conduit patch uncommitted and five releases behind HEAD; the 569.9 MB Conduit health store with no retention; and a call that has never been placed since the memory fix.