30 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Scheduled Tasks | system | active | 2026-07-22 | 2026-10-11 | 2026-10-11 | 2026-10-25 | high |
|
|
Scheduled Tasks
Count
- Total jobs: 30 — 27 enabled, 3 disabled (verified from live
~/.hermes/cron/jobs.json, 2026-10-06) - Failing (last run failed outright): 0 — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider
custom:omni-101using routing aliases, and everyfailure_streakinjobs.jsonnow reads 0. Four of the seven jobs had still carried a pre-repinlast_status: error(1b23c042e17d,fad5a86979d4,0e753e9720fd,1a8e90ea2271) — a stale field, not a live failure; it clears on each job's next scheduled run.1a8e90ea2271cleared on 2026-10-06 03:00, running on schedule for the first time since 2026-09-25; the remaining three (1b23c042e17d,fad5a86979d4,0e753e9720fd) are due Fri 08:00 / Mon 03:00 / 1st 09:00. See incidents/2026-10-05-dead-model-pin-repin.- Historical (2026-10-01 → 2026-10-05), now all remediated:
dca8482e4f76403MODEL_NOT_IN_PLAN×7;1a8e90ea2271HTTP 410 ×11 (retiredqwen3.5:397b);fad5a86979d4HTTP 400 ×3;0e753e9720fdHTTP 400 ×1;06a0a1c8704f,1b23c042e17dand79f9bb41ecf5HTTP 401 (thecliprox/terraconnection auth expiry, settled 3/3 on 2026-10-04).
- Historical (2026-10-01 → 2026-10-05), now all remediated:
- Failing (delivery only): 1 —
066bb5191280: the report itself generated; the send failed against the DM thread (baretelegramtarget, 2026-10-01). Unchanged this window. - Repin applied 2026-10-05:
free-stack×6 (06a0a1c8704f,1b23c042e17d,79f9bb41ecf5,1a8e90ea2271,fad5a86979d4,0e753e9720fd) andsmart-route×1 (dca8482e4f76), all oncustom:omni-101. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it.free-stackis in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs. - Verified live, not assumed:
79f9bb41ecf5succeeded on a manual run 2026-10-05 23:39 and06a0a1c8704fat 23:45 (both incidents auto-resolved);dca8482e4f76completed 23:56 and delivered totelegram:-1004321904721:1030(incident resolved 2026-10-06 00:05:07); and1a8e90ea2271completed automatically on 2026-10-06 03:00. Four of the seven are therefore verified by live run. The remaining three (1b23c042e17d,fad5a86979d4,0e753e9720fd) are repinned but not yet re-run — they are due on their normal schedules.1a8e90ea2271has since run twice more on schedule (2026-10-06 03:00 →33f4e95; 2026-10-07 03:00 →8c63ef2), so the stocktake repin is verified across consecutive automatic runs. - Streak counting note: the
N failures in a rowfigure inhermes cron listcomes from thefailure_streakcounter injobs.json.executions.dbretains only rows from 2026-09-30 onward, so DB-based recounts undercount older streaks. Live counters as of 2026-10-06: all 0. - Re-verified 2026-10-06 AWST: job count unchanged at 30 (27 enabled / 3 disabled).
066bb5191280and0e753e9720fdstill carry the bare-telegramtarget warning (configured thread_id 34795 for telegram:1793951355 was not found). - Re-verified 2026-10-07 AWST: job count unchanged at 30 (27 enabled / 3 disabled); 0 failing — all 30
failure_streakcounters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176completed, 0failed, 1running(the audit itself); 14 sessions, allcron— no interactive (non-cron) sessions. The stocktake job1a8e90ea2271ran on schedule at 03:00 and regenerated systems/skills-index (344 → 346 skills) plusraw/stocktake/freshness-20261006.txt, committing to Gitea as33f4e95— the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repinlast_status: error(1b23c042e17d,fad5a86979d4,0e753e9720fd). - A stale notice, not a live outage, drove this review. The interactive session on 2026-10-05 23:16 began because Anthony replied to a
85edf6772bcafailure notice that had been sitting in topic346since the 2026-09-29 07:00 run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were alreadyresolvedon 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Checkexecutions.started_at(and thecron_incidentsstate) before acting on a notice. - Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00): job count unchanged at 30 (27 enabled / 3 disabled); 0 failing — every
failure_streakreads 0. Window tally (executions.db): 176completed, 1failed, 1running(the audit itself). The single failure is62619aa6bcb3(DHL Tracking, 08:00) — a Camofox HTTP read timeout (ensure_browser()→TimeoutError: timed out), notified 08:00:58 and auto-resolved 11:01:13 when the next 3-hourly run succeeded (incident62619a_f485ccfd4a0d). 13 sessions started in the window, allsource = cron— no interactive sessions; 12 deliveries, 0 delivery failures. The same three stale pre-repinlast_status: errorjobs remain (1b23c042e17d,fad5a86979d4,0e753e9720fd); the stocktake job1a8e90ea2271ran on schedule at 03:00 for the second consecutive day. - Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, 0 failing (2026-10-06) → 30, 0 failing (2026-10-07) → 30, 0 failing (2026-10-08)
Job Table
| Name | ID | Schedule | Delivery | Skills | Script | Notes |
|---|---|---|---|---|---|---|
| AgentMail Rhino Inbox Poll | 0a7f933d06e2 |
* * * * * |
origin | terminal | agentmail-poll.py |
disabled since 2026-07-27 (legacy) |
| AI Newsletter Curation for Patrick (Weekly) | 06a0a1c8704f |
0 10 * * 4 |
telegram:-1004321904721:346 | — | — | ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was custom:Omniroute:20128 / cliprox/terra, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live last_status: ok, streak 0 |
| Daily Birthday Checker | a432898b55a4 |
30 7 * * * |
telegram:1793951355 | — | birthday_check.py |
|
| Daily Marketing Job Radar | dca8482e4f76 |
0 7 * * 1-5 |
telegram:-1004321904721:1030 | job-search-automation | — | ✅ Repinned 2026-10-05 → custom:omni-101 / smart-route (was command-code / gpt-5.6-terra, 403 MODEL_NOT_IN_PLAN ×7). A manual run completed at 23:56 and delivered to topic 1030 — first successful run since 2026-09-25. Live last_status: ok, streak 0 |
| DHL Tracking - Parcel 8052838382 | 62619aa6bcb3 |
0 8,11,14,17,20 * * * |
telegram:1793951355 | — | dhl_tracking_monitor.py |
no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (ensure_browser() → TimeoutError: timed out) — incident 62619a_f485ccfd4a0d, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 Delivered — the parcel has arrived, so the monitor is now redundant |
| Enhanced Weekend Adventure Planner | 1b23c042e17d |
0 8 * * 5 |
telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix confirmed working (delivered 2026-10-02, flag cleared). ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the failed cliprox/terra pin, HTTP 401 ×1). Not yet re-run — carries a stale last_status: error until Fri 08:00; streak 0 |
| Evening Stand-Down Briefing | e530c89f2be0 |
45 18 * * 1-5 |
telegram:-1004321904721 | — | evening_standdown.py |
no-agent |
| Facebook Birthday Monthly Scrape | 0e753e9720fd |
0 9 1 * * |
telegram | — | — | ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the catalog-missing nvidia/deepseek-ai/deepseek-v4-flash pin, HTTP 400 ×1). Streak 0; stale last_status: error until the 1st. ⚠ Bare telegram still targets the home DM — retarget to a group topic |
| Facebook Birthday Weekly Registry Sync | fad5a86979d4 |
0 3 * * 1 |
telegram | — | birthday_check.py |
✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the catalog-missing nvidia/deepseek-ai/deepseek-v4-flash pin, HTTP 400 ×3). Streak 0; stale last_status: error until Mon 03:00. ⚠ Bare telegram target should move to a group topic |
| FreshRSS Midday News Briefing | ecc1fc13655a |
0 12 * * * |
telegram:-1004321904721:346 | — | freshrss_intelligence_v2.py |
|
| GitHub Trending Radar — Weekly Roundup | c0a9fa314542 |
0 9 * * 1 |
telegram:1793951355 | — | — | ✅ Token-revocation claim refuted. The scheduled run on 2026-10-05 09:00 completed cleanly — a second consecutive success after the off-schedule 2026-09-30 19:22 run — so the getMe → 401 diagnosis no longer applies and no token replacement is needed. Still delivers to the bare home DM, which works |
| Gmail Action Digest | 08eb86f02d37 |
30 8 * * * |
telegram:-1004321904721:notifications | — | gmail_action_digest_v3.py |
|
| Important-mail monitor | d40085da631c |
every 30m |
telegram:1793951355 | — | _cron_silent_monitor.py |
disabled since 2026-08-29 (Gmail read-state incident) |
| Job Tracker Dashboard Refresh | 2fd50c87b0d0 |
0 */2 * * * |
telegram:-1004321904721:1030 | — | dashboard_watchdog.py |
no-agent |
| Mnemosyne Auto-Consolidate | 0c771463c3cd |
every 30m |
local | — | consolidate_mnemosyne.sh |
no-agent |
| Mnemosyne DB Backup | e5ed215fc91c |
0 3 * * 0 |
telegram:1793951355 | — | mnemosyne-backup.sh |
no-agent |
| Mnemosyne Embedding Coverage Monitor | b63ed5817212 |
0 */6 * * * |
telegram:-1004321904721:102 | — | check_embedding_coverage.py |
Alerts if coverage < 95% |
| Mnemosyne Health Monitor | ea79a2107e36 |
0 */12 * * * |
telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
| Mnemosyne Watchdog | 96eb95ce12f2 |
*/15 * * * * |
telegram:-1004321904721:102 | — | mnemosyne-watchdog.py |
no-agent, silent ≤10 conflicts; also runs hermes-runtime-deps-guard.sh every 15 min (added 2026-09-30) |
| Mosquito Tracker | 1b31187397a5 |
0 7,15,22 * * * |
telegram:-1004321904721:273 | — | mosquito_tracker.py |
Excludes 23:00–07:00 |
| Perth New Restaurant & Cafe Openings (Weekly) | c8d52904e1d6 |
0 18 * * 4 |
telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified delivered, status ok |
| Political News Digest (Mon/Thu) | 0e519874d280 |
0 8 * * 1,4 |
telegram:-1004321904721:346 | — | — | |
| Rhino AgentMail Inbox Watcher | f77b7e46bd09 |
every 5m |
origin | — | agentmail-rhino-watch.sh |
disabled since 2026-07-27 (legacy) |
| Tech & AI Newsletter Digest (Analytical) | 85edf6772bca |
0 7 * * * |
telegram:-1004321904721:346 | — | — | |
| Uber Eats Monthly Spend Report | 066bb5191280 |
0 2 1 * * |
telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (configured thread_id 34795 for telegram:1793951355 was not found) — the 2026-09 report generated fine, only the send failed. Bare telegram → home DM; retarget to a group topic |
| Weekly Review | 79f9bb41ecf5 |
0 18 * * 0 |
origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the cliprox/terra pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ Repinned 2026-10-05 23:39 → custom:omni-101 / free-stack and verified by a live manual run; the incident auto-resolved. Now last_status: ok, streak 0 |
| weekly-save-restricted-bot-restart | a688ed7b7ab0 |
0 3 * * 0 |
local | — | restart-save-restricted-bot.sh |
no-agent; script restored 2026-09-30 |
| Wiki Audit — Nightly | cbfe434a3b5e |
59 23 * * * |
local | task-approach | — | Reviews past 24h for wiki changes |
| wiki-stocktake-maintenance | 1a8e90ea2271 |
0 3 * * * |
local | — | wiki-maintain.sh |
✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the retired qwen3.5:397b, HTTP 410 ×11). ✅ Verified by live run 2026-10-06 03:00 — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as 33f4e95. Live last_status: ok, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ The Quartz rebuild step is a no-op in practice — no quartz.config.ts exists and npx is unavailable, so the static site in public/ is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit 8c63ef2) — a second consecutive automatic success, verified. The script itself was never broken |
| 🦝 Hermes Config Backup to Gitea (Daily) | e27da6c34bbb |
0 11 * * * |
origin | — | hermes-config-backup.sh |
no-agent |
Removed Jobs
| Name | Removed | Reason |
|---|---|---|
| Workday start reminder | 2026-08-26 | No longer present in live cron list |
| CT 221 Miner Watchdog | 2026-08-26 | Job d5516d0a5147 removed; CT221 status unknown/inspected |
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
Disabled Jobs (not removed)
| Name | ID | Disabled since | Reason |
|---|---|---|---|
| Important-mail monitor | d40085da631c |
2026-08-29 | Paused during Gmail read-state incident. See incidents/2026-08-29-gmail-read-state. |
| Rhino AgentMail Inbox Watcher | f77b7e46bd09 |
2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| AgentMail Rhino Inbox Poll | 0a7f933d06e2 |
2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
Known Issues
Resolved 2026-10-07
- DHL Tracking
62619aa6bcb3— single Camofox timeout, self-recovered (2026-10-07 08:00 → 11:01): the 08:00 run died inensure_browser()withTimeoutError: timed outon a browser HTTP call (dhl_tracking_monitor.py:233), i.e. Camofox was momentarily unresponsive. The failure notice was sent at 08:00:58 and the incident (62619a_f485ccfd4a0d,failure_type: timeout) auto-resolved at 11:01:13 when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. This is the first detect → notify → auto-resolve incident cycle since the 2026-09-30 managed-runtime event — the incident machinery itself worked as designed. No action needed; a repeat would justify a Camofox health probe inside the script. - The tracked parcel arrived. The 14:00 run on 2026-10-07 reported waybill 8052838382
Delivered(Kensington → Perth). The monitor has therefore achieved its purpose and runs 5×/day for no remaining benefit — a candidate for removal, flagged rather than removed (the job is Anthony's to retire).
Resolved 2026-10-05
- Dead model pins across 7 jobs — repinned to routing aliases (resolved 2026-10-05 23:16 → 2026-10-06 00:05): Seven jobs were each pinned to a raw model name that had died upstream —
cliprox/terra(HTTP 401, connection auth expired, 3 jobs),ollama-cloud/qwen3.5:397b(HTTP 410, retired 2026-09-25),nvidia/deepseek-ai/deepseek-v4-flash(HTTP 400, dropped from the live catalog, 2 jobs) andcommand-code/gpt-5.6-terra(HTTP 403MODEL_NOT_IN_PLAN) — across 2026-10-01 → 2026-10-05. All seven were repinned to providercustom:omni-101using routing aliases —free-stack×6,smart-route×1 — deliberately, because the fault was raw-pin brittleness and re-pinning to another raw name would have re-created it.failure_streakis now 0 on all 30 jobs, down from 7 failing. Three were verified by live run:79f9bb41ecf5(23:39),06a0a1c8704f(23:45) anddca8482e4f76(23:56, delivered to topic 1030 — its first success since 2026-09-25). The remaining four are repinned but not yet re-run; the stalelast_status: errorfield clears on each one's next scheduled run. See incidents/2026-10-05-dead-model-pin-repin.- The
cliprox/terraincident is closed for cron purposes: no enabled job references that pin any more. The underlying proxy connection was never re-authenticated — the jobs were moved off it instead. See incidents/2026-10-01-cliproxy-connection-auth-expired. wiki-stocktake-maintenanceresumes its 03:00 schedule, sowiki-maintain.shno longer needs a daily manual run.
- The
- GitHub Trending Radar
c0a9fa314542— token-revocation claim refuted (2026-10-05): the scheduled Mon 09:00 run completed cleanly — its second consecutive success after the off-schedule 2026-09-30 19:22 run. ThegetMe→ 401 revoked-token diagnosis no longer applies and no token replacement is needed. It still delivers to the bare home DM, which works. - A stale failure notice, not a live outage, opened the 2026-10-05 review: the interactive session began because Anthony acted on a
85edf6772bcafailure notice sitting in topic346since the 2026-09-29 07:00 run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were alreadyresolvedon 2026-09-30 17:55:13. The notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. Checkexecutions.started_atand thecron_incidentsstate before acting on a notice.
Resolved 2026-09-30
- Telegram thread delivery failures — root cause found and fixed: jobs
fad5a86979d4,1b23c042e17d,c8d52904e1d6reportedconfigured thread_id 34795 for telegram:1793951355 was not found/Message thread not found. The target was wrong, not the topic: a baredeliver: "telegram"resolves to the home DM (1793951355), where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong).1b23c042e17dandc8d52904e1d6were retargeted totelegram:-1004321904721:102and verified end-to-end. See systems/messaging-integrations. - Managed-runtime dependency failure (2026-09-28 → 30): Jobs failed with
Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1). Real cause wasModuleNotFoundError: No module named 'dotenv'/'mnemosyne'in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed byhermes-runtime-deps-guard.shv2, run every 15 min by job96eb95ce12f2. All managed venvs verified. See incidents/2026-09-28-hermes-managed-runtime-deps. weekly-save-restricted-bot-restartmissing script (2026-08-30): Previously failed with "Script not found:restart-save-restricted-bot.sh". The script existed at/home/hermes/scripts/but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job nowok, streak 0.- Residual pre-fix dispatch errors cleared:
cbfe434a3b5ereturned tookon its next run as predicted. The1a8e90ea2271error of the same era also cleared — its current failure is a different, upstream problem (below). - Stale
delivery_failedflag on1b23c042e17dcleared (confirmed 2026-10-02): the 2026-09-30 retarget held — the job's next run delivered to topic102.
Open
- Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07):
wiki-maintain.shdocuments a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: noquartz.config.tsis present andnpxis unavailable, sopublic/has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description. skills-indexunder-counts by one skill (NEW 2026-10-07): livefind ~/.hermes/skills -name SKILL.md -type freturns 347, but the generated page says 346 (70 categories). The generator de-dups on skill basename within a category, so the nested duplicateproductivity/obsidian-git-sync/obsidian-git-sync/SKILL.mdcollapses into its parent and is silently dropped. The other duplicated basename (hermes-api-vault, at the root and underdevops/) survives because the two land in different categories. Remove the nested duplicate directory, or key the generator on path rather than basename.- A delivered notice is not a successful job:
cliprox/terrafailures recordeddelivery_outcome: deliveredbecause the failure notice is itself a deliverable. Read the executionstatuscolumn, not the delivery outcome, when judging whether a job produced its output. - Stale provider key
custom:omni(1 enabled + 1 disabled): the provider was removed fromcustom_providers(live set on 2026-10-01:litellm,guacano,Omniroute:20128,command-code,omni-101). Re-verified 2026-10-06:e530c89f2be0(enabled) andd40085da631c(disabled) still name it; both runsmart-route, and the defaultcustomprovider resolves to the same base URL (http://100.78.63.20:20128), so no failure has been observed — but the references should be normalised tocustom:omni-101. The 2026-10-05 repin moved seven jobs to an explicitcustom:omni-101, leaving these two as the only remainingcustom:omnireferences. - Bare
telegramdelivery targets (3 jobs) — demonstrably breaking sends:0e753e9720fd,fad5a86979d4,066bb5191280deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical:066bb5191280(02:11) and0e753e9720fd(09:01) both recordeddelivery_outcome: failedwithconfigured thread_id 34795 for telegram:1793951355 was not found. Work generated correctly; the delivery did not. Retarget all three totelegram:-1004321904721:<topic>. See systems/messaging-integrations. - Two wiki trees, drifted in both directions (NEW 2026-10-02):
/home/hermes/workspace/agent-estate-wikiis the canonical source (Gitea remoteAnthony/agent-estate-wiki, Quartz build,wiki-maintain.shtarget)./home/hermes/wiki/agent-estate-wikiis the Obsidian/Ignis vault copy the iPad reads — pages nested underwiki/, and no git remote at all. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged backwards (systems/ignis.md,systems/nextcloud.md, threeincidents/pages exist only in the vault;raw/stocktake/*,agents/,incidents/2026-09-02-sftpgo-outage.mdonly in canonical) — a naive one-way sync deletes pages. See systems/hermes-agent. - Work mailbox unreachable (non-cron; affects on-demand lookups): the
workOutlook account is configured with password auth while the tenant advertises onlyXOAuth2, sohimalaya -a workfails withlogin not supported. No cron job polls it. See incidents/2026-08-26-work-outlook-basic-auth-disabled.
Historical
- Mnemosyne Watchdog incident (2026-08-19): Job
96eb95ce12f2cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See systems/mnemosyne. - Session storage failures (2026-08-10): Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See current-state.
- Model repinning (2026-08-08): 13 jobs repinned from dead/stale provider combos to
smart-routeorgpt-5.6-terra. - Wiki Audit model drift (2026-08-07): Global inference config change caused
RuntimeError: Skipped to prevent unintended spendon an unpinned job.
Related
- systems/mnemosyne
- runbooks/job-radar
- systems/hermes-agent
- systems/messaging-integrations
- incidents/2026-09-28-hermes-managed-runtime-deps
- incidents/2026-10-01-cliproxy-connection-auth-expired
- incidents/2026-08-26-work-outlook-basic-auth-disabled
Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09)
Fleet: 30 jobs · 27 enabled · 3 disabled · 10 carrying a non-clean last_status at
2026-10-10 00:22 (jobs.json.updated_at 2026-10-10T00:22:17+08:00). Executions by AWST day:
10-07 {'completed': 177, 'failed': 1} · 10-08 {'completed': 178, 'failed': 2} · 10-09 {'completed': 162, 'failed': 15, 'running': 1}.
The 2026-10-09 timeout storm. 16 failed executions across 11 distinct jobs, every one
RuntimeError: Request timed out. on the routing aliases behind custom:omni-101, 00:00 → 15:00 AWST;
every run from 18:00 onward completed. Full timeline, job list and incident table:
incidents/2026-10-09-cron-provider-timeout-storm.
- Four incidents auto-closed on the next clean run (
96eb9507:01,b63ed518:01,1b311822:05,ea79a22026-10-10 00:03); eight remain open —1a8e90_789224ad589c,cbfe43_fa81052baaf7,1a8e90_dc6b2d40ec44,dca848_23fcd2b18aaa,85edf6_18bedbeec5cd,1b23c0_c35703f637b8,08eb86_634f1a296f78,ecc1fc_58b6c38296f3. 1a8e90ea2271is the only job still atfailure_streak: 2(10-08 HTTP 403 + 10-09 timeout); its 03:00 run today decides whether the streak clears.96eb95ce12f2(watchdog) failed twice as a cascade — it exits 1 when it finds ERROR lines inagent.log, so it reported the storm rather than suffering it.
The 2026-10-08 window has no audit entry. The nightly audit's own 23:59 run on 2026-10-08 failed
(RuntimeError: Request timed out., incident cbfe43_fa81052baaf7, first seen 2026-10-09T00:32:09) and
committed nothing; the last committed audit covers the 2026-10-07 window (commit 659356a, 2026-10-08
00:12). The 2026-10-09 run covers both days.
Stocktake: two-day gap, artifacts recovered. The 2026-10-08 03:01 run left systems/skills-index.md
modified and raw/stocktake/freshness-20261008.txt untracked; both were stranded (the job's own commit
step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator
ran, so no freshness-20261009.txt exists and the skills index is still the 2026-10-08 output.
DHL monitor — retirement instructed, still enabled. Anthony replied "Delete this job" to the 20:26
DHL delivery notice on 2026-10-09; the session replied that 62619aa6bcb3 was "stopped and deleted".
Live jobs.json at 2026-10-10T00:22 shows 62619aa6bcb3 still enabled — paused_at: null,
last_status: ok, schedule 0 8,11,14,17,20 * * * — so it runs again at 08:00. The parcel
(8052838382) was reported Delivered on 2026-10-07, making the monitor redundant. This audit did not
mutate cron state; retiring the job is open item #1 below.
Memory recall relevance. "Are you up to date" returned a stale Uber Eats episode on 2026-10-09; capture and vector health were verified good (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is not in live config. See incidents/2026-10-09-memory-recall-relevance.
Open items (2026-10-09)
- Retire
62619aa6bcb3(DHL Tracking — Parcel8052838382) — instructed 2026-10-09 21:24, not executed. - Decide and persist the recall-ranking change in
memory.mnemosyne; re-run the trigger question as a regression test. - Close the eight open cron incidents once their next clean runs land; watch
1a8e90ea2271(streak 2). - Stale delivery targets —
0e753e9720fd,fad5a86979d4(last_status: error) and066bb5191280(delivery_failed) unchanged since the 2026-10-07 audit. - Quartz site rebuild is skipped by
wiki-maintain.shstep 3 (noquartz.config.ts, nonpx) — carried from 2026-10-07, still open. - Skills index off by one (347
SKILL.mdon disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.
Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST)
Fleet: 29 jobs - 26 enabled - 3 disabled (jobs.json.updated_at 2026-10-11T00:19:44+08:00).
The count fell from 30 to 29 because 62619aa6bcb3 (DHL Tracking - Parcel 8052838382) is gone:
open item #1 of the 2026-10-09 audit is CLOSED (below). Disabled: d40085da631c (Important-mail
monitor), f77b7e46bd09 (Rhino AgentMail Inbox Watcher), 0a7f933d06e2 (AgentMail Rhino Inbox Poll).
Executions by AWST day (executions.db): 10-07 177 completed / 1 failed - 10-08 178 / 2 -
10-09 163 / 15 - 10-10 174 completed / 0 failed (+1 left running). The 2026-10-10 fleet was
wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm.
Non-clean last_status: 5 jobs, down from 10.
| Job | ID | last_status |
streak |
|---|---|---|---|
| Facebook Birthday Monthly Scrape | 0e753e9720fd |
error |
0 |
| Facebook Birthday Weekly Registry Sync | fad5a86979d4 |
error |
0 |
| Daily Marketing Job Radar | dca8482e4f76 |
error |
1 |
| Enhanced Weekend Adventure Planner | 1b23c042e17d |
error |
1 |
| Uber Eats Monthly Spend Report | 066bb5191280 |
delivery_failed |
0 |
The three streak 0 rows are stale last_status fields, not live failure streaks. 1a8e90ea2271
(stocktake) is back to last_status: ok.
Incidents: 7 closed during 2026-10-10, 5 open. The 10-09 storm unwound as each job next ran clean
(ea79a2 00:03, cbfe43 00:27, 1a8e90ea2271 x2 03:06, 85edf6 07:04, 08eb86 08:32, ecc1fc
12:05). Still open: 1b23c042e17d + dca8482e4f76 (timeout, alerted, both first seen
2026-10-09) and 0e753e9720fd + fad5a86979d4 (agent; open since 2026-10-01 / 2026-09-30).
Open item #1 CLOSED - the DHL monitor is retired. Anthony instructed removal again on the evening
of 2026-10-10 and the job is absent from live jobs.json, verified 2026-10-11 00:19 by direct ID
lookup (no job id contains 62619; no job name matches dhl/parcel). This is the third "deleted"
claim for this job - the 2026-10-07 and 2026-10-09 reports were both false because the check was
the absence of job output rather than a positive lookup. Removals must be verified by ID lookup.
Still open: the Quartz rebuild step in wiki-maintain.sh (no quartz.config.ts) - skills-index
off-by-one (347 SKILL.md vs 346 reported) - the 5 non-clean last_status rows above - Mattermost
platform unreachable (systems/messaging-integrations) - the gateway runtime env is outside the
15-minute dependency guard (incidents/2026-09-28-hermes-managed-runtime-deps).