Files
agent-estate-wiki/systems/scheduled-tasks.md
T

28 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Scheduled Tasks system active 2026-07-22 2026-10-09 2026-10-09 2026-10-23 high
cron
scheduled
automation
live-cron-list
live-jobs-json
cron-executions-db

Scheduled Tasks

Count

  • Total jobs: 30 — 27 enabled, 3 disabled (verified from live ~/.hermes/cron/jobs.json, 2026-10-06)
  • Failing (last run failed outright): 0 — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider custom:omni-101 using routing aliases, and every failure_streak in jobs.json now reads 0. Four of the seven jobs had still carried a pre-repin last_status: error (1b23c042e17d, fad5a86979d4, 0e753e9720fd, 1a8e90ea2271) — a stale field, not a live failure; it clears on each job's next scheduled run. 1a8e90ea2271 cleared on 2026-10-06 03:00, running on schedule for the first time since 2026-09-25; the remaining three (1b23c042e17d, fad5a86979d4, 0e753e9720fd) are due Fri 08:00 / Mon 03:00 / 1st 09:00. See incidents/2026-10-05-dead-model-pin-repin.
    • Historical (2026-10-01 → 2026-10-05), now all remediated: dca8482e4f76 403 MODEL_NOT_IN_PLAN ×7; 1a8e90ea2271 HTTP 410 ×11 (retired qwen3.5:397b); fad5a86979d4 HTTP 400 ×3; 0e753e9720fd HTTP 400 ×1; 06a0a1c8704f, 1b23c042e17d and 79f9bb41ecf5 HTTP 401 (the cliprox/terra connection auth expiry, settled 3/3 on 2026-10-04).
  • Failing (delivery only): 1 — 066bb5191280: the report itself generated; the send failed against the DM thread (bare telegram target, 2026-10-01). Unchanged this window.
  • Repin applied 2026-10-05: free-stack ×6 (06a0a1c8704f, 1b23c042e17d, 79f9bb41ecf5, 1a8e90ea2271, fad5a86979d4, 0e753e9720fd) and smart-route ×1 (dca8482e4f76), all on custom:omni-101. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. free-stack is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs.
  • Verified live, not assumed: 79f9bb41ecf5 succeeded on a manual run 2026-10-05 23:39 and 06a0a1c8704f at 23:45 (both incidents auto-resolved); dca8482e4f76 completed 23:56 and delivered to telegram:-1004321904721:1030 (incident resolved 2026-10-06 00:05:07); and 1a8e90ea2271 completed automatically on 2026-10-06 03:00. Four of the seven are therefore verified by live run. The remaining three (1b23c042e17d, fad5a86979d4, 0e753e9720fd) are repinned but not yet re-run — they are due on their normal schedules. 1a8e90ea2271 has since run twice more on schedule (2026-10-06 03:00 → 33f4e95; 2026-10-07 03:00 → 8c63ef2), so the stocktake repin is verified across consecutive automatic runs.
  • Streak counting note: the N failures in a row figure in hermes cron list comes from the failure_streak counter in jobs.json. executions.db retains only rows from 2026-09-30 onward, so DB-based recounts undercount older streaks. Live counters as of 2026-10-06: all 0.
  • Re-verified 2026-10-06 AWST: job count unchanged at 30 (27 enabled / 3 disabled). 066bb5191280 and 0e753e9720fd still carry the bare-telegram target warning (configured thread_id 34795 for telegram:1793951355 was not found).
  • Re-verified 2026-10-07 AWST: job count unchanged at 30 (27 enabled / 3 disabled); 0 failing — all 30 failure_streak counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 completed, 0 failed, 1 running (the audit itself); 14 sessions, all cron — no interactive (non-cron) sessions. The stocktake job 1a8e90ea2271 ran on schedule at 03:00 and regenerated systems/skills-index (344 → 346 skills) plus raw/stocktake/freshness-20261006.txt, committing to Gitea as 33f4e95 — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin last_status: error (1b23c042e17d, fad5a86979d4, 0e753e9720fd).
  • A stale notice, not a live outage, drove this review. The interactive session on 2026-10-05 23:16 began because Anthony replied to a 85edf6772bca failure notice that had been sitting in topic 346 since the 2026-09-29 07:00 run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already resolved on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check executions.started_at (and the cron_incidents state) before acting on a notice.
  • Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00): job count unchanged at 30 (27 enabled / 3 disabled); 0 failing — every failure_streak reads 0. Window tally (executions.db): 176 completed, 1 failed, 1 running (the audit itself). The single failure is 62619aa6bcb3 (DHL Tracking, 08:00) — a Camofox HTTP read timeout (ensure_browser() → TimeoutError: timed out), notified 08:00:58 and auto-resolved 11:01:13 when the next 3-hourly run succeeded (incident 62619a_f485ccfd4a0d). 13 sessions started in the window, all source = cron — no interactive sessions; 12 deliveries, 0 delivery failures. The same three stale pre-repin last_status: error jobs remain (1b23c042e17d, fad5a86979d4, 0e753e9720fd); the stocktake job 1a8e90ea2271 ran on schedule at 03:00 for the second consecutive day.
  • Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, 0 failing (2026-10-06) → 30, 0 failing (2026-10-07) → 30, 0 failing (2026-10-08)

Job Table

Name ID Schedule Delivery Skills Script Notes
AgentMail Rhino Inbox Poll 0a7f933d06e2 * * * * * origin terminal agentmail-poll.py disabled since 2026-07-27 (legacy)
AI Newsletter Curation for Patrick (Weekly) 06a0a1c8704f 0 10 * * 4 telegram:-1004321904721:346 — — ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was custom:Omniroute:20128 / cliprox/terra, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live last_status: ok, streak 0
Daily Birthday Checker a432898b55a4 30 7 * * * telegram:1793951355 — birthday_check.py
Daily Marketing Job Radar dca8482e4f76 0 7 * * 1-5 telegram:-1004321904721:1030 job-search-automation — ✅ Repinned 2026-10-05 → custom:omni-101 / smart-route (was command-code / gpt-5.6-terra, 403 MODEL_NOT_IN_PLAN ×7). A manual run completed at 23:56 and delivered to topic 1030 — first successful run since 2026-09-25. Live last_status: ok, streak 0
DHL Tracking - Parcel 8052838382 62619aa6bcb3 0 8,11,14,17,20 * * * telegram:1793951355 — dhl_tracking_monitor.py no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (ensure_browser() → TimeoutError: timed out) — incident 62619a_f485ccfd4a0d, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 Delivered — the parcel has arrived, so the monitor is now redundant
Enhanced Weekend Adventure Planner 1b23c042e17d 0 8 * * 5 telegram:-1004321904721:102 — — 2026-09-30 delivery-target fix confirmed working (delivered 2026-10-02, flag cleared). ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the failed cliprox/terra pin, HTTP 401 ×1). Not yet re-run — carries a stale last_status: error until Fri 08:00; streak 0
Evening Stand-Down Briefing e530c89f2be0 45 18 * * 1-5 telegram:-1004321904721 — evening_standdown.py no-agent
Facebook Birthday Monthly Scrape 0e753e9720fd 0 9 1 * * telegram — — ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the catalog-missing nvidia/deepseek-ai/deepseek-v4-flash pin, HTTP 400 ×1). Streak 0; stale last_status: error until the 1st. ⚠ Bare telegram still targets the home DM — retarget to a group topic
Facebook Birthday Weekly Registry Sync fad5a86979d4 0 3 * * 1 telegram — birthday_check.py ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the catalog-missing nvidia/deepseek-ai/deepseek-v4-flash pin, HTTP 400 ×3). Streak 0; stale last_status: error until Mon 03:00. ⚠ Bare telegram target should move to a group topic
FreshRSS Midday News Briefing ecc1fc13655a 0 12 * * * telegram:-1004321904721:346 — freshrss_intelligence_v2.py
GitHub Trending Radar — Weekly Roundup c0a9fa314542 0 9 * * 1 telegram:1793951355 — — ✅ Token-revocation claim refuted. The scheduled run on 2026-10-05 09:00 completed cleanly — a second consecutive success after the off-schedule 2026-09-30 19:22 run — so the getMe → 401 diagnosis no longer applies and no token replacement is needed. Still delivers to the bare home DM, which works
Gmail Action Digest 08eb86f02d37 30 8 * * * telegram:-1004321904721:notifications — gmail_action_digest_v3.py
Important-mail monitor d40085da631c every 30m telegram:1793951355 — _cron_silent_monitor.py disabled since 2026-08-29 (Gmail read-state incident)
Job Tracker Dashboard Refresh 2fd50c87b0d0 0 */2 * * * telegram:-1004321904721:1030 — dashboard_watchdog.py no-agent
Mnemosyne Auto-Consolidate 0c771463c3cd every 30m local — consolidate_mnemosyne.sh no-agent
Mnemosyne DB Backup e5ed215fc91c 0 3 * * 0 telegram:1793951355 — mnemosyne-backup.sh no-agent
Mnemosyne Embedding Coverage Monitor b63ed5817212 0 */6 * * * telegram:-1004321904721:102 — check_embedding_coverage.py Alerts if coverage < 95%
Mnemosyne Health Monitor ea79a2107e36 0 */12 * * * telegram:-1004321904721:102 mnemosyne-operations — Reports only on exception
Mnemosyne Watchdog 96eb95ce12f2 */15 * * * * telegram:-1004321904721:102 — mnemosyne-watchdog.py no-agent, silent ≤10 conflicts; also runs hermes-runtime-deps-guard.sh every 15 min (added 2026-09-30)
Mosquito Tracker 1b31187397a5 0 7,15,22 * * * telegram:-1004321904721:273 — mosquito_tracker.py Excludes 23:00–07:00
Perth New Restaurant & Cafe Openings (Weekly) c8d52904e1d6 0 18 * * 4 telegram:-1004321904721:102 — — delivery target corrected 2026-09-30; verified delivered, status ok
Political News Digest (Mon/Thu) 0e519874d280 0 8 * * 1,4 telegram:-1004321904721:346 — —
Rhino AgentMail Inbox Watcher f77b7e46bd09 every 5m origin — agentmail-rhino-watch.sh disabled since 2026-07-27 (legacy)
Tech & AI Newsletter Digest (Analytical) 85edf6772bca 0 7 * * * telegram:-1004321904721:346 — —
Uber Eats Monthly Spend Report 066bb5191280 0 2 1 * * telegram camofox-browser-automation, browser-backend-verification — ⚠ delivery failed 2026-10-01 (configured thread_id 34795 for telegram:1793951355 was not found) — the 2026-09 report generated fine, only the send failed. Bare telegram → home DM; retarget to a group topic
Weekly Review 79f9bb41ecf5 0 18 * * 0 origin weekly-review — ⚠ Failed 2026-10-04 18:00 on the cliprox/terra pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ Repinned 2026-10-05 23:39 → custom:omni-101 / free-stack and verified by a live manual run; the incident auto-resolved. Now last_status: ok, streak 0
weekly-save-restricted-bot-restart a688ed7b7ab0 0 3 * * 0 local — restart-save-restricted-bot.sh no-agent; script restored 2026-09-30
Wiki Audit — Nightly cbfe434a3b5e 59 23 * * * local task-approach — Reviews past 24h for wiki changes
wiki-stocktake-maintenance 1a8e90ea2271 0 3 * * * local — wiki-maintain.sh ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the retired qwen3.5:397b, HTTP 410 ×11). ✅ Verified by live run 2026-10-06 03:00 — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as 33f4e95. Live last_status: ok, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ The Quartz rebuild step is a no-op in practice — no quartz.config.ts exists and npx is unavailable, so the static site in public/ is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit 8c63ef2) — a second consecutive automatic success, verified. The script itself was never broken
🦝 Hermes Config Backup to Gitea (Daily) e27da6c34bbb 0 11 * * * origin — hermes-config-backup.sh no-agent

Removed Jobs

Name Removed Reason
Workday start reminder 2026-08-26 No longer present in live cron list
CT 221 Miner Watchdog 2026-08-26 Job d5516d0a5147 removed; CT221 status unknown/inspected
Review Agent Estate Phase One 2026-08-26 One-time job; completed 2026-08-08, now cleaned up

Disabled Jobs (not removed)

Name ID Disabled since Reason
Important-mail monitor d40085da631c 2026-08-29 Paused during Gmail read-state incident. See incidents/2026-08-29-gmail-read-state.
Rhino AgentMail Inbox Watcher f77b7e46bd09 2026-07-27 Legacy AgentMail polling; superseded by newer patterns.
AgentMail Rhino Inbox Poll 0a7f933d06e2 2026-07-27 Legacy AgentMail polling; superseded by newer patterns.

Known Issues

Resolved 2026-10-07

  • DHL Tracking 62619aa6bcb3 — single Camofox timeout, self-recovered (2026-10-07 08:00 → 11:01): the 08:00 run died in ensure_browser() with TimeoutError: timed out on a browser HTTP call (dhl_tracking_monitor.py:233), i.e. Camofox was momentarily unresponsive. The failure notice was sent at 08:00:58 and the incident (62619a_f485ccfd4a0d, failure_type: timeout) auto-resolved at 11:01:13 when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. This is the first detect → notify → auto-resolve incident cycle since the 2026-09-30 managed-runtime event — the incident machinery itself worked as designed. No action needed; a repeat would justify a Camofox health probe inside the script.
  • The tracked parcel arrived. The 14:00 run on 2026-10-07 reported waybill 8052838382 Delivered (Kensington → Perth). The monitor has therefore achieved its purpose and runs 5×/day for no remaining benefit — a candidate for removal, flagged rather than removed (the job is Anthony's to retire).

Resolved 2026-10-05

  • Dead model pins across 7 jobs — repinned to routing aliases (resolved 2026-10-05 23:16 → 2026-10-06 00:05): Seven jobs were each pinned to a raw model name that had died upstream — cliprox/terra (HTTP 401, connection auth expired, 3 jobs), ollama-cloud/qwen3.5:397b (HTTP 410, retired 2026-09-25), nvidia/deepseek-ai/deepseek-v4-flash (HTTP 400, dropped from the live catalog, 2 jobs) and command-code/gpt-5.6-terra (HTTP 403 MODEL_NOT_IN_PLAN) — across 2026-10-01 → 2026-10-05. All seven were repinned to provider custom:omni-101 using routing aliases — free-stack ×6, smart-route ×1 — deliberately, because the fault was raw-pin brittleness and re-pinning to another raw name would have re-created it. failure_streak is now 0 on all 30 jobs, down from 7 failing. Three were verified by live run: 79f9bb41ecf5 (23:39), 06a0a1c8704f (23:45) and dca8482e4f76 (23:56, delivered to topic 1030 — its first success since 2026-09-25). The remaining four are repinned but not yet re-run; the stale last_status: error field clears on each one's next scheduled run. See incidents/2026-10-05-dead-model-pin-repin.
    • The cliprox/terra incident is closed for cron purposes: no enabled job references that pin any more. The underlying proxy connection was never re-authenticated — the jobs were moved off it instead. See incidents/2026-10-01-cliproxy-connection-auth-expired.
    • wiki-stocktake-maintenance resumes its 03:00 schedule, so wiki-maintain.sh no longer needs a daily manual run.
  • GitHub Trending Radar c0a9fa314542 — token-revocation claim refuted (2026-10-05): the scheduled Mon 09:00 run completed cleanly — its second consecutive success after the off-schedule 2026-09-30 19:22 run. The getMe → 401 revoked-token diagnosis no longer applies and no token replacement is needed. It still delivers to the bare home DM, which works.
  • A stale failure notice, not a live outage, opened the 2026-10-05 review: the interactive session began because Anthony acted on a 85edf6772bca failure notice sitting in topic 346 since the 2026-09-29 07:00 run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already resolved on 2026-09-30 17:55:13. The notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. Check executions.started_at and the cron_incidents state before acting on a notice.

Resolved 2026-09-30

  • Telegram thread delivery failures — root cause found and fixed: jobs fad5a86979d4, 1b23c042e17d, c8d52904e1d6 reported configured thread_id 34795 for telegram:1793951355 was not found / Message thread not found. The target was wrong, not the topic: a bare deliver: "telegram" resolves to the home DM (1793951355), where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong). 1b23c042e17d and c8d52904e1d6 were retargeted to telegram:-1004321904721:102 and verified end-to-end. See systems/messaging-integrations.
  • Managed-runtime dependency failure (2026-09-28 → 30): Jobs failed with Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1). Real cause was ModuleNotFoundError: No module named 'dotenv' / 'mnemosyne' in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by hermes-runtime-deps-guard.sh v2, run every 15 min by job 96eb95ce12f2. All managed venvs verified. See incidents/2026-09-28-hermes-managed-runtime-deps.
  • weekly-save-restricted-bot-restart missing script (2026-08-30): Previously failed with "Script not found: restart-save-restricted-bot.sh". The script existed at /home/hermes/scripts/ but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job now ok, streak 0.
  • Residual pre-fix dispatch errors cleared: cbfe434a3b5e returned to ok on its next run as predicted. The 1a8e90ea2271 error of the same era also cleared — its current failure is a different, upstream problem (below).
  • Stale delivery_failed flag on 1b23c042e17d cleared (confirmed 2026-10-02): the 2026-09-30 retarget held — the job's next run delivered to topic 102.

Open

  • Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07): wiki-maintain.sh documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no quartz.config.ts is present and npx is unavailable, so public/ has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description.
  • skills-index under-counts by one skill (NEW 2026-10-07): live find ~/.hermes/skills -name SKILL.md -type f returns 347, but the generated page says 346 (70 categories). The generator de-dups on skill basename within a category, so the nested duplicate productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md collapses into its parent and is silently dropped. The other duplicated basename (hermes-api-vault, at the root and under devops/) survives because the two land in different categories. Remove the nested duplicate directory, or key the generator on path rather than basename.
  • A delivered notice is not a successful job: cliprox/terra failures recorded delivery_outcome: delivered because the failure notice is itself a deliverable. Read the execution status column, not the delivery outcome, when judging whether a job produced its output.
  • Stale provider key custom:omni (1 enabled + 1 disabled): the provider was removed from custom_providers (live set on 2026-10-01: litellm, guacano, Omniroute:20128, command-code, omni-101). Re-verified 2026-10-06: e530c89f2be0 (enabled) and d40085da631c (disabled) still name it; both run smart-route, and the default custom provider resolves to the same base URL (http://100.78.63.20:20128), so no failure has been observed — but the references should be normalised to custom:omni-101. The 2026-10-05 repin moved seven jobs to an explicit custom:omni-101, leaving these two as the only remaining custom:omni references.
  • Bare telegram delivery targets (3 jobs) — demonstrably breaking sends: 0e753e9720fd, fad5a86979d4, 066bb5191280 deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical: 066bb5191280 (02:11) and 0e753e9720fd (09:01) both recorded delivery_outcome: failed with configured thread_id 34795 for telegram:1793951355 was not found. Work generated correctly; the delivery did not. Retarget all three to telegram:-1004321904721:<topic>. See systems/messaging-integrations.
  • Two wiki trees, drifted in both directions (NEW 2026-10-02): /home/hermes/workspace/agent-estate-wiki is the canonical source (Gitea remote Anthony/agent-estate-wiki, Quartz build, wiki-maintain.sh target). /home/hermes/wiki/agent-estate-wiki is the Obsidian/Ignis vault copy the iPad reads — pages nested under wiki/, and no git remote at all. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged backwards (systems/ignis.md, systems/nextcloud.md, three incidents/ pages exist only in the vault; raw/stocktake/*, agents/, incidents/2026-09-02-sftpgo-outage.md only in canonical) — a naive one-way sync deletes pages. See systems/hermes-agent.
  • Work mailbox unreachable (non-cron; affects on-demand lookups): the work Outlook account is configured with password auth while the tenant advertises only XOAuth2, so himalaya -a work fails with login not supported. No cron job polls it. See incidents/2026-08-26-work-outlook-basic-auth-disabled.

Historical

  • Mnemosyne Watchdog incident (2026-08-19): Job 96eb95ce12f2 cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See systems/mnemosyne.
  • Session storage failures (2026-08-10): Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See current-state.
  • Model repinning (2026-08-08): 13 jobs repinned from dead/stale provider combos to smart-route or gpt-5.6-terra.
  • Wiki Audit model drift (2026-08-07): Global inference config change caused RuntimeError: Skipped to prevent unintended spend on an unpinned job.

Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09)

Fleet: 30 jobs · 27 enabled · 3 disabled · 10 carrying a non-clean last_status at 2026-10-10 00:22 (jobs.json.updated_at 2026-10-10T00:22:17+08:00). Executions by AWST day: 10-07 {'completed': 177, 'failed': 1} · 10-08 {'completed': 178, 'failed': 2} · 10-09 {'completed': 162, 'failed': 15, 'running': 1}.

The 2026-10-09 timeout storm. 16 failed executions across 11 distinct jobs, every one RuntimeError: Request timed out. on the routing aliases behind custom:omni-101, 00:00 → 15:00 AWST; every run from 18:00 onward completed. Full timeline, job list and incident table: incidents/2026-10-09-cron-provider-timeout-storm.

  • Four incidents auto-closed on the next clean run (96eb95 07:01, b63ed5 18:01, 1b3118 22:05, ea79a2 2026-10-10 00:03); eight remain open — 1a8e90_789224ad589c, cbfe43_fa81052baaf7, 1a8e90_dc6b2d40ec44, dca848_23fcd2b18aaa, 85edf6_18bedbeec5cd, 1b23c0_c35703f637b8, 08eb86_634f1a296f78, ecc1fc_58b6c38296f3.
  • 1a8e90ea2271 is the only job still at failure_streak: 2 (10-08 HTTP 403 + 10-09 timeout); its 03:00 run today decides whether the streak clears.
  • 96eb95ce12f2 (watchdog) failed twice as a cascade — it exits 1 when it finds ERROR lines in agent.log, so it reported the storm rather than suffering it.

The 2026-10-08 window has no audit entry. The nightly audit's own 23:59 run on 2026-10-08 failed (RuntimeError: Request timed out., incident cbfe43_fa81052baaf7, first seen 2026-10-09T00:32:09) and committed nothing; the last committed audit covers the 2026-10-07 window (commit 659356a, 2026-10-08 00:12). The 2026-10-09 run covers both days.

Stocktake: two-day gap, artifacts recovered. The 2026-10-08 03:01 run left systems/skills-index.md modified and raw/stocktake/freshness-20261008.txt untracked; both were stranded (the job's own commit step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator ran, so no freshness-20261009.txt exists and the skills index is still the 2026-10-08 output.

DHL monitor — retirement instructed, still enabled. Anthony replied "Delete this job" to the 20:26 DHL delivery notice on 2026-10-09; the session replied that 62619aa6bcb3 was "stopped and deleted". Live jobs.json at 2026-10-10T00:22 shows 62619aa6bcb3 still enabled — paused_at: null, last_status: ok, schedule 0 8,11,14,17,20 * * * — so it runs again at 08:00. The parcel (8052838382) was reported Delivered on 2026-10-07, making the monitor redundant. This audit did not mutate cron state; retiring the job is open item #1 below.

Memory recall relevance. "Are you up to date" returned a stale Uber Eats episode on 2026-10-09; capture and vector health were verified good (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is not in live config. See incidents/2026-10-09-memory-recall-relevance.

Open items (2026-10-09)

  1. Retire 62619aa6bcb3 (DHL Tracking — Parcel 8052838382) — instructed 2026-10-09 21:24, not executed.
  2. Decide and persist the recall-ranking change in memory.mnemosyne; re-run the trigger question as a regression test.
  3. Close the eight open cron incidents once their next clean runs land; watch 1a8e90ea2271 (streak 2).
  4. Stale delivery targets — 0e753e9720fd, fad5a86979d4 (last_status: error) and 066bb5191280 (delivery_failed) unchanged since the 2026-10-07 audit.
  5. Quartz site rebuild is skipped by wiki-maintain.sh step 3 (no quartz.config.ts, no npx) — carried from 2026-10-07, still open.
  6. Skills index off by one (347 SKILL.md on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.