--- title: Scheduled Tasks type: system status: active created: 2026-07-22 updated: 2026-10-13 verified_on: 2026-10-13 review_after: 2026-10-25 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] --- # Scheduled Tasks ## Count - **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, re-verified 2026-10-13 — unchanged; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10) - **Non-clean `last_status`:** **3** (was 5 at the 2026-10-12 audit) — `0e753e9720fd` (`error`, streak 0), `1b23c042e17d` (`error`, streak 1), `066bb5191280` (`delivery_failed`, streak 0). `fad5a86979d4` and `dca8482e4f76` cleared on **2026-10-12** when both ran clean on schedule. - **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]]. - Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04). - **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window. - **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs. - **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07); and `1a8e90ea2271` completed automatically on **2026-10-06 03:00**. Four of the seven are therefore verified by live run. The remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are repinned but not yet re-run — they are due on their normal schedules. `1a8e90ea2271` has since run **twice more on schedule** (2026-10-06 03:00 → `33f4e95`; 2026-10-07 03:00 → `8c63ef2`), so the stocktake repin is verified across consecutive automatic runs. - **Streak counting note:** the `N failures in a row` figure in `hermes cron list` comes from the `failure_streak` counter in `jobs.json`. `executions.db` retains only rows from **2026-09-30 onward**, so DB-based recounts undercount older streaks. Live counters as of 2026-10-06: **all 0**. - **Re-verified 2026-10-06 AWST:** job count unchanged at 30 (27 enabled / 3 disabled). `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found`). - **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 `completed`, 0 `failed`, 1 `running` (the audit itself); 14 sessions, all `cron` — **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`). - **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice. - **Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00):** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — every `failure_streak` reads 0. Window tally (`executions.db`): **176 `completed`, 1 `failed`, 1 `running`** (the audit itself). The single failure is `62619aa6bcb3` (DHL Tracking, 08:00) — a Camofox HTTP read timeout (`ensure_browser()` → `TimeoutError: timed out`), notified 08:00:58 and **auto-resolved 11:01:13** when the next 3-hourly run succeeded (incident `62619a_f485ccfd4a0d`). 13 sessions started in the window, all `source = cron` — **no interactive sessions**; 12 deliveries, **0 delivery failures**. The same three stale pre-repin `last_status: error` jobs remain (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`); the stocktake job `1a8e90ea2271` ran on schedule at 03:00 for the second consecutive day. - Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08) → **29, 0 failing** (2026-10-10) → **29, 0 failing** (2026-10-11) → **29, 0 failing** (2026-10-12) - **Re-verified 2026-10-12 AWST (window 2026-10-11 00:00 → 2026-10-12 00:00):** 29 jobs (26/3), **0 failing**, **5** non-clean `last_status`; window tally **172 `completed`, 0 `failed`** (the 2026-10-11 day finalised one run higher than the previous audit's snapshot, which caught it mid-flight at 171 + 1 `running`). 18 sessions — 17 `cron` + 1 interactive. - **Re-verified 2026-10-13 AWST (window 2026-10-12 00:00 → 2026-10-13 00:00):** 29 jobs (26/3), **0 failing**, non-clean `last_status` down to **3**. Window tally **174 `completed`, 0 `failed`** — **third consecutive clean day** (10-10 `175/0`, 10-11 `172/0`, 10-12 `174/0`). **5 open incidents → 2:** three closed on 2026-10-12 (see [[#Resolved 2026-10-12]]). 18 sessions in the window, **all `cron`, no interactive sessions**; 0 new incident pages required. Two findings led the window: the **GitHub Trending Radar delivery token is dead** (`getMe` → 401, live-verified) and the **Daily Marketing Job Radar run hit Camofox memory pressure** (2.5 GB peak, heavy swap, self-recovered by restart). ## Job Table | Name | ID | Schedule | Delivery | Skills | Script | Notes | |---|---|---|---|---|---|---| | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) | | AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 | | Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | | | Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. Live `last_status: ok`, streak 0. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open) | | DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant | | Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1. **Next scheduled slot: Fri 2026-10-16 08:00**, which should clear it | | Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent | | Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic | | Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). Live `last_status: ok`, streak 0. ⚠ Bare `telegram` target should move to a group topic | | FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | | | GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. `last_status: ok` because the 401 is inside the script, not the cron run. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** | | Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | | | Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) | | Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent | | Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent | | Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent | | Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95% | | Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception | | Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30) | | Mosquito Tracker | `1b31187397a5` | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 | | Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` | | Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) | | Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | — | | | Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic | | Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 | | weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 | | Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes | | wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken | | 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent | ## Removed Jobs | Name | Removed | Reason | |---|---|---| | Workday start reminder | 2026-08-26 | No longer present in live cron list | | CT 221 Miner Watchdog | 2026-08-26 | Job `d5516d0a5147` removed; CT221 status unknown/inspected | | Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up | ## Disabled Jobs (not removed) | Name | ID | Disabled since | Reason | |---|---|---|---| | Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | ## Known Issues ### Resolved 2026-10-12 - **Both `deepseek-v4-flash` incidents closed by a clean scheduled run (`fad5a86979d4`, 03:00):** the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents `fad5a8_a13bbc837e1c` (open since 2026-09-28, every fallback provider exhausted) and `fad5a8_c80b59a907f0` (since 2026-09-30, `deepseek-v4-flash` HTTP 400) **both auto-closed at 03:05:39**. Its stale pre-repin `last_status: error` cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one. - **`dca8482e4f76` timeout incident closed (`dca848_23fcd2b18aaa`):** the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm **auto-closed at 07:33:41**. Of the ten storm incidents, nine have now closed and **one remains**: `1b23c0_c35703f637b8` (`1b23c042e17d`, still `alerted`, next slot Fri 2026-10-16 08:00). See [[incidents/2026-10-09-cron-provider-timeout-storm]]. - **Net effect: open cron incidents 5 → 2.** The two survivors are `0e753e_e7f8194a7d06` (`detected` since 2026-10-01, untouched because its job does not run until 2026-11-01) and `1b23c0_c35703f637b8` (`alerted`, cleared on the next Friday slot). ### Resolved 2026-10-07 - **DHL Tracking `62619aa6bcb3` — single Camofox timeout, self-recovered (2026-10-07 08:00 → 11:01):** the 08:00 run died in `ensure_browser()` with `TimeoutError: timed out` on a browser HTTP call (`dhl_tracking_monitor.py:233`), i.e. Camofox was momentarily unresponsive. The failure notice was sent at 08:00:58 and the incident (`62619a_f485ccfd4a0d`, `failure_type: timeout`) **auto-resolved at 11:01:13** when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. **This is the first detect → notify → auto-resolve incident cycle since the 2026-09-30 managed-runtime event** — the incident machinery itself worked as designed. No action needed; a repeat would justify a Camofox health probe inside the script. - **The tracked parcel arrived.** The 14:00 run on 2026-10-07 reported waybill **8052838382 `Delivered`** (Kensington → Perth). The monitor has therefore achieved its purpose and runs 5×/day for no remaining benefit — a candidate for removal, flagged rather than removed (the job is Anthony's to retire). ### Resolved 2026-10-05 - **Dead model pins across 7 jobs — repinned to routing aliases (resolved 2026-10-05 23:16 → 2026-10-06 00:05):** Seven jobs were each pinned to a raw model name that had died upstream — `cliprox/terra` (HTTP 401, connection auth expired, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`) — across 2026-10-01 → 2026-10-05. All seven were repinned to provider `custom:omni-101` using **routing aliases** — `free-stack` ×6, `smart-route` ×1 — deliberately, because the fault was raw-pin brittleness and re-pinning to another raw name would have re-created it. `failure_streak` is now 0 on all 30 jobs, down from 7 failing. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The remaining four are repinned but not yet re-run; the stale `last_status: error` field clears on each one's next scheduled run. See [[incidents/2026-10-05-dead-model-pin-repin]]. - The `cliprox/terra` incident is **closed for cron purposes**: no enabled job references that pin any more. The underlying proxy connection was never re-authenticated — the jobs were moved off it instead. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. - `wiki-stocktake-maintenance` resumes its 03:00 schedule, so `wiki-maintain.sh` no longer needs a daily manual run. - **GitHub Trending Radar `c0a9fa314542` — token-revocation claim refuted (2026-10-05): SUPERSEDED 2026-10-13.** The Mon 09:00 run *completed* on 2026-10-05 and 2026-10-12, which was read as proof the `getMe` → 401 diagnosis was wrong. It was not: a cron run can be green while the script's own delivery hop is dead, and the 2026-10-13 live `getMe` against the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **401**. Corrected above and in Open items. **Lesson: a completed run does not clear a credential fault unless the run exercised the same call that failed.** - **A stale failure notice, not a live outage, opened the 2026-10-05 review:** the interactive session began because Anthony acted on a `85edf6772bca` failure notice sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.** ### Resolved 2026-09-30 - **Telegram thread delivery failures — root cause found and fixed:** jobs `fad5a86979d4`, `1b23c042e17d`, `c8d52904e1d6` reported `configured thread_id 34795 for telegram:1793951355 was not found` / `Message thread not found`. The target was wrong, not the topic: a **bare `deliver: "telegram"` resolves to the home DM (`1793951355`)**, where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong). `1b23c042e17d` and `c8d52904e1d6` were retargeted to `telegram:-1004321904721:102` and verified end-to-end. See [[systems/messaging-integrations]]. - **Managed-runtime dependency failure (2026-09-28 → 30):** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause was `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2, run every 15 min by job `96eb95ce12f2`. All managed venvs verified. See [[incidents/2026-09-28-hermes-managed-runtime-deps]]. - **`weekly-save-restricted-bot-restart` missing script (2026-08-30):** Previously failed with "Script not found: `restart-save-restricted-bot.sh`". The script existed at `/home/hermes/scripts/` but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job now `ok`, streak 0. - **Residual pre-fix dispatch errors cleared:** `cbfe434a3b5e` returned to `ok` on its next run as predicted. The `1a8e90ea2271` error of the same era also cleared — its current failure is a *different*, upstream problem (below). - **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`. ### Open - **GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13):** live `getMe` on the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**, so `digest.py`'s Telegram send fails every Monday while the cron run itself stays green (`last_status: ok`). A *completed* run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with `--dry-run` and its own response was delivered. Fix: issue a fresh token from @BotFather, replace it in `config.json`, and re-run `getMe` to confirm; do not re-read a green cron status as proof of delivery. **This corrects the 2026-10-05 "claim refuted" entry, which was wrong.** See [[systems/scheduled-tasks]] job `c0a9fa314542`. - **Camofox memory pressure during long browser runs (NEW 2026-10-13):** the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`: `MemoryHigh=2G`, `MemoryMax=3G`, swap max 1G). Live probe mid-run read `Memory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G)` with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the `/tabs` endpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See [[systems/browser-backend]]. - **Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07):** `wiki-maintain.sh` documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no `quartz.config.ts` is present and `npx` is unavailable, so `public/` has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description. - **`skills-index` under-counts by one skill (NEW 2026-10-07):** live `find ~/.hermes/skills -name SKILL.md -type f` returns **347**, but the generated page says **346** (70 categories). The generator de-dups on skill basename within a category, so the nested duplicate `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` collapses into its parent and is silently dropped. The other duplicated basename (`hermes-api-vault`, at the root and under `devops/`) survives because the two land in different categories. Remove the nested duplicate directory, or key the generator on path rather than basename. - **A delivered notice is not a successful job:** `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output. - **Stale provider key `custom:omni` (1 enabled + 1 disabled):** the provider was removed from `custom_providers` (live set on 2026-10-01: `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101`). Re-verified 2026-10-06: `e530c89f2be0` (enabled) and `d40085da631c` (disabled) still name it; both run `smart-route`, and the default `custom` provider resolves to the same base URL (`http://100.78.63.20:20128`), so no failure has been observed — but the references should be normalised to `custom:omni-101`. The 2026-10-05 repin moved seven jobs to an explicit `custom:omni-101`, leaving these two as the only remaining `custom:omni` references. - **Bare `telegram` delivery targets (3 jobs) — demonstrably breaking sends:** `0e753e9720fd`, `fad5a86979d4`, `066bb5191280` deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical: `066bb5191280` (02:11) and `0e753e9720fd` (09:01) both recorded `delivery_outcome: failed` with `configured thread_id 34795 for telegram:1793951355 was not found`. Work generated correctly; the delivery did not. Retarget all three to `telegram:-1004321904721:`. See [[systems/messaging-integrations]]. - **Two wiki trees, drifted in both directions (NEW 2026-10-02):** `/home/hermes/workspace/agent-estate-wiki` is the canonical source (Gitea remote `Anthony/agent-estate-wiki`, Quartz build, `wiki-maintain.sh` target). `/home/hermes/wiki/agent-estate-wiki` is the **Obsidian/Ignis vault copy** the iPad reads — pages nested under `wiki/`, and **no git remote at all**. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged *backwards* (`systems/ignis.md`, `systems/nextcloud.md`, three `incidents/` pages exist only in the vault; `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` only in canonical) — a naive one-way sync deletes pages. See [[systems/hermes-agent]]. - **Work mailbox unreachable (non-cron; affects on-demand lookups):** the `work` Outlook account is configured with password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` fails with `login not supported`. No cron job polls it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]]. ### Historical - **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]]. - **Session storage failures (2026-08-10):** Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See [[current-state]]. - **Model repinning (2026-08-08):** 13 jobs repinned from dead/stale provider combos to `smart-route` or `gpt-5.6-terra`. - **Wiki Audit model drift (2026-08-07):** Global inference config change caused `RuntimeError: Skipped to prevent unintended spend` on an unpinned job. ## Related - [[systems/mnemosyne]] - [[runbooks/job-radar]] - [[systems/hermes-agent]] - [[systems/messaging-integrations]] - [[incidents/2026-09-28-hermes-managed-runtime-deps]] - [[incidents/2026-10-01-cliproxy-connection-auth-expired]] - [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] --- ## Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09) **Fleet:** 30 jobs · 27 enabled · 3 disabled · **10 carrying a non-clean `last_status`** at 2026-10-10 00:22 (`jobs.json.updated_at 2026-10-10T00:22:17+08:00`). Executions by AWST day: **10-07** {'completed': 177, 'failed': 1} · **10-08** {'completed': 178, 'failed': 2} · **10-09** {'completed': 162, 'failed': 15, 'running': 1}. **The 2026-10-09 timeout storm.** 16 failed executions across 11 distinct jobs, every one `RuntimeError: Request timed out.` on the routing aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; every run from 18:00 onward completed. Full timeline, job list and incident table: [[incidents/2026-10-09-cron-provider-timeout-storm]]. - Four incidents auto-closed on the next clean run (`96eb95` 07:01, `b63ed5` 18:01, `1b3118` 22:05, `ea79a2` **2026-10-10 00:03**); **eight remain open** — `1a8e90_789224ad589c`, `cbfe43_fa81052baaf7`, `1a8e90_dc6b2d40ec44`, `dca848_23fcd2b18aaa`, `85edf6_18bedbeec5cd`, `1b23c0_c35703f637b8`, `08eb86_634f1a296f78`, `ecc1fc_58b6c38296f3`. - `1a8e90ea2271` is the only job still at **`failure_streak: 2`** (10-08 HTTP 403 + 10-09 timeout); its 03:00 run today decides whether the streak clears. - `96eb95ce12f2` (watchdog) failed twice as a **cascade** — it exits 1 when it finds ERROR lines in `agent.log`, so it reported the storm rather than suffering it. **The 2026-10-08 window has no audit entry.** The nightly audit's own 23:59 run on 2026-10-08 failed (`RuntimeError: Request timed out.`, incident `cbfe43_fa81052baaf7`, first seen 2026-10-09T00:32:09) and committed nothing; the last committed audit covers the 2026-10-07 window (commit `659356a`, 2026-10-08 00:12). The 2026-10-09 run covers both days. **Stocktake: two-day gap, artifacts recovered.** The 2026-10-08 03:01 run left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261008.txt` untracked; both were stranded (the job's own commit step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator ran, so **no `freshness-20261009.txt` exists** and the skills index is still the 2026-10-08 output. **DHL monitor — retirement instructed, still enabled.** Anthony replied "Delete this job" to the 20:26 DHL delivery notice on 2026-10-09; the session replied that `62619aa6bcb3` was *"stopped and deleted"*. Live `jobs.json` at 2026-10-10T00:22 shows `62619aa6bcb3` **still enabled** — `paused_at: null`, `last_status: ok`, schedule `0 8,11,14,17,20 * * *` — so it runs again at 08:00. The parcel (`8052838382`) was reported `Delivered` on 2026-10-07, making the monitor redundant. **This audit did not mutate cron state**; retiring the job is open item #1 below. **Memory recall relevance.** "Are you up to date" returned a stale Uber Eats episode on 2026-10-09; capture and vector health were verified good (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is **not in live config**. See [[incidents/2026-10-09-memory-recall-relevance]]. ### Open items (2026-10-09) 1. **Retire `62619aa6bcb3`** (DHL Tracking — Parcel `8052838382`) — instructed 2026-10-09 21:24, not executed. 2. **Decide and persist the recall-ranking change** in `memory.mnemosyne`; re-run the trigger question as a regression test. 3. **Close the eight open cron incidents** once their next clean runs land; watch `1a8e90ea2271` (streak 2). 4. **Stale delivery targets** — `0e753e9720fd`, `fad5a86979d4` (`last_status: error`) and `066bb5191280` (`delivery_failed`) unchanged since the 2026-10-07 audit. 5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open. 6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open. --- ## Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST) **Fleet: 29 jobs - 26 enabled - 3 disabled** (`jobs.json.updated_at 2026-10-11T00:19:44+08:00`). The count fell from 30 to 29 because **`62619aa6bcb3` (DHL Tracking - Parcel `8052838382`) is gone**: open item #1 of the 2026-10-09 audit is **CLOSED** (below). Disabled: `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher), `0a7f933d06e2` (AgentMail Rhino Inbox Poll). **Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` - 10-08 `178 / 2` - 10-09 `163 / 15` - **10-10 `174 completed / 0 failed`** (+1 left `running`). The 2026-10-10 fleet was wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm. **Non-clean `last_status`: 5 jobs, down from 10.** | Job | ID | `last_status` | streak | |---|---|---|---| | Facebook Birthday Monthly Scrape | `0e753e9720fd` | `error` | 0 | | Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `error` | 0 | | Daily Marketing Job Radar | `dca8482e4f76` | `error` | 1 | | Enhanced Weekend Adventure Planner | `1b23c042e17d` | `error` | 1 | | Uber Eats Monthly Spend Report | `066bb5191280` | `delivery_failed` | 0 | The three `streak 0` rows are stale `last_status` fields, not live failure streaks. `1a8e90ea2271` (stocktake) is back to `last_status: ok`. **Incidents: 7 closed during 2026-10-10, 5 open.** The 10-09 storm unwound as each job next ran clean (`ea79a2` 00:03, `cbfe43` 00:27, `1a8e90ea2271` x2 03:06, `85edf6` 07:04, `08eb86` 08:32, `ecc1fc` 12:05). Still open: `1b23c042e17d` + `dca8482e4f76` (`timeout`, `alerted`, both first seen 2026-10-09) and `0e753e9720fd` + `fad5a86979d4` (`agent`; open since 2026-10-01 / 2026-09-30). **Open item #1 CLOSED - the DHL monitor is retired.** Anthony instructed removal again on the evening of 2026-10-10 and the job is **absent from live `jobs.json`**, verified 2026-10-11 00:19 by direct ID lookup (no job id contains `62619`; no job name matches dhl/parcel). This is the third "deleted" claim for this job - the 2026-10-07 and 2026-10-09 reports were both **false** because the check was the absence of job output rather than a positive lookup. Removals must be verified by ID lookup. **Still open:** the Quartz rebuild step in `wiki-maintain.sh` (no `quartz.config.ts`) - skills-index off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the 15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]). --- ## Re-verified 2026-10-12 (audit window 2026-10-11 00:00 → 2026-10-12 00:2x AWST; live probes 00:0x-00:2x) **Fleet:** **29 jobs — 26 enabled / 3 disabled** (`jobs.json` `updated_at` 2026-10-12T00:10:42+08:00). Disabled set unchanged: `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher), `0a7f933d06e2` (AgentMail Rhino Inbox Poll — legacy). **Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` — 10-08 `178 / 2` — 10-09 `163 / 15` — 10-10 **`175 completed / 0 failed`** (finalised; the 10-11 audit saw 174 + 1 still `running`) — 10-11 **`171 completed / 0 failed`** + 1 `running` (this audit). **Two consecutive zero-failure days** after the 2026-10-09 provider-timeout storm. Heaviest jobs on 10-11: runtime-deps guard `96eb95ce12f2` 96 runs (15-min cadence), `0c771463c3cd` 46, `2fd50c87b0d0` 12, coverage monitor `b63ed5817212` 3, health monitor `ea79a2107e36` 2, stocktake `1a8e90ea2271` 1. **Non-clean `last_status`: 5 jobs — unchanged from 2026-10-10.** `0e753e9720fd` (`error`, streak 0), `fad5a86979d4` (`error`, streak 0), `dca8482e4f76` (`error`, streak 1), `1b23c042e17d` (`error`, streak 1), `066bb5191280` (`delivery_failed`, streak 0). The two streak-1 jobs are the Mon-Fri and Friday-only schedulers: their `last_error` is `RuntimeError: Request timed out.` from **Friday 2026-10-09** (the storm) and neither has had a scheduled slot since — `dca8482e4f76` next runs 07:00 Mon, `1b23c042e17d` Fri 08:00, so both should clear themselves. Everything else reads `last_status: ok` (this build reports `ok`, not `success`). **Incidents: 0 opened / 0 closed in the window — 5 unresolved, unchanged.** `alerted` 3: `fad5a8_c80b59a907f0` (weekly registry sync, scope `agent`, first seen 2026-09-30, `deepseek-v4-flash` HTTP 400), `dca848_23fcd2b18aaa` + `1b23c0_c35703f637b8` (`timeout`, both first seen 2026-10-09 — the last two left from the storm). `detected` 2: `fad5a8_a13bbc837e1c` (2026-09-28, every provider in the fallback chain exhausted), `0e753e_e7f8194a7d06` (2026-10-01, same `deepseek-v4-flash` 400). Table totals: alerted 3 / detected 2 / closed 8 / resolved 63. Last closures remain the seven on 2026-10-10. **Sessions in the window:** 18 — 17 `cron` + 1 interactive (`talaria-ios`, 2026-10-11 19:13 → 19:51, 41 messages, "best VNC clients" research). The 03:00 stocktake `1a8e90ea2271` ran clean on schedule (`completed` 03:00:40 → 03:11:29) and committed `d981e30`. **False alarm — the 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION was a timezone mislabel, not a capture stall.** The run compared a naive-UTC `working_memory.created_at` (`2026-10-10 13:24:53`) against AWST `now` and reported "10.7 h of no captures"; the true gap was 2.7 h (the row landed 21:24:53 AWST, inside the session that produced it). No pipeline fault. The same job's 12:05 and 2026-10-12 00:02 runs returned `[SILENT]` (green). → [[systems/mnemosyne]] **Still open (unchanged):** Quartz rebuild step inert (`wiki-maintain.sh` step 3 — no `quartz.config.ts`); skills-index off-by-one (347 `SKILL.md` on disk vs 346 reported); the 5 non-clean `last_status` rows; Mattermost platform unreachable ([[systems/messaging-integrations]]); gateway runtime env outside the 15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]). ## Re-verified 2026-10-13 (audit window 2026-10-12 00:00 → 2026-10-12 23:59 AWST; live probes 2026-10-13 00:05–00:45) **Headline: a clean run *hid* a dead credential.** The cron table was green — 174 completed / 0 failed — and two of the three non-clean `last_status` rows cleared overnight, yet one job that reported success on paper had its delivery hop dead the whole time. Detail below. **Cron tally:** 2026-10-11 **172 completed / 0 failed** (the one row still `running` at the last audit finished clean); 2026-10-12 **174 completed / 0 failed / 1 running** — and the running row is *this audit itself* (`cbfe434a3b5e`, started 23:59:41). **Third consecutive clean day.** The execution-log retention floor has moved up to **2026-10-07 08:15**, so 10-07 rows are now pruned (their earlier tally is no longer reproducible from this DB). **Incidents: open 5 → 2.** Three closed on 2026-10-12 — `fad5a8_a13bbc837e1c` and `fad5a8_c80b59a907f0` at **03:05:39** (both `deepseek-v4-flash`, closed by the first clean scheduled run of `fad5a86979d4`); `dca848_23fcd2b18aaa` at **07:33:41** (closed by `dca8482e4f76`'s own successful 07:00 run). Table totals now **alerted 1 / detected 1 / closed 8 / resolved 66** (76 rows). Survivors: `0e753e_e7f8194a7d06` (`detected` since 2026-10-01 — cannot clear until its job next runs on **2026-11-01**) and `1b23c0_c35703f637b8` (`alerted` since 2026-10-09, next slot **Fri 2026-10-16 08:00**). **Job-by-job (verified against the executions log, `delivery_outcome` included):** - `fad5a86979d4` — Facebook Birthday Weekly Registry Sync **03:00:45 → 03:05:47, completed, delivered**. Output: Birthday Dashboard, **448 unique birthdays**, 1 contact match. This is the repin's live proof. - `dca8482e4f76` — Daily Marketing Job Radar **07:00:47 → 07:33:50, completed, delivered** to topic 1030 (32-minute run; the 2026-10-09 timeout incident closed 9 seconds before it finished). Run was messy — see the Camofox finding — but it produced and delivered. - `c0a9fa314542` — GitHub Trending Radar **09:00:41 → 09:08:33, completed, `delivered`** — and yet **its own send failed** (finding 1). A `completed` row plus `delivery_outcome: delivered` is *not* proof the job's payload went out; here the delivered artifact was the agent's recovered digest, not the script's. - `1b23c042e17d` — Enhanced Weekend Adventure Planner **2026-10-09 08:00:38 → 08:17:46, failed** (`RuntimeError: Request timed out.`, notice delivered). No run since; next slot Fri 2026-10-16 08:00. - `0e753e9720fd` — Facebook Birthday Monthly Scrape: **no runs at all inside the retained window** (monthly, next 2026-11-01), so its `error`/streak-0 row is still untested rather than fixed. **Sessions in the window:** **18, all `cron`, no interactive session** — a quiet day, in contrast to the previous window's `talaria-ios` research session. Busiest by message count: `dca8482e4f76` (140), `1a8e90ea2271` stocktake (97), Wiki Audit (28), `c0a9fa314542` (25). The 03:00 stocktake ran clean and pushed **`0eee5f1`** to origin/master: skills-index regenerated (346 skills / 70 categories, 347 files on disk — the known off-by-one), freshness audit **54 stale / 32 fresh / 7 no-date of 93 pages**, Quartz step skipped on the documented path. **Mnemosyne (12:02 health run, green):** working **760/786 embedded = 96.7%**, episodic **1050/1056 = 99.4%** — both above the 95% floor, so no `reindex` needed. Orphan embeddings 0. FTS5 aligned exactly (`fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876). 0 unconsolidated working rows; 1 unresolved conflict (threshold 10); last consolidation 0.3 h earlier. 2252 memories total (786 working / 1056 episodic / 1 triple), 395 consolidated facts. Disk 15 GB free (81% used). No exceptions → nothing to act on, and nothing to change on [[systems/mnemosyne]] beyond the coverage figures. **Finding 1 — GitHub Trending Radar's Telegram token is revoked (live-verified).** `getMe` against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The token is a plausible length (46 chars), so it is revoked/invalid rather than malformed, and there is **no valid replacement anywhere on the box**. The fetch path is healthy — GitHub trending pulled 100 repos and wrote 26 cache files in 24 h — so only `digest.py`'s `send()` fails. Consequence: the Monday digest reached the DM **only because the agent noticed the failure and re-ran with `--dry-run`** and its own response was delivered. **The 2026-10-05 "token-revocation claim refuted" entry was wrong** and has been corrected: two green runs were read as proof the token worked, when a green cron row only means the *agent* finished. Fix: new token from @BotFather → `config.json` → re-run `getMe`. Credential present, value not recorded here. **Finding 2 — Camofox hit its memory ceiling during the Job Radar run.** Mid-run probe of `camofox-browser.service`: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M)` with 383 tasks and the service up 9 days. Swap fully consumed, zero reclaimable headroom → `/tabs` and page evals hung. The agent restarted the service (back to ~2.1 GB used of 14 GB on CT460) and completed the extraction **one tab at a time with immediate close**. Same containment drop-in as documented (`resource-containment.conf`); the limits held, the *workload* is what needs bounding. → [[systems/browser-backend]] **Nothing added/removed:** no new system, no retired job, so `index.md` is unchanged this cycle. No page archived: the DHL monitor stays where the 2026-10-10 audit put it (redundant, in Removed Jobs). **Still open after this cycle:** GitHub Trending Radar token (new 2026-10-13); Camofox containment vs. multi-tab workloads (new 2026-10-13); Quartz rebuild step inert; skills-index off-by-one; **3** non-clean `last_status` rows (`0e753e9720fd`, `1b23c042e17d`, `066bb5191280`); Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.