From ebf57146f17ecec45b9f18e4cbc4a52d0507e727 Mon Sep 17 00:00:00 2001 From: hermes-audit Date: Tue, 13 Oct 2026 00:19:15 +0800 Subject: [PATCH] audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline. NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450). Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07. --- comparisons/steel-vs-camoufox.md | 20 ++-- concepts/browser-automation.md | 6 +- current-state.md | 20 +++- .../2026-10-09-cron-provider-timeout-storm.md | 32 ++++-- index.md | 10 +- log.md | 64 +++++++++++ runbooks/job-radar.md | 23 +++- systems/browser-backend.md | 83 +++++++++++--- systems/mnemosyne.md | 6 +- systems/scheduled-tasks.md | 105 ++++++++++++++++-- 10 files changed, 304 insertions(+), 65 deletions(-) diff --git a/comparisons/steel-vs-camoufox.md b/comparisons/steel-vs-camoufox.md index 0df59c7..7a8c4b1 100644 --- a/comparisons/steel-vs-camoufox.md +++ b/comparisons/steel-vs-camoufox.md @@ -3,8 +3,8 @@ title: Steel Browser vs Camoufox type: comparison status: active created: 2026-07-29 -updated: 2026-08-02 -verified_on: 2026-08-02 +updated: 2026-10-13 +verified_on: 2026-10-13 confidence: high tags: [comparison, browser, steel, camofox, decision] sources: [live-probe, session-history] @@ -16,14 +16,14 @@ sources: [live-probe, session-history] Compare the two browser-automation backends available to Hermes, including the Jul 29–30 experiment and the decision to revert. ## At a glance -| | Steel Browser (CT201) | Camoufox (CT450) | +| | Steel Browser (CT201) | Camofox (systemd --user on CT460) | |---|---|---| | **In use** | Disabled since 2026-07-30 | **Yes** — primary backend | | **Engine** | Chromium 140 (Playwright) | Camoufox (Firefox-based) | -| **Connection** | CDP (`ws://` via port 9223) | REST API (localhost:9093) | +| **Connection** | CDP (`ws://` via port 9223) | REST API + CDP WebSocket, both `localhost:9377` | | **API** | REST at `:3000` — create/destroy/query sessions | Camoufox REST + CDP WebSocket | | **Headless** | Native headless | Xvfb + VNC (headed, anti-fingerprint) | -| **Container** | Docker on CT201 | Systemd service on CT450 | +| **Container** | Docker on CT201 | `systemd --user` unit on the Hermes host (CT460) | | **Stealth** | Standard Chromium | Anti-detection fingerprinting | | **Session viewer** | Built-in UI at `/steel` | VNC at `/vnc-camofox` | | **Plugin** | `browser-steel` plugin (disabled) | Native Hermes `browser.cloud_provider: camofox` | @@ -47,13 +47,15 @@ Compare the two browser-automation backends available to Hermes, including the J - Tool name validation on DeepSeek 0731 was stricter — this affected all providers, not just Steel ### Lesson learned -Steel is a capable backend but adds a network hop and another plugin layer. Camoufox runs locally on CT450 with fewer failure modes. The `browser-steel` plugin still exists (disabled) if needed in future. +Steel is a capable backend but adds a network hop and another plugin layer. Camofox runs locally on the +Hermes host (CT460) with fewer failure modes. The `browser-steel` plugin still exists (disabled) if needed in future. ## Detail -### Camoufox (current primary) -- **Deployed:** Persistent systemd service on CT450 -- **Connection:** REST API at `localhost:9093`, profile `anthony`, sessionKey `anthony` +### Camofox (current primary) +- **Deployed:** Persistent `systemd --user` service on the Hermes host (CT460), `camofox-browser.service` +- **Connection:** REST API **and** CDP WebSocket at `localhost:9377`, profile `anthony`, sessionKey `anthony` +- **Limits:** drop-in `MemoryHigh=2G` / `MemoryMax=3G` / `MemorySwapMax=1G` — a 2026-10-12 browser-heavy cron run hit `2.5G` with swap fully consumed and had to be restarted (see [[systems/browser-backend]]) - **Key advantage:** Runs on the same host as Hermes — no network hop, no container networking to debug - **Auth:** Already authenticated into services (Uber Eats, LinkedIn, etc.) — no cookie files needed - **Limitation:** Headed via Xvfb, consumes display resources diff --git a/concepts/browser-automation.md b/concepts/browser-automation.md index a33b695..2da8396 100644 --- a/concepts/browser-automation.md +++ b/concepts/browser-automation.md @@ -3,8 +3,8 @@ title: Browser Automation type: concept status: active created: 2026-07-22 -updated: 2026-08-02 -verified_on: 2026-08-02 +updated: 2026-10-13 +verified_on: 2026-10-13 confidence: high tags: [concept, browser, camofox, steel] sources: [live-probe, session-history] @@ -17,7 +17,7 @@ Giving an agent a real, controllable web browser — as opposed to just an HTTP- ## How this install implements it - **Engine:** Camoufox (Firefox-based, fingerprint-resistant) — see [[comparisons/camoufox-vs-chrome]] for why this over stock Chrome. -- **Connection:** REST API at `localhost:9093` (profile `anthony`) + CDP WebSocket at `localhost:9377` +- **Connection:** REST API **and** CDP WebSocket on the same service at `localhost:9377` (profile `anthony`). *Corrected 2026-10-13 — earlier revisions split this across `:9093` (REST) and `:9377` (WS); nothing listens on `:9093`.* - **Display:** headed, not truly headless — runs under `Xvfb :99` (1600×1200×24) with `fluxbox` as window manager, viewable live over VNC (see [[systems/headless-server]]). - **Config:** `cloud_provider: camofox`, `camera_provider: camofox` in `~/.hermes/config.yaml` - **Auth:** Persistent profile `anthony` — authenticated into Uber Eats, LinkedIn, etc. No cookie files needed. diff --git a/current-state.md b/current-state.md index 4d9c697..590b998 100644 --- a/current-state.md +++ b/current-state.md @@ -3,16 +3,26 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-12 -verified_on: 2026-10-12 +updated: 2026-10-13 +verified_on: 2026-10-13 review_after: 2026-10-25 confidence: high tags: [system] -sources: [live-config, live-cron-list, live-hermes-version] +sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] --- # Current State -Updated: 2026-10-12 +Updated: 2026-10-13 + +## Recent Changes (2026-10-12 → 2026-10-13, nightly audit; live probes 2026-10-13 00:05-00:45 AWST) + +- **A clean cron day hid a dead credential.** 2026-10-12 ran **174 completed / 0 failed** (third consecutive clean day; 10-11 finalised at `172 / 0`) — yet `c0a9fa314542` (GitHub Trending Radar) reported `completed` **with `delivery_outcome: delivered` while its own Telegram send was dying on an invalid token**. The delivered artifact was the *agent's* recovered digest, not the script's. → [[systems/scheduled-tasks]] +- **GitHub Trending Radar's bot token is REVOKED — live-verified, and the 2026-10-06 "refuted" entry was WRONG.** `getMe` against `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**; the token is a valid 46-char shape, so revoked rather than malformed, and **no valid replacement exists on the box**. The fetch path is healthy (100 repos, 26 cache files in 24 h), so only `digest.py`'s `send()` fails. **Correction:** the 2026-10-06 entry below treated two green runs as proof the token worked; a green cron row only means the *agent* finished, and neither run exercised the failing call. Fix = new @BotFather token → `config.json` → re-run `getMe`. Credential present, value not recorded. → [[systems/scheduled-tasks]] +- **Open cron incidents 5 → 2.** Closed on 2026-10-12: `fad5a8_a13bbc837e1c` + `fad5a8_c80b59a907f0` at **03:05:39** (both `deepseek-v4-flash`, cleared by `fad5a86979d4`'s first clean scheduled run — 03:00:45 → 03:05:47, 448 unique birthdays, delivered) and `dca848_23fcd2b18aaa` at **07:33:41** (cleared by `dca8482e4f76`'s own 07:00 run, delivered to topic 1030 after a 32-minute run). Survivors: `0e753e_e7f8194a7d06` (`detected` since 2026-10-01 — cannot clear until 2026-11-01, the job has **no retained executions at all**) and `1b23c0_c35703f637b8` (`alerted` since 2026-10-09, next slot Fri 2026-10-16 08:00). Table: alerted 1 / detected 1 / closed 8 / resolved 66. Fleet **29 jobs — 26 enabled / 3 disabled**; non-clean `last_status` **5 → 3** (`0e753e9720fd`, `1b23c042e17d`, `066bb5191280`). → [[systems/scheduled-tasks]] +- **Camofox hit its memory containment ceiling inside a cron run (NEW).** The 2026-10-12 07:01 Daily Marketing Job Radar drove `camofox-browser.service` to **`2.5G` with swap fully consumed, `available: 0B`, 383 tasks** (drop-in `MemoryHigh=2G`/`MemoryMax=3G`/`MemorySwapMax=1G`); `/tabs` and page evals hung. The agent restarted the service at **07:23:50** and finished **one tab at a time**, and the job still delivered. A green cron row does not mean the browser layer was healthy — check the unit's peak/swap/tasks when a browser-heavy job runs long. → [[systems/browser-backend]] +- **Browser page corrected against a live probe.** REST API is **`:9377`**, not `:9093` (nothing listened on 9093); `:9378` does not exist; Camofox is a `systemd --user` unit **on the Hermes host (CT460)**, not CT450. Containment drop-in documented. → [[systems/browser-backend]] +- **Mnemosyne healthy at 12:02** — working **760/786 = 96.7 %**, episodic **1050/1056 = 99.4 %** (both above the 95 % floor, no `reindex` needed); orphan embeddings 0; FTS5 aligned exactly (786/786, 1056/1056, 876/876); 0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → [[systems/mnemosyne]] +- **Window shape:** **18 sessions, all `cron`, zero interactive** — the other bookend to the previous window's `talaria-ios` research session. The 03:00 stocktake `1a8e90ea2271` ran clean and pushed **`0eee5f1`** (freshness **54 stale / 32 fresh / 7 no-date of 93 pages**; skills index 346 / 70 categories — the known 347-vs-346 off-by-one persists). The execution log's retention floor has moved to **2026-10-07 08:15**, so 10-07 rows are now pruned. ## Recent Changes (2026-10-12, 03:00 stocktake; live probes 2026-10-12 03:0x-03:1x AWST) @@ -91,7 +101,7 @@ _The manual 2026-10-10 stocktake entry follows beneath this one._ ## Recent Changes (2026-10-06) - **Dead model pins cleared across the whole fleet — failing set 7 → 0.** All seven jobs that had been failing since 2026-10-01 were repinned on 2026-10-05 23:16 → 2026-10-06 00:05. Each had been pinned to a raw model name that died upstream: `cliprox/terra` (HTTP 401 connection auth, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`). They were moved to `custom:omni-101` on **routing aliases** — `free-stack` ×6, `smart-route` ×1 — because the fault was raw-pin brittleness, not any one model. Every `failure_streak` in `jobs.json` now reads 0. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The other four are repinned but not yet re-run, so they carry a stale `last_status: error` until their next scheduled slot. See [[incidents/2026-10-05-dead-model-pin-repin]]. - **The `cliprox/terra` connection incident is closed for cron purposes.** No enabled job references that pin any more — the jobs were moved off it rather than the proxy connection being re-authenticated. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. -- **GitHub Trending Radar's revoked-token diagnosis is refuted.** `c0a9fa314542` ran on schedule at **2026-10-05 09:00 and completed cleanly** — a second consecutive success after the off-schedule 2026-09-30 19:22 run. No `getMe` → 401, and no BotFather token replacement is needed. The previous audit correctly refused to treat the claim as standing fact; it can now be retired. +- ~~**GitHub Trending Radar's revoked-token diagnosis is refuted.**~~ **SUPERSEDED 2026-10-13 — this conclusion was WRONG.** `c0a9fa314542` ran on schedule at **2026-10-05 09:00 and completed cleanly** — a second consecutive success after the off-schedule 2026-09-30 19:22 run. No `getMe` → 401, and no BotFather token replacement is needed. The previous audit correctly refused to treat the claim as standing fact; it can now be retired. - **A stale failure notice, not a live outage, opened the review.** Anthony acted on a `85edf6772bca` notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job succeeded on every run from 2026-09-30 17:53 onward, and its three incidents were resolved on 2026-09-30 17:55:13; the notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.** - **Window tally** (`executions.db`, 2026-10-05 00:00 → 2026-10-06 00:15 AWST): 185 `completed`, 3 `failed` (all pre-repin, 03:00–07:00), 2 `unknown` (aborted first attempts of `79f9bb41ecf5` / `dca8482e4f76`, superseded by their successful re-runs) and 1 `running` (the nightly audit itself). No interactive sessions other than the repin session. diff --git a/incidents/2026-10-09-cron-provider-timeout-storm.md b/incidents/2026-10-09-cron-provider-timeout-storm.md index b28daad..433c417 100644 --- a/incidents/2026-10-09-cron-provider-timeout-storm.md +++ b/incidents/2026-10-09-cron-provider-timeout-storm.md @@ -3,8 +3,8 @@ title: Fleet-Wide Provider Timeout Storm — 2026-10-09 type: incident status: unresolved created: 2026-10-09 -updated: 2026-10-09 -verified_on: 2026-10-09 +updated: 2026-10-13 +verified_on: 2026-10-13 review_after: 2026-10-16 confidence: high tags: [cron, omniroute, router, timeout, incident, unresolved] @@ -14,7 +14,9 @@ sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-2026100 # Fleet-Wide Provider Timeout Storm — 2026-10-09 **Window:** 2026-10-09 00:00 → 15:00 AWST (last failure 15:00; every run from 18:00 onward completed) -**Status:** Unresolved — the storm stopped and four incidents auto-closed, but **eight incidents opened on 2026-10-08/09 are still open** and `1a8e90ea2271` is the only job left at `failure_streak: 2` +**Status:** Unresolved — the storm ended and **11 of its 12 incidents are now closed**; only +`1b23c0_c35703f637b8` remains open, and it cannot clear before that job's next scheduled slot +(**Fri 2026-10-16 08:00**). Nothing here is still symptomatic. **Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08** ## What happened @@ -79,18 +81,26 @@ therefore serving normally again well before the window closed. | Incident | Job | State | Failure type | First seen | Closed | |----------|-----|-------|--------------|------------|--------| -| `1a8e90_789224ad589c` | `1a8e90ea2271` (wiki-stocktake-maintenance) | detected | unknown | 2026-10-08 03:01 | — | +| `1a8e90_789224ad589c` | `1a8e90ea2271` (wiki-stocktake-maintenance) | resolved | unknown | 2026-10-08 03:01 | 2026-10-10 03:06 | | `ea79a2_6fb70b032305` | `ea79a2107e36` (Mnemosyne Health Monitor) | resolved | timeout | 2026-10-09 00:18 | 2026-10-10 00:03 | | `b63ed5_f2c93dd24181` | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | resolved | timeout | 2026-10-09 00:18 | 2026-10-09 18:01 | -| `cbfe43_fa81052baaf7` | `cbfe434a3b5e` (Wiki Audit — Nightly) | detected | timeout | 2026-10-09 00:32 | — | -| `1a8e90_dc6b2d40ec44` | `1a8e90ea2271` (wiki-stocktake-maintenance) | detected | timeout | 2026-10-09 03:17 | — | +| `cbfe43_fa81052baaf7` | `cbfe434a3b5e` (Wiki Audit — Nightly) | resolved | timeout | 2026-10-09 00:32 | 2026-10-10 00:27 | +| `1a8e90_dc6b2d40ec44` | `1a8e90ea2271` (wiki-stocktake-maintenance) | resolved | timeout | 2026-10-09 03:17 | 2026-10-10 03:06 | | `96eb95_acb3ae5ec223` | `96eb95ce12f2` (Mnemosyne Watchdog) | resolved | timeout | 2026-10-09 06:30 | 2026-10-09 07:01 | -| `dca848_23fcd2b18aaa` | `dca8482e4f76` (Daily Marketing Job Radar) | alerted | timeout | 2026-10-09 07:18 | — | +| `dca848_23fcd2b18aaa` | `dca8482e4f76` (Daily Marketing Job Radar) | resolved | timeout | 2026-10-09 07:18 | **2026-10-12 07:33** | | `1b3118_f49c8d2c6e7f` | `1b31187397a5` (Mosquito Tracker) | resolved | timeout | 2026-10-09 07:18 | 2026-10-09 22:05 | -| `85edf6_18bedbeec5cd` | `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | alerted | timeout | 2026-10-09 07:18 | — | -| `1b23c0_c35703f637b8` | `1b23c042e17d` (Enhanced Weekend Adventure Planner) | alerted | timeout | 2026-10-09 08:17 | — | -| `08eb86_634f1a296f78` | `08eb86f02d37` (Gmail Action Digest) | alerted | timeout | 2026-10-09 08:48 | — | -| `ecc1fc_58b6c38296f3` | `ecc1fc13655a` (FreshRSS Midday News Briefing) | alerted | timeout | 2026-10-09 12:21 | — | +| `85edf6_18bedbeec5cd` | `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | resolved | timeout | 2026-10-09 07:18 | 2026-10-10 07:04 | +| `1b23c0_c35703f637b8` | `1b23c042e17d` (Enhanced Weekend Adventure Planner) | **alerted — still open** | timeout | 2026-10-09 08:17 | — (next slot Fri 2026-10-16 08:00) | +| `08eb86_634f1a296f78` | `08eb86f02d37` (Gmail Action Digest) | resolved | timeout | 2026-10-09 08:48 | 2026-10-10 08:32 | +| `ecc1fc_58b6c38296f3` | `ecc1fc13655a` (FreshRSS Midday News Briefing) | resolved | timeout | 2026-10-09 12:21 | 2026-10-10 12:05 | + +**Closure timeline (re-read 2026-10-13):** four closed inside the storm window itself (2026-10-09 +07:01 → 22:05); five more closed on 2026-10-10 (00:03, 00:27, 03:06 ×2, 07:04, 08:32, 12:05); the +in-window `dca848_23fcd2b18aaa` — the **last one to hold out** — only cleared on **2026-10-12 07:33:41**, +nine seconds before its own job's 07:00 run finished successfully. That is the pattern worth remembering: +each incident clears on the *next successful scheduled run of its own job*, so a weekly job carries its +incident for up to a week after the underlying fault is gone. `1b23c042e17d` is the remaining example — +its fault is long gone, but the row cannot close until Friday. ## Why it reached Anthony diff --git a/index.md b/index.md index 6418af1..d7fd8e2 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-12 +> model routing and automation. Last updated: 2026-10-13 ## Start here - [[current-state]] — Current systems status @@ -20,7 +20,7 @@ - [[systems/openrouter]] — OpenRouter current state and history - [[systems/opencode-go]] — OpenCode Go subscription, model limits, pricing - [[systems/nanobot]] — Nanobot current state and history -- [[systems/browser-backend]] — Camoufox (primary) + Steel (disabled), CDP config +- [[systems/browser-backend]] — Camofox (primary, `systemd --user` on CT460, REST `:9377`) + Steel (disabled); memory-containment drop-in and the 2026-10-12 ceiling event documented - [[systems/hermes-ui]] — Browser Hermes Desktop (przbadu/hermes-ui) - [[systems/paseo]] — Paseo coding-agent daemon + web terminal workspace (CT460, :6767) - [[systems/ignis]] — Ignis browser-native Obsidian (wiki interface) @@ -29,11 +29,11 @@ ### Integrations - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. -- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **5 carry a non-clean `last_status`** as of 2026-10-12 00:10. **Two consecutive clean days: 2026-10-11 ran `171 completed` / 0 `failed`** (2026-10-10 finalised `175 / 0`), ending the 2026-10-09 provider timeout storm (15 failed executions / 10 jobs; 7 of its incidents auto-closed on 2026-10-10, 2 of its jobs remain `alerted`). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. See [[incidents/2026-10-09-cron-provider-timeout-storm]] -- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-12 by the stocktake run itself (`6fc109e` follow-on: 346 skills / 70 categories). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **regenerated daily — the 10-10, 10-11 and 10-12 stocktake runs all committed**) +- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **3 carry a non-clean `last_status`** as of 2026-10-13 00:15. **Three consecutive clean days: 2026-10-12 ran `174 completed` / 0 `failed`** (2026-10-11 finalised `172 / 0`), and **open incidents fell 5 → 2** on 2026-10-12 (three closed: both `deepseek-v4-flash` rows + the Job Radar timeout). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. ⚠ The 2026-10-12 clean sheet is *not* an all-clear — GitHub Trending Radar's Telegram token is revoked (`getMe` → 401) while the cron row stays green. See [[incidents/2026-10-09-cron-provider-timeout-storm]] +- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-13 against the stocktake's own `0eee5f1` commit: 346 skills / 70 categories). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **regenerated daily — the 10-10, 10-11 and 10-12 stocktake runs all committed**) ## Incidents -- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; 7 of its incidents auto-closed on 2026-10-10, **2 remain `alerted`** (`dca8482e4f76`, `1b23c042e17d` — both weekly jobs still carrying the Friday `RuntimeError: Request timed out.`) +- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00) - [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved) - [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05) - [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic diff --git a/log.md b/log.md index 3121aa9..73afacb 100644 --- a/log.md +++ b/log.md @@ -534,3 +534,67 @@ nothing contradicting was found, so both stay in the stale list rather than bein `raw/stocktake/freshness-20261012.txt`, `log.md`. No secrets written; job/incident IDs, paths and tailnet addresses only. + +## [2026-10-13] audit | nightly review of 2026-10-12 — clean cron day hid a dead credential; Camofox containment ceiling; two pages corrected against live probes + +**Window.** 2026-10-12 00:00 → 23:59 AWST (sessions from `~/.hermes/state.db`, cron from +`~/.hermes/cron/executions.db` + `cron_incidents`; live probes 2026-10-13 00:05–00:45). One former +compaction of this job left part of the review written but the log entry and cross-page updates +unfinished — completed here. + +**Headline 1 — a clean run *hid* a dead credential.** `c0a9fa314542` (GitHub Trending Radar — Weekly +Roundup, Mondays 09:00) reported **`completed` with `delivery_outcome: delivered`** on 2026-10-12, but its +fetch-then-send script died at `digest.py` `send()`. Live probe of the bot token in +`/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 from `getMe`** — the token is +a valid 46-character shape, i.e. **revoked**, not malformed. The fetch path is healthy (100 repos, 26 cache +files in 24 h). Critically, this **supersedes the 2026-10-06 entry on [[current-state]]**, which concluded +the revoked-token diagnosis was "refuted" because two runs had finished green. Lesson recorded in +[[systems/scheduled-tasks]]: **a completed run does not clear a credential fault unless the run exercised +the same call that failed**, and green cron status is not delivery proof. (Credential was present; value +neither recorded nor repeated.) + +**Headline 2 — Camofox hit its cgroup ceiling inside a cron run (new).** The 2026-10-12 07:01 Daily +Marketing Job Radar run drove `camofox-browser.service` to **`2.5G`, swap fully consumed, `available: 0B`, +383 tasks** against the drop-in ceiling `MemoryHigh=2G` / `MemoryMax=3G` / `MemorySwapMax=1G`. REST `/tabs` +and page evals hung (indistinguishable from ordinary tool timeouts). The job restarted the service at +**07:23:50**, switched to one-tab-at-a-time extraction with immediate closes, and still delivered. Diagnosis: +**workload exceeded the cgroup, not a leak** — `OOMPolicy=kill` never fired. + +**Headline 3 — [[systems/browser-backend]] corrected against live probes.** Prior revisions were wrong on +three counts: the REST API listens on **`:9377`** only (`:9093` closed, `:9378` does not exist — REST and +WebSocket are the same service, not two), Camofox runs as a **`systemd --user` unit on the Hermes host +(CT460)**, not CT450, and the containment drop-in was undocumented. `current-state.md` also still described +the old port model; both pages now match the probe. + +**Cron state.** 2026-10-12: **174 completed / 0 failed / 1 running** (this audit, `cbfe434a3b5e`, started +23:59:41) — **third consecutive clean day**; 2026-10-11 finalised at `172 / 0`. Retention floor moved to +2026-10-07 08:15. Fleet **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3** +(`0e753e9720fd` error, `1b23c042e17d` error, `066bb5191280` delivery_failed). `fad5a86979d4` (Facebook +Birthday Weekly Registry Sync) and `dca8482e4f76` (Daily Marketing Job Radar) both cleared on 2026-10-12 — +the former at 03:00:45 → 03:05:47 (448 birthdays, delivered), the latter 07:00:47 → 07:33:50. + +**Incidents — open 5 → 2.** Closed on 2026-10-12: `fad5a8_a13bbc837e1c` + `fad5a8_c80b59a907f0` at +**03:05:39** (the two `deepseek-v4-flash` HTTP-400 rows, cleared by the registry sync's clean run) and +`dca848_23fcd2b18aaa` at **07:33:41**. Remaining: `1b23c0_c35703f637b8` (`alerted` — weekly job, cannot +clear before Fri 2026-10-16 08:00) and `0e753e_e7f8194a7d06` (`detected` since 2026-10-01 — the job has no +retained executions, so it cannot clear until 2026-11-01). → [[incidents/2026-10-09-cron-provider-timeout-storm]] +updated from "8 still open" to **11 of 12 closed** with the full closure timeline; the pattern recorded +there is that **each incident clears only on the next successful run of its own job**, so weekly jobs carry +their rows for up to a week after the fault is gone. + +**Mnemosyne (2026-10-12 12:02 health run, green).** Working **760/786 = 96.7 %**, episodic +**1050/1056 = 99.4 %** — both above the 95 % floor, no `reindex`; orphan embeddings 0; FTS5 exactly aligned; +0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → [[systems/mnemosyne]]. + +**Sessions.** **18 in the window, all `cron`, zero interactive.** The 03:00 stocktake `1a8e90ea2271` ran +clean and pushed `0eee5f1` (freshness 54 stale / 32 fresh / 7 no-date of 93 pages; skills index 346 / 70 +categories — the known 347-vs-346 on-disk off-by-one persists). + +**Pages touched:** `current-state.md` (new Recent Changes section + superseded-correction tag), +`systems/scheduled-tasks.md` (frontmatter, job inventory, non-clean statuses, re-verified history, five +job rows, `### Resolved 2026-10-12`, corrected incident IDs, `### Open` additions, new +`## Re-verified 2026-10-13` block), `systems/browser-backend.md` (port + host + containment rewrite, +history entries), `systems/mnemosyne.md` (12:02 coverage), `incidents/2026-10-09-cron-provider-timeout-storm.md` +(status + full closure table + timeline), `index.md` (date, four blurbs), `log.md`. + +No secrets written; job/incident IDs, paths and tailnet addresses only. diff --git a/runbooks/job-radar.md b/runbooks/job-radar.md index e1a1271..4b2bee1 100644 --- a/runbooks/job-radar.md +++ b/runbooks/job-radar.md @@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery type: runbook status: active created: 2026-07-27 -updated: 2026-07-28 +updated: 2026-10-13 review_after: 2026-10-25 -verified_on: 2026-07-28 +verified_on: 2026-10-13 version_applies_to: Hermes Agent cron dca8482e4f76 confidence: high tags: [automation, cron, jobs, notion, dashboard, runbook] @@ -262,8 +262,27 @@ hermes cron run dca8482e4f76 # reproduce and capture the 403 Do **not** allow a silent fallback — the radar must surface a model failure rather than emit an empty "no jobs found" digest. +## Browser-layer hazard inside the run (2026-10-12) + +This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove +`camofox-browser.service` to its memory ceiling — `2.5G`, swap fully consumed (`available: 0B`), 383 tasks — +after which REST `/tabs` and page evals hung in a way indistinguishable from ordinary tool timeouts. The +run recovered by restarting the service at **07:23:50** and finishing **one tab at a time with an immediate +close**, and it delivered normally. + +Operational implications for this runbook: + +- A **green cron row is not proof the browser layer was healthy** — check + `systemctl --user status camofox-browser.service` (peak / swap / task count) after any long run. +- Extraction must be bounded: one tab in flight, close before opening the next. Multi-tab pre-fetch is what + pushed it over. +- Limits live in `~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf` + (`MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G`, `OOMPolicy=kill`). Raise them only if the host has + real headroom — full detail in [[systems/browser-backend]]. + ## Related - [[systems/scheduled-tasks]] +- [[systems/browser-backend]] - [[infrastructure/tailscale]] - [[systems/model-providers]] diff --git a/systems/browser-backend.md b/systems/browser-backend.md index 3fc559c..205fcd5 100644 --- a/systems/browser-backend.md +++ b/systems/browser-backend.md @@ -3,8 +3,8 @@ title: Browser Backend type: system status: active created: 2026-07-22 -updated: 2026-08-02 -verified_on: 2026-08-02 +updated: 2026-10-13 +verified_on: 2026-10-13 confidence: high tags: [browser, camofox, automation] sources: [live-probe, config, session-history] @@ -15,36 +15,77 @@ sources: [live-probe, config, session-history] ## Purpose Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping. -## Current Configuration (as of 2026-08-02) -- **Primary backend:** Camoufox (Firefox-based, anti-fingerprint) on CT450 +## Current Configuration (re-verified 2026-10-13) +- **Primary backend:** Camofox (Firefox-based, anti-fingerprint) — `systemd --user` unit `camofox-browser.service` **on the Hermes host itself** (the unit path is `/home/hermes/.config/systemd/user/…`, and the VNC route is named after CT460). The older `CT450` attribution below was not reproducible on 2026-10-13 — see the correction note. - **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox` -- **REST API:** `localhost:9093` (profile `anthony`, sessionKey `anthony`) -- **CDP WebSocket:** `localhost:9377` (via Camoufox's Playwright bridge) -- **VNC viewer:** `https://ct460.kangaroo-eel.ts.net/vnc-camofox` (port 6080) +- **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13. +- **VNC viewer:** NovNC `websockify` on `127.0.0.1:6080` → `127.0.0.1:5900`, published as `https://ct460.kangaroo-eel.ts.net/vnc-camofox` - **Steel Browser (CT201):** Still running but Hermes not connected — plugin disabled -## Camoufox (CT450) -- **Host:** CT450 (same host as Hermes — no network hop) +## Camofox (local `systemd --user` service) + +> ⚠ Earlier revisions titled this section "Camoufox (CT450)". Live-verified 2026-10-13: the service Hermes +> talks to is a `systemd --user` unit on the Hermes host (`/home/hermes/.config/systemd/user/camofox-browser.service`), +> reached at `localhost:9377`, with its VNC route published as `…/vnc-camofox`. Treat any "runs on CT450 / +> CT201" claim as unverified until re-probed. + +- **Host:** the Hermes host itself (CT460) — `systemd --user` unit resolved on `hostname = hermes`, no network hop - **Deployment:** Systemd service `camofox-browser.service` - **Engine:** Camoufox (custom Firefox build with anti-detection fingerprinting) - **Display:** Headed via Xvfb `:99` (1600×1200×24) + fluxbox window manager - **Auth:** Persistent profile `anthony` — authenticated into Uber Eats, LinkedIn, etc. -- **Connection:** REST API at `localhost:9093` for session management, CDP WebSocket at `localhost:9377` for browser tools +- **Connection:** REST API **and** WebSocket bridge both at `localhost:9377` (see the port correction above) - **Last verified:** Jul 30 — `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s on scroll/navigate ### Access | Endpoint | URL | Used by | |---|---|---| -| REST API | `http://localhost:9093` | Hermes browser tools (navigate, click, snapshot) | -| CDP WebSocket | `ws://localhost:9377` | Direct CDP control | -| VNC | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer | -| Camoufox UI | `http://localhost:9378` | Internal management | +| REST API | `http://localhost:9377` | Hermes browser tools + direct REST (`/tabs`, `/start`, `/sessions`) | +| VNC (NovNC) | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer (`websockify` `127.0.0.1:6080` → `127.0.0.1:5900`) | + +> ⚠ **Port correction (2026-10-13):** earlier revisions listed the REST API on `:9093` and a separate CDP +> WebSocket on `:9377`. Live probe: the only Camofox listener is `node server.js` on `*:9377` (with +> `CAMOFOX_URL=http://localhost:9377` in the unit). `:9093` and `:9378` were both closed. `:9377` is the +> REST API **and** the WebSocket bridge — there is no separate 9093 service. ### Config - `cloud_provider: camofox` in `~/.hermes/config.yaml` -- `CAMOFOX_URL=http://localhost:9093` in `~/.hermes/.env` -- `CAMOFOX_API_KEY` in `~/.hermes/.env` -- Profile: `anthony` (502913bf) — NOT `persist-anthony` +- `CAMOFOX_URL=http://localhost:9377` and `CAMOFOX_API_KEY` in `~/.hermes/.env` +- Unit environment (`~/.config/systemd/user/camofox-browser.service`): `CAMOFOX_PORT=9377`, + `CAMOFOX_PROFILE_DIR=/home/hermes/.camofox/profiles`, `ENABLE_VNC=1`, `HANDLER_TIMEOUT_MS=90000`, + `NAVIGATE_TIMEOUT_MS=60000`, `SESSION_TIMEOUT_MS=600000`, `TAB_INACTIVITY_MS=300000`, + `BROWSER_IDLE_TIMEOUT_MS=300000`, `Restart=always`, `RestartSec=5`. *(The unit also carries an API key + value — deliberately not reproduced here.)* +- Profile: `anthony` (502913bf) — NOT `persist-anthony`. A clean public profile `jobradar` is used for job-board scraping. + +## Resource containment (live-verified 2026-10-13) + +The unit carries a drop-in **`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`**: + +``` +[Service] +MemoryHigh=2G +MemoryMax=3G +MemorySwapMax=1G +OOMPolicy=kill +``` + +**Observed ceiling event — 2026-10-12 07:01–07:23 (Daily Marketing Job Radar).** A long browser run drove +the service to the top of that containment window: + +- Mid-run: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M, + swap peak: 1G)`, **383 tasks**, uptime 9 days. Swap fully consumed, no reclaimable headroom. +- Symptom: the REST `/tabs` endpoint and page evals began hanging (they look like ordinary tool timeouts). +- Recovery: restart `camofox-browser.service`, then work **one tab at a time, closing each before opening + the next**. Post-restart the host sat at ~2.1 GB of 14 GB. + +**Diagnosis:** the limits held and `OOMPolicy=kill` never had to fire — the *workload* exceeded the cgroup, +not a leak. A green cron row therefore does **not** mean the browser layer was healthy; check +`systemctl --user status camofox-browser.service` (peak/swap/tasks) whenever a browser-heavy job runs long. + +Restart (`systemctl --user restart camofox-browser.service`) is measurable roughly 16 h later as a healthy +baseline: `Memory: 16.5M` cgroup, `peak: 1.4G`, `swap peak: 234.5M`, 16 tasks, `browserConnected: false` +until the next session starts. ## Steel Browser (CT201) — Disabled - **Status:** Still running on CT201, Hermes not connected since 2026-07-30 @@ -59,6 +100,14 @@ Browser automation for Hermes — web search, form filling, screenshot, CDP cont - **2026-07-29:** Steel Browser deployed on CT201, became primary - **2026-07-30:** Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary. - **2026-08-02:** Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API) +- **2026-10-12 — containment ceiling hit under a browser-heavy cron run.** Daily Marketing Job Radar drove + `camofox-browser.service` to `2.5G` with swap fully consumed (`available: 0B`, 383 tasks, 9-day uptime); + `/tabs` and evals hung until the service was restarted at 07:23:50 and the job finished one tab at a time. + Workload problem, not a leak — limits (`MemoryHigh=2G`/`MemoryMax=3G`/`MemorySwapMax=1G`) held. +- **2026-10-13 — page corrected against a live probe.** REST port `:9093` → **`:9377`**; `:9378` (claimed + "Camoufox UI") does not exist; `CT450` host attribution replaced with the local `systemd --user` unit on + the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited + end-to-end and its memory-pressure evidence added above. ## Related - [[comparisons/steel-vs-camoufox]] diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index 809315a..4bde911 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,8 +3,8 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-10-12 -verified_on: 2026-10-12 +updated: 2026-10-13 +verified_on: 2026-10-13 confidence: high tags: [memory, mnemosyne, backend] sources: [live-probe, mnemosyne-db, config, ops-profile-config] @@ -51,6 +51,8 @@ Primary agent memory store for Hermes. **Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]]. +**Tier counts (live, 2026-10-12 12:02 — health job `ea79a2107e36`, green):** working **760/786 embedded (96.7%)**, episodic **1050/1056 (99.4%)** — both above the 95% floor, no `reindex` needed (`b63ed5817212` continues to backfill). Orphan embeddings **0**. FTS5 exactly aligned: `fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876. **0 unconsolidated** working rows; **1 unresolved conflict** (threshold 10); last consolidation 0.3 h earlier; 2,252 memories total (786 working / 1,056 episodic / 1 triple); disk 15 GB free (81% used). Interpretation: coverage is *not* the live issue — ranking relevance is (see [[incidents/2026-10-09-memory-recall-relevance]]). + **Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check. **Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index acc47bc..c598f20 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,8 +3,8 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-12 -verified_on: 2026-10-12 +updated: 2026-10-13 +verified_on: 2026-10-13 review_after: 2026-10-25 confidence: high tags: [cron, scheduled, automation] @@ -14,8 +14,9 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, 2026-10-12; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10) -- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. See [[incidents/2026-10-05-dead-model-pin-repin]]. +- **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, re-verified 2026-10-13 — unchanged; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10) +- **Non-clean `last_status`:** **3** (was 5 at the 2026-10-12 audit) — `0e753e9720fd` (`error`, streak 0), `1b23c042e17d` (`error`, streak 1), `066bb5191280` (`delivery_failed`, streak 0). `fad5a86979d4` and `dca8482e4f76` cleared on **2026-10-12** when both ran clean on schedule. +- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]]. - Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04). - **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window. - **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs. @@ -25,7 +26,9 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 `completed`, 0 `failed`, 1 `running` (the audit itself); 14 sessions, all `cron` — **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`). - **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice. - **Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00):** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — every `failure_streak` reads 0. Window tally (`executions.db`): **176 `completed`, 1 `failed`, 1 `running`** (the audit itself). The single failure is `62619aa6bcb3` (DHL Tracking, 08:00) — a Camofox HTTP read timeout (`ensure_browser()` → `TimeoutError: timed out`), notified 08:00:58 and **auto-resolved 11:01:13** when the next 3-hourly run succeeded (incident `62619a_f485ccfd4a0d`). 13 sessions started in the window, all `source = cron` — **no interactive sessions**; 12 deliveries, **0 delivery failures**. The same three stale pre-repin `last_status: error` jobs remain (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`); the stocktake job `1a8e90ea2271` ran on schedule at 03:00 for the second consecutive day. -- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08) → **29, 0 failing** (2026-10-10) → **29, 0 failing** (2026-10-11) +- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08) → **29, 0 failing** (2026-10-10) → **29, 0 failing** (2026-10-11) → **29, 0 failing** (2026-10-12) +- **Re-verified 2026-10-12 AWST (window 2026-10-11 00:00 → 2026-10-12 00:00):** 29 jobs (26/3), **0 failing**, **5** non-clean `last_status`; window tally **172 `completed`, 0 `failed`** (the 2026-10-11 day finalised one run higher than the previous audit's snapshot, which caught it mid-flight at 171 + 1 `running`). 18 sessions — 17 `cron` + 1 interactive. +- **Re-verified 2026-10-13 AWST (window 2026-10-12 00:00 → 2026-10-13 00:00):** 29 jobs (26/3), **0 failing**, non-clean `last_status` down to **3**. Window tally **174 `completed`, 0 `failed`** — **third consecutive clean day** (10-10 `175/0`, 10-11 `172/0`, 10-12 `174/0`). **5 open incidents → 2:** three closed on 2026-10-12 (see [[#Resolved 2026-10-12]]). 18 sessions in the window, **all `cron`, no interactive sessions**; 0 new incident pages required. Two findings led the window: the **GitHub Trending Radar delivery token is dead** (`getMe` → 401, live-verified) and the **Daily Marketing Job Radar run hit Camofox memory pressure** (2.5 GB peak, heavy swap, self-recovered by restart). ## Job Table @@ -34,14 +37,14 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) | | AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 | | Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | | -| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). A manual run completed at 23:56 and delivered to topic 1030 — first successful run since 2026-09-25. Live `last_status: ok`, streak 0 | +| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. Live `last_status: ok`, streak 0. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open) | | DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant | -| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). Not yet re-run — carries a stale `last_status: error` until Fri 08:00; streak 0 | +| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1. **Next scheduled slot: Fri 2026-10-16 08:00**, which should clear it | | Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent | -| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). Streak 0; stale `last_status: error` until the 1st. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic | -| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). Streak 0; stale `last_status: error` until Mon 03:00. ⚠ Bare `telegram` target should move to a group topic | +| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic | +| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). Live `last_status: ok`, streak 0. ⚠ Bare `telegram` target should move to a group topic | | FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | | -| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ✅ **Token-revocation claim refuted.** The scheduled run on 2026-10-05 09:00 completed cleanly — a second consecutive success after the off-schedule 2026-09-30 19:22 run — so the `getMe` → 401 diagnosis no longer applies and no token replacement is needed. Still delivers to the bare home DM, which works | +| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. `last_status: ok` because the 401 is inside the script, not the cron run. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** | | Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | | | Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) | | Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent | @@ -78,6 +81,11 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] ## Known Issues +### Resolved 2026-10-12 +- **Both `deepseek-v4-flash` incidents closed by a clean scheduled run (`fad5a86979d4`, 03:00):** the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents `fad5a8_a13bbc837e1c` (open since 2026-09-28, every fallback provider exhausted) and `fad5a8_c80b59a907f0` (since 2026-09-30, `deepseek-v4-flash` HTTP 400) **both auto-closed at 03:05:39**. Its stale pre-repin `last_status: error` cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one. +- **`dca8482e4f76` timeout incident closed (`dca848_23fcd2b18aaa`):** the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm **auto-closed at 07:33:41**. Of the ten storm incidents, nine have now closed and **one remains**: `1b23c0_c35703f637b8` (`1b23c042e17d`, still `alerted`, next slot Fri 2026-10-16 08:00). See [[incidents/2026-10-09-cron-provider-timeout-storm]]. +- **Net effect: open cron incidents 5 → 2.** The two survivors are `0e753e_e7f8194a7d06` (`detected` since 2026-10-01, untouched because its job does not run until 2026-11-01) and `1b23c0_c35703f637b8` (`alerted`, cleared on the next Friday slot). + ### Resolved 2026-10-07 - **DHL Tracking `62619aa6bcb3` — single Camofox timeout, self-recovered (2026-10-07 08:00 → 11:01):** the 08:00 run died in `ensure_browser()` with `TimeoutError: timed out` on a browser HTTP call (`dhl_tracking_monitor.py:233`), i.e. Camofox was momentarily unresponsive. The failure notice was sent at 08:00:58 and the incident (`62619a_f485ccfd4a0d`, `failure_type: timeout`) **auto-resolved at 11:01:13** when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. **This is the first detect → notify → auto-resolve incident cycle since the 2026-09-30 managed-runtime event** — the incident machinery itself worked as designed. No action needed; a repeat would justify a Camofox health probe inside the script. - **The tracked parcel arrived.** The 14:00 run on 2026-10-07 reported waybill **8052838382 `Delivered`** (Kensington → Perth). The monitor has therefore achieved its purpose and runs 5×/day for no remaining benefit — a candidate for removal, flagged rather than removed (the job is Anthony's to retire). @@ -86,7 +94,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - **Dead model pins across 7 jobs — repinned to routing aliases (resolved 2026-10-05 23:16 → 2026-10-06 00:05):** Seven jobs were each pinned to a raw model name that had died upstream — `cliprox/terra` (HTTP 401, connection auth expired, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`) — across 2026-10-01 → 2026-10-05. All seven were repinned to provider `custom:omni-101` using **routing aliases** — `free-stack` ×6, `smart-route` ×1 — deliberately, because the fault was raw-pin brittleness and re-pinning to another raw name would have re-created it. `failure_streak` is now 0 on all 30 jobs, down from 7 failing. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The remaining four are repinned but not yet re-run; the stale `last_status: error` field clears on each one's next scheduled run. See [[incidents/2026-10-05-dead-model-pin-repin]]. - The `cliprox/terra` incident is **closed for cron purposes**: no enabled job references that pin any more. The underlying proxy connection was never re-authenticated — the jobs were moved off it instead. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. - `wiki-stocktake-maintenance` resumes its 03:00 schedule, so `wiki-maintain.sh` no longer needs a daily manual run. -- **GitHub Trending Radar `c0a9fa314542` — token-revocation claim refuted (2026-10-05):** the scheduled Mon 09:00 run **completed cleanly** — its second consecutive success after the off-schedule 2026-09-30 19:22 run. The `getMe` → 401 revoked-token diagnosis no longer applies and no token replacement is needed. It still delivers to the bare home DM, which works. +- **GitHub Trending Radar `c0a9fa314542` — token-revocation claim refuted (2026-10-05): SUPERSEDED 2026-10-13.** The Mon 09:00 run *completed* on 2026-10-05 and 2026-10-12, which was read as proof the `getMe` → 401 diagnosis was wrong. It was not: a cron run can be green while the script's own delivery hop is dead, and the 2026-10-13 live `getMe` against the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **401**. Corrected above and in Open items. **Lesson: a completed run does not clear a credential fault unless the run exercised the same call that failed.** - **A stale failure notice, not a live outage, opened the 2026-10-05 review:** the interactive session began because Anthony acted on a `85edf6772bca` failure notice sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.** ### Resolved 2026-09-30 @@ -97,6 +105,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`. ### Open +- **GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13):** live `getMe` on the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**, so `digest.py`'s Telegram send fails every Monday while the cron run itself stays green (`last_status: ok`). A *completed* run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with `--dry-run` and its own response was delivered. Fix: issue a fresh token from @BotFather, replace it in `config.json`, and re-run `getMe` to confirm; do not re-read a green cron status as proof of delivery. **This corrects the 2026-10-05 "claim refuted" entry, which was wrong.** See [[systems/scheduled-tasks]] job `c0a9fa314542`. +- **Camofox memory pressure during long browser runs (NEW 2026-10-13):** the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`: `MemoryHigh=2G`, `MemoryMax=3G`, swap max 1G). Live probe mid-run read `Memory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G)` with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the `/tabs` endpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See [[systems/browser-backend]]. - **Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07):** `wiki-maintain.sh` documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no `quartz.config.ts` is present and `npx` is unavailable, so `public/` has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description. - **`skills-index` under-counts by one skill (NEW 2026-10-07):** live `find ~/.hermes/skills -name SKILL.md -type f` returns **347**, but the generated page says **346** (70 categories). The generator de-dups on skill basename within a category, so the nested duplicate `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` collapses into its parent and is silently dropped. The other duplicated basename (`hermes-api-vault`, at the root and under `devops/`) survives because the two land in different categories. Remove the nested duplicate directory, or key the generator on path rather than basename. - **A delivered notice is not a successful job:** `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output. @@ -258,3 +268,76 @@ session that produced it). No pipeline fault. The same job's 12:05 and 2026-10-1 skills-index off-by-one (347 `SKILL.md` on disk vs 346 reported); the 5 non-clean `last_status` rows; Mattermost platform unreachable ([[systems/messaging-integrations]]); gateway runtime env outside the 15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]). + +## Re-verified 2026-10-13 (audit window 2026-10-12 00:00 → 2026-10-12 23:59 AWST; live probes 2026-10-13 00:05–00:45) + +**Headline: a clean run *hid* a dead credential.** The cron table was green — 174 completed / 0 failed — +and two of the three non-clean `last_status` rows cleared overnight, yet one job that reported success on +paper had its delivery hop dead the whole time. Detail below. + +**Cron tally:** 2026-10-11 **172 completed / 0 failed** (the one row still `running` at the last audit +finished clean); 2026-10-12 **174 completed / 0 failed / 1 running** — and the running row is *this audit +itself* (`cbfe434a3b5e`, started 23:59:41). **Third consecutive clean day.** The execution-log retention +floor has moved up to **2026-10-07 08:15**, so 10-07 rows are now pruned (their earlier tally is no longer +reproducible from this DB). + +**Incidents: open 5 → 2.** Three closed on 2026-10-12 — `fad5a8_a13bbc837e1c` and `fad5a8_c80b59a907f0` at +**03:05:39** (both `deepseek-v4-flash`, closed by the first clean scheduled run of `fad5a86979d4`); +`dca848_23fcd2b18aaa` at **07:33:41** (closed by `dca8482e4f76`'s own successful 07:00 run). Table totals +now **alerted 1 / detected 1 / closed 8 / resolved 66** (76 rows). Survivors: `0e753e_e7f8194a7d06` +(`detected` since 2026-10-01 — cannot clear until its job next runs on **2026-11-01**) and +`1b23c0_c35703f637b8` (`alerted` since 2026-10-09, next slot **Fri 2026-10-16 08:00**). + +**Job-by-job (verified against the executions log, `delivery_outcome` included):** +- `fad5a86979d4` — Facebook Birthday Weekly Registry Sync **03:00:45 → 03:05:47, completed, delivered**. + Output: Birthday Dashboard, **448 unique birthdays**, 1 contact match. This is the repin's live proof. +- `dca8482e4f76` — Daily Marketing Job Radar **07:00:47 → 07:33:50, completed, delivered** to topic 1030 + (32-minute run; the 2026-10-09 timeout incident closed 9 seconds before it finished). Run was messy — + see the Camofox finding — but it produced and delivered. +- `c0a9fa314542` — GitHub Trending Radar **09:00:41 → 09:08:33, completed, `delivered`** — and yet **its + own send failed** (finding 1). A `completed` row plus `delivery_outcome: delivered` is *not* proof the + job's payload went out; here the delivered artifact was the agent's recovered digest, not the script's. +- `1b23c042e17d` — Enhanced Weekend Adventure Planner **2026-10-09 08:00:38 → 08:17:46, failed** + (`RuntimeError: Request timed out.`, notice delivered). No run since; next slot Fri 2026-10-16 08:00. +- `0e753e9720fd` — Facebook Birthday Monthly Scrape: **no runs at all inside the retained window** + (monthly, next 2026-11-01), so its `error`/streak-0 row is still untested rather than fixed. + +**Sessions in the window:** **18, all `cron`, no interactive session** — a quiet day, in contrast to the +previous window's `talaria-ios` research session. Busiest by message count: `dca8482e4f76` (140), +`1a8e90ea2271` stocktake (97), Wiki Audit (28), `c0a9fa314542` (25). The 03:00 stocktake ran clean and +pushed **`0eee5f1`** to origin/master: skills-index regenerated (346 skills / 70 categories, 347 files on +disk — the known off-by-one), freshness audit **54 stale / 32 fresh / 7 no-date of 93 pages**, Quartz step +skipped on the documented path. + +**Mnemosyne (12:02 health run, green):** working **760/786 embedded = 96.7%**, episodic **1050/1056 = +99.4%** — both above the 95% floor, so no `reindex` needed. Orphan embeddings 0. FTS5 aligned exactly +(`fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876). 0 unconsolidated working rows; +1 unresolved conflict (threshold 10); last consolidation 0.3 h earlier. 2252 memories total (786 working / +1056 episodic / 1 triple), 395 consolidated facts. Disk 15 GB free (81% used). No exceptions → nothing to +act on, and nothing to change on [[systems/mnemosyne]] beyond the coverage figures. + +**Finding 1 — GitHub Trending Radar's Telegram token is revoked (live-verified).** `getMe` against the +`telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 +Unauthorized**. The token is a plausible length (46 chars), so it is revoked/invalid rather than malformed, +and there is **no valid replacement anywhere on the box**. The fetch path is healthy — GitHub trending +pulled 100 repos and wrote 26 cache files in 24 h — so only `digest.py`'s `send()` fails. Consequence: the +Monday digest reached the DM **only because the agent noticed the failure and re-ran with `--dry-run`** and +its own response was delivered. **The 2026-10-05 "token-revocation claim refuted" entry was wrong** and has +been corrected: two green runs were read as proof the token worked, when a green cron row only means the +*agent* finished. Fix: new token from @BotFather → `config.json` → re-run `getMe`. Credential present, value +not recorded here. + +**Finding 2 — Camofox hit its memory ceiling during the Job Radar run.** Mid-run probe of +`camofox-browser.service`: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, +swap: 1023.9M)` with 383 tasks and the service up 9 days. Swap fully consumed, zero reclaimable headroom → +`/tabs` and page evals hung. The agent restarted the service (back to ~2.1 GB used of 14 GB on CT460) and +completed the extraction **one tab at a time with immediate close**. Same containment drop-in as documented +(`resource-containment.conf`); the limits held, the *workload* is what needs bounding. → [[systems/browser-backend]] + +**Nothing added/removed:** no new system, no retired job, so `index.md` is unchanged this cycle. No page +archived: the DHL monitor stays where the 2026-10-10 audit put it (redundant, in Removed Jobs). + +**Still open after this cycle:** GitHub Trending Radar token (new 2026-10-13); Camofox containment vs. +multi-tab workloads (new 2026-10-13); Quartz rebuild step inert; skills-index off-by-one; **3** non-clean +`last_status` rows (`0e753e9720fd`, `1b23c042e17d`, `066bb5191280`); Mattermost platform unreachable; +gateway runtime env outside the 15-minute dependency guard.