audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry

Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline.

NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450).

Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
This commit is contained in:
hermes-audit
2026-10-13 00:19:15 +08:00
parent 0eee5f101e
commit ebf57146f1
10 changed files with 304 additions and 65 deletions
+64
View File
@@ -534,3 +534,67 @@ nothing contradicting was found, so both stay in the stale list rather than bein
`raw/stocktake/freshness-20261012.txt`, `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-13] audit | nightly review of 2026-10-12 — clean cron day hid a dead credential; Camofox containment ceiling; two pages corrected against live probes
**Window.** 2026-10-12 00:00 → 23:59 AWST (sessions from `~/.hermes/state.db`, cron from
`~/.hermes/cron/executions.db` + `cron_incidents`; live probes 2026-10-13 00:05–00:45). One former
compaction of this job left part of the review written but the log entry and cross-page updates
unfinished — completed here.
**Headline 1 — a clean run *hid* a dead credential.** `c0a9fa314542` (GitHub Trending Radar — Weekly
Roundup, Mondays 09:00) reported **`completed` with `delivery_outcome: delivered`** on 2026-10-12, but its
fetch-then-send script died at `digest.py` `send()`. Live probe of the bot token in
`/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 from `getMe`** — the token is
a valid 46-character shape, i.e. **revoked**, not malformed. The fetch path is healthy (100 repos, 26 cache
files in 24 h). Critically, this **supersedes the 2026-10-06 entry on [[current-state]]**, which concluded
the revoked-token diagnosis was "refuted" because two runs had finished green. Lesson recorded in
[[systems/scheduled-tasks]]: **a completed run does not clear a credential fault unless the run exercised
the same call that failed**, and green cron status is not delivery proof. (Credential was present; value
neither recorded nor repeated.)
**Headline 2 — Camofox hit its cgroup ceiling inside a cron run (new).** The 2026-10-12 07:01 Daily
Marketing Job Radar run drove `camofox-browser.service` to **`2.5G`, swap fully consumed, `available: 0B`,
383 tasks** against the drop-in ceiling `MemoryHigh=2G` / `MemoryMax=3G` / `MemorySwapMax=1G`. REST `/tabs`
and page evals hung (indistinguishable from ordinary tool timeouts). The job restarted the service at
**07:23:50**, switched to one-tab-at-a-time extraction with immediate closes, and still delivered. Diagnosis:
**workload exceeded the cgroup, not a leak** — `OOMPolicy=kill` never fired.
**Headline 3 — [[systems/browser-backend]] corrected against live probes.** Prior revisions were wrong on
three counts: the REST API listens on **`:9377`** only (`:9093` closed, `:9378` does not exist — REST and
WebSocket are the same service, not two), Camofox runs as a **`systemd --user` unit on the Hermes host
(CT460)**, not CT450, and the containment drop-in was undocumented. `current-state.md` also still described
the old port model; both pages now match the probe.
**Cron state.** 2026-10-12: **174 completed / 0 failed / 1 running** (this audit, `cbfe434a3b5e`, started
23:59:41) — **third consecutive clean day**; 2026-10-11 finalised at `172 / 0`. Retention floor moved to
2026-10-07 08:15. Fleet **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**
(`0e753e9720fd` error, `1b23c042e17d` error, `066bb5191280` delivery_failed). `fad5a86979d4` (Facebook
Birthday Weekly Registry Sync) and `dca8482e4f76` (Daily Marketing Job Radar) both cleared on 2026-10-12 —
the former at 03:00:45 → 03:05:47 (448 birthdays, delivered), the latter 07:00:47 → 07:33:50.
**Incidents — open 5 → 2.** Closed on 2026-10-12: `fad5a8_a13bbc837e1c` + `fad5a8_c80b59a907f0` at
**03:05:39** (the two `deepseek-v4-flash` HTTP-400 rows, cleared by the registry sync's clean run) and
`dca848_23fcd2b18aaa` at **07:33:41**. Remaining: `1b23c0_c35703f637b8` (`alerted` — weekly job, cannot
clear before Fri 2026-10-16 08:00) and `0e753e_e7f8194a7d06` (`detected` since 2026-10-01 — the job has no
retained executions, so it cannot clear until 2026-11-01). → [[incidents/2026-10-09-cron-provider-timeout-storm]]
updated from "8 still open" to **11 of 12 closed** with the full closure timeline; the pattern recorded
there is that **each incident clears only on the next successful run of its own job**, so weekly jobs carry
their rows for up to a week after the fault is gone.
**Mnemosyne (2026-10-12 12:02 health run, green).** Working **760/786 = 96.7 %**, episodic
**1050/1056 = 99.4 %** — both above the 95 % floor, no `reindex`; orphan embeddings 0; FTS5 exactly aligned;
0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → [[systems/mnemosyne]].
**Sessions.** **18 in the window, all `cron`, zero interactive.** The 03:00 stocktake `1a8e90ea2271` ran
clean and pushed `0eee5f1` (freshness 54 stale / 32 fresh / 7 no-date of 93 pages; skills index 346 / 70
categories — the known 347-vs-346 on-disk off-by-one persists).
**Pages touched:** `current-state.md` (new Recent Changes section + superseded-correction tag),
`systems/scheduled-tasks.md` (frontmatter, job inventory, non-clean statuses, re-verified history, five
job rows, `### Resolved 2026-10-12`, corrected incident IDs, `### Open` additions, new
`## Re-verified 2026-10-13` block), `systems/browser-backend.md` (port + host + containment rewrite,
history entries), `systems/mnemosyne.md` (12:02 coverage), `incidents/2026-10-09-cron-provider-timeout-storm.md`
(status + full closure table + timeline), `index.md` (date, four blurbs), `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.