audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry

Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline.

NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450).

Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
This commit is contained in:
hermes-audit
2026-10-13 00:19:15 +08:00
parent 0eee5f101e
commit ebf57146f1
10 changed files with 304 additions and 65 deletions
+5 -5
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-12
> model routing and automation. Last updated: 2026-10-13
## Start here
- [[current-state]] — Current systems status
@@ -20,7 +20,7 @@
- [[systems/openrouter]] — OpenRouter current state and history
- [[systems/opencode-go]] — OpenCode Go subscription, model limits, pricing
- [[systems/nanobot]] — Nanobot current state and history
- [[systems/browser-backend]] — Camoufox (primary) + Steel (disabled), CDP config
- [[systems/browser-backend]] — Camofox (primary, `systemd --user` on CT460, REST `:9377`) + Steel (disabled); memory-containment drop-in and the 2026-10-12 ceiling event documented
- [[systems/hermes-ui]] — Browser Hermes Desktop (przbadu/hermes-ui)
- [[systems/paseo]] — Paseo coding-agent daemon + web terminal workspace (CT460, :6767)
- [[systems/ignis]] — Ignis browser-native Obsidian (wiki interface)
@@ -29,11 +29,11 @@
### Integrations
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **5 carry a non-clean `last_status`** as of 2026-10-12 00:10. **Two consecutive clean days: 2026-10-11 ran `171 completed` / 0 `failed`** (2026-10-10 finalised `175 / 0`), ending the 2026-10-09 provider timeout storm (15 failed executions / 10 jobs; 7 of its incidents auto-closed on 2026-10-10, 2 of its jobs remain `alerted`). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-12 by the stocktake run itself (`6fc109e` follow-on: 346 skills / 70 categories). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **regenerated daily — the 10-10, 10-11 and 10-12 stocktake runs all committed**)
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **3 carry a non-clean `last_status`** as of 2026-10-13 00:15. **Three consecutive clean days: 2026-10-12 ran `174 completed` / 0 `failed`** (2026-10-11 finalised `172 / 0`), and **open incidents fell 5 → 2** on 2026-10-12 (three closed: both `deepseek-v4-flash` rows + the Job Radar timeout). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. ⚠ The 2026-10-12 clean sheet is *not* an all-clear — GitHub Trending Radar's Telegram token is revoked (`getMe` → 401) while the cron row stays green. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-13 against the stocktake's own `0eee5f1` commit: 346 skills / 70 categories). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **regenerated daily — the 10-10, 10-11 and 10-12 stocktake runs all committed**)
## Incidents
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; 7 of its incidents auto-closed on 2026-10-10, **2 remain `alerted`** (`dca8482e4f76`, `1b23c042e17d` — both weekly jobs still carrying the Friday `RuntimeError: Request timed out.`)
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)
- [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05)
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic