audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry
Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline. NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450). Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
This commit is contained in:
+15
-5
@@ -3,16 +3,26 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-12
|
||||
verified_on: 2026-10-12
|
||||
updated: 2026-10-13
|
||||
verified_on: 2026-10-13
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [system]
|
||||
sources: [live-config, live-cron-list, live-hermes-version]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-12
|
||||
Updated: 2026-10-13
|
||||
|
||||
## Recent Changes (2026-10-12 → 2026-10-13, nightly audit; live probes 2026-10-13 00:05-00:45 AWST)
|
||||
|
||||
- **A clean cron day hid a dead credential.** 2026-10-12 ran **174 completed / 0 failed** (third consecutive clean day; 10-11 finalised at `172 / 0`) — yet `c0a9fa314542` (GitHub Trending Radar) reported `completed` **with `delivery_outcome: delivered` while its own Telegram send was dying on an invalid token**. The delivered artifact was the *agent's* recovered digest, not the script's. → [[systems/scheduled-tasks]]
|
||||
- **GitHub Trending Radar's bot token is REVOKED — live-verified, and the 2026-10-06 "refuted" entry was WRONG.** `getMe` against `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**; the token is a valid 46-char shape, so revoked rather than malformed, and **no valid replacement exists on the box**. The fetch path is healthy (100 repos, 26 cache files in 24 h), so only `digest.py`'s `send()` fails. **Correction:** the 2026-10-06 entry below treated two green runs as proof the token worked; a green cron row only means the *agent* finished, and neither run exercised the failing call. Fix = new @BotFather token → `config.json` → re-run `getMe`. Credential present, value not recorded. → [[systems/scheduled-tasks]]
|
||||
- **Open cron incidents 5 → 2.** Closed on 2026-10-12: `fad5a8_a13bbc837e1c` + `fad5a8_c80b59a907f0` at **03:05:39** (both `deepseek-v4-flash`, cleared by `fad5a86979d4`'s first clean scheduled run — 03:00:45 → 03:05:47, 448 unique birthdays, delivered) and `dca848_23fcd2b18aaa` at **07:33:41** (cleared by `dca8482e4f76`'s own 07:00 run, delivered to topic 1030 after a 32-minute run). Survivors: `0e753e_e7f8194a7d06` (`detected` since 2026-10-01 — cannot clear until 2026-11-01, the job has **no retained executions at all**) and `1b23c0_c35703f637b8` (`alerted` since 2026-10-09, next slot Fri 2026-10-16 08:00). Table: alerted 1 / detected 1 / closed 8 / resolved 66. Fleet **29 jobs — 26 enabled / 3 disabled**; non-clean `last_status` **5 → 3** (`0e753e9720fd`, `1b23c042e17d`, `066bb5191280`). → [[systems/scheduled-tasks]]
|
||||
- **Camofox hit its memory containment ceiling inside a cron run (NEW).** The 2026-10-12 07:01 Daily Marketing Job Radar drove `camofox-browser.service` to **`2.5G` with swap fully consumed, `available: 0B`, 383 tasks** (drop-in `MemoryHigh=2G`/`MemoryMax=3G`/`MemorySwapMax=1G`); `/tabs` and page evals hung. The agent restarted the service at **07:23:50** and finished **one tab at a time**, and the job still delivered. A green cron row does not mean the browser layer was healthy — check the unit's peak/swap/tasks when a browser-heavy job runs long. → [[systems/browser-backend]]
|
||||
- **Browser page corrected against a live probe.** REST API is **`:9377`**, not `:9093` (nothing listened on 9093); `:9378` does not exist; Camofox is a `systemd --user` unit **on the Hermes host (CT460)**, not CT450. Containment drop-in documented. → [[systems/browser-backend]]
|
||||
- **Mnemosyne healthy at 12:02** — working **760/786 = 96.7 %**, episodic **1050/1056 = 99.4 %** (both above the 95 % floor, no `reindex` needed); orphan embeddings 0; FTS5 aligned exactly (786/786, 1056/1056, 876/876); 0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → [[systems/mnemosyne]]
|
||||
- **Window shape:** **18 sessions, all `cron`, zero interactive** — the other bookend to the previous window's `talaria-ios` research session. The 03:00 stocktake `1a8e90ea2271` ran clean and pushed **`0eee5f1`** (freshness **54 stale / 32 fresh / 7 no-date of 93 pages**; skills index 346 / 70 categories — the known 347-vs-346 off-by-one persists). The execution log's retention floor has moved to **2026-10-07 08:15**, so 10-07 rows are now pruned.
|
||||
|
||||
## Recent Changes (2026-10-12, 03:00 stocktake; live probes 2026-10-12 03:0x-03:1x AWST)
|
||||
|
||||
@@ -91,7 +101,7 @@ _The manual 2026-10-10 stocktake entry follows beneath this one._
|
||||
## Recent Changes (2026-10-06)
|
||||
- **Dead model pins cleared across the whole fleet — failing set 7 → 0.** All seven jobs that had been failing since 2026-10-01 were repinned on 2026-10-05 23:16 → 2026-10-06 00:05. Each had been pinned to a raw model name that died upstream: `cliprox/terra` (HTTP 401 connection auth, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`). They were moved to `custom:omni-101` on **routing aliases** — `free-stack` ×6, `smart-route` ×1 — because the fault was raw-pin brittleness, not any one model. Every `failure_streak` in `jobs.json` now reads 0. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The other four are repinned but not yet re-run, so they carry a stale `last_status: error` until their next scheduled slot. See [[incidents/2026-10-05-dead-model-pin-repin]].
|
||||
- **The `cliprox/terra` connection incident is closed for cron purposes.** No enabled job references that pin any more — the jobs were moved off it rather than the proxy connection being re-authenticated. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||
- **GitHub Trending Radar's revoked-token diagnosis is refuted.** `c0a9fa314542` ran on schedule at **2026-10-05 09:00 and completed cleanly** — a second consecutive success after the off-schedule 2026-09-30 19:22 run. No `getMe` → 401, and no BotFather token replacement is needed. The previous audit correctly refused to treat the claim as standing fact; it can now be retired.
|
||||
- ~~**GitHub Trending Radar's revoked-token diagnosis is refuted.**~~ **SUPERSEDED 2026-10-13 — this conclusion was WRONG.** `c0a9fa314542` ran on schedule at **2026-10-05 09:00 and completed cleanly** — a second consecutive success after the off-schedule 2026-09-30 19:22 run. No `getMe` → 401, and no BotFather token replacement is needed. The previous audit correctly refused to treat the claim as standing fact; it can now be retired.
|
||||
- **A stale failure notice, not a live outage, opened the review.** Anthony acted on a `85edf6772bca` notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job succeeded on every run from 2026-09-30 17:53 onward, and its three incidents were resolved on 2026-09-30 17:55:13; the notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.**
|
||||
- **Window tally** (`executions.db`, 2026-10-05 00:00 → 2026-10-06 00:15 AWST): 185 `completed`, 3 `failed` (all pre-repin, 03:00–07:00), 2 `unknown` (aborted first attempts of `79f9bb41ecf5` / `dca8482e4f76`, superseded by their successful re-runs) and 1 `running` (the nightly audit itself). No interactive sessions other than the repin session.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user