Stocktake 2026-10-30: 0 stale by age, but 5 pages drifted from live state

Ran wiki-maintain.sh (skills-index regenerated: 302 skills / 69 categories,
unchanged; freshness report 0 stale / 109 fresh / 3 no-date of 112 pages;
Quartz rebuilt and deployed).

Freshness is not accuracy. A live-vs-page diff found real drift on five pages.
All corrections verified against live sources first.

- infrastructure/tailscale: node table rebuilt from tailscale status --json.
  50 non-funnel nodes (49 IPv4 peers + hermes), not 54. 2 IP changes, 4 peers
  expired out of the tailnet. Zero IP mismatches, zero missing/extra rows on
  re-verification. m93p records online (peer record chronically stale, third
  occurrence -- ping + SSH confirm). CT460 :443 route set (12 paths) and
  CT201 :443/:8443 tables re-probed and match.
- systems/scheduled-tasks: fleet is 29 jobs (25 enabled / 4 disabled), not
  28/25/3. Mnemosyne Health Monitor ea79a2107e36 was auto-paused 2026-10-30
  01:04:22 after its 00:01:08 slot closed "unknown" on a gateway-connection
  failure. Non-clean last_status is 4, not 3. "Failing (delivery only)" is 2.
  Seven job rows carried stale error claims for jobs since cleared to ok --
  each marked superseded with a dated correction, not rewritten (SCHEMA rule 9).
- NEW incidents/2026-10-30-mnemosyne-health-monitor-paused: the job's own
  failure notice was never delivered (deliveries.db has zero rows for
  2026-10-30), so the incident row is the only trace. Job not re-enabled and
  gateway not restarted by this run; root cause recorded as unverified.
- infrastructure/hosts: CT201 LAN is 192.168.178.77 (was .166); CT450 Hermes
  is v0.22.5 / bf0203a794a0, same build as CT460 (page said v0.22.3 /
  5b147ab036d6); cron count 29/25/4; orphaned "omni" peer has expired out of
  the tailnet entirely. Skills count of 302 left alone -- it is correct.
- systems/homepage-dashboard: live app count is 123 (was 114); grid is a
  responsive repeat(auto-fill, minmax(230px, 1fr)), not a fixed 5-column grid;
  tab bar is built from the service groups (renderTabs iterates state.groups),
  so the settings.yaml layout block and its tab:/columns: keys are inert --
  there is no "Overview" tab; saved theme is Boarding Pass / light, not dark
  amber; manager.py runs inside the homepage-manager container; and the "no
  Traefik container" evidence was wrong -- the container exists but is
  stopped (Exited (255) 2 months ago). The conclusion that Traefik does not
  occupy 443 stands. Same evidence error corrected in tailscale.md.

Also corrected two of my own first-pass errors before committing: the
homepage tab list (groups vs tabs) and the CT450 version (a stale root-level
/usr/local/bin/hermes reports v0.19.0; the live agent is the hermes-user
install at v0.22.5).

No secrets written -- provider names, key names and HTTP status codes only.
This commit is contained in:
Tony0410
2026-10-30 05:41:11 +08:00
parent 1d335ca8e6
commit 987eba72e6
10 changed files with 448 additions and 73 deletions
+17 -5
View File
@@ -39,11 +39,23 @@ Updated: 2026-10-30
then no-oped as designed (10 m 46 s / 1 m 51 s).
→ [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]],
[[runbooks/job-radar]]
- **Cron: 2026-10-29 settled 176 `completed` / 0 `failed`** — the first zero-failure day since 2026-10-24 — with
**10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; every `failure_streak`
reads **0**; unclosed incident rows **4 → 1**. Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**, pins
unchanged (19 of the 26 enabled jobs on `custom:omni-101`: 12 `smart-route`, 6 `free-stack`, 1 explicit; 7
unpinned on the global default).
- **Cron: 2026-10-29 settled 177 `completed` / 0 `failed`** — the first zero-failure day since 2026-10-24 — with
**10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; unclosed incident rows
**4 → 1** (the two radar rows closed `resolved` at 10:16:12). Fleet unchanged at **29 jobs (26 enabled / 3
disabled)** at that audit.
- **Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure
notice was never delivered.** `ea79a2107e36`'s 00:01:08 slot closed **`unknown`** at 01:04:22 on a
gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`:
Gateway connection failed; no local fallback"*), opening incident `ea79a2_d93c3acdc4f2` (`detected`), and the
scheduler **auto-paused the job**. Fleet is now **29 jobs (25 enabled / 4 disabled)**; non-clean
`last_status` is **4** (2 `error` + 2 `delivery_failed`); `ea79a2107e36` carries `failure_streak: 1`.
⚠ **The failure notice itself failed to send** — `hermes cron list` reports `send_path_degraded; live adapter
delivery to telegram:-1004321904721:102 failed`, and `deliveries.db` holds **zero rows for 2026-10-30**, so
the only trace is the incident row. **Unclosed incident rows 1 → 2.** Every other previously-red job
(`dca8482e4f76`, `1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`,
`c0a9fa314542`) has **cleared to `ok` with streak 0** on its own slot. Pins: **18 of the 25 enabled jobs pin
`custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); 7 enabled rows are `no_agent`
jobs with no pin. → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST)