Stocktake 2026-11-12: tailscale live-vs-page drift corrected; sftpgo incident re-verified; pre-dispatch-filter signature found recurring on a second job/pin
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json. mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online. omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout). 9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed. - incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared. - incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved 2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded. - Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits), work-outlook (himalaya still login not supported), omniroute-free-deepseek (good-free/free-stack both HTTP 200, non-deepseek resolution). - Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md. - wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
This commit is contained in:
@@ -3,8 +3,8 @@ title: Anthony's Systems Wiki
|
||||
type: collection
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
confidence: high
|
||||
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
|
||||
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
|
||||
@@ -12,7 +12,11 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
|
||||
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. Last updated: 2026-11-09.
|
||||
> model routing and automation. Last updated: 2026-11-12.
|
||||
>
|
||||
> **2026-11-12 (03:00 stocktake) — the freshness scan was clean, the live-vs-page diff was not.** The age scan flagged **1 stale / 129 fresh / 3 no-date of 133** (the sftpgo incident at 22d, re-verified live and cleared). But [[infrastructure/tailscale.md]] — only **2 days old** — had drifted: **`mailflow`, `qwenpaw` and `springroll` all read `Online: false` in the peer record while all three ping fine** (CT1262 and CT465 `running` in `pct list`; springroll answers via **DERP(sin)**, relay not direct), and `omarchymac` was stale in the *other* direction (page said `offline 2d / last seen 2026-11-08 00:45Z`; live reads `LastSeen 2026-11-07T06:41:48Z` = **offline 4d**, ping timeout). Node table rebuilt by script, all IPs matching, no rows missing, **58 rows = 57 live IPv4 peers + `hermes`**, 80 peers total (57 IPv4 / 23 IPv6-only) unchanged. Also re-probed: CT299 `sftpgo` running/online/healthz-401; both DEAD Serve targets (8769, 9123) still listener-less; CT201's Serve table unchanged; container counts **17 LXCs + 4 VMs on `pve`** and **20 LXCs + 2 VMs on `m93p`**; and **new guest CT1007 `homebridge` on m93p — running but `tailscaled` `inactive`**, so it does not join the tailnet.
|
||||
>
|
||||
> **2026-11-12 (03:00 stocktake) — a "router" failure mode turned out to be fleet-level.** [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] recorded *"all targets were skipped by pre-dispatch filters"* as a Job Radar one-off and left three open questions. Live re-read of `executions.db` shows the **same signature on a second job and a second pin**: `cbfe434a3b5e` (Wiki Audit, `custom:omni-101` / **`free-stack`**) failed on **2026-11-10** with `HTTP 429 … (credits 100%)`, incident row `cbfe43_ed01845310d8` (`rate_limit`) closed `resolved` **2026-11-12T00:25**. The original subject has **not** repeated — `dca8482e4f76` ran clean 11-06/09/10/11. So open question 1 (a job-specific mis-configured filter) now looks **less likely** than a router-side guard that blanks every target, and open question 3 (**no failover pin**) applies to the whole free-tier cohort, not just the radar. ⚠ Caveat recorded on the page: `executions.db`'s retention floor is **2026-11-06T05:09:30+08:00**, so the 2026-10-21 row is not re-readable today — the recurrence is an in-window read.
|
||||
>
|
||||
> **2026-11-09 (nightly audit — window 2026-11-08) — the day's fault was a hook, not a schedule, and it is fixed.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`), and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** — `errors.log` holds **12** lines on 11-07 and **46** on 11-08, plus **89** *"skipped after previous timeout or while still running"* suppression lines. Every tool call degraded to slow-then-suppressed, which is what stretched the day. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 → 120** in `~/.hermes/config.yaml` — verified live, and verified *effective* (the last timeout line is still 08:21:03, re-checked at audit time). ⚠ **The obvious suspect was measured and acquitted:** the plugin's caching is fine (`session_titles.py:21 CACHE_SECONDS = 30.0`; timed in-process at **3.1 s cold, 0.00 s warm**), so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong** — the trigger is **load** on the 2-core CT460 (`load average 5.89` at 13:27:50, `state.db` checkpointing a **40 MB WAL** on the same spindle) → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. **Cost of the fault:** *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)** — its two slowest slots on record, both delivered; *FreshRSS Midday Briefing* took **2 h 09 m 53 s**; and **the stocktake failed a third consecutive day** (`1a8e90ea2271`, 03:01:11 → 04:36:08, `failure_streak: 3`) on a **new terminal signature** — `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` where the previous two red days read *"lost its durable fire claim ownership"*. *Working hypothesis, not proven:* an open tool result is exactly what a suppressed `pre_tool_call` would produce, but the string does not name the hook. **11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33); deliveries **8 / 8 `delivered`**; fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count with nothing added, retired, disabled or repinned, though `updated_at` moved to `2026-11-09T00:06:50+08:00` as a run-state stamp. **Unclosed incident rows 2 → 3**, all three the stocktake's and all silent (`detected`, `alerted_at: null`) — but **the alerting path is not the fault**: the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window. ✅ **Two verified resolutions:** the **Mnemosyne Health Monitor's no-agent conversion works** — its first script slot ran **2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, silent, writing a **159 B** output file against **105,982 B** for the last agent-mode run (coverage live at **working 1030/1030 = 100.0 %, episodic 1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet; the script uses `awk` because **`bc` is not installed on this host**); and the **1.89 GB `state.db`** was re-measured (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL) with the 2026-08-11 remnant **`state.db.corrupted.20260811_151052` (624.1 MB) deleted**. **New page:** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
|
||||
>
|
||||
|
||||
Reference in New Issue
Block a user