Stocktake 2026-11-12: tailscale live-vs-page drift corrected; sftpgo incident re-verified; pre-dispatch-filter signature found recurring on a second job/pin

- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json.
  mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping
  (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online.
  omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout).
  9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge
  (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table
  unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed.
- incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer
  Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared.
- incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on
  cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved
  2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded.
- Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits),
  work-outlook (himalaya still login not supported), omniroute-free-deepseek
  (good-free/free-stack both HTTP 200, non-deepseek resolution).
- Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md.
- wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
This commit is contained in:
Hermes
2026-11-12 03:19:12 +08:00
parent b94de554f2
commit ab3652aa20
9 changed files with 158 additions and 57 deletions
+15
View File
@@ -2480,3 +2480,18 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
- **Post-audit housekeeping (00:2x AWST 2026-11-12), vault sync.** Touched files copied to **both** vault conventions — `…/agent-estate-wiki/<rel>` and `…/agent-estate-wiki/wiki/<rel>` — with `sha256sum` confirmed equal across all three copies; vault committed locally in `/home/hermes/wiki` (`master`, **no remote — never pushed**), explicit paths only, so the unrelated `reports/uber-eats/*` dirt stayed out. Before copying, each vault file was compared against the **pre-edit** authoring blob (`c108eb1`): every difference was a *stale-but-clean* older version (vault `index.md`/`skills-index.md` still 11-10/11-09, `log.md` still missing the 11-11 stocktake entry, the nous incident page still 2026-10-26 at 48 lines vs 96) — **no direct-to-vault edits to preserve**, so forward-sync was safe. The 2026-10-22 nous incident page was **created** in vault B's `incidents/` (it had never been mirrored); the new 11-10 audit page went into **both** `incidents/` dirs.
- **Vault drift re-measured (hash sweep over all 133 authoring pages, this run's 10 touched files excluded):** **Vault A — 13 stale / 56 missing / 64 in sync. Vault B — 58 stale / 8 missing / 67 in sync.** Unchanged in character from the 2026-10-19 measurement (the copy bridge still only syncs what an audit touches), so the missing-page backlog is recorded here and **left to an explicit request** rather than silently swept — a 133-page mirror pass is bigger than a nightly audit's mandate. ⚠ This is now a two-night-old gap: because the 2026-11-10 audit died at 00:12, **nothing was mirrored for that window either.**
- **Freshness report:** `raw/stocktake/freshness-20261112.txt` (this run, 6,325 B) — **1 stale / 129 fresh / 3 no-date of 133** pages (was 132; **+1 = the new incident page**). The single stale entry is **`log.md` itself**, which carries no frontmatter `updated:` — expected, not a finding. `systems/skills-index.md` regenerates identically at **299 skills / 67 categories**. Quartz rebuilt and `hermes-wiki-serve.service` restarted; the new page returns **200** at the served site with its body content present.
## 2026-11-12 03:0x–04:1x AWST — Stocktake (03:00): tailscale drift corrected; 1 stale + 5 threshold pages re-verified live
- **Freshness:** `wiki-maintain.sh` → **1 stale / 129 fresh / 3 no-date of 133** (was 132, +1 = the new 2026-11-10 incident page). The single stale page was **[[incidents/2026-09-02-sftpgo-outage]] at 22d** — re-verified live and cleared. The 3 no-date pages are unchanged (`SCHEMA.md`, `experiments/active/example.md`, `log.md`). `systems/skills-index.md` regenerates identically at **299 skills / 67 categories**.
- **⚠ Live-vs-page diff found real drift in [[infrastructure/tailscale.md]] (2d old — age scan would have missed it):** three peers read `Online: false` in the peer record while **all three ping fine** — `mailflow` (CT1262 m93p, pong `192.168.178.142:41641` 2 ms), `qwenpaw` (CT465 pve, pong `192.168.178.186:41641` 7 ms) and `springroll` (pong via **DERP(sin)** 210 ms, relay not direct) — with both LXCs `running` in `pct list`. All three rows stay **online**; the peer record is stale (the m93p pattern again). One **genuine** correction in the other direction: `omarchymac`'s row claimed `offline 2d / last seen 2026-11-08 00:45Z` when the live record reads `LastSeen 2026-11-07T06:41:48Z` = **offline 4d** and ping → **timeout**. 9 named offline ages + the six `codex-environment-*` ephemerals refreshed (+2d). No IP changes, no missing rows, no new peers (58 rows = 57 live IPv4 peers + `hermes`; 80 peers total / 57 IPv4 / 23 IPv6-only, unchanged). Table rebuilt by script, not by hand.
- **Re-probed and re-confirmed:** CT299 `sftpgo` **running**, peer `Online: true`, `pong via 192.168.178.92:41641 in 0s`, healthz **401** (auth-gated, service up) — closing [[incidents/2026-09-02-sftpgo-outage]] as still-resolved, and noting that `pct config 299` now reads **`swap: 512`** (the 43 Gi over-provisioning in that root cause is gone). Both DEAD Serve targets (`8769` `/agent-estate`, `9123` `/synergy-prep`) still have **no listener** — re-confirmed a fifth time. CT201's Serve table re-probed unchanged (`/`→8085, `/mgr`→8084 on both `:443` and `:8443`). Container counts re-confirmed on **both** hypervisors incl. `qm list`: `pve` **17 LXCs + 4 VMs**, `m93p` **20 LXCs + 2 VMs**. **New guest found: CT1007 `homebridge` on m93p — running but `tailscaled` `inactive`**, so it runs without joining the tailnet (same class as CT105 conduit, CT1264 openmausbot).
- **Incident page re-opened for a **recurrence** — [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] (21d, tomorrow's stale list).** The *"all targets were skipped by pre-dispatch filters"* signature has now fired on a **second job and a second pin**: `cbfe434a3b5e` (Wiki Audit) failed on **2026-11-10** with `RuntimeError: HTTP 429: … pre-dispatch filters … (credits 100%)`, `failure_type: rate_limit`, and its incident row `cbfe43_ed01845310d8` closed `resolved` **2026-11-12T00:25**. The original subject has **not** repeated — `dca8482e4f76` ran clean on 11-06/09/10/11. Caveat recorded on the page: `executions.db`'s retention floor is **2026-11-06T05:09:30+08:00**, so the 2026-10-21 row is *not* re-readable from the DB — the new read is in-window only.
- **Four threshold/near-threshold pages re-verified against live state (skill pattern 22), each with a dated note — no date bump without a probe:**
- [[incidents/2026-07-29-mnemosyne-relay]] (19d) — `tools: None` absent from `config.yaml`; **0** occurrences of `Unknown Mnemosyne tool` across 4 rotated gateway logs and **0** of the relay `scope handle` error; `mnemosyne stats` → 2782 memories and recall returns ranked results.
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] (18d) — `himalaya envelope list -a work -s 1` still fails on `login not supported`; no OAuth2 token in the himalaya config, so path 1 (Anthony's sign-in) remains the only route.
- [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]] (19d) — keyed probes on `omniroute-101`: **`good-free` → HTTP 200 (`glm-5.2-cheap:free`)** and **`free-stack` → HTTP 200 (`stepfun-ai/step-5-preview`)**. The mitigation holds and neither combo now resolves to deepseek. **The three free-DeepSeek upstreams themselves were NOT probed**, so "upstream still degraded" is neither confirmed nor refuted and restoring deepseek-first ordering is still unjustified — recorded as such on the page.
- [[reports/communities-wa-social-media-audit-2026]] (18d) — **not re-verified this pass** (live Department of Communities pages not re-fetched); its `review_after: 2026-11-26` still stands and its own note already scopes the unverified claims. Flagged, left as written.
- **Order of operations:** script → page corrections → **re-run script** → log/index → commit. `wiki-maintain.sh` re-run after every page edit; Quartz rebuilt and `hermes-wiki-serve.service` restarted.
- **No page deleted, no decision reversed, no secrets written.** No credential value appears in any page; the OmniRoute probe used the `key_env` name, never the value.
- **Correction to the entry above (2026-11-12 00:1x AWST audit), verified live rather than inherited:** that entry states *"The single stale entry is **`log.md` itself**, which carries no frontmatter `updated:` — expected, not a finding."* **That is wrong.** The committed `raw/stocktake/freshness-20261112.txt` from that run (sha256 `da5f9e2e…`, byte-identical to this run's pre-fix output) reads **`- [ 22d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-10-21)`** as its single `## STALE (1)` line — the real page, which this stocktake has now re-verified and cleared. `log.md` sits in the `## NO updated DATE` bucket, as always (pattern 13). The stale page was therefore **left unflagged in prose for one cycle while sitting stale in the tree** — the exact "fresh-looking prose, stale content" failure the report exists to prevent.