stocktake 2026-11-05: fix skills-index generator divergence at the root; correct Docker + Tailscale drift against live probes

ROOT CAUSE FIXED - systems/skills-index.md had TWO WRITERS:
the repo wiki-maintain.sh (302 skills / 68 categories) and the stale
~/.hermes/scripts/wiki-maintain.sh (mtime 2026-08-17, 349 skills / 26
categories - it used Path.rglob() and counted the parked .archive/ tree) were
different scripts with different generators, and wiki-maintenance/SKILL.md
line 569 pointed the 23:59 nightly audit at the stale path while the 03:00
stocktake ran the repo path. Each job overwrote the other's page; neither
raised an error. The stale copy is now a delegating wrapper with no generator
logic, and the repo script gained --no-build so the wrapper honours the old
interface without a second generator. Verified by execution.

Live-probe corrections:
- infrastructure/docker-services.md: CT200 61 running / 68 total (was 62);
  worldmonitor-seeder-scheduled does not exist on ANY host (zero 'seeder'
  matches across CT200/201/333/460); CT333 first container renamed again
  (nostalgic_leavitt -> flamboyant_mahavira); CT201 20/26, CT460 1/1,
  CT1001 1/9.
- infrastructure/tailscale.md: node table verified programmatically - 52 rows
  vs 52 live IPv4 peers, no live peer missing, no orphan row, every
  online/offline state matching. conduit 59d->60d, ubuntu-server-xfce
  75d->76d. Serve table still matches tailscale serve status exactly.
- infrastructure/hosts.md: skills figure corrected to 302 / 68 categories.
- index.md: skills-index bullet refreshed to the 2026-11-05 figures.

New incident: incidents/2026-11-05-skills-index-generator-divergence.md
(closes the still-open item on the 2026-08-15 skills-index drift incident).

Freshness: 0 stale / 123 fresh / 3 no-date of 126 pages.
This commit is contained in:
Tony0410
2026-11-05 05:55:45 +08:00
parent 20aa11b0c4
commit 2f3fc9952b
11 changed files with 722 additions and 457 deletions
+53
View File
@@ -2027,3 +2027,56 @@ embedded credential was seen during verification and is not reproduced here ([RE
- **Not done / unchanged:** no job added, removed, disabled or re-pinned; no page removed or archived; no
decision reversed. **No secrets, passwords, tokens or keys written or quoted** - the API key was read from
`~/.hermes/.env` into the shell for the probe only.
## [2026-11-05] stocktake | Skills-index generator divergence FIXED at the root; Docker + Tailscale drift corrected against live probes
- **Ran `wiki-maintain.sh`** (repo canonical path) - exit 0. Regenerated `systems/skills-index.md`
(**302 skills / 68 categories**, `.archive/` excluded, 46 archived), wrote
`raw/stocktake/freshness-20261105.txt` (**0 stale / 122 fresh / 3 no-date of 125 pages**), rebuilt
Quartz, deployed to `public/`, restarted `hermes-wiki-serve.service`.
- **ROOT CAUSE FOUND AND FIXED - the `systems/skills-index.md` flip-flop had TWO WRITERS.** The repo
`wiki-maintain.sh` (mtime 2026-11-03) and `~/.hermes/scripts/wiki-maintain.sh` (mtime **2026-08-17**) are
**different scripts**, each with its own generator: the repo one scans with `find` excluding
`*/.archive/*` (-> **302 skills / 68 categories**); the stale one uses `Path.rglob('SKILL.md')` and a
2-segment category heuristic (-> **349 skills / 26 categories**, because it counts the parked archive).
`wiki-maintenance/SKILL.md` line 569 pointed the **23:59 nightly audit** at the stale path while the
**03:00 stocktake** ran the repo path - so each job overwrote the other's page and the wiki oscillated
nightly with **no error raised by either job**.
- **Fix:** `~/.hermes/scripts/wiki-maintain.sh` is now a **delegating wrapper** with **no generator logic**
- it `exec`s the repo script. The repo script gained **`--no-build`** so the wrapper honours the
interface its callers were written against without carrying a second copy of the generator.
- **Verified by execution, not inspection:** `bash ~/.hermes/scripts/wiki-maintain.sh --no-build` ->
`Written ... with 302 skills in 68 categories`; both scripts pass `bash -n`.
- **SKILL.md** repointed at the repo path, with a note that a generator must live in exactly one place.
-> [[incidents/2026-11-05-skills-index-generator-divergence]]; the "still open" item on
[[incidents/2026-08-15-skills-index-drift]] is closed.
- **`infrastructure/docker-services.md` corrected against live `docker ps` (all hosts, 2026-11-05 05:2x):**
- CT200 **61 running / 68 total** (was "62"); the **`worldmonitor-seeder-scheduled`** entry recorded on
2026-11-04 as "unhealthy" **does not exist on any host** - `docker ps -a --filter name=seeder` across
CT200/201/333/460 returns **zero** matches. Live CT200 shows exactly three worldmonitor containers
(`worldmonitor` `Up 5 days` **healthy**, `worldmonitor-redis`, `worldmonitor-redis-rest`). There is **no
seeder container and no worldmonitor-seeder cron job** anywhere in the estate.
- CT333 first container renamed again: `nostalgic_leavitt` -> **`flamboyant_mahavira`** (image
`ghcr.io/github/github-mcp-server`, created 2026-11-04T10:53:53Z). CT333 = **8 running / 9 total**.
- CT201 **20 running / 26 total**; CT460 **1 running / 1 total** (`ignis` `Up 5 days`); CT1001
**1 running / 9 total** (`sshwifty`). Per-host "Last verified 2026-11-05" added.
- **`infrastructure/tailscale.md` re-verified against live `tailscale status --json`:** the node table is
**exact** - 52 rows, 52 live IPv4 peers, **no live peer missing from the table, no table row without a live
peer, and every Online/offline state matches** (checked programmatically, not by eye). Two offline ages
corrected: `conduit` 59d -> **60d**, `ubuntu-server-xfce` 75d -> **76d**. `tailscale serve status` still
matches the recorded table exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443`
tailnet route).
- **`infrastructure/hosts.md`:** skills figure corrected to **302 skills / 68 categories** (was "69
categories"); `updated` / `verified_on` -> 2026-11-05.
- **`index.md`:** skills-index bullet refreshed to the 2026-11-05 figures (302 / **68 categories**,
freshness **0 stale / 122 fresh / 3 no-date of 125**), the `_archive/` typo corrected to `.archive/`, and
the new incident linked.
- **Freshness:** the 8 pages stale on 2026-11-04 were all re-verified and refreshed, so the 2026-11-05 report
reads **0 stale**. The 3 pages with no `updated` date are `SCHEMA.md`, `log.md` and
`experiments/active/example.md` - `log.md` is append-only by design, the other two are templates.
- **Files touched this pass:** `systems/skills-index.md`, `infrastructure/docker-services.md`,
`infrastructure/tailscale.md`, `infrastructure/hosts.md`, `index.md`,
`incidents/2026-11-05-skills-index-generator-divergence.md` (new),
`incidents/2026-08-15-skills-index-drift.md`, `wiki-maintain.sh`, `log.md`.
- **Not done / unchanged:** no job added, removed, disabled or re-pinned; no page removed or archived; no
decision reversed. **No secrets, passwords, tokens or keys written or quoted.**