stocktake 2026-11-05: fix skills-index generator divergence at the root; correct Docker + Tailscale drift against live probes

ROOT CAUSE FIXED - systems/skills-index.md had TWO WRITERS:
the repo wiki-maintain.sh (302 skills / 68 categories) and the stale
~/.hermes/scripts/wiki-maintain.sh (mtime 2026-08-17, 349 skills / 26
categories - it used Path.rglob() and counted the parked .archive/ tree) were
different scripts with different generators, and wiki-maintenance/SKILL.md
line 569 pointed the 23:59 nightly audit at the stale path while the 03:00
stocktake ran the repo path. Each job overwrote the other's page; neither
raised an error. The stale copy is now a delegating wrapper with no generator
logic, and the repo script gained --no-build so the wrapper honours the old
interface without a second generator. Verified by execution.

Live-probe corrections:
- infrastructure/docker-services.md: CT200 61 running / 68 total (was 62);
  worldmonitor-seeder-scheduled does not exist on ANY host (zero 'seeder'
  matches across CT200/201/333/460); CT333 first container renamed again
  (nostalgic_leavitt -> flamboyant_mahavira); CT201 20/26, CT460 1/1,
  CT1001 1/9.
- infrastructure/tailscale.md: node table verified programmatically - 52 rows
  vs 52 live IPv4 peers, no live peer missing, no orphan row, every
  online/offline state matching. conduit 59d->60d, ubuntu-server-xfce
  75d->76d. Serve table still matches tailscale serve status exactly.
- infrastructure/hosts.md: skills figure corrected to 302 / 68 categories.
- index.md: skills-index bullet refreshed to the 2026-11-05 figures.

New incident: incidents/2026-11-05-skills-index-generator-divergence.md
(closes the still-open item on the 2026-08-15 skills-index drift incident).

Freshness: 0 stale / 123 fresh / 3 no-date of 126 pages.
This commit is contained in:
Tony0410
2026-11-05 05:55:45 +08:00
parent 20aa11b0c4
commit 2f3fc9952b
11 changed files with 722 additions and 457 deletions
@@ -0,0 +1,90 @@
---
title: "Skills Index Generator Divergence — Two Scripts, One Page"
type: incident
status: resolved
created: 2026-11-05
updated: 2026-11-05
resolved_on: 2026-11-05
verified_on: 2026-11-05
confidence: high
tags: [skills, wiki, drift, auto-regeneration, root-cause]
sources: [live-filesystem-probe, script-diff, wiki-maintain-run, agent-estate-wiki]
---
# Skills Index Generator Divergence — Two Scripts, One Page
## Summary
`systems/skills-index.md` flipped between two different shapes on alternating days — **302 skills / 68
categories** on some runs and **349 skills / 26 categories** on others. The cause was **two different
scripts** writing the same page, each with its own generator, and the two scheduled jobs pointing at
different ones. Whichever job ran last won, so the page oscillated nightly.
This is the **same class of defect** as [[incidents/2026-08-15-skills-index-drift]], but a different and
previously-unidentified root cause. The 2026-08-15 incident attributed the drift to archived skills being
counted; that was a *symptom* of one of the two generators, not the mechanism.
## Root Cause
Two files both named `wiki-maintain.sh`, each containing a **complete, independent copy** of the page
generator:
| Script | Generator technique | Output |
|---|---|---|
| `/home/hermes/workspace/agent-estate-wiki/wiki-maintain.sh` (repo, canonical, mtime 2026-11-03) | `find` scan **excluding** `*/.archive/*`; dedupes by `(category, skill_name)` | **302 skills / 68 categories** |
| `/home/hermes/.hermes/scripts/wiki-maintain.sh` (stale copy, mtime **2026-08-17**) | `Path.rglob('SKILL.md')` — **includes** the parked `~/.hermes/skills/.archive/` tree; 2-segment category heuristic | **349 skills / 26 categories** |
The two scripts are **not** identical — `diff` between them returns a non-zero exit and shows entirely
different generator bodies, different category heuristics and different output formats.
**The trigger:** `~/.hermes/skills/devops/wiki-maintenance/SKILL.md` (line 569) instructed the nightly audit
job (`cbfe434a3b5e`, daily 23:59) to run:
```bash
run `bash ~/.hermes/scripts/wiki-maintain.sh` (it regenerates `systems/skills-index.md`, writes a freshness
```
— i.e. the **stale** copy. The 03:00 stocktake job (`1a8e90ea2271`) runs the **repo** script. So:
- 03:00 stocktake → repo script → page becomes **302 / 68**
- 23:59 nightly audit → stale script → page becomes **349 / 26**
- repeat, forever
Each job was individually "successful" — neither reported an error. The defect was only visible as a
day-over-day oscillation in the committed page.
**Live counts verified 2026-11-05 05:1x AWST** (filesystem probes):
- `find ~/.hermes/skills -name SKILL.md` → **349** (includes archive)
- `find ~/.hermes/skills -name SKILL.md -not -path '*/.archive/*'` → **303**
- the generator de-dupes one nested alias duplicate → **302 unique** ✅
- `~/.hermes/skills/.archive/` → **46** parked skills (excluded by the canonical generator)
## Impact
- The published skills inventory was **wrong roughly half the time**, understating the real category
topology (26 vs 68 categories) and overstating the count by including 47 archived skills.
- Any agent reading the page to reason about the estate's skill surface got a materially different
picture depending on which run last wrote it.
- **No functional impact** — skill loading was never affected; only the documentation was wrong.
## Resolution
1. **`~/.hermes/scripts/wiki-maintain.sh` is now a delegating wrapper.** It contains **no generator logic** —
it `exec`s the canonical repo script and forwards arguments. The wrapper's header comment explains why
it must stay that way, so the divergent copy is not recreated.
2. **`--no-build` is now supported by the repo script.** The stale copy accepted `--no-build` (regenerate +
audit, skip the Quartz build/deploy); the repo script did not. The repo script gained the flag so the
wrapper can honour the interface its callers were written against **without** carrying a second copy of
the generator. This closes the only functional gap the delegation could otherwise have introduced.
3. **Verified by execution, not by inspection:** running the wrapper with `--no-build` produces the canonical
shape — `302 skills in 68 categories` — and the generated page reports `46 archived skills` excluded.
Both scripts pass `bash -n`.
4. **`systems/skills-index.md` regenerated through the canonical path** and committed in the canonical shape.
## Lesson
**A generator must live in exactly one place.** Two copies of the same script, differing in one technique
(`rglob` vs `find`), produced two valid-looking outputs and a page that flipped daily — with **no error
anywhere**, because each job did exactly what it was told.
When a scheduled job's output disagrees with another job's output for the same artefact, the cause is
almost never the artefact — it is **two writers**. Diff the scripts before diffing the data.
Related: [[incidents/2026-08-15-skills-index-drift]] (the original drift), [[systems/skills-index]],
[[systems/scheduled-tasks]].