stocktake 2026-11-05: fix skills-index generator divergence at the root; correct Docker + Tailscale drift against live probes
ROOT CAUSE FIXED - systems/skills-index.md had TWO WRITERS: the repo wiki-maintain.sh (302 skills / 68 categories) and the stale ~/.hermes/scripts/wiki-maintain.sh (mtime 2026-08-17, 349 skills / 26 categories - it used Path.rglob() and counted the parked .archive/ tree) were different scripts with different generators, and wiki-maintenance/SKILL.md line 569 pointed the 23:59 nightly audit at the stale path while the 03:00 stocktake ran the repo path. Each job overwrote the other's page; neither raised an error. The stale copy is now a delegating wrapper with no generator logic, and the repo script gained --no-build so the wrapper honours the old interface without a second generator. Verified by execution. Live-probe corrections: - infrastructure/docker-services.md: CT200 61 running / 68 total (was 62); worldmonitor-seeder-scheduled does not exist on ANY host (zero 'seeder' matches across CT200/201/333/460); CT333 first container renamed again (nostalgic_leavitt -> flamboyant_mahavira); CT201 20/26, CT460 1/1, CT1001 1/9. - infrastructure/tailscale.md: node table verified programmatically - 52 rows vs 52 live IPv4 peers, no live peer missing, no orphan row, every online/offline state matching. conduit 59d->60d, ubuntu-server-xfce 75d->76d. Serve table still matches tailscale serve status exactly. - infrastructure/hosts.md: skills figure corrected to 302 / 68 categories. - index.md: skills-index bullet refreshed to the 2026-11-05 figures. New incident: incidents/2026-11-05-skills-index-generator-divergence.md (closes the still-open item on the 2026-08-15 skills-index drift incident). Freshness: 0 stale / 123 fresh / 3 no-date of 126 pages.
This commit is contained in:
@@ -3,10 +3,10 @@ title: "Skills Index Drift — 4x Undercount"
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-08-15
|
||||
updated: 2026-10-23
|
||||
updated: 2026-11-05
|
||||
resolved_on: 2026-08-15
|
||||
reopened_on: 2026-10-23
|
||||
verified_on: 2026-10-23
|
||||
verified_on: 2026-11-05
|
||||
confidence: high
|
||||
tags: [skills, wiki, drift, auto-regeneration]
|
||||
sources: [session-investigation, agent-estate-wiki, live-filesystem-probe, cron-executions-db]
|
||||
@@ -88,9 +88,14 @@ from the scan, prints the archived count as an explicit exclusion, and states th
|
||||
filesystem inventory. The page was regenerated through the patched script and
|
||||
committed with the fix.
|
||||
|
||||
**Still open:**
|
||||
- The stale `~/.hermes/scripts/wiki-maintain.sh` copy must never be run — it produces
|
||||
the divergent shape. It should be deleted or made to delegate to the repo script.
|
||||
**Still open (as of 2026-10-23):**
|
||||
- ~~The stale `~/.hermes/scripts/wiki-maintain.sh` copy must never be run — it produces
|
||||
the divergent shape. It should be deleted or made to delegate to the repo script.~~
|
||||
**✅ CLOSED 2026-11-05** — the stale copy is now a **delegating wrapper** with no generator logic of its
|
||||
own; it `exec`s the repo script. The repo script gained `--no-build` so the wrapper can honour the
|
||||
interface its callers used without carrying a second copy of the generator. Verified by running the
|
||||
wrapper: it now produces the canonical **302 skills / 68 categories** shape.
|
||||
→ [[incidents/2026-11-05-skills-index-generator-divergence]]
|
||||
- The page will still read higher than the loader's 329: nested collections
|
||||
(`context-engineering-collection/…`), `apple/` and hub-installed trees are counted as
|
||||
installed files. Whether the page should instead report the loader's figure is a
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
title: "Skills Index Generator Divergence — Two Scripts, One Page"
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-11-05
|
||||
updated: 2026-11-05
|
||||
resolved_on: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
confidence: high
|
||||
tags: [skills, wiki, drift, auto-regeneration, root-cause]
|
||||
sources: [live-filesystem-probe, script-diff, wiki-maintain-run, agent-estate-wiki]
|
||||
---
|
||||
|
||||
# Skills Index Generator Divergence — Two Scripts, One Page
|
||||
|
||||
## Summary
|
||||
`systems/skills-index.md` flipped between two different shapes on alternating days — **302 skills / 68
|
||||
categories** on some runs and **349 skills / 26 categories** on others. The cause was **two different
|
||||
scripts** writing the same page, each with its own generator, and the two scheduled jobs pointing at
|
||||
different ones. Whichever job ran last won, so the page oscillated nightly.
|
||||
|
||||
This is the **same class of defect** as [[incidents/2026-08-15-skills-index-drift]], but a different and
|
||||
previously-unidentified root cause. The 2026-08-15 incident attributed the drift to archived skills being
|
||||
counted; that was a *symptom* of one of the two generators, not the mechanism.
|
||||
|
||||
## Root Cause
|
||||
Two files both named `wiki-maintain.sh`, each containing a **complete, independent copy** of the page
|
||||
generator:
|
||||
|
||||
| Script | Generator technique | Output |
|
||||
|---|---|---|
|
||||
| `/home/hermes/workspace/agent-estate-wiki/wiki-maintain.sh` (repo, canonical, mtime 2026-11-03) | `find` scan **excluding** `*/.archive/*`; dedupes by `(category, skill_name)` | **302 skills / 68 categories** |
|
||||
| `/home/hermes/.hermes/scripts/wiki-maintain.sh` (stale copy, mtime **2026-08-17**) | `Path.rglob('SKILL.md')` — **includes** the parked `~/.hermes/skills/.archive/` tree; 2-segment category heuristic | **349 skills / 26 categories** |
|
||||
|
||||
The two scripts are **not** identical — `diff` between them returns a non-zero exit and shows entirely
|
||||
different generator bodies, different category heuristics and different output formats.
|
||||
|
||||
**The trigger:** `~/.hermes/skills/devops/wiki-maintenance/SKILL.md` (line 569) instructed the nightly audit
|
||||
job (`cbfe434a3b5e`, daily 23:59) to run:
|
||||
|
||||
```bash
|
||||
run `bash ~/.hermes/scripts/wiki-maintain.sh` (it regenerates `systems/skills-index.md`, writes a freshness
|
||||
```
|
||||
|
||||
— i.e. the **stale** copy. The 03:00 stocktake job (`1a8e90ea2271`) runs the **repo** script. So:
|
||||
|
||||
- 03:00 stocktake → repo script → page becomes **302 / 68**
|
||||
- 23:59 nightly audit → stale script → page becomes **349 / 26**
|
||||
- repeat, forever
|
||||
|
||||
Each job was individually "successful" — neither reported an error. The defect was only visible as a
|
||||
day-over-day oscillation in the committed page.
|
||||
|
||||
**Live counts verified 2026-11-05 05:1x AWST** (filesystem probes):
|
||||
|
||||
- `find ~/.hermes/skills -name SKILL.md` → **349** (includes archive)
|
||||
- `find ~/.hermes/skills -name SKILL.md -not -path '*/.archive/*'` → **303**
|
||||
- the generator de-dupes one nested alias duplicate → **302 unique** ✅
|
||||
- `~/.hermes/skills/.archive/` → **46** parked skills (excluded by the canonical generator)
|
||||
|
||||
## Impact
|
||||
- The published skills inventory was **wrong roughly half the time**, understating the real category
|
||||
topology (26 vs 68 categories) and overstating the count by including 47 archived skills.
|
||||
- Any agent reading the page to reason about the estate's skill surface got a materially different
|
||||
picture depending on which run last wrote it.
|
||||
- **No functional impact** — skill loading was never affected; only the documentation was wrong.
|
||||
|
||||
## Resolution
|
||||
1. **`~/.hermes/scripts/wiki-maintain.sh` is now a delegating wrapper.** It contains **no generator logic** —
|
||||
it `exec`s the canonical repo script and forwards arguments. The wrapper's header comment explains why
|
||||
it must stay that way, so the divergent copy is not recreated.
|
||||
2. **`--no-build` is now supported by the repo script.** The stale copy accepted `--no-build` (regenerate +
|
||||
audit, skip the Quartz build/deploy); the repo script did not. The repo script gained the flag so the
|
||||
wrapper can honour the interface its callers were written against **without** carrying a second copy of
|
||||
the generator. This closes the only functional gap the delegation could otherwise have introduced.
|
||||
3. **Verified by execution, not by inspection:** running the wrapper with `--no-build` produces the canonical
|
||||
shape — `302 skills in 68 categories` — and the generated page reports `46 archived skills` excluded.
|
||||
Both scripts pass `bash -n`.
|
||||
4. **`systems/skills-index.md` regenerated through the canonical path** and committed in the canonical shape.
|
||||
|
||||
## Lesson
|
||||
**A generator must live in exactly one place.** Two copies of the same script, differing in one technique
|
||||
(`rglob` vs `find`), produced two valid-looking outputs and a page that flipped daily — with **no error
|
||||
anywhere**, because each job did exactly what it was told.
|
||||
|
||||
When a scheduled job's output disagrees with another job's output for the same artefact, the cause is
|
||||
almost never the artefact — it is **two writers**. Diff the scripts before diffing the data.
|
||||
|
||||
Related: [[incidents/2026-08-15-skills-index-drift]] (the original drift), [[systems/skills-index]],
|
||||
[[systems/scheduled-tasks]].
|
||||
Reference in New Issue
Block a user