ROOT CAUSE FIXED - systems/skills-index.md had TWO WRITERS: the repo wiki-maintain.sh (302 skills / 68 categories) and the stale ~/.hermes/scripts/wiki-maintain.sh (mtime 2026-08-17, 349 skills / 26 categories - it used Path.rglob() and counted the parked .archive/ tree) were different scripts with different generators, and wiki-maintenance/SKILL.md line 569 pointed the 23:59 nightly audit at the stale path while the 03:00 stocktake ran the repo path. Each job overwrote the other's page; neither raised an error. The stale copy is now a delegating wrapper with no generator logic, and the repo script gained --no-build so the wrapper honours the old interface without a second generator. Verified by execution. Live-probe corrections: - infrastructure/docker-services.md: CT200 61 running / 68 total (was 62); worldmonitor-seeder-scheduled does not exist on ANY host (zero 'seeder' matches across CT200/201/333/460); CT333 first container renamed again (nostalgic_leavitt -> flamboyant_mahavira); CT201 20/26, CT460 1/1, CT1001 1/9. - infrastructure/tailscale.md: node table verified programmatically - 52 rows vs 52 live IPv4 peers, no live peer missing, no orphan row, every online/offline state matching. conduit 59d->60d, ubuntu-server-xfce 75d->76d. Serve table still matches tailscale serve status exactly. - infrastructure/hosts.md: skills figure corrected to 302 / 68 categories. - index.md: skills-index bullet refreshed to the 2026-11-05 figures. New incident: incidents/2026-11-05-skills-index-generator-divergence.md (closes the still-open item on the 2026-08-15 skills-index drift incident). Freshness: 0 stale / 123 fresh / 3 no-date of 126 pages.
91 lines
5.0 KiB
Markdown
91 lines
5.0 KiB
Markdown
---
|
|
title: "Skills Index Generator Divergence — Two Scripts, One Page"
|
|
type: incident
|
|
status: resolved
|
|
created: 2026-11-05
|
|
updated: 2026-11-05
|
|
resolved_on: 2026-11-05
|
|
verified_on: 2026-11-05
|
|
confidence: high
|
|
tags: [skills, wiki, drift, auto-regeneration, root-cause]
|
|
sources: [live-filesystem-probe, script-diff, wiki-maintain-run, agent-estate-wiki]
|
|
---
|
|
|
|
# Skills Index Generator Divergence — Two Scripts, One Page
|
|
|
|
## Summary
|
|
`systems/skills-index.md` flipped between two different shapes on alternating days — **302 skills / 68
|
|
categories** on some runs and **349 skills / 26 categories** on others. The cause was **two different
|
|
scripts** writing the same page, each with its own generator, and the two scheduled jobs pointing at
|
|
different ones. Whichever job ran last won, so the page oscillated nightly.
|
|
|
|
This is the **same class of defect** as [[incidents/2026-08-15-skills-index-drift]], but a different and
|
|
previously-unidentified root cause. The 2026-08-15 incident attributed the drift to archived skills being
|
|
counted; that was a *symptom* of one of the two generators, not the mechanism.
|
|
|
|
## Root Cause
|
|
Two files both named `wiki-maintain.sh`, each containing a **complete, independent copy** of the page
|
|
generator:
|
|
|
|
| Script | Generator technique | Output |
|
|
|---|---|---|
|
|
| `/home/hermes/workspace/agent-estate-wiki/wiki-maintain.sh` (repo, canonical, mtime 2026-11-03) | `find` scan **excluding** `*/.archive/*`; dedupes by `(category, skill_name)` | **302 skills / 68 categories** |
|
|
| `/home/hermes/.hermes/scripts/wiki-maintain.sh` (stale copy, mtime **2026-08-17**) | `Path.rglob('SKILL.md')` — **includes** the parked `~/.hermes/skills/.archive/` tree; 2-segment category heuristic | **349 skills / 26 categories** |
|
|
|
|
The two scripts are **not** identical — `diff` between them returns a non-zero exit and shows entirely
|
|
different generator bodies, different category heuristics and different output formats.
|
|
|
|
**The trigger:** `~/.hermes/skills/devops/wiki-maintenance/SKILL.md` (line 569) instructed the nightly audit
|
|
job (`cbfe434a3b5e`, daily 23:59) to run:
|
|
|
|
```bash
|
|
run `bash ~/.hermes/scripts/wiki-maintain.sh` (it regenerates `systems/skills-index.md`, writes a freshness
|
|
```
|
|
|
|
— i.e. the **stale** copy. The 03:00 stocktake job (`1a8e90ea2271`) runs the **repo** script. So:
|
|
|
|
- 03:00 stocktake → repo script → page becomes **302 / 68**
|
|
- 23:59 nightly audit → stale script → page becomes **349 / 26**
|
|
- repeat, forever
|
|
|
|
Each job was individually "successful" — neither reported an error. The defect was only visible as a
|
|
day-over-day oscillation in the committed page.
|
|
|
|
**Live counts verified 2026-11-05 05:1x AWST** (filesystem probes):
|
|
|
|
- `find ~/.hermes/skills -name SKILL.md` → **349** (includes archive)
|
|
- `find ~/.hermes/skills -name SKILL.md -not -path '*/.archive/*'` → **303**
|
|
- the generator de-dupes one nested alias duplicate → **302 unique** ✅
|
|
- `~/.hermes/skills/.archive/` → **46** parked skills (excluded by the canonical generator)
|
|
|
|
## Impact
|
|
- The published skills inventory was **wrong roughly half the time**, understating the real category
|
|
topology (26 vs 68 categories) and overstating the count by including 47 archived skills.
|
|
- Any agent reading the page to reason about the estate's skill surface got a materially different
|
|
picture depending on which run last wrote it.
|
|
- **No functional impact** — skill loading was never affected; only the documentation was wrong.
|
|
|
|
## Resolution
|
|
1. **`~/.hermes/scripts/wiki-maintain.sh` is now a delegating wrapper.** It contains **no generator logic** —
|
|
it `exec`s the canonical repo script and forwards arguments. The wrapper's header comment explains why
|
|
it must stay that way, so the divergent copy is not recreated.
|
|
2. **`--no-build` is now supported by the repo script.** The stale copy accepted `--no-build` (regenerate +
|
|
audit, skip the Quartz build/deploy); the repo script did not. The repo script gained the flag so the
|
|
wrapper can honour the interface its callers were written against **without** carrying a second copy of
|
|
the generator. This closes the only functional gap the delegation could otherwise have introduced.
|
|
3. **Verified by execution, not by inspection:** running the wrapper with `--no-build` produces the canonical
|
|
shape — `302 skills in 68 categories` — and the generated page reports `46 archived skills` excluded.
|
|
Both scripts pass `bash -n`.
|
|
4. **`systems/skills-index.md` regenerated through the canonical path** and committed in the canonical shape.
|
|
|
|
## Lesson
|
|
**A generator must live in exactly one place.** Two copies of the same script, differing in one technique
|
|
(`rglob` vs `find`), produced two valid-looking outputs and a page that flipped daily — with **no error
|
|
anywhere**, because each job did exactly what it was told.
|
|
|
|
When a scheduled job's output disagrees with another job's output for the same artefact, the cause is
|
|
almost never the artefact — it is **two writers**. Diff the scripts before diffing the data.
|
|
|
|
Related: [[incidents/2026-08-15-skills-index-drift]] (the original drift), [[systems/skills-index]],
|
|
[[systems/scheduled-tasks]].
|