ROOT CAUSE FIXED - systems/skills-index.md had TWO WRITERS: the repo wiki-maintain.sh (302 skills / 68 categories) and the stale ~/.hermes/scripts/wiki-maintain.sh (mtime 2026-08-17, 349 skills / 26 categories - it used Path.rglob() and counted the parked .archive/ tree) were different scripts with different generators, and wiki-maintenance/SKILL.md line 569 pointed the 23:59 nightly audit at the stale path while the 03:00 stocktake ran the repo path. Each job overwrote the other's page; neither raised an error. The stale copy is now a delegating wrapper with no generator logic, and the repo script gained --no-build so the wrapper honours the old interface without a second generator. Verified by execution. Live-probe corrections: - infrastructure/docker-services.md: CT200 61 running / 68 total (was 62); worldmonitor-seeder-scheduled does not exist on ANY host (zero 'seeder' matches across CT200/201/333/460); CT333 first container renamed again (nostalgic_leavitt -> flamboyant_mahavira); CT201 20/26, CT460 1/1, CT1001 1/9. - infrastructure/tailscale.md: node table verified programmatically - 52 rows vs 52 live IPv4 peers, no live peer missing, no orphan row, every online/offline state matching. conduit 59d->60d, ubuntu-server-xfce 75d->76d. Serve table still matches tailscale serve status exactly. - infrastructure/hosts.md: skills figure corrected to 302 / 68 categories. - index.md: skills-index bullet refreshed to the 2026-11-05 figures. New incident: incidents/2026-11-05-skills-index-generator-divergence.md (closes the still-open item on the 2026-08-15 skills-index drift incident). Freshness: 0 stale / 123 fresh / 3 no-date of 126 pages.
5.0 KiB
title, type, status, created, updated, resolved_on, verified_on, confidence, tags, sources
| title | type | status | created | updated | resolved_on | verified_on | confidence | tags | sources | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Skills Index Generator Divergence — Two Scripts, One Page | incident | resolved | 2026-11-05 | 2026-11-05 | 2026-11-05 | 2026-11-05 | high |
|
|
Skills Index Generator Divergence — Two Scripts, One Page
Summary
systems/skills-index.md flipped between two different shapes on alternating days — 302 skills / 68
categories on some runs and 349 skills / 26 categories on others. The cause was two different
scripts writing the same page, each with its own generator, and the two scheduled jobs pointing at
different ones. Whichever job ran last won, so the page oscillated nightly.
This is the same class of defect as incidents/2026-08-15-skills-index-drift, but a different and previously-unidentified root cause. The 2026-08-15 incident attributed the drift to archived skills being counted; that was a symptom of one of the two generators, not the mechanism.
Root Cause
Two files both named wiki-maintain.sh, each containing a complete, independent copy of the page
generator:
| Script | Generator technique | Output |
|---|---|---|
/home/hermes/workspace/agent-estate-wiki/wiki-maintain.sh (repo, canonical, mtime 2026-11-03) |
find scan excluding */.archive/*; dedupes by (category, skill_name) |
302 skills / 68 categories |
/home/hermes/.hermes/scripts/wiki-maintain.sh (stale copy, mtime 2026-08-17) |
Path.rglob('SKILL.md') — includes the parked ~/.hermes/skills/.archive/ tree; 2-segment category heuristic |
349 skills / 26 categories |
The two scripts are not identical — diff between them returns a non-zero exit and shows entirely
different generator bodies, different category heuristics and different output formats.
The trigger: ~/.hermes/skills/devops/wiki-maintenance/SKILL.md (line 569) instructed the nightly audit
job (cbfe434a3b5e, daily 23:59) to run:
run `bash ~/.hermes/scripts/wiki-maintain.sh` (it regenerates `systems/skills-index.md`, writes a freshness
— i.e. the stale copy. The 03:00 stocktake job (1a8e90ea2271) runs the repo script. So:
- 03:00 stocktake → repo script → page becomes 302 / 68
- 23:59 nightly audit → stale script → page becomes 349 / 26
- repeat, forever
Each job was individually "successful" — neither reported an error. The defect was only visible as a day-over-day oscillation in the committed page.
Live counts verified 2026-11-05 05:1x AWST (filesystem probes):
find ~/.hermes/skills -name SKILL.md→ 349 (includes archive)find ~/.hermes/skills -name SKILL.md -not -path '*/.archive/*'→ 303- the generator de-dupes one nested alias duplicate → 302 unique ✅
~/.hermes/skills/.archive/→ 46 parked skills (excluded by the canonical generator)
Impact
- The published skills inventory was wrong roughly half the time, understating the real category topology (26 vs 68 categories) and overstating the count by including 47 archived skills.
- Any agent reading the page to reason about the estate's skill surface got a materially different picture depending on which run last wrote it.
- No functional impact — skill loading was never affected; only the documentation was wrong.
Resolution
~/.hermes/scripts/wiki-maintain.shis now a delegating wrapper. It contains no generator logic — itexecs the canonical repo script and forwards arguments. The wrapper's header comment explains why it must stay that way, so the divergent copy is not recreated.--no-buildis now supported by the repo script. The stale copy accepted--no-build(regenerate + audit, skip the Quartz build/deploy); the repo script did not. The repo script gained the flag so the wrapper can honour the interface its callers were written against without carrying a second copy of the generator. This closes the only functional gap the delegation could otherwise have introduced.- Verified by execution, not by inspection: running the wrapper with
--no-buildproduces the canonical shape —302 skills in 68 categories— and the generated page reports46 archived skillsexcluded. Both scripts passbash -n. systems/skills-index.mdregenerated through the canonical path and committed in the canonical shape.
Lesson
A generator must live in exactly one place. Two copies of the same script, differing in one technique
(rglob vs find), produced two valid-looking outputs and a page that flipped daily — with no error
anywhere, because each job did exactly what it was told.
When a scheduled job's output disagrees with another job's output for the same artefact, the cause is almost never the artefact — it is two writers. Diff the scripts before diffing the data.
Related: incidents/2026-08-15-skills-index-drift (the original drift), systems/skills-index, systems/scheduled-tasks.