ROOT CAUSE FIXED - systems/skills-index.md had TWO WRITERS: the repo wiki-maintain.sh (302 skills / 68 categories) and the stale ~/.hermes/scripts/wiki-maintain.sh (mtime 2026-08-17, 349 skills / 26 categories - it used Path.rglob() and counted the parked .archive/ tree) were different scripts with different generators, and wiki-maintenance/SKILL.md line 569 pointed the 23:59 nightly audit at the stale path while the 03:00 stocktake ran the repo path. Each job overwrote the other's page; neither raised an error. The stale copy is now a delegating wrapper with no generator logic, and the repo script gained --no-build so the wrapper honours the old interface without a second generator. Verified by execution. Live-probe corrections: - infrastructure/docker-services.md: CT200 61 running / 68 total (was 62); worldmonitor-seeder-scheduled does not exist on ANY host (zero 'seeder' matches across CT200/201/333/460); CT333 first container renamed again (nostalgic_leavitt -> flamboyant_mahavira); CT201 20/26, CT460 1/1, CT1001 1/9. - infrastructure/tailscale.md: node table verified programmatically - 52 rows vs 52 live IPv4 peers, no live peer missing, no orphan row, every online/offline state matching. conduit 59d->60d, ubuntu-server-xfce 75d->76d. Serve table still matches tailscale serve status exactly. - infrastructure/hosts.md: skills figure corrected to 302 / 68 categories. - index.md: skills-index bullet refreshed to the 2026-11-05 figures. New incident: incidents/2026-11-05-skills-index-generator-divergence.md (closes the still-open item on the 2026-08-15 skills-index drift incident). Freshness: 0 stale / 123 fresh / 3 no-date of 126 pages.
6.0 KiB
title, type, status, created, updated, resolved_on, reopened_on, verified_on, confidence, tags, sources
| title | type | status | created | updated | resolved_on | reopened_on | verified_on | confidence | tags | sources | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Skills Index Drift — 4x Undercount | incident | active | 2026-08-15 | 2026-11-05 | 2026-08-15 | 2026-10-23 | 2026-11-05 | high |
|
|
Skills Index Drift — 4x Undercount
Summary
The systems/skills-index.md page in both wikis claimed 66 skills when 330 were actually installed — a 4x undercount. The page had been hand-written once on 2026-07-22/25 and never regenerated as the estate grew.
Root Cause
Static documentation for a mutable inventory. The skills tree at ~/.hermes/skills/ grew from ~66 to 330 skills over three weeks, but the wiki page was never updated because there was no automation to keep it in sync.
Impact
- Wiki page was 4x out of date
- Any inventory-based decisions (skill loading, capacity planning) would have been based on wrong numbers
- No functional impact — skills still loaded correctly via Tool Search
Resolution
- Regenerated
systems/skills-index.mdfrom the live skill tree (all 330 skills, 26 categories) - Created a regeneration script for automated regeneration — superseded on 2026-10-21: the
regeneration now lives inside
wiki-maintain.sh(step 1/3) in the wiki repo; no standaloneregenerate-skills-index.shexists on disk any more. - Set up cron job
1a8e90ea2271to run daily at 03:00 AWST - Script auto-commits and pushes to Gitea (
Anthony/agent-estate-wiki) - Quartz site at
http://100.118.5.51:9120/systems/skills-indexrebuilt and redeployed
Lesson
Static wiki pages that reflect mutable inventory (skills, services, cron jobs) MUST be auto-regenerated on a schedule, not hand-maintained. The moment the estate grows, a hand-written page becomes stale.
Re-verified 2026-10-21
The prevention mechanism is live and working. wiki-maintain.sh step 1/3 regenerated
systems/skills-index.md from ~/.hermes/skills/ on the 2026-10-21 03:00 scheduled run:
347 skills across 70 categories (the "26 categories" figure in the original write-up was
the count at the time of the incident, not the current topology — the skill tree has since
nested deeper). The daily cron job 1a8e90ea2271 is active with last_status: ok.
Prevention
- Daily cron job regenerates skills-index.md from live tree
- Page includes "auto-regenerated, do not hand-edit" warning
- Same script pattern is reusable for other inventory pages (cron jobs, services, agents)
Reopened 2026-10-23 — the count includes archived skills, and two generators disagree
The 2026-10-21 all-clear above was premature. Two defects were confirmed on the night of 2026-10-22/23:
1. The stocktake ran six times long and committed nothing. Cron job
1a8e90ea2271 ran 03:00:28 → 04:38:19 AWST on 2026-10-22 (97 m 58 s), against
16 m 01 s on 2026-10-21, and finished completed with delivery suppressed —
but git status in the wiki repo still shows the run's regenerated
systems/skills-index.md (mtime 10-22 03:00) uncommitted, and none of the fixes
the run had diagnosed were applied. A completed row is therefore not evidence that
the stocktake did its job.
2. Two scripts generate the same page with different shapes.
| Generator | Skill scan | 2026-10-22 output |
|---|---|---|
/home/hermes/workspace/agent-estate-wiki/wiki-maintain.sh (the one the cron job executes) |
find … -name SKILL.md — does not prune dot-directories |
347 skills, with a ## .Archive (41) section |
/home/hermes/.hermes/scripts/wiki-maintain.sh (stale copy, mtime 2026-08-17) |
Path.rglob('SKILL.md') |
348 skills, no archive section |
The 00:18 maintenance step committed the rglob shape; the 03:00 stocktake wrote the
find shape. Whoever runs last wins, so the page oscillates.
Live counts, verified 2026-10-23 00:1x AWST (filesystem probes + hermes skills list):
find ~/.hermes/skills -name SKILL.md→ 348find -L ~/.hermes/skills -name SKILL.md→ 376 (28 more skills reachable only through symlinks)- excluding
.archive/(41) andapple/(4) → 331 - the loader's own catalogue —
hermes skills list→ 329 enabled (34 hub-installed, 52 builtin, 243 local — 329 enabled, 0 disabled)
So the page total counts 45 artefacts the loader does not expose (the parked
.archive/ set plus apple/), which is why its number has never matched the loader.
Fix applied 2026-10-23: the repo's wiki-maintain.sh now excludes */.archive/*
from the scan, prints the archived count as an explicit exclusion, and states that
hermes skills list is the authoritative loadable figure while the page remains a
filesystem inventory. The page was regenerated through the patched script and
committed with the fix.
Still open (as of 2026-10-23):
The stale✅ CLOSED 2026-11-05 — the stale copy is now a delegating wrapper with no generator logic of its own; it~/.hermes/scripts/wiki-maintain.shcopy must never be run — it produces the divergent shape. It should be deleted or made to delegate to the repo script.execs the repo script. The repo script gained--no-buildso the wrapper can honour the interface its callers used without carrying a second copy of the generator. Verified by running the wrapper: it now produces the canonical 302 skills / 68 categories shape. → incidents/2026-11-05-skills-index-generator-divergence- The page will still read higher than the loader's 329: nested collections
(
context-engineering-collection/…),apple/and hub-installed trees are counted as installed files. Whether the page should instead report the loader's figure is a decision for Anthony. - Watch the 2026-10-23 03:00 stocktake run: it should complete in ~16 minutes and commit the regenerated page.
Related
- systems/skills-index — The updated page
- systems/scheduled-tasks — The new cron job