ROOT CAUSE FIXED - systems/skills-index.md had TWO WRITERS: the repo wiki-maintain.sh (302 skills / 68 categories) and the stale ~/.hermes/scripts/wiki-maintain.sh (mtime 2026-08-17, 349 skills / 26 categories - it used Path.rglob() and counted the parked .archive/ tree) were different scripts with different generators, and wiki-maintenance/SKILL.md line 569 pointed the 23:59 nightly audit at the stale path while the 03:00 stocktake ran the repo path. Each job overwrote the other's page; neither raised an error. The stale copy is now a delegating wrapper with no generator logic, and the repo script gained --no-build so the wrapper honours the old interface without a second generator. Verified by execution. Live-probe corrections: - infrastructure/docker-services.md: CT200 61 running / 68 total (was 62); worldmonitor-seeder-scheduled does not exist on ANY host (zero 'seeder' matches across CT200/201/333/460); CT333 first container renamed again (nostalgic_leavitt -> flamboyant_mahavira); CT201 20/26, CT460 1/1, CT1001 1/9. - infrastructure/tailscale.md: node table verified programmatically - 52 rows vs 52 live IPv4 peers, no live peer missing, no orphan row, every online/offline state matching. conduit 59d->60d, ubuntu-server-xfce 75d->76d. Serve table still matches tailscale serve status exactly. - infrastructure/hosts.md: skills figure corrected to 302 / 68 categories. - index.md: skills-index bullet refreshed to the 2026-11-05 figures. New incident: incidents/2026-11-05-skills-index-generator-divergence.md (closes the still-open item on the 2026-08-15 skills-index drift incident). Freshness: 0 stale / 123 fresh / 3 no-date of 126 pages.
109 lines
6.0 KiB
Markdown
109 lines
6.0 KiB
Markdown
---
|
|
title: "Skills Index Drift — 4x Undercount"
|
|
type: incident
|
|
status: active
|
|
created: 2026-08-15
|
|
updated: 2026-11-05
|
|
resolved_on: 2026-08-15
|
|
reopened_on: 2026-10-23
|
|
verified_on: 2026-11-05
|
|
confidence: high
|
|
tags: [skills, wiki, drift, auto-regeneration]
|
|
sources: [session-investigation, agent-estate-wiki, live-filesystem-probe, cron-executions-db]
|
|
---
|
|
|
|
# Skills Index Drift — 4x Undercount
|
|
|
|
## Summary
|
|
The `systems/skills-index.md` page in both wikis claimed **66 skills** when **330 were actually installed** — a 4x undercount. The page had been hand-written once on 2026-07-22/25 and never regenerated as the estate grew.
|
|
|
|
## Root Cause
|
|
Static documentation for a mutable inventory. The skills tree at `~/.hermes/skills/` grew from ~66 to 330 skills over three weeks, but the wiki page was never updated because there was no automation to keep it in sync.
|
|
|
|
## Impact
|
|
- Wiki page was 4x out of date
|
|
- Any inventory-based decisions (skill loading, capacity planning) would have been based on wrong numbers
|
|
- No functional impact — skills still loaded correctly via Tool Search
|
|
|
|
## Resolution
|
|
1. Regenerated `systems/skills-index.md` from the live skill tree (all 330 skills, 26 categories)
|
|
2. Created a regeneration script for automated regeneration — **superseded on 2026-10-21:** the
|
|
regeneration now lives inside `wiki-maintain.sh` (step 1/3) in the wiki repo; no standalone
|
|
`regenerate-skills-index.sh` exists on disk any more.
|
|
3. Set up cron job `1a8e90ea2271` to run daily at 03:00 AWST
|
|
4. Script auto-commits and pushes to Gitea (`Anthony/agent-estate-wiki`)
|
|
5. Quartz site at `http://100.118.5.51:9120/systems/skills-index` rebuilt and redeployed
|
|
|
|
## Lesson
|
|
Static wiki pages that reflect **mutable inventory** (skills, services, cron jobs) **MUST be auto-regenerated on a schedule**, not hand-maintained. The moment the estate grows, a hand-written page becomes stale.
|
|
|
|
## Re-verified 2026-10-21
|
|
The prevention mechanism is live and working. `wiki-maintain.sh` step 1/3 regenerated
|
|
`systems/skills-index.md` from `~/.hermes/skills/` on the 2026-10-21 03:00 scheduled run:
|
|
**347 skills across 70 categories** (the "26 categories" figure in the original write-up was
|
|
the count at the time of the incident, not the current topology — the skill tree has since
|
|
nested deeper). The daily cron job `1a8e90ea2271` is `active` with `last_status: ok`.
|
|
|
|
## Prevention
|
|
- Daily cron job regenerates skills-index.md from live tree
|
|
- Page includes "auto-regenerated, do not hand-edit" warning
|
|
- Same script pattern is reusable for other inventory pages (cron jobs, services, agents)
|
|
|
|
## Reopened 2026-10-23 — the count includes archived skills, and two generators disagree
|
|
|
|
The 2026-10-21 all-clear above was premature. Two defects were confirmed on the
|
|
night of 2026-10-22/23:
|
|
|
|
**1. The stocktake ran six times long and committed nothing.** Cron job
|
|
`1a8e90ea2271` ran **03:00:28 → 04:38:19 AWST on 2026-10-22 (97 m 58 s)**, against
|
|
**16 m 01 s** on 2026-10-21, and finished `completed` with delivery `suppressed` —
|
|
but `git status` in the wiki repo still shows the run's regenerated
|
|
`systems/skills-index.md` (mtime 10-22 03:00) **uncommitted**, and none of the fixes
|
|
the run had diagnosed were applied. A `completed` row is therefore not evidence that
|
|
the stocktake did its job.
|
|
|
|
**2. Two scripts generate the same page with different shapes.**
|
|
|
|
| Generator | Skill scan | 2026-10-22 output |
|
|
|---|---|---|
|
|
| `/home/hermes/workspace/agent-estate-wiki/wiki-maintain.sh` (the one the cron job executes) | `find … -name SKILL.md` — does **not** prune dot-directories | **347** skills, with a `## .Archive (41)` section |
|
|
| `/home/hermes/.hermes/scripts/wiki-maintain.sh` (stale copy, mtime 2026-08-17) | `Path.rglob('SKILL.md')` | **348** skills, no archive section |
|
|
|
|
The 00:18 maintenance step committed the `rglob` shape; the 03:00 stocktake wrote the
|
|
`find` shape. Whoever runs last wins, so the page oscillates.
|
|
|
|
**Live counts, verified 2026-10-23 00:1x AWST** (filesystem probes + `hermes skills list`):
|
|
|
|
- `find ~/.hermes/skills -name SKILL.md` → **348**
|
|
- `find -L ~/.hermes/skills -name SKILL.md` → **376** (28 more skills reachable only through symlinks)
|
|
- excluding `.archive/` (**41**) and `apple/` (**4**) → **331**
|
|
- the loader's own catalogue — `hermes skills list` → **329 enabled** (`34 hub-installed, 52 builtin, 243 local — 329 enabled, 0 disabled`)
|
|
|
|
So the page total counts **45 artefacts the loader does not expose** (the parked
|
|
`.archive/` set plus `apple/`), which is why its number has never matched the loader.
|
|
|
|
**Fix applied 2026-10-23:** the repo's `wiki-maintain.sh` now excludes `*/.archive/*`
|
|
from the scan, prints the archived count as an explicit exclusion, and states that
|
|
`hermes skills list` is the authoritative *loadable* figure while the page remains a
|
|
filesystem inventory. The page was regenerated through the patched script and
|
|
committed with the fix.
|
|
|
|
**Still open (as of 2026-10-23):**
|
|
- ~~The stale `~/.hermes/scripts/wiki-maintain.sh` copy must never be run — it produces
|
|
the divergent shape. It should be deleted or made to delegate to the repo script.~~
|
|
**✅ CLOSED 2026-11-05** — the stale copy is now a **delegating wrapper** with no generator logic of its
|
|
own; it `exec`s the repo script. The repo script gained `--no-build` so the wrapper can honour the
|
|
interface its callers used without carrying a second copy of the generator. Verified by running the
|
|
wrapper: it now produces the canonical **302 skills / 68 categories** shape.
|
|
→ [[incidents/2026-11-05-skills-index-generator-divergence]]
|
|
- The page will still read higher than the loader's 329: nested collections
|
|
(`context-engineering-collection/…`), `apple/` and hub-installed trees are counted as
|
|
installed files. Whether the page should instead report the loader's figure is a
|
|
decision for Anthony.
|
|
- Watch the **2026-10-23 03:00** stocktake run: it should complete in ~16 minutes and
|
|
commit the regenerated page.
|
|
|
|
## Related
|
|
- [[systems/skills-index]] — The updated page
|
|
- [[systems/scheduled-tasks]] — The new cron job
|