wiki audit 2026-10-07: stocktake repin verified by live run (344->346 skills), fleet 0 failing; vault skills-index re-mirrored

This commit is contained in:
Tony0410
2026-10-07 00:15:06 +08:00
parent 33f4e95d95
commit 01f1db3aae
4 changed files with 37 additions and 12 deletions
+9 -3
View File
@@ -3,8 +3,8 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-10-06
verified_on: 2026-10-06
updated: 2026-10-07
verified_on: 2026-10-07
review_after: 2026-10-31
confidence: high
tags: [system]
@@ -12,7 +12,13 @@ sources: [live-config, live-cron-list, live-hermes-version]
---
# Current State
Updated: 2026-10-06
Updated: 2026-10-07
## Recent Changes (2026-10-07)
- **Cron fleet green for a second consecutive day — 0 failing.** Job count unchanged at 30 (27 enabled / 3 disabled) and all 30 `failure_streak` counters read 0. Window tally (`executions.db`, 2026-10-06 00:00 → 2026-10-07 00:00 AWST): **180 `completed`, 0 `failed`, 1 `running`** (the nightly audit itself). **No interactive (non-cron) sessions** in the window — every session was cron. No new incidents raised or resolved in the window.
- **The stocktake job proved its repin — first fully automatic run since 2026-09-25.** `1a8e90ea2271` (`wiki-stocktake-maintenance`, 03:00 daily) ran on schedule **2026-10-06 03:00:31 → 03:03:32** on its new `custom:omni-101` / `free-stack` pin and completed cleanly. Two of its own incidents (`069d7834bd68`, `7061381d4259`) flipped to `resolved` at 03:03:31. It regenerated `systems/skills-index.md` (**344 → 346 skills**), wrote `raw/stocktake/freshness-20261006.txt`, rebuilt Quartz and committed to Gitea as `33f4e95`. Its stale pre-repin `last_status: error` is now cleared — live `last_status: ok`, streak 0. **The daily manual `wiki-maintain.sh` run is no longer needed.** See [[systems/scheduled-tasks]].
- **Three jobs still carry a stale pre-repin `last_status: error`** — `1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd` (down from four). Each clears on its next scheduled slot (Fri 08:00 / Mon 03:00 / 1st 09:00); none is a live failure. The bare-`telegram` target warning still stands on `066bb5191280` and `0e753e9720fd` (unchanged).
- **Wiki vault mirror refreshed.** `systems/skills-index.md` in `/home/hermes/wiki/agent-estate-wiki/` had drifted to 2026-10-03 / 344 skills while the canonical copy regenerated to 2026-10-06 / 346; this audit re-mirrored it along with the pages below. The two-wiki-tree drift remains a standing open item.
## Recent Changes (2026-10-06)
- **Dead model pins cleared across the whole fleet — failing set 7 → 0.** All seven jobs that had been failing since 2026-10-01 were repinned on 2026-10-05 23:16 → 2026-10-06 00:05. Each had been pinned to a raw model name that died upstream: `cliprox/terra` (HTTP 401 connection auth, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`). They were moved to `custom:omni-101` on **routing aliases** — `free-stack` ×6, `smart-route` ×1 — because the fault was raw-pin brittleness, not any one model. Every `failure_streak` in `jobs.json` now reads 0. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The other four are repinned but not yet re-run, so they carry a stale `last_status: error` until their next scheduled slot. See [[incidents/2026-10-05-dead-model-pin-repin]].
+3 -3
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-06
> model routing and automation. Last updated: 2026-10-07
## Start here
- [[current-state]] — Current systems status
@@ -28,8 +28,8 @@
### Integrations
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; **0 failing as of 2026-10-06** — 7 dead-model-pin jobs repinned 2026-10-05)
- [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily; being refreshed by hand while the stocktake job is blocked)
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; **0 failing as of 2026-10-07** — 7 dead-model-pin jobs repinned 2026-10-05, 4 now verified by live run)
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-06)
## Incidents
- [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05)
+18
View File
@@ -357,3 +357,21 @@ Reviewed window: 2026-10-05 00:04 → 2026-10-06 00:15 AWST.
**Vault mirror:** the new incident page plus `systems/scheduled-tasks.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by connection ID only.
## [2026-10-07] update | Nightly wiki audit — stocktake job verified by live run, fleet green a second day
Reviewed window: 2026-10-06 00:00 → 2026-10-07 00:00 AWST.
**Updated**
- `systems/scheduled-tasks.md` — Count block corrected: the stale pre-repin `last_status: error` set is now **three** (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`), not four — `1a8e90ea2271` cleared its own when it ran at 03:00. Added a **Re-verified 2026-10-07 AWST** bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 180 `completed` / 0 `failed` / 1 `running`; no interactive sessions). The stocktake job-table row and the "Verified live, not assumed" bullet now record the live-run verification. Historic count progression extended.
- `index.md` — last-updated → 2026-10-07; cron-fleet line re-dated (**0 failing as of 2026-10-07**, 4 of the 7 verified by live run); the skills-index line no longer reads "being refreshed by hand while the stocktake job is blocked" — it now names 346 skills and the auto-regeneration.
- `current-state.md` — new Recent Changes (2026-10-07) block; frontmatter dates bumped.
**Findings worth the ink**
- **`1a8e90ea2271` ran itself at 03:00 and proved the repin.** `wiki-stocktake-maintenance` completed **2026-10-06 03:00:31 → 03:03:32** on `custom:omni-101` / `free-stack` — its first automatic success since 2026-09-25 (it had failed HTTP 410 ×11 on the retired `qwen3.5:397b`). It regenerated `systems/skills-index.md` (**344 → 346 skills**), wrote `raw/stocktake/freshness-20261006.txt`, rebuilt Quartz and committed to Gitea as `33f4e95`. Two of its own incidents (`069d7834bd68`, `7061381d4259`) auto-resolved at 03:03:31. **The daily manual `wiki-maintain.sh` run can stop.**
- **Vault mirror drift found and partly repaired.** `systems/skills-index.md` in `/home/hermes/wiki/agent-estate-wiki/` was still 2026-10-03 / 344 skills while the canonical copy had regenerated to 2026-10-06 / 346 — a knock-on of the stocktake having been blocked. Re-mirrored in this audit; the underlying two-wiki-tree split remains a standing open item.
- **Window tally** (`executions.db`, 2026-10-06 00:00 → 2026-10-07 00:00): 180 `completed`, 0 `failed`, 1 `running` (this audit). No interactive (non-cron) sessions; no new incidents raised or resolved in the window.
**Vault mirror:** `systems/scheduled-tasks.md`, `systems/skills-index.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by connection ID only.
+7 -6
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-06
verified_on: 2026-10-06
updated: 2026-10-07
verified_on: 2026-10-07
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,15 +14,16 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
## Count
- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-10-06)
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven still carry a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. See [[incidents/2026-10-05-dead-model-pin-repin]].
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. See [[incidents/2026-10-05-dead-model-pin-repin]].
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window.
- **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs.
- **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07). The other four are repinned but not yet re-run — they are due on their normal schedules.
- **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07); and `1a8e90ea2271` completed automatically on **2026-10-06 03:00**. Four of the seven are therefore verified by live run. The remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are repinned but not yet re-run — they are due on their normal schedules.
- **Streak counting note:** the `N failures in a row` figure in `hermes cron list` comes from the `failure_streak` counter in `jobs.json`. `executions.db` retains only rows from **2026-09-30 onward**, so DB-based recounts undercount older streaks. Live counters as of 2026-10-06: **all 0**.
- **Re-verified 2026-10-06 AWST:** job count unchanged at 30 (27 enabled / 3 disabled). `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found`).
- **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 180 `completed`, 0 `failed`, 1 `running` (the audit itself); **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`).
- **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice.
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06)
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07)
## Job Table
@@ -56,7 +57,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 |
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). Resumes its 03:00 schedule, so `wiki-maintain.sh` no longer needs a daily manual run. Streak 0; stale `last_status: error` until 03:00. Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea — **the script itself was never broken** |
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea — **the script itself was never broken** |
| 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
## Removed Jobs