audit(2026-10-14): 10-13 broke the clean streak — wiki-stocktake failed on a dead opencode credential (HTTP 404, incident 1a8e90_d11698bd7275); stranded skills-index + freshness-20261013 recovered; retention floor moved to 2026-10-08 07:11

This commit is contained in:
Tony0410
2026-10-14 00:17:31 +08:00
parent 1df564e152
commit 828913e23b
6 changed files with 244 additions and 10 deletions
+11 -3
View File
@@ -3,8 +3,8 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-10-13
verified_on: 2026-10-13
updated: 2026-10-14
verified_on: 2026-10-14
review_after: 2026-10-25
confidence: high
tags: [system]
@@ -12,7 +12,15 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
---
# Current State
Updated: 2026-10-13
Updated: 2026-10-14
## Recent Changes (2026-10-13 → 2026-10-14, nightly audit; live probes 2026-10-14 00:00-00:30 AWST)
- **The four-day clean streak broke: 2026-10-13 = `167 completed` / 1 `failed`** (10-12 finalised `175 / 0`, 10-11 `172 / 0`, 10-10 `175 / 0`). The failure is `1a8e90ea2271` (wiki-stocktake-maintenance, 03:00) on `RuntimeError: HTTP 404: No active credentials for provider: opencode` — the **third** failure of that job in six days, all three on the opencode path (10-08 403 free-tier, 10-09 timeout, 10-13 404). New incident `1a8e90_d11698bd7275` (`detected`, streak 1, not alerted); it clears only on a successful 2026-10-14 03:00 run. → [[systems/scheduled-tasks]]
- **The failing run stranded its own generated artifacts.** Step 1 (skills-index) and step 2 (freshness report) wrote to disk, then the agent turn died → no commit, no push. `systems/skills-index.md` (date-only change; **346 skills / 70 categories** unchanged) and `raw/stocktake/freshness-20261013.txt` (**48 stale / 38 fresh / 7 no-date of 93 pages**, down from 54 stale) were committed by the 2026-10-14 audit. → [[systems/skills-index]]
- **`executions.db` retention floor moved to 2026-10-08 07:11** (`MAX_TERMINAL_EXECUTIONS = 1000`; 1,001 rows = 1,000 terminal + 1 in flight). Days older than the floor now undercount — 10-08 reads `126 completed / 1 failed`, a partial tally.
- **Non-clean `last_status` rows: 3 → 4** (cleared `fad5a86979d4`, `dca8482e4f76`; added `1a8e90ea2271`). **Open incidents: 2 → 3.** Fleet unchanged at 29 jobs / 26 enabled.
- **No interactive sessions for the third window running:** 15 sessions in the window, all `cron`, zero interactive.
## Recent Changes (2026-10-12 → 2026-10-13, nightly audit; live probes 2026-10-13 00:05-00:45 AWST)
+3 -3
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-13
> model routing and automation. Last updated: 2026-10-14
## Start here
- [[current-state]] — Current systems status
@@ -29,8 +29,8 @@
### Integrations
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **3 carry a non-clean `last_status`** as of 2026-10-13 00:15. **Three consecutive clean days: 2026-10-12 ran `174 completed` / 0 `failed`** (2026-10-11 finalised `172 / 0`), and **open incidents fell 5 → 2** on 2026-10-12 (three closed: both `deepseek-v4-flash` rows + the Job Radar timeout). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. ⚠ The 2026-10-12 clean sheet is *not* an all-clear — GitHub Trending Radar's Telegram token is revoked (`getMe` → 401) while the cron row stays green. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-13 against the stocktake's own `0eee5f1` commit: 346 skills / 70 categories). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **regenerated daily — the 10-10, 10-11 and 10-12 stocktake runs all committed**)
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **4 carry a non-clean `last_status`** as of 2026-10-14 00:15. **The clean streak broke on 2026-10-13: `167 completed` / 1 `failed`**, after 10-12 finalised `175 / 0` (10-11 `172 / 0`, 10-10 `175 / 0`) — `1a8e90ea2271` (wiki-stocktake-maintenance, 03:00) hit `HTTP 404: No active credentials for provider: opencode`, its **third failure in six days**, so **open incidents rose 2 → 3**. Two rows cleared: `fad5a86979d4` (10-12 03:05), `dca8482e4f76` (10-13 07:02). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. ⚠ A green cron row is not proof of delivery: GitHub Trending Radar's Telegram token is revoked (`getMe` → 401) while the row stays green — it next runs Mon 09:00. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
## Incidents
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)
+55
View File
@@ -598,3 +598,58 @@ history entries), `systems/mnemosyne.md` (12:02 coverage), `incidents/2026-10-09
(status + full closure table + timeline), `index.md` (date, four blurbs), `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-14] audit | nightly review of 2026-10-13 — the clean streak broke on a dead opencode credential
Window 2026-10-13 00:00 → 23:59 AWST (ran 2026-10-13 23:59 → 2026-10-14 00:2x AWST). Live probes only.
**Cron.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row in flight → the day finalises
at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0` — **the first failed run since 2026-10-09**
(which recorded 15). Retention floor moved to **2026-10-08 07:11** (`MAX_TERMINAL_EXECUTIONS = 1000`; 1,001 rows
= 1,000 terminal + 1 in flight), so days above the floor are complete and older ones undercount (10-08 reads
`126 / 1`, partial).
**Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.** 03:00:43 →
03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.` Pinned
`custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`, `failure_type: agent`,
`alerted_at: NULL` — streak 1, under the 3-strike alert rule). Third failure of this job in six days, all three
on the opencode path (10-08 `HTTP 403: OpenCode's free tier can only be used from within OpenCode`; 10-09
timeout; 10-13 404). The first two resolved on 2026-10-10; this one clears only on a successful 2026-10-14 03:00
run. → [[incidents/2026-10-05-dead-model-pin-repin]]
**Finding 2 — the failure stranded its own generated artifacts (now committed).** Step 1 (skills-index) and
step 2 (freshness report) both completed, then the agent turn failed → no commit, no push. Left in the working
tree: `systems/skills-index.md` (modified, **date-only**: `updated`/`verified_on` 10-12 → 10-13; **346 skills /
70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked; **48 stale / 38 fresh /
7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are committed by this audit.
Consequence: the stocktake published nothing on 10-13 — and because the *agent* row is what commits, even a
perfect script run leaves nothing behind when the LLM turn dies.
**Delivery.** `159 suppressed` / **9 delivered** / 1 no-record. Delivered: Daily Marketing Job Radar,
Mosquito Tracker ×3, Tech & AI Newsletter Digest, Daily Birthday Checker, Gmail Action Digest, FreshRSS Midday,
Evening Stand-Down Briefing. No unexpected delivery; the two dead-thread faults (`0e753e9720fd`,
`066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 2026-10-01.
**Mnemosyne.** Six monitor runs, all silent: Health Monitor 00:02 + 12:02, Embedding Coverage Monitor 00:02 /
06:02 / 12:01 / 18:01 → no exception, so both coverages stayed at/above the 95 % floor and no `reindex` was
needed. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic
1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the silent token markdown-bolded rather than
bare; the delivery layer still suppressed it (`delivery_outcome: suppressed`) — the matcher tolerates wrapping
rather than requiring the literal token as a strict control. → [[systems/mnemosyne]]
**Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive
session.
**Fleet.** 29 jobs (26 enabled / 3 disabled), pins unchanged; **non-clean `last_status` 3 → 4** (cleared
`fad5a86979d4` 10-12 03:05 and `dca8482e4f76` 10-13 07:02; added `1a8e90ea2271`); **open incidents 2 → 3**
(1 `alerted`, 2 `detected`, 8 `closed`, 66 `resolved`; 77 rows).
**Pages touched:** `systems/scheduled-tasks.md` (frontmatter, retention floor, non-clean list,
`## Re-verified 2026-10-14`), `systems/skills-index.md` (committed stocktake artifact), `current-state.md`
(new Recent Changes + date), `index.md` (date + two blurbs), `log.md`.
**Vault mirror.** The 2026-10-13 audit was never mirrored to `/home/hermes/wiki` (last mirror `bb09581`,
2026-10-12), so this run mirrored both nights: the 10-13 audit's ten pages plus `skills-index.md`, whose vault
copy was still the 2026-10-08 version. The vault's own `raw/stocktake/` still holds only the 2026-10-03 report.
No secrets written; job/incident IDs, paths and tailnet addresses only.
+106
View File
@@ -0,0 +1,106 @@
# Wiki freshness audit — 2026-10-13
Threshold: >21 days since `updated`
## STALE (48)
- [ 83d] comparisons/camoufox-vs-chrome.md (updated 2026-07-22)
- [ 83d] comparisons/hermes-vs-nanobot.md (updated 2026-07-22)
- [ 83d] concepts/agent-memory.md (updated 2026-07-22)
- [ 83d] concepts/model-routing.md (updated 2026-07-22)
- [ 83d] concepts/persistent-shells.md (updated 2026-07-22)
- [ 83d] concepts/tool-calling.md (updated 2026-07-22)
- [ 83d] decisions/index.md (updated 2026-07-22)
- [ 83d] decisions/keep-openrouter-as-fallback.md (updated 2026-07-22)
- [ 83d] decisions/persistent-shell-enabled.md (updated 2026-07-22)
- [ 83d] experiments/index.md (updated 2026-07-22)
- [ 83d] infrastructure/backups.md (updated 2026-07-22)
- [ 83d] infrastructure/credentials-map.md (updated 2026-07-22)
- [ 83d] infrastructure/domains-and-tunnels.md (updated 2026-07-22)
- [ 83d] infrastructure/network-map.md (updated 2026-07-22)
- [ 83d] runbooks/backup-wiki.md (updated 2026-07-22)
- [ 83d] runbooks/diagnose-docker.md (updated 2026-07-22)
- [ 83d] runbooks/recover-docker-service.md (updated 2026-07-22)
- [ 83d] runbooks/restart-browser.md (updated 2026-07-22)
- [ 83d] runbooks/restart-hermes.md (updated 2026-07-22)
- [ 83d] runbooks/restore-openrouter.md (updated 2026-07-22)
- [ 83d] runbooks/test-openrouter.md (updated 2026-07-22)
- [ 83d] runbooks/troubleshoot-tool-loop.md (updated 2026-07-22)
- [ 83d] systems/cloudflare-tunnel.md (updated 2026-07-22)
- [ 83d] systems/headless-server.md (updated 2026-07-22)
- [ 83d] systems/openrouter.md (updated 2026-07-22)
- [ 83d] systems/terminal-backend.md (updated 2026-07-22)
- [ 80d] runbooks/gateway-resource-resilience.md (updated 2026-07-25)
- [ 80d] systems/tool-search.md (updated 2026-07-25)
- [ 79d] systems/homepage-dashboard.md (updated 2026-07-26)
- [ 79d] systems/obsidian.md (updated 2026-07-26)
- [ 77d] runbooks/multiplexer-setup.md (updated 2026-07-28)
- [ 76d] systems/opencode-go.md (updated 2026-07-29)
- [ 75d] systems/hermes-ui.md (updated 2026-07-30)
- [ 73d] incidents/2026-08-01-mnemosyne-capture-failure.md (updated 2026-08-01)
- [ 72d] decisions/rollback-steel-to-camofox.md (updated 2026-08-02)
- [ 72d] reports/uber-eats/index.md (updated 2026-08-02)
- [ 62d] comparisons/memory-backends.md (updated 2026-08-12)
- [ 62d] comparisons/model-routers.md (updated 2026-08-12)
- [ 60d] systems/guanaco.md (updated 2026-08-14)
- [ 59d] agents/index.md (updated 2026-08-15)
- [ 59d] agents/open-claw.md (updated 2026-08-15)
- [ 59d] agents/zeroclaw.md (updated 2026-08-15)
- [ 59d] incidents/2026-08-15-skills-index-drift.md (updated 2026-08-15)
- [ 58d] CONTRIBUTING.md (updated 2026-08-16)
- [ 57d] systems/nextcloud.md (updated 2026-08-17)
- [ 46d] reports/communities-wa-social-media-audit-2026.md (updated 2026-08-28)
- [ 45d] incidents/2026-08-29-gmail-read-state.md (updated 2026-08-29)
- [ 39d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-09-04)
## FRESH (38)
- [ 9d] agents/hermes-chaos.md
- [ 9d] agents/hermes-production.md
- [ 2d] agents/nanobot.md
- [ 2d] agents/qwenpaw.md
- [ 0d] comparisons/steel-vs-camoufox.md
- [ 0d] concepts/browser-automation.md
- [ 0d] current-state.md
- [ 1d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 10d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 2d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 7d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 9d] incidents/2026-10-03-ct450-hermes-update.md
- [ 9d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 7d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 0d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 4d] incidents/2026-10-09-memory-recall-relevance.md
- [ 2d] infrastructure/docker-services.md
- [ 9d] infrastructure/hosts.md
- [ 1d] infrastructure/proxmox.md
- [ 2d] infrastructure/tailscale.md
- [ 1d] queries/retained-research-answers.md
- [ 0d] runbooks/job-radar.md
- [ 9d] runbooks/provider-health.md
- [ 9d] runbooks/update-hermes-safely.md
- [ 0d] systems/browser-backend.md
- [ 11d] systems/current-profile.md
- [ 2d] systems/hermes-agent.md
- [ 1d] systems/home-assistant.md
- [ 2d] systems/ignis.md
- [ 9d] systems/memory-backend.md
- [ 2d] systems/messaging-integrations.md
- [ 0d] systems/mnemosyne.md
- [ 12d] systems/model-providers.md
- [ 1d] systems/nanobot.md
- [ 2d] systems/omniroute.md
- [ 2d] systems/paseo.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
## NO updated DATE (7)
- SCHEMA.md
- experiments/active/example.md
- incidents/2026-07-29-mnemosyne-relay.md
- incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- index.md
- log.md
- reports/uber-eats/2026-07.md
TOTAL: 48 stale, 38 fresh, 7 no-date of 93 pages
+67 -2
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-13
verified_on: 2026-10-13
updated: 2026-10-14
verified_on: 2026-10-14
review_after: 2026-10-25
confidence: high
tags: [cron, scheduled, automation]
@@ -341,3 +341,68 @@ archived: the DHL monitor stays where the 2026-10-10 audit put it (redundant, in
multi-tab workloads (new 2026-10-13); Quartz rebuild step inert; skills-index off-by-one; **3** non-clean
`last_status` rows (`0e753e9720fd`, `1b23c042e17d`, `066bb5191280`); Mattermost platform unreachable;
gateway runtime env outside the 15-minute dependency guard.
## Re-verified 2026-10-14
Window: **2026-10-13 00:00 → 23:59 AWST**; live probes 2026-10-14 00:00–00:30 AWST.
Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged this cycle. **Open incidents 2 → 3**.
**Cron tally — the clean streak broke.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row
in flight at probe time, which finalises the day at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0`
→ **the first failed cron run since 2026-10-09** (a day that recorded 15).
**Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.**
03:00:43 → 03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.`
Pinned `custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`,
`failure_type: agent`, `alerted_at: NULL` — failure streak 1, so under the 3-strike alert rule).
This is the job's **third failure in six days, all three on the opencode path**: 10-08
`HTTP 403: OpenCode's free tier can only be used from within OpenCode` (`1a8e90_789224ad589c`), 10-09 timeout
(`1a8e90_dc6b2d40ec44`), 10-13 404 no credentials (`1a8e90_d11698bd7275`). The first two resolved on
2026-10-10; this one clears only when the 2026-10-14 03:00 run succeeds — i.e. **each incident clears only on
the next successful run of its own job**. → [[incidents/2026-10-05-dead-model-pin-repin]]
**Finding 2 — the failure stranded its own generated artifacts; this audit committed them.**
The script's step 1 (skills-index regeneration) and step 2 (freshness report) both completed and wrote to disk,
then the agent/LLM turn that follows died → nothing committed, nothing pushed. Left in the working tree:
`systems/skills-index.md` (modified — **date-only change: `updated`/`verified_on` 10-12 → 10-13; 346 skills /
70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked — **48 stale / 38 fresh /
7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are in this audit's commit.
Structural point: the *agent* row is what commits, so a perfectly healthy script run still publishes nothing
when the LLM turn dies. There is no green-row alternative — the row went red.
**Retention floor moved.** `executions.db` now holds **1,001 rows** (1,000 terminal + 1 in flight);
`MAX_TERMINAL_EXECUTIONS = 1000` in `cron/executions.py`. Oldest retained terminal row: **2026-10-08 07:11**.
Days above the floor are complete; older ones undercount — 10-08 now reads `126 completed / 1 failed` where its
finalised tally was higher. Run-level evidence older than the floor is pruned, not recoverable.
**Delivery.** 10-13: `159 suppressed` / **9 delivered** / 1 no-record. Delivered = Daily Marketing Job Radar
(`dca8482e4f76`), Mosquito Tracker (`1b31187397a5` ×3), Tech & AI Newsletter Digest (`85edf6772bca`),
Daily Birthday Checker (`a432898b55a4`), Gmail Action Digest (`08eb86f02d37`), FreshRSS Midday News Briefing
(`ecc1fc13655a`), Evening Stand-Down Briefing (`e530c89f2be0`). No unexpected delivery; the two dead-thread
delivery faults (`0e753e9720fd`, `066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 10-01.
**Mnemosyne.** All six monitor runs returned the silent token — Health Monitor 00:02 and 12:02, Embedding
Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, i.e. both coverages at/above the 95 % floor and
no `reindex`. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic
1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the token **markdown-bolded** rather than bare;
the delivery layer still suppressed it (`delivery_outcome: suppressed`), so the guard held — but the matcher is
tolerant of wrapping rather than literal. → [[systems/mnemosyne]]
**Non-clean `last_status`: 3 → 4.** Cleared: `fad5a86979d4` (success 10-12 03:05), `dca8482e4f76` (success
10-13 07:02). Added: `1a8e90ea2271`. Remaining: `0e753e9720fd` (10-01), `1b23c042e17d` (10-09),
`066bb5191280` (10-01), `1a8e90ea2271` (10-13).
**Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive
session.
**Pages touched:** `systems/scheduled-tasks.md` (this block, frontmatter, non-clean list, retention note),
`systems/skills-index.md` (committed stocktake artifact), `current-state.md`, `index.md`, `log.md`.
**Nothing added/removed:** no new system, no retired job, no page archived → `index.md` structure unchanged.
**Still open after this cycle:** `wiki-stocktake-maintenance` opencode credential (new 2026-10-13 — watch the
10-14 03:00 run: streak 3 triggers an alert, and the durable fix is to repin the job off `free-stack`-opencode
or repair OmniRoute's `opencode` provider credentials); GitHub Trending Radar token (2026-10-13, next run
Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert;
skills-index off-by-one; **4** non-clean `last_status` rows (above); Mattermost platform unreachable;
gateway runtime env outside the 15-minute dependency guard.
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system
status: active
created: 2026-07-22
updated: 2026-10-12
verified_on: 2026-10-12
updated: 2026-10-13
verified_on: 2026-10-13
confidence: high
tags: [skills, hermes]
sources: []