audit(2026-10-14): 10-13 broke the clean streak — wiki-stocktake failed on a dead opencode credential (HTTP 404, incident 1a8e90_d11698bd7275); stranded skills-index + freshness-20261013 recovered; retention floor moved to 2026-10-08 07:11
This commit is contained in:
+11
-3
@@ -3,8 +3,8 @@ title: Current State
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-13
|
updated: 2026-10-14
|
||||||
verified_on: 2026-10-13
|
verified_on: 2026-10-14
|
||||||
review_after: 2026-10-25
|
review_after: 2026-10-25
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system]
|
tags: [system]
|
||||||
@@ -12,7 +12,15 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
|
|||||||
---
|
---
|
||||||
|
|
||||||
# Current State
|
# Current State
|
||||||
Updated: 2026-10-13
|
Updated: 2026-10-14
|
||||||
|
|
||||||
|
## Recent Changes (2026-10-13 → 2026-10-14, nightly audit; live probes 2026-10-14 00:00-00:30 AWST)
|
||||||
|
|
||||||
|
- **The four-day clean streak broke: 2026-10-13 = `167 completed` / 1 `failed`** (10-12 finalised `175 / 0`, 10-11 `172 / 0`, 10-10 `175 / 0`). The failure is `1a8e90ea2271` (wiki-stocktake-maintenance, 03:00) on `RuntimeError: HTTP 404: No active credentials for provider: opencode` — the **third** failure of that job in six days, all three on the opencode path (10-08 403 free-tier, 10-09 timeout, 10-13 404). New incident `1a8e90_d11698bd7275` (`detected`, streak 1, not alerted); it clears only on a successful 2026-10-14 03:00 run. → [[systems/scheduled-tasks]]
|
||||||
|
- **The failing run stranded its own generated artifacts.** Step 1 (skills-index) and step 2 (freshness report) wrote to disk, then the agent turn died → no commit, no push. `systems/skills-index.md` (date-only change; **346 skills / 70 categories** unchanged) and `raw/stocktake/freshness-20261013.txt` (**48 stale / 38 fresh / 7 no-date of 93 pages**, down from 54 stale) were committed by the 2026-10-14 audit. → [[systems/skills-index]]
|
||||||
|
- **`executions.db` retention floor moved to 2026-10-08 07:11** (`MAX_TERMINAL_EXECUTIONS = 1000`; 1,001 rows = 1,000 terminal + 1 in flight). Days older than the floor now undercount — 10-08 reads `126 completed / 1 failed`, a partial tally.
|
||||||
|
- **Non-clean `last_status` rows: 3 → 4** (cleared `fad5a86979d4`, `dca8482e4f76`; added `1a8e90ea2271`). **Open incidents: 2 → 3.** Fleet unchanged at 29 jobs / 26 enabled.
|
||||||
|
- **No interactive sessions for the third window running:** 15 sessions in the window, all `cron`, zero interactive.
|
||||||
|
|
||||||
## Recent Changes (2026-10-12 → 2026-10-13, nightly audit; live probes 2026-10-13 00:05-00:45 AWST)
|
## Recent Changes (2026-10-12 → 2026-10-13, nightly audit; live probes 2026-10-13 00:05-00:45 AWST)
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Anthony's Systems Wiki
|
# Anthony's Systems Wiki
|
||||||
> Operational knowledge for Anthony's agents, infrastructure,
|
> Operational knowledge for Anthony's agents, infrastructure,
|
||||||
> model routing and automation. Last updated: 2026-10-13
|
> model routing and automation. Last updated: 2026-10-14
|
||||||
|
|
||||||
## Start here
|
## Start here
|
||||||
- [[current-state]] — Current systems status
|
- [[current-state]] — Current systems status
|
||||||
@@ -29,8 +29,8 @@
|
|||||||
|
|
||||||
### Integrations
|
### Integrations
|
||||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
||||||
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **3 carry a non-clean `last_status`** as of 2026-10-13 00:15. **Three consecutive clean days: 2026-10-12 ran `174 completed` / 0 `failed`** (2026-10-11 finalised `172 / 0`), and **open incidents fell 5 → 2** on 2026-10-12 (three closed: both `deepseek-v4-flash` rows + the Job Radar timeout). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. ⚠ The 2026-10-12 clean sheet is *not* an all-clear — GitHub Trending Radar's Telegram token is revoked (`getMe` → 401) while the cron row stays green. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
|
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **4 carry a non-clean `last_status`** as of 2026-10-14 00:15. **The clean streak broke on 2026-10-13: `167 completed` / 1 `failed`**, after 10-12 finalised `175 / 0` (10-11 `172 / 0`, 10-10 `175 / 0`) — `1a8e90ea2271` (wiki-stocktake-maintenance, 03:00) hit `HTTP 404: No active credentials for provider: opencode`, its **third failure in six days**, so **open incidents rose 2 → 3**. Two rows cleared: `fad5a86979d4` (10-12 03:05), `dca8482e4f76` (10-13 07:02). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. ⚠ A green cron row is not proof of delivery: GitHub Trending Radar's Telegram token is revoked (`getMe` → 401) while the row stays green — it next runs Mon 09:00. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
|
||||||
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-13 against the stocktake's own `0eee5f1` commit: 346 skills / 70 categories). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **regenerated daily — the 10-10, 10-11 and 10-12 stocktake runs all committed**)
|
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
|
||||||
|
|
||||||
## Incidents
|
## Incidents
|
||||||
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)
|
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)
|
||||||
|
|||||||
@@ -598,3 +598,58 @@ history entries), `systems/mnemosyne.md` (12:02 coverage), `incidents/2026-10-09
|
|||||||
(status + full closure table + timeline), `index.md` (date, four blurbs), `log.md`.
|
(status + full closure table + timeline), `index.md` (date, four blurbs), `log.md`.
|
||||||
|
|
||||||
No secrets written; job/incident IDs, paths and tailnet addresses only.
|
No secrets written; job/incident IDs, paths and tailnet addresses only.
|
||||||
|
|
||||||
|
## [2026-10-14] audit | nightly review of 2026-10-13 — the clean streak broke on a dead opencode credential
|
||||||
|
|
||||||
|
Window 2026-10-13 00:00 → 23:59 AWST (ran 2026-10-13 23:59 → 2026-10-14 00:2x AWST). Live probes only.
|
||||||
|
|
||||||
|
**Cron.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row in flight → the day finalises
|
||||||
|
at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0` — **the first failed run since 2026-10-09**
|
||||||
|
(which recorded 15). Retention floor moved to **2026-10-08 07:11** (`MAX_TERMINAL_EXECUTIONS = 1000`; 1,001 rows
|
||||||
|
= 1,000 terminal + 1 in flight), so days above the floor are complete and older ones undercount (10-08 reads
|
||||||
|
`126 / 1`, partial).
|
||||||
|
|
||||||
|
**Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.** 03:00:43 →
|
||||||
|
03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.` Pinned
|
||||||
|
`custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`, `failure_type: agent`,
|
||||||
|
`alerted_at: NULL` — streak 1, under the 3-strike alert rule). Third failure of this job in six days, all three
|
||||||
|
on the opencode path (10-08 `HTTP 403: OpenCode's free tier can only be used from within OpenCode`; 10-09
|
||||||
|
timeout; 10-13 404). The first two resolved on 2026-10-10; this one clears only on a successful 2026-10-14 03:00
|
||||||
|
run. → [[incidents/2026-10-05-dead-model-pin-repin]]
|
||||||
|
|
||||||
|
**Finding 2 — the failure stranded its own generated artifacts (now committed).** Step 1 (skills-index) and
|
||||||
|
step 2 (freshness report) both completed, then the agent turn failed → no commit, no push. Left in the working
|
||||||
|
tree: `systems/skills-index.md` (modified, **date-only**: `updated`/`verified_on` 10-12 → 10-13; **346 skills /
|
||||||
|
70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked; **48 stale / 38 fresh /
|
||||||
|
7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are committed by this audit.
|
||||||
|
Consequence: the stocktake published nothing on 10-13 — and because the *agent* row is what commits, even a
|
||||||
|
perfect script run leaves nothing behind when the LLM turn dies.
|
||||||
|
|
||||||
|
**Delivery.** `159 suppressed` / **9 delivered** / 1 no-record. Delivered: Daily Marketing Job Radar,
|
||||||
|
Mosquito Tracker ×3, Tech & AI Newsletter Digest, Daily Birthday Checker, Gmail Action Digest, FreshRSS Midday,
|
||||||
|
Evening Stand-Down Briefing. No unexpected delivery; the two dead-thread faults (`0e753e9720fd`,
|
||||||
|
`066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 2026-10-01.
|
||||||
|
|
||||||
|
**Mnemosyne.** Six monitor runs, all silent: Health Monitor 00:02 + 12:02, Embedding Coverage Monitor 00:02 /
|
||||||
|
06:02 / 12:01 / 18:01 → no exception, so both coverages stayed at/above the 95 % floor and no `reindex` was
|
||||||
|
needed. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic
|
||||||
|
1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the silent token markdown-bolded rather than
|
||||||
|
bare; the delivery layer still suppressed it (`delivery_outcome: suppressed`) — the matcher tolerates wrapping
|
||||||
|
rather than requiring the literal token as a strict control. → [[systems/mnemosyne]]
|
||||||
|
|
||||||
|
**Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive
|
||||||
|
session.
|
||||||
|
|
||||||
|
**Fleet.** 29 jobs (26 enabled / 3 disabled), pins unchanged; **non-clean `last_status` 3 → 4** (cleared
|
||||||
|
`fad5a86979d4` 10-12 03:05 and `dca8482e4f76` 10-13 07:02; added `1a8e90ea2271`); **open incidents 2 → 3**
|
||||||
|
(1 `alerted`, 2 `detected`, 8 `closed`, 66 `resolved`; 77 rows).
|
||||||
|
|
||||||
|
**Pages touched:** `systems/scheduled-tasks.md` (frontmatter, retention floor, non-clean list,
|
||||||
|
`## Re-verified 2026-10-14`), `systems/skills-index.md` (committed stocktake artifact), `current-state.md`
|
||||||
|
(new Recent Changes + date), `index.md` (date + two blurbs), `log.md`.
|
||||||
|
|
||||||
|
**Vault mirror.** The 2026-10-13 audit was never mirrored to `/home/hermes/wiki` (last mirror `bb09581`,
|
||||||
|
2026-10-12), so this run mirrored both nights: the 10-13 audit's ten pages plus `skills-index.md`, whose vault
|
||||||
|
copy was still the 2026-10-08 version. The vault's own `raw/stocktake/` still holds only the 2026-10-03 report.
|
||||||
|
|
||||||
|
No secrets written; job/incident IDs, paths and tailnet addresses only.
|
||||||
|
|||||||
@@ -0,0 +1,106 @@
|
|||||||
|
# Wiki freshness audit — 2026-10-13
|
||||||
|
Threshold: >21 days since `updated`
|
||||||
|
|
||||||
|
## STALE (48)
|
||||||
|
|
||||||
|
- [ 83d] comparisons/camoufox-vs-chrome.md (updated 2026-07-22)
|
||||||
|
- [ 83d] comparisons/hermes-vs-nanobot.md (updated 2026-07-22)
|
||||||
|
- [ 83d] concepts/agent-memory.md (updated 2026-07-22)
|
||||||
|
- [ 83d] concepts/model-routing.md (updated 2026-07-22)
|
||||||
|
- [ 83d] concepts/persistent-shells.md (updated 2026-07-22)
|
||||||
|
- [ 83d] concepts/tool-calling.md (updated 2026-07-22)
|
||||||
|
- [ 83d] decisions/index.md (updated 2026-07-22)
|
||||||
|
- [ 83d] decisions/keep-openrouter-as-fallback.md (updated 2026-07-22)
|
||||||
|
- [ 83d] decisions/persistent-shell-enabled.md (updated 2026-07-22)
|
||||||
|
- [ 83d] experiments/index.md (updated 2026-07-22)
|
||||||
|
- [ 83d] infrastructure/backups.md (updated 2026-07-22)
|
||||||
|
- [ 83d] infrastructure/credentials-map.md (updated 2026-07-22)
|
||||||
|
- [ 83d] infrastructure/domains-and-tunnels.md (updated 2026-07-22)
|
||||||
|
- [ 83d] infrastructure/network-map.md (updated 2026-07-22)
|
||||||
|
- [ 83d] runbooks/backup-wiki.md (updated 2026-07-22)
|
||||||
|
- [ 83d] runbooks/diagnose-docker.md (updated 2026-07-22)
|
||||||
|
- [ 83d] runbooks/recover-docker-service.md (updated 2026-07-22)
|
||||||
|
- [ 83d] runbooks/restart-browser.md (updated 2026-07-22)
|
||||||
|
- [ 83d] runbooks/restart-hermes.md (updated 2026-07-22)
|
||||||
|
- [ 83d] runbooks/restore-openrouter.md (updated 2026-07-22)
|
||||||
|
- [ 83d] runbooks/test-openrouter.md (updated 2026-07-22)
|
||||||
|
- [ 83d] runbooks/troubleshoot-tool-loop.md (updated 2026-07-22)
|
||||||
|
- [ 83d] systems/cloudflare-tunnel.md (updated 2026-07-22)
|
||||||
|
- [ 83d] systems/headless-server.md (updated 2026-07-22)
|
||||||
|
- [ 83d] systems/openrouter.md (updated 2026-07-22)
|
||||||
|
- [ 83d] systems/terminal-backend.md (updated 2026-07-22)
|
||||||
|
- [ 80d] runbooks/gateway-resource-resilience.md (updated 2026-07-25)
|
||||||
|
- [ 80d] systems/tool-search.md (updated 2026-07-25)
|
||||||
|
- [ 79d] systems/homepage-dashboard.md (updated 2026-07-26)
|
||||||
|
- [ 79d] systems/obsidian.md (updated 2026-07-26)
|
||||||
|
- [ 77d] runbooks/multiplexer-setup.md (updated 2026-07-28)
|
||||||
|
- [ 76d] systems/opencode-go.md (updated 2026-07-29)
|
||||||
|
- [ 75d] systems/hermes-ui.md (updated 2026-07-30)
|
||||||
|
- [ 73d] incidents/2026-08-01-mnemosyne-capture-failure.md (updated 2026-08-01)
|
||||||
|
- [ 72d] decisions/rollback-steel-to-camofox.md (updated 2026-08-02)
|
||||||
|
- [ 72d] reports/uber-eats/index.md (updated 2026-08-02)
|
||||||
|
- [ 62d] comparisons/memory-backends.md (updated 2026-08-12)
|
||||||
|
- [ 62d] comparisons/model-routers.md (updated 2026-08-12)
|
||||||
|
- [ 60d] systems/guanaco.md (updated 2026-08-14)
|
||||||
|
- [ 59d] agents/index.md (updated 2026-08-15)
|
||||||
|
- [ 59d] agents/open-claw.md (updated 2026-08-15)
|
||||||
|
- [ 59d] agents/zeroclaw.md (updated 2026-08-15)
|
||||||
|
- [ 59d] incidents/2026-08-15-skills-index-drift.md (updated 2026-08-15)
|
||||||
|
- [ 58d] CONTRIBUTING.md (updated 2026-08-16)
|
||||||
|
- [ 57d] systems/nextcloud.md (updated 2026-08-17)
|
||||||
|
- [ 46d] reports/communities-wa-social-media-audit-2026.md (updated 2026-08-28)
|
||||||
|
- [ 45d] incidents/2026-08-29-gmail-read-state.md (updated 2026-08-29)
|
||||||
|
- [ 39d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-09-04)
|
||||||
|
|
||||||
|
## FRESH (38)
|
||||||
|
|
||||||
|
- [ 9d] agents/hermes-chaos.md
|
||||||
|
- [ 9d] agents/hermes-production.md
|
||||||
|
- [ 2d] agents/nanobot.md
|
||||||
|
- [ 2d] agents/qwenpaw.md
|
||||||
|
- [ 0d] comparisons/steel-vs-camoufox.md
|
||||||
|
- [ 0d] concepts/browser-automation.md
|
||||||
|
- [ 0d] current-state.md
|
||||||
|
- [ 1d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||||
|
- [ 10d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||||
|
- [ 2d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||||
|
- [ 7d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||||
|
- [ 9d] incidents/2026-10-03-ct450-hermes-update.md
|
||||||
|
- [ 9d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||||
|
- [ 7d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||||
|
- [ 0d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||||
|
- [ 4d] incidents/2026-10-09-memory-recall-relevance.md
|
||||||
|
- [ 2d] infrastructure/docker-services.md
|
||||||
|
- [ 9d] infrastructure/hosts.md
|
||||||
|
- [ 1d] infrastructure/proxmox.md
|
||||||
|
- [ 2d] infrastructure/tailscale.md
|
||||||
|
- [ 1d] queries/retained-research-answers.md
|
||||||
|
- [ 0d] runbooks/job-radar.md
|
||||||
|
- [ 9d] runbooks/provider-health.md
|
||||||
|
- [ 9d] runbooks/update-hermes-safely.md
|
||||||
|
- [ 0d] systems/browser-backend.md
|
||||||
|
- [ 11d] systems/current-profile.md
|
||||||
|
- [ 2d] systems/hermes-agent.md
|
||||||
|
- [ 1d] systems/home-assistant.md
|
||||||
|
- [ 2d] systems/ignis.md
|
||||||
|
- [ 9d] systems/memory-backend.md
|
||||||
|
- [ 2d] systems/messaging-integrations.md
|
||||||
|
- [ 0d] systems/mnemosyne.md
|
||||||
|
- [ 12d] systems/model-providers.md
|
||||||
|
- [ 1d] systems/nanobot.md
|
||||||
|
- [ 2d] systems/omniroute.md
|
||||||
|
- [ 2d] systems/paseo.md
|
||||||
|
- [ 0d] systems/scheduled-tasks.md
|
||||||
|
- [ 0d] systems/skills-index.md
|
||||||
|
|
||||||
|
## NO updated DATE (7)
|
||||||
|
|
||||||
|
- SCHEMA.md
|
||||||
|
- experiments/active/example.md
|
||||||
|
- incidents/2026-07-29-mnemosyne-relay.md
|
||||||
|
- incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||||
|
- index.md
|
||||||
|
- log.md
|
||||||
|
- reports/uber-eats/2026-07.md
|
||||||
|
|
||||||
|
TOTAL: 48 stale, 38 fresh, 7 no-date of 93 pages
|
||||||
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-13
|
updated: 2026-10-14
|
||||||
verified_on: 2026-10-13
|
verified_on: 2026-10-14
|
||||||
review_after: 2026-10-25
|
review_after: 2026-10-25
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, scheduled, automation]
|
tags: [cron, scheduled, automation]
|
||||||
@@ -341,3 +341,68 @@ archived: the DHL monitor stays where the 2026-10-10 audit put it (redundant, in
|
|||||||
multi-tab workloads (new 2026-10-13); Quartz rebuild step inert; skills-index off-by-one; **3** non-clean
|
multi-tab workloads (new 2026-10-13); Quartz rebuild step inert; skills-index off-by-one; **3** non-clean
|
||||||
`last_status` rows (`0e753e9720fd`, `1b23c042e17d`, `066bb5191280`); Mattermost platform unreachable;
|
`last_status` rows (`0e753e9720fd`, `1b23c042e17d`, `066bb5191280`); Mattermost platform unreachable;
|
||||||
gateway runtime env outside the 15-minute dependency guard.
|
gateway runtime env outside the 15-minute dependency guard.
|
||||||
|
|
||||||
|
|
||||||
|
## Re-verified 2026-10-14
|
||||||
|
|
||||||
|
Window: **2026-10-13 00:00 → 23:59 AWST**; live probes 2026-10-14 00:00–00:30 AWST.
|
||||||
|
Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged this cycle. **Open incidents 2 → 3**.
|
||||||
|
|
||||||
|
**Cron tally — the clean streak broke.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row
|
||||||
|
in flight at probe time, which finalises the day at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0`
|
||||||
|
→ **the first failed cron run since 2026-10-09** (a day that recorded 15).
|
||||||
|
|
||||||
|
**Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.**
|
||||||
|
03:00:43 → 03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.`
|
||||||
|
Pinned `custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`,
|
||||||
|
`failure_type: agent`, `alerted_at: NULL` — failure streak 1, so under the 3-strike alert rule).
|
||||||
|
This is the job's **third failure in six days, all three on the opencode path**: 10-08
|
||||||
|
`HTTP 403: OpenCode's free tier can only be used from within OpenCode` (`1a8e90_789224ad589c`), 10-09 timeout
|
||||||
|
(`1a8e90_dc6b2d40ec44`), 10-13 404 no credentials (`1a8e90_d11698bd7275`). The first two resolved on
|
||||||
|
2026-10-10; this one clears only when the 2026-10-14 03:00 run succeeds — i.e. **each incident clears only on
|
||||||
|
the next successful run of its own job**. → [[incidents/2026-10-05-dead-model-pin-repin]]
|
||||||
|
|
||||||
|
**Finding 2 — the failure stranded its own generated artifacts; this audit committed them.**
|
||||||
|
The script's step 1 (skills-index regeneration) and step 2 (freshness report) both completed and wrote to disk,
|
||||||
|
then the agent/LLM turn that follows died → nothing committed, nothing pushed. Left in the working tree:
|
||||||
|
`systems/skills-index.md` (modified — **date-only change: `updated`/`verified_on` 10-12 → 10-13; 346 skills /
|
||||||
|
70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked — **48 stale / 38 fresh /
|
||||||
|
7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are in this audit's commit.
|
||||||
|
Structural point: the *agent* row is what commits, so a perfectly healthy script run still publishes nothing
|
||||||
|
when the LLM turn dies. There is no green-row alternative — the row went red.
|
||||||
|
|
||||||
|
**Retention floor moved.** `executions.db` now holds **1,001 rows** (1,000 terminal + 1 in flight);
|
||||||
|
`MAX_TERMINAL_EXECUTIONS = 1000` in `cron/executions.py`. Oldest retained terminal row: **2026-10-08 07:11**.
|
||||||
|
Days above the floor are complete; older ones undercount — 10-08 now reads `126 completed / 1 failed` where its
|
||||||
|
finalised tally was higher. Run-level evidence older than the floor is pruned, not recoverable.
|
||||||
|
|
||||||
|
**Delivery.** 10-13: `159 suppressed` / **9 delivered** / 1 no-record. Delivered = Daily Marketing Job Radar
|
||||||
|
(`dca8482e4f76`), Mosquito Tracker (`1b31187397a5` ×3), Tech & AI Newsletter Digest (`85edf6772bca`),
|
||||||
|
Daily Birthday Checker (`a432898b55a4`), Gmail Action Digest (`08eb86f02d37`), FreshRSS Midday News Briefing
|
||||||
|
(`ecc1fc13655a`), Evening Stand-Down Briefing (`e530c89f2be0`). No unexpected delivery; the two dead-thread
|
||||||
|
delivery faults (`0e753e9720fd`, `066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 10-01.
|
||||||
|
|
||||||
|
**Mnemosyne.** All six monitor runs returned the silent token — Health Monitor 00:02 and 12:02, Embedding
|
||||||
|
Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, i.e. both coverages at/above the 95 % floor and
|
||||||
|
no `reindex`. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic
|
||||||
|
1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the token **markdown-bolded** rather than bare;
|
||||||
|
the delivery layer still suppressed it (`delivery_outcome: suppressed`), so the guard held — but the matcher is
|
||||||
|
tolerant of wrapping rather than literal. → [[systems/mnemosyne]]
|
||||||
|
|
||||||
|
**Non-clean `last_status`: 3 → 4.** Cleared: `fad5a86979d4` (success 10-12 03:05), `dca8482e4f76` (success
|
||||||
|
10-13 07:02). Added: `1a8e90ea2271`. Remaining: `0e753e9720fd` (10-01), `1b23c042e17d` (10-09),
|
||||||
|
`066bb5191280` (10-01), `1a8e90ea2271` (10-13).
|
||||||
|
|
||||||
|
**Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive
|
||||||
|
session.
|
||||||
|
|
||||||
|
**Pages touched:** `systems/scheduled-tasks.md` (this block, frontmatter, non-clean list, retention note),
|
||||||
|
`systems/skills-index.md` (committed stocktake artifact), `current-state.md`, `index.md`, `log.md`.
|
||||||
|
**Nothing added/removed:** no new system, no retired job, no page archived → `index.md` structure unchanged.
|
||||||
|
|
||||||
|
**Still open after this cycle:** `wiki-stocktake-maintenance` opencode credential (new 2026-10-13 — watch the
|
||||||
|
10-14 03:00 run: streak 3 triggers an alert, and the durable fix is to repin the job off `free-stack`-opencode
|
||||||
|
or repair OmniRoute's `opencode` provider credentials); GitHub Trending Radar token (2026-10-13, next run
|
||||||
|
Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert;
|
||||||
|
skills-index off-by-one; **4** non-clean `last_status` rows (above); Mattermost platform unreachable;
|
||||||
|
gateway runtime env outside the 15-minute dependency guard.
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-12
|
updated: 2026-10-13
|
||||||
verified_on: 2026-10-12
|
verified_on: 2026-10-13
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [skills, hermes]
|
tags: [skills, hermes]
|
||||||
sources: []
|
sources: []
|
||||||
|
|||||||
Reference in New Issue
Block a user