audit(2026-10-06): dead-model-pin repin resolved; failing set 7 -> 0

- new incident: incidents/2026-10-05-dead-model-pin-repin.md
- scheduled-tasks.md: Count block -> 0 failing, all streaks 0; 8 job rows updated;
  new 'Resolved 2026-10-05' subsection; 5 resolved bullets pruned from Open
- cliprox incident closed for cron (status: open -> resolved) + resolution section
- index.md: new incident linked, cron line + cliprox line updated
- current-state.md + log.md: 2026-10-06 entries
This commit is contained in:
Tony0410
2026-10-06 00:20:53 +08:00
parent 543e7782f8
commit 14b467e91b
6 changed files with 207 additions and 47 deletions
+10 -3
View File
@@ -3,8 +3,8 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-10-05
verified_on: 2026-10-05
updated: 2026-10-06
verified_on: 2026-10-06
review_after: 2026-10-31
confidence: high
tags: [system]
@@ -12,7 +12,14 @@ sources: [live-config, live-cron-list, live-hermes-version]
---
# Current State
Updated: 2026-10-05
Updated: 2026-10-06
## Recent Changes (2026-10-06)
- **Dead model pins cleared across the whole fleet — failing set 7 → 0.** All seven jobs that had been failing since 2026-10-01 were repinned on 2026-10-05 23:16 → 2026-10-06 00:05. Each had been pinned to a raw model name that died upstream: `cliprox/terra` (HTTP 401 connection auth, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`). They were moved to `custom:omni-101` on **routing aliases** — `free-stack` ×6, `smart-route` ×1 — because the fault was raw-pin brittleness, not any one model. Every `failure_streak` in `jobs.json` now reads 0. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The other four are repinned but not yet re-run, so they carry a stale `last_status: error` until their next scheduled slot. See [[incidents/2026-10-05-dead-model-pin-repin]].
- **The `cliprox/terra` connection incident is closed for cron purposes.** No enabled job references that pin any more — the jobs were moved off it rather than the proxy connection being re-authenticated. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **GitHub Trending Radar's revoked-token diagnosis is refuted.** `c0a9fa314542` ran on schedule at **2026-10-05 09:00 and completed cleanly** — a second consecutive success after the off-schedule 2026-09-30 19:22 run. No `getMe` → 401, and no BotFather token replacement is needed. The previous audit correctly refused to treat the claim as standing fact; it can now be retired.
- **A stale failure notice, not a live outage, opened the review.** Anthony acted on a `85edf6772bca` notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job succeeded on every run from 2026-09-30 17:53 onward, and its three incidents were resolved on 2026-09-30 17:55:13; the notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.**
- **Window tally** (`executions.db`, 2026-10-05 00:00 → 2026-10-06 00:15 AWST): 185 `completed`, 3 `failed` (all pre-repin, 03:00–07:00), 2 `unknown` (aborted first attempts of `79f9bb41ecf5` / `dca8482e4f76`, superseded by their successful re-runs) and 1 `running` (the nightly audit itself). No interactive sessions other than the repin session.
## Recent Changes (2026-10-05)
- **Cron fleet re-verified 2026-10-05 00:01 AWST — failing set 6 → 7.** Job count unchanged at 30 (27 enabled / 3 disabled). The one new failure event in the audit window is `79f9bb41ecf5` (Weekly Review, Sun 18:00), which ran on schedule **2026-10-04 18:00** and failed with the *identical* `cliprox/terra` error the incident predicted: `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That is the last untested job on the pin, so the provider-connection auth expiry is now **3 of 3 confirmed** — settled as a credential fault, not per-job. No review was produced; `delivery_outcome: delivered` again because the failure notice is the deliverable. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].