audit(2026-10-06): dead-model-pin repin resolved; failing set 7 -> 0

- new incident: incidents/2026-10-05-dead-model-pin-repin.md
- scheduled-tasks.md: Count block -> 0 failing, all streaks 0; 8 job rows updated;
  new 'Resolved 2026-10-05' subsection; 5 resolved bullets pruned from Open
- cliprox incident closed for cron (status: open -> resolved) + resolution section
- index.md: new incident linked, cron line + cliprox line updated
- current-state.md + log.md: 2026-10-06 entries
This commit is contained in:
Tony0410
2026-10-06 00:20:53 +08:00
parent 543e7782f8
commit 14b467e91b
6 changed files with 207 additions and 47 deletions
@@ -0,0 +1,114 @@
---
title: Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route
type: incident
status: resolved
created: 2026-10-06
updated: 2026-10-06
verified_on: 2026-10-06
confidence: high
tags: [cron, model-pin, omniroute, incident, resolved]
sources: [live-jobs-json, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c]
---
# Seven Cron Jobs on Dead Model Pins Repinned to `free-stack` / `smart-route`
**Window:** failures 2026-10-01 → 2026-10-05; remediated 2026-10-05 23:16 → 2026-10-06 00:05 AWST
**Status:** Resolved — all seven repinned, every `failure_streak` cleared to 0; three verified by live run
**Trigger:** an interactive review on 2026-10-05 23:16 that began from a *stale* cron failure notice
## Trigger: a six-day-old alert
The review started because Anthony replied to a cron failure notice for `85edf6772bca`
(**Tech & AI Newsletter Digest (Analytical)**) reading:
```
Restart-safe cron worker dispatch failed: cron external worker exited before ownership
acknowledgement (exit 1)
```
```
This job has failed 4 runs in a row — worth a review.
```
**That alert was stale and the job was healthy.** `85edf6772bca` had completed on every run
from 2026-09-30 17:53 through 2026-10-05 07:00, and all three of its incidents were already
`resolved` as of **2026-09-30 17:55:13**. The notice came from the **2026-09-29 07:00** run
(incident `85edf6_7ab60a6b6d4a`, first seen 2026-09-29 07:00:45) and had been sitting
unread in the topic for six days.
It looked current because the *wording* matches the live 2026-09-28 managed-runtime
incident signature (`cron external worker exited before ownership acknowledgement`) — see
[[incidents/2026-09-28-hermes-managed-runtime-deps]].
**Rule: a failure notice in a topic is not evidence of a current failure.** Check the job's
`executions` row timestamp (and its `cron_incidents` state) before acting on a notice —
a resolved incident still leaves its old message behind.
## What was actually broken — seven jobs, four dead pins
The check prompted by the stale alert found the set the 2026-10-01 sweep had flagged, all
still pinned to upstreams that no longer work:
| Job | Name | Dead pin | Error |
|---|---|---|---|
| `1a8e90ea2271` | wiki-stocktake-maintenance | `ollama-cloud / qwen3.5:397b` | HTTP 410 — retired 2026-09-25 |
| `dca8482e4f76` | Daily Marketing Job Radar | `command-code / gpt-5.6-terra` | 403 `MODEL_NOT_IN_PLAN` |
| `fad5a86979d4` | Facebook Birthday Weekly Registry Sync | `nvidia / deepseek-v4-flash` | HTTP 400 — absent from live catalog |
| `0e753e9720fd` | Facebook Birthday Monthly Scrape | `nvidia / deepseek-v4-flash` | HTTP 400 — same pin |
| `06a0a1c8704f` | AI Newsletter Curation for Patrick | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — connection auth expired |
| `1b23c042e17d` | Enhanced Weekend Adventure Planner | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — same pin |
| `79f9bb41ecf5` | Weekly Review | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — same pin |
Four distinct fault classes, none self-recovering: retired model (410), plan-gated model
(403), catalog-missing model (400 ×2 jobs), provider-connection credential expiry (401 ×3 jobs).
The 401 pair is documented separately in
[[incidents/2026-10-01-cliproxy-connection-auth-expired]].
## Fix — repin to routing aliases
All seven moved to provider **`custom:omni-101`** (OmniRoute on the estate), using routing
aliases rather than another raw upstream model name:
| Alias | Jobs |
|---|---|
| `free-stack` | `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd` |
| `smart-route` | `dca8482e4f76` |
The root cause was **raw-pin brittleness** — a raw model name dies silently when the upstream
retires it, the catalog drops it, or a plan changes. An alias absorbs that. This is the same
lesson as the 2026-08-04 free-DeepSeek outage.
`free-stack` is in service again: it was on the broken-combo list as recently as 2026-08-14.
Its route was re-verified before being handed six jobs — do not assume the 2026-08 list
is still current.
## Verification
| Job | Evidence | Outcome |
|---|---|---|
| `79f9bb41ecf5` | manual run 2026-10-05 23:39 | succeeded; incident auto-resolved 23:39:12 |
| `06a0a1c8704f` | manual run 2026-10-05 23:45 | succeeded; incident auto-resolved 23:45:01 |
| `dca8482e4f76` | manual run 23:48 → completed 23:56 | delivered to `telegram:-1004321904721:1030` (message_id 9367); incident resolved 2026-10-06 00:05:07 |
Fleet-wide re-read after the repin: **30 jobs, 27 enabled / 3 disabled, `failure_streak` 0 on
every job** — down from seven jobs carrying streaks up to ×11.
Enabled fleet by pin after the fix: `smart-route` ×10, no-agent scripts ×8, `free-stack` ×6,
`deepseek-paid-smart` ×2, stale `custom:omni` ×1.
## Residual
- Four repinned jobs still carry the *pre-repin* `last_status: error` — `1b23c042e17d`,
`fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`. Their streaks are already 0; the status
clears on the next scheduled run. **Do not read those stale statuses as live failures.**
- `066bb5191280` still records `delivery_failed` (bare `telegram` target — separate open issue).
- The `cliprox/terra` connection itself was never re-authenticated. That pin remains unusable.
- The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can
run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day.
## Related
- [[systems/scheduled-tasks]]
- [[systems/model-providers]]
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
- [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]]