Files
agent-estate-wiki/incidents/2026-10-05-dead-model-pin-repin.md
T

140 lines
8.6 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route
type: incident
status: resolved
created: 2026-10-06
updated: 2026-11-20
verified_on: 2026-11-20
confidence: high
tags: [cron, model-pin, omniroute, incident, resolved]
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c]
---
# Seven Cron Jobs on Dead Model Pins Repinned to `free-stack` / `smart-route`
**Window:** failures 2026-10-01 → 2026-10-05; remediated 2026-10-05 23:16 → 2026-10-06 00:05 AWST
**Status:** Resolved — all seven repinned, every `failure_streak` cleared to 0; three verified by live run
**Trigger:** an interactive review on 2026-10-05 23:16 that began from a *stale* cron failure notice
## Trigger: a six-day-old alert
The review started because Anthony replied to a cron failure notice for `85edf6772bca`
(**Tech & AI Newsletter Digest (Analytical)**) reading:
```
Restart-safe cron worker dispatch failed: cron external worker exited before ownership
acknowledgement (exit 1)
```
```
This job has failed 4 runs in a row — worth a review.
```
**That alert was stale and the job was healthy.** `85edf6772bca` had completed on every run
from 2026-09-30 17:53 through 2026-10-05 07:00, and all three of its incidents were already
`resolved` as of **2026-09-30 17:55:13**. The notice came from the **2026-09-29 07:00** run
(incident `85edf6_7ab60a6b6d4a`, first seen 2026-09-29 07:00:45) and had been sitting
unread in the topic for six days.
It looked current because the *wording* matches the live 2026-09-28 managed-runtime
incident signature (`cron external worker exited before ownership acknowledgement`) — see
[[incidents/2026-09-28-hermes-managed-runtime-deps]].
**Rule: a failure notice in a topic is not evidence of a current failure.** Check the job's
`executions` row timestamp (and its `cron_incidents` state) before acting on a notice —
a resolved incident still leaves its old message behind.
## What was actually broken — seven jobs, four dead pins
The check prompted by the stale alert found the set the 2026-10-01 sweep had flagged, all
still pinned to upstreams that no longer work:
| Job | Name | Dead pin | Error |
|---|---|---|---|
| `1a8e90ea2271` | wiki-stocktake-maintenance | `ollama-cloud / qwen3.5:397b` | HTTP 410 — retired 2026-09-25 |
| `dca8482e4f76` | Daily Marketing Job Radar | `command-code / gpt-5.6-terra` | 403 `MODEL_NOT_IN_PLAN` |
| `fad5a86979d4` | Facebook Birthday Weekly Registry Sync | `nvidia / deepseek-v4-flash` | HTTP 400 — absent from live catalog |
| `0e753e9720fd` | Facebook Birthday Monthly Scrape | `nvidia / deepseek-v4-flash` | HTTP 400 — same pin |
| `06a0a1c8704f` | AI Newsletter Curation for Patrick | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — connection auth expired |
| `1b23c042e17d` | Enhanced Weekend Adventure Planner | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — same pin |
| `79f9bb41ecf5` | Weekly Review | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — same pin |
Four distinct fault classes, none self-recovering: retired model (410), plan-gated model
(403), catalog-missing model (400 ×2 jobs), provider-connection credential expiry (401 ×3 jobs).
The 401 pair is documented separately in
[[incidents/2026-10-01-cliproxy-connection-auth-expired]].
## Fix — repin to routing aliases
All seven moved to provider **`custom:omni-101`** (OmniRoute on the estate), using routing
aliases rather than another raw upstream model name:
| Alias | Jobs |
|---|---|
| `free-stack` | `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd` |
| `smart-route` | `dca8482e4f76` |
The root cause was **raw-pin brittleness** — a raw model name dies silently when the upstream
retires it, the catalog drops it, or a plan changes. An alias absorbs that. This is the same
lesson as the 2026-08-04 free-DeepSeek outage.
`free-stack` is in service again: it was on the broken-combo list as recently as 2026-08-14.
Its route was re-verified before being handed six jobs — do not assume the 2026-08 list
is still current.
## Verification
| Job | Evidence | Outcome |
|---|---|---|
| `79f9bb41ecf5` | manual run 2026-10-05 23:39 | succeeded; incident auto-resolved 23:39:12 |
| `06a0a1c8704f` | manual run 2026-10-05 23:45 | succeeded; incident auto-resolved 23:45:01 |
| `dca8482e4f76` | manual run 23:48 → completed 23:56 | delivered to `telegram:-1004321904721:1030` (message_id 9367); incident resolved 2026-10-06 00:05:07 |
Fleet-wide re-read after the repin: **30 jobs, 27 enabled / 3 disabled, `failure_streak` 0 on
every job** — down from seven jobs carrying streaks up to ×11.
Enabled fleet by pin after the fix: `smart-route` ×10, no-agent scripts ×8, `free-stack` ×6,
`deepseek-paid-smart` ×2, stale `custom:omni` ×1.
## Re-verification — 2026-10-29 03:0x AWST (stocktake)
This incident page had aged past the 21-day freshness threshold, so its claims were re-probed against live
state rather than left on trust.
| Claim | Live probe 2026-10-29 | Verdict |
|---|---|---|
| All 7 jobs repinned to `custom:omni-101` with routing aliases | `jobs.json`: `06a0a1c8704f` `free-stack`, `1b23c042e17d` `free-stack`, `79f9bb41ecf5` `free-stack`, `1a8e90ea2271` `free-stack`, `fad5a86979d4` `free-stack`, `0e753e9720fd` `free-stack`, `dca8482e4f76` `smart-route` | **Holds** |
| `free-stack` / `smart-route` are live aliases | `POST /v1/chat/completions` on CT600 → **HTTP 200** for both | **Holds** |
| `ollama-cloud/qwen3.5:397b` retired | **HTTP 410** — *"qwen3.5:397b was retired on September 25, 2026. Try glm-5.3-flash:cloud."* | **Holds** |
| `nvidia/deepseek-v4-flash` absent from live catalog | **HTTP 400** — *"Model 'deepseek-v4-flash' is not available in the active live catalog for provider 'nvidia'."* | **Holds** |
| `command-code/gpt-5.6-terra` plan-gated | **HTTP 401** — *"Invalid 'Authorization' header or token."* | **Pin still dead, but the error class changed** — the upstream now rejects the credential rather than the plan (403 → 401). The remediation verdict is unchanged; only the failure signature differs. |
| `cliprox/terra` connection never re-authenticated | **HTTP 200 on CT600**; **HTTP 401** (`No active credentials for provider: cliprox.`) on CT101 | **No longer holds** — see the SUPERSEDED marker in Residual above |
## Residual
- Four repinned jobs still carry the *pre-repin* `last_status: error` — `1b23c042e17d`,
`fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`. Their streaks are already 0; the status
clears on the next scheduled run. **Do not read those stale statuses as live failures.**
- `066bb5191280` still records `delivery_failed` (bare `telegram` target — separate open issue).
- ~~The `cliprox/terra` connection itself was never re-authenticated. That pin remains unusable.~~ **⚠ SUPERSEDED 2026-10-29 — this is no longer true.** A live probe of `cliprox/terra` against `custom:Omniroute:20128` (CT600) returned **HTTP 200**, so the connection has since been re-authenticated on CT600. The same pin on `custom:omni-101` (CT101) still returns **HTTP 401** (`No active credentials for provider: cliprox.`) — the nodes carry separate credentials. The repin itself was never reversed and remains correct. → [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
- The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can
run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day.
> **Re-verified 2026-11-20 (stocktake).** All seven jobs still on `custom:omni-101` with the repinned
> aliases (`free-stack` ×6, `smart-route` ×1 for `dca8482e4f76`); **every one now reads
> `last_status: ok`, `failure_streak: 0`** — the stale pre-repin `error` statuses noted in Residual
> have cleared. `free-stack` and `smart-route` probed live on **both** OmniRoute nodes → HTTP 200
> (CT600: free-stack → `openrouter/dots-studio/dots-3-note-preview:free`; CT101: both →
> `qwen/qwen4-max:free`). `cliprox/terra`: 200 on CT600 / 401 on CT101 (unchanged).
> **Error-class drift again on `command-code/gpt-5.6-terra`:** now **HTTP 403 `error code: 1010`**
> (Cloudflare access-denied signature) where 2026-10-29 saw 401 and the incident recorded 403
> `MODEL_NOT_IN_PLAN`. The pin remains dead; only the rejection layer differs. Fleet is now
> **29 jobs / 26 enabled** (was 30/27 at the fix) — composition changes are tracked in
> [[systems/scheduled-tasks]], not here.
## Related
- [[systems/scheduled-tasks]]
- [[systems/model-providers]]
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
- [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]]