--- title: Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route type: incident status: resolved created: 2026-10-06 updated: 2026-11-20 verified_on: 2026-11-20 confidence: high tags: [cron, model-pin, omniroute, incident, resolved] sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c] --- # Seven Cron Jobs on Dead Model Pins Repinned to `free-stack` / `smart-route` **Window:** failures 2026-10-01 → 2026-10-05; remediated 2026-10-05 23:16 → 2026-10-06 00:05 AWST **Status:** Resolved — all seven repinned, every `failure_streak` cleared to 0; three verified by live run **Trigger:** an interactive review on 2026-10-05 23:16 that began from a *stale* cron failure notice ## Trigger: a six-day-old alert The review started because Anthony replied to a cron failure notice for `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) reading: ``` Restart-safe cron worker dispatch failed: cron external worker exited before ownership acknowledgement (exit 1) ``` ``` This job has failed 4 runs in a row — worth a review. ``` **That alert was stale and the job was healthy.** `85edf6772bca` had completed on every run from 2026-09-30 17:53 through 2026-10-05 07:00, and all three of its incidents were already `resolved` as of **2026-09-30 17:55:13**. The notice came from the **2026-09-29 07:00** run (incident `85edf6_7ab60a6b6d4a`, first seen 2026-09-29 07:00:45) and had been sitting unread in the topic for six days. It looked current because the *wording* matches the live 2026-09-28 managed-runtime incident signature (`cron external worker exited before ownership acknowledgement`) — see [[incidents/2026-09-28-hermes-managed-runtime-deps]]. **Rule: a failure notice in a topic is not evidence of a current failure.** Check the job's `executions` row timestamp (and its `cron_incidents` state) before acting on a notice — a resolved incident still leaves its old message behind. ## What was actually broken — seven jobs, four dead pins The check prompted by the stale alert found the set the 2026-10-01 sweep had flagged, all still pinned to upstreams that no longer work: | Job | Name | Dead pin | Error | |---|---|---|---| | `1a8e90ea2271` | wiki-stocktake-maintenance | `ollama-cloud / qwen3.5:397b` | HTTP 410 — retired 2026-09-25 | | `dca8482e4f76` | Daily Marketing Job Radar | `command-code / gpt-5.6-terra` | 403 `MODEL_NOT_IN_PLAN` | | `fad5a86979d4` | Facebook Birthday Weekly Registry Sync | `nvidia / deepseek-v4-flash` | HTTP 400 — absent from live catalog | | `0e753e9720fd` | Facebook Birthday Monthly Scrape | `nvidia / deepseek-v4-flash` | HTTP 400 — same pin | | `06a0a1c8704f` | AI Newsletter Curation for Patrick | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — connection auth expired | | `1b23c042e17d` | Enhanced Weekend Adventure Planner | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — same pin | | `79f9bb41ecf5` | Weekly Review | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — same pin | Four distinct fault classes, none self-recovering: retired model (410), plan-gated model (403), catalog-missing model (400 ×2 jobs), provider-connection credential expiry (401 ×3 jobs). The 401 pair is documented separately in [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. ## Fix — repin to routing aliases All seven moved to provider **`custom:omni-101`** (OmniRoute on the estate), using routing aliases rather than another raw upstream model name: | Alias | Jobs | |---|---| | `free-stack` | `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd` | | `smart-route` | `dca8482e4f76` | The root cause was **raw-pin brittleness** — a raw model name dies silently when the upstream retires it, the catalog drops it, or a plan changes. An alias absorbs that. This is the same lesson as the 2026-08-04 free-DeepSeek outage. `free-stack` is in service again: it was on the broken-combo list as recently as 2026-08-14. Its route was re-verified before being handed six jobs — do not assume the 2026-08 list is still current. ## Verification | Job | Evidence | Outcome | |---|---|---| | `79f9bb41ecf5` | manual run 2026-10-05 23:39 | succeeded; incident auto-resolved 23:39:12 | | `06a0a1c8704f` | manual run 2026-10-05 23:45 | succeeded; incident auto-resolved 23:45:01 | | `dca8482e4f76` | manual run 23:48 → completed 23:56 | delivered to `telegram:-1004321904721:1030` (message_id 9367); incident resolved 2026-10-06 00:05:07 | Fleet-wide re-read after the repin: **30 jobs, 27 enabled / 3 disabled, `failure_streak` 0 on every job** — down from seven jobs carrying streaks up to ×11. Enabled fleet by pin after the fix: `smart-route` ×10, no-agent scripts ×8, `free-stack` ×6, `deepseek-paid-smart` ×2, stale `custom:omni` ×1. ## Re-verification — 2026-10-29 03:0x AWST (stocktake) This incident page had aged past the 21-day freshness threshold, so its claims were re-probed against live state rather than left on trust. | Claim | Live probe 2026-10-29 | Verdict | |---|---|---| | All 7 jobs repinned to `custom:omni-101` with routing aliases | `jobs.json`: `06a0a1c8704f` `free-stack`, `1b23c042e17d` `free-stack`, `79f9bb41ecf5` `free-stack`, `1a8e90ea2271` `free-stack`, `fad5a86979d4` `free-stack`, `0e753e9720fd` `free-stack`, `dca8482e4f76` `smart-route` | **Holds** | | `free-stack` / `smart-route` are live aliases | `POST /v1/chat/completions` on CT600 → **HTTP 200** for both | **Holds** | | `ollama-cloud/qwen3.5:397b` retired | **HTTP 410** — *"qwen3.5:397b was retired on September 25, 2026. Try glm-5.3-flash:cloud."* | **Holds** | | `nvidia/deepseek-v4-flash` absent from live catalog | **HTTP 400** — *"Model 'deepseek-v4-flash' is not available in the active live catalog for provider 'nvidia'."* | **Holds** | | `command-code/gpt-5.6-terra` plan-gated | **HTTP 401** — *"Invalid 'Authorization' header or token."* | **Pin still dead, but the error class changed** — the upstream now rejects the credential rather than the plan (403 → 401). The remediation verdict is unchanged; only the failure signature differs. | | `cliprox/terra` connection never re-authenticated | **HTTP 200 on CT600**; **HTTP 401** (`No active credentials for provider: cliprox.`) on CT101 | **No longer holds** — see the SUPERSEDED marker in Residual above | ## Residual - Four repinned jobs still carry the *pre-repin* `last_status: error` — `1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`. Their streaks are already 0; the status clears on the next scheduled run. **Do not read those stale statuses as live failures.** - `066bb5191280` still records `delivery_failed` (bare `telegram` target — separate open issue). - ~~The `cliprox/terra` connection itself was never re-authenticated. That pin remains unusable.~~ **⚠ SUPERSEDED 2026-10-29 — this is no longer true.** A live probe of `cliprox/terra` against `custom:Omniroute:20128` (CT600) returned **HTTP 200**, so the connection has since been re-authenticated on CT600. The same pin on `custom:omni-101` (CT101) still returns **HTTP 401** (`No active credentials for provider: cliprox.`) — the nodes carry separate credentials. The repin itself was never reversed and remains correct. → [[incidents/2026-10-01-cliproxy-connection-auth-expired]] - The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day. > **Re-verified 2026-11-20 (stocktake).** All seven jobs still on `custom:omni-101` with the repinned > aliases (`free-stack` ×6, `smart-route` ×1 for `dca8482e4f76`); **every one now reads > `last_status: ok`, `failure_streak: 0`** — the stale pre-repin `error` statuses noted in Residual > have cleared. `free-stack` and `smart-route` probed live on **both** OmniRoute nodes → HTTP 200 > (CT600: free-stack → `openrouter/dots-studio/dots-3-note-preview:free`; CT101: both → > `qwen/qwen4-max:free`). `cliprox/terra`: 200 on CT600 / 401 on CT101 (unchanged). > **Error-class drift again on `command-code/gpt-5.6-terra`:** now **HTTP 403 `error code: 1010`** > (Cloudflare access-denied signature) where 2026-10-29 saw 401 and the incident recorded 403 > `MODEL_NOT_IN_PLAN`. The pin remains dead; only the rejection layer differs. Fleet is now > **29 jobs / 26 enabled** (was 30/27 at the fix) — composition changes are tracked in > [[systems/scheduled-tasks]], not here. ## Related - [[systems/scheduled-tasks]] - [[systems/model-providers]] - [[incidents/2026-10-01-cliproxy-connection-auth-expired]] - [[incidents/2026-09-28-hermes-managed-runtime-deps]] - [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]]