8.6 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route | incident | resolved | 2026-10-06 | 2026-11-20 | 2026-11-20 | high |
|
|
Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route
Window: failures 2026-10-01 → 2026-10-05; remediated 2026-10-05 23:16 → 2026-10-06 00:05 AWST
Status: Resolved — all seven repinned, every failure_streak cleared to 0; three verified by live run
Trigger: an interactive review on 2026-10-05 23:16 that began from a stale cron failure notice
Trigger: a six-day-old alert
The review started because Anthony replied to a cron failure notice for 85edf6772bca
(Tech & AI Newsletter Digest (Analytical)) reading:
Restart-safe cron worker dispatch failed: cron external worker exited before ownership
acknowledgement (exit 1)
This job has failed 4 runs in a row — worth a review.
That alert was stale and the job was healthy. 85edf6772bca had completed on every run
from 2026-09-30 17:53 through 2026-10-05 07:00, and all three of its incidents were already
resolved as of 2026-09-30 17:55:13. The notice came from the 2026-09-29 07:00 run
(incident 85edf6_7ab60a6b6d4a, first seen 2026-09-29 07:00:45) and had been sitting
unread in the topic for six days.
It looked current because the wording matches the live 2026-09-28 managed-runtime
incident signature (cron external worker exited before ownership acknowledgement) — see
incidents/2026-09-28-hermes-managed-runtime-deps.
Rule: a failure notice in a topic is not evidence of a current failure. Check the job's
executions row timestamp (and its cron_incidents state) before acting on a notice —
a resolved incident still leaves its old message behind.
What was actually broken — seven jobs, four dead pins
The check prompted by the stale alert found the set the 2026-10-01 sweep had flagged, all still pinned to upstreams that no longer work:
| Job | Name | Dead pin | Error |
|---|---|---|---|
1a8e90ea2271 |
wiki-stocktake-maintenance | ollama-cloud / qwen3.5:397b |
HTTP 410 — retired 2026-09-25 |
dca8482e4f76 |
Daily Marketing Job Radar | command-code / gpt-5.6-terra |
403 MODEL_NOT_IN_PLAN |
fad5a86979d4 |
Facebook Birthday Weekly Registry Sync | nvidia / deepseek-v4-flash |
HTTP 400 — absent from live catalog |
0e753e9720fd |
Facebook Birthday Monthly Scrape | nvidia / deepseek-v4-flash |
HTTP 400 — same pin |
06a0a1c8704f |
AI Newsletter Curation for Patrick | custom:Omniroute:20128 / cliprox/terra |
HTTP 401 — connection auth expired |
1b23c042e17d |
Enhanced Weekend Adventure Planner | custom:Omniroute:20128 / cliprox/terra |
HTTP 401 — same pin |
79f9bb41ecf5 |
Weekly Review | custom:Omniroute:20128 / cliprox/terra |
HTTP 401 — same pin |
Four distinct fault classes, none self-recovering: retired model (410), plan-gated model (403), catalog-missing model (400 ×2 jobs), provider-connection credential expiry (401 ×3 jobs). The 401 pair is documented separately in incidents/2026-10-01-cliproxy-connection-auth-expired.
Fix — repin to routing aliases
All seven moved to provider custom:omni-101 (OmniRoute on the estate), using routing
aliases rather than another raw upstream model name:
| Alias | Jobs |
|---|---|
free-stack |
06a0a1c8704f, 1b23c042e17d, 79f9bb41ecf5, 1a8e90ea2271, fad5a86979d4, 0e753e9720fd |
smart-route |
dca8482e4f76 |
The root cause was raw-pin brittleness — a raw model name dies silently when the upstream retires it, the catalog drops it, or a plan changes. An alias absorbs that. This is the same lesson as the 2026-08-04 free-DeepSeek outage.
free-stack is in service again: it was on the broken-combo list as recently as 2026-08-14.
Its route was re-verified before being handed six jobs — do not assume the 2026-08 list
is still current.
Verification
| Job | Evidence | Outcome |
|---|---|---|
79f9bb41ecf5 |
manual run 2026-10-05 23:39 | succeeded; incident auto-resolved 23:39:12 |
06a0a1c8704f |
manual run 2026-10-05 23:45 | succeeded; incident auto-resolved 23:45:01 |
dca8482e4f76 |
manual run 23:48 → completed 23:56 | delivered to telegram:-1004321904721:1030 (message_id 9367); incident resolved 2026-10-06 00:05:07 |
Fleet-wide re-read after the repin: 30 jobs, 27 enabled / 3 disabled, failure_streak 0 on
every job — down from seven jobs carrying streaks up to ×11.
Enabled fleet by pin after the fix: smart-route ×10, no-agent scripts ×8, free-stack ×6,
deepseek-paid-smart ×2, stale custom:omni ×1.
Re-verification — 2026-10-29 03:0x AWST (stocktake)
This incident page had aged past the 21-day freshness threshold, so its claims were re-probed against live state rather than left on trust.
| Claim | Live probe 2026-10-29 | Verdict |
|---|---|---|
All 7 jobs repinned to custom:omni-101 with routing aliases |
jobs.json: 06a0a1c8704f free-stack, 1b23c042e17d free-stack, 79f9bb41ecf5 free-stack, 1a8e90ea2271 free-stack, fad5a86979d4 free-stack, 0e753e9720fd free-stack, dca8482e4f76 smart-route |
Holds |
free-stack / smart-route are live aliases |
POST /v1/chat/completions on CT600 → HTTP 200 for both |
Holds |
ollama-cloud/qwen3.5:397b retired |
HTTP 410 — "qwen3.5:397b was retired on September 25, 2026. Try glm-5.3-flash:cloud." | Holds |
nvidia/deepseek-v4-flash absent from live catalog |
HTTP 400 — "Model 'deepseek-v4-flash' is not available in the active live catalog for provider 'nvidia'." | Holds |
command-code/gpt-5.6-terra plan-gated |
HTTP 401 — "Invalid 'Authorization' header or token." | Pin still dead, but the error class changed — the upstream now rejects the credential rather than the plan (403 → 401). The remediation verdict is unchanged; only the failure signature differs. |
cliprox/terra connection never re-authenticated |
HTTP 200 on CT600; HTTP 401 (No active credentials for provider: cliprox.) on CT101 |
No longer holds — see the SUPERSEDED marker in Residual above |
Residual
- Four repinned jobs still carry the pre-repin
last_status: error—1b23c042e17d,fad5a86979d4,0e753e9720fd,1a8e90ea2271. Their streaks are already 0; the status clears on the next scheduled run. Do not read those stale statuses as live failures. 066bb5191280still recordsdelivery_failed(baretelegramtarget — separate open issue).The⚠ SUPERSEDED 2026-10-29 — this is no longer true. A live probe ofcliprox/terraconnection itself was never re-authenticated. That pin remains unusable.cliprox/terraagainstcustom:Omniroute:20128(CT600) returned HTTP 200, so the connection has since been re-authenticated on CT600. The same pin oncustom:omni-101(CT101) still returns HTTP 401 (No active credentials for provider: cliprox.) — the nodes carry separate credentials. The repin itself was never reversed and remains correct. → incidents/2026-10-01-cliproxy-connection-auth-expired- The daily stocktake (
1a8e90ea2271) had been blocked since 2026-09-25; with the repin it can run again at its 03:00 schedule, sowiki-maintain.shno longer needs a manual run each day.
Re-verified 2026-11-20 (stocktake). All seven jobs still on
custom:omni-101with the repinned aliases (free-stack×6,smart-route×1 fordca8482e4f76); every one now readslast_status: ok,failure_streak: 0— the stale pre-repinerrorstatuses noted in Residual have cleared.free-stackandsmart-routeprobed live on both OmniRoute nodes → HTTP 200 (CT600: free-stack →openrouter/dots-studio/dots-3-note-preview:free; CT101: both →qwen/qwen4-max:free).cliprox/terra: 200 on CT600 / 401 on CT101 (unchanged). Error-class drift again oncommand-code/gpt-5.6-terra: now HTTP 403error code: 1010(Cloudflare access-denied signature) where 2026-10-29 saw 401 and the incident recorded 403MODEL_NOT_IN_PLAN. The pin remains dead; only the rejection layer differs. Fleet is now 29 jobs / 26 enabled (was 30/27 at the fix) — composition changes are tracked in systems/scheduled-tasks, not here.