Files
agent-estate-wiki/incidents/2026-10-05-dead-model-pin-repin.md
T
Tony0410 14b467e91b audit(2026-10-06): dead-model-pin repin resolved; failing set 7 -> 0
- new incident: incidents/2026-10-05-dead-model-pin-repin.md
- scheduled-tasks.md: Count block -> 0 failing, all streaks 0; 8 job rows updated;
  new 'Resolved 2026-10-05' subsection; 5 resolved bullets pruned from Open
- cliprox incident closed for cron (status: open -> resolved) + resolution section
- index.md: new incident linked, cron line + cliprox line updated
- current-state.md + log.md: 2026-10-06 entries
2026-10-06 00:20:53 +08:00

115 lines
5.6 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route
type: incident
status: resolved
created: 2026-10-06
updated: 2026-10-06
verified_on: 2026-10-06
confidence: high
tags: [cron, model-pin, omniroute, incident, resolved]
sources: [live-jobs-json, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c]
---
# Seven Cron Jobs on Dead Model Pins Repinned to `free-stack` / `smart-route`
**Window:** failures 2026-10-01 → 2026-10-05; remediated 2026-10-05 23:16 → 2026-10-06 00:05 AWST
**Status:** Resolved — all seven repinned, every `failure_streak` cleared to 0; three verified by live run
**Trigger:** an interactive review on 2026-10-05 23:16 that began from a *stale* cron failure notice
## Trigger: a six-day-old alert
The review started because Anthony replied to a cron failure notice for `85edf6772bca`
(**Tech & AI Newsletter Digest (Analytical)**) reading:
```
Restart-safe cron worker dispatch failed: cron external worker exited before ownership
acknowledgement (exit 1)
```
```
This job has failed 4 runs in a row — worth a review.
```
**That alert was stale and the job was healthy.** `85edf6772bca` had completed on every run
from 2026-09-30 17:53 through 2026-10-05 07:00, and all three of its incidents were already
`resolved` as of **2026-09-30 17:55:13**. The notice came from the **2026-09-29 07:00** run
(incident `85edf6_7ab60a6b6d4a`, first seen 2026-09-29 07:00:45) and had been sitting
unread in the topic for six days.
It looked current because the *wording* matches the live 2026-09-28 managed-runtime
incident signature (`cron external worker exited before ownership acknowledgement`) — see
[[incidents/2026-09-28-hermes-managed-runtime-deps]].
**Rule: a failure notice in a topic is not evidence of a current failure.** Check the job's
`executions` row timestamp (and its `cron_incidents` state) before acting on a notice —
a resolved incident still leaves its old message behind.
## What was actually broken — seven jobs, four dead pins
The check prompted by the stale alert found the set the 2026-10-01 sweep had flagged, all
still pinned to upstreams that no longer work:
| Job | Name | Dead pin | Error |
|---|---|---|---|
| `1a8e90ea2271` | wiki-stocktake-maintenance | `ollama-cloud / qwen3.5:397b` | HTTP 410 — retired 2026-09-25 |
| `dca8482e4f76` | Daily Marketing Job Radar | `command-code / gpt-5.6-terra` | 403 `MODEL_NOT_IN_PLAN` |
| `fad5a86979d4` | Facebook Birthday Weekly Registry Sync | `nvidia / deepseek-v4-flash` | HTTP 400 — absent from live catalog |
| `0e753e9720fd` | Facebook Birthday Monthly Scrape | `nvidia / deepseek-v4-flash` | HTTP 400 — same pin |
| `06a0a1c8704f` | AI Newsletter Curation for Patrick | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — connection auth expired |
| `1b23c042e17d` | Enhanced Weekend Adventure Planner | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — same pin |
| `79f9bb41ecf5` | Weekly Review | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — same pin |
Four distinct fault classes, none self-recovering: retired model (410), plan-gated model
(403), catalog-missing model (400 ×2 jobs), provider-connection credential expiry (401 ×3 jobs).
The 401 pair is documented separately in
[[incidents/2026-10-01-cliproxy-connection-auth-expired]].
## Fix — repin to routing aliases
All seven moved to provider **`custom:omni-101`** (OmniRoute on the estate), using routing
aliases rather than another raw upstream model name:
| Alias | Jobs |
|---|---|
| `free-stack` | `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd` |
| `smart-route` | `dca8482e4f76` |
The root cause was **raw-pin brittleness** — a raw model name dies silently when the upstream
retires it, the catalog drops it, or a plan changes. An alias absorbs that. This is the same
lesson as the 2026-08-04 free-DeepSeek outage.
`free-stack` is in service again: it was on the broken-combo list as recently as 2026-08-14.
Its route was re-verified before being handed six jobs — do not assume the 2026-08 list
is still current.
## Verification
| Job | Evidence | Outcome |
|---|---|---|
| `79f9bb41ecf5` | manual run 2026-10-05 23:39 | succeeded; incident auto-resolved 23:39:12 |
| `06a0a1c8704f` | manual run 2026-10-05 23:45 | succeeded; incident auto-resolved 23:45:01 |
| `dca8482e4f76` | manual run 23:48 → completed 23:56 | delivered to `telegram:-1004321904721:1030` (message_id 9367); incident resolved 2026-10-06 00:05:07 |
Fleet-wide re-read after the repin: **30 jobs, 27 enabled / 3 disabled, `failure_streak` 0 on
every job** — down from seven jobs carrying streaks up to ×11.
Enabled fleet by pin after the fix: `smart-route` ×10, no-agent scripts ×8, `free-stack` ×6,
`deepseek-paid-smart` ×2, stale `custom:omni` ×1.
## Residual
- Four repinned jobs still carry the *pre-repin* `last_status: error` — `1b23c042e17d`,
`fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`. Their streaks are already 0; the status
clears on the next scheduled run. **Do not read those stale statuses as live failures.**
- `066bb5191280` still records `delivery_failed` (bare `telegram` target — separate open issue).
- The `cliprox/terra` connection itself was never re-authenticated. That pin remains unusable.
- The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can
run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day.
## Related
- [[systems/scheduled-tasks]]
- [[systems/model-providers]]
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
- [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]]