Files
agent-estate-wiki/incidents/2026-10-05-dead-model-pin-repin.md
T
Tony0410 cc95dcc54d stocktake: re-verify Tailscale table, correct falsified cliprox/terra claim, regenerate skills-index
Wiki maintenance (wiki-maintain.sh, exit 0) + live-vs-page drift audit.

Script:
- systems/skills-index.md regenerated from the live skill tree: 302 skills in 69
  categories; 45 archived skills under ~/.hermes/skills/.archive/ now excluded and
  noted. (303 SKILL.md files exist outside .archive/, but
  productivity/obsidian-git-sync/obsidian-git-sync is a nested alias duplicate,
  so 302 unique is correct.)
- Freshness report raw/stocktake/freshness-20261029.txt: 2 stale / 105 fresh /
  3 no-date. After the corrections below and a second script run: 0 stale /
  107 fresh / 3 no-date.
- Quartz rebuilt (110 pages -> 644 files) and deployed to public/.

Drift corrections (live-verified):
- infrastructure/tailscale.md: State column rebuilt from tailscale status --json,
  11 rows changed. Two were state transitions, not age drift:
  * sftpgo (CT299, pve) is back online - tailscale ping pong in 0s,
    systemctl is-active sftpgo -> active, tailscale ip -4 inside the guest.
  * m93p is online despite a stale peer record (Online: false in the JSON) -
    tailscale ping pong in 1ms, SSH over the tailnet returned up 11 days.
    Second occurrence of this pattern after 2026-10-26.
  The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged
  at 54 (53 live IPv4 peers + hermes); 76 peers total, 23 IPv6-only funnel
  ingress. No missing peers, no IP mismatches. Container identity re-confirmed
  from inside all 34 guests via pct exec tailscale ip -4 on both hypervisors;
  CT601 still absent from both.

- incidents/2026-10-01-cliproxy-connection-auth-expired.md: the residual claim
  "the connection credential itself was never re-authenticated" is falsified.
  Live POST /v1/chat/completions for cliprox/terra against custom:Omniroute:20128
  (CT600) returns HTTP 200 - the connection has since been re-authenticated. The
  same pin on custom:omni-101 (CT101) still returns HTTP 401, so the nodes carry
  separate credentials. Claim marked superseded, not deleted (SCHEMA rule 9); the
  remediation was never reversed and remains correct.

- incidents/2026-10-05-dead-model-pin-repin.md: every claim re-probed. All 7 jobs
  still on custom:omni-101 with routing aliases; both aliases HTTP 200;
  ollama-cloud/qwen3.5:397b still HTTP 410; nvidia/deepseek-v4-flash still
  HTTP 400; command-code/gpt-5.6-terra now HTTP 401 rather than the original 403
  - still dead, different failure signature.

- systems/scheduled-tasks.md: the Known-Issues entry repeating the
  "never re-authenticated" claim now carries the correction. Fleet figures
  re-checked against live jobs.json and remain accurate (29 jobs / 26 enabled /
  3 disabled, 5 non-clean last_status).

- log.md / index.md: stocktake entry appended and index last-updated refreshed.

Verified correct, no change: systems/homepage-dashboard (CT201 has 8084 and 8085
listening, no 8082, Homarr on 8102); CT201 Serve routes (/ -> 8085, /mgr -> 8084
on both :443 and :8443, Steel routes still gone); CT460 Serve routes (12 paths
on :443, plus :8443/webhook Funnel and :9443).

Noted for a human, not performed: vault mirror at /home/hermes/wiki/agent-estate-wiki
has drifted on both conventions (top-level 35 match / 3 stale / 72 missing;
wiki/ 41 match / 53 stale / 16 missing). A full mirror pass is larger than a
stocktake's mandate.
2026-10-29 03:42:54 +08:00

129 lines
7.7 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route
type: incident
status: resolved
created: 2026-10-06
updated: 2026-10-29
verified_on: 2026-10-29
confidence: high
tags: [cron, model-pin, omniroute, incident, resolved]
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c]
---
# Seven Cron Jobs on Dead Model Pins Repinned to `free-stack` / `smart-route`
**Window:** failures 2026-10-01 → 2026-10-05; remediated 2026-10-05 23:16 → 2026-10-06 00:05 AWST
**Status:** Resolved — all seven repinned, every `failure_streak` cleared to 0; three verified by live run
**Trigger:** an interactive review on 2026-10-05 23:16 that began from a *stale* cron failure notice
## Trigger: a six-day-old alert
The review started because Anthony replied to a cron failure notice for `85edf6772bca`
(**Tech & AI Newsletter Digest (Analytical)**) reading:
```
Restart-safe cron worker dispatch failed: cron external worker exited before ownership
acknowledgement (exit 1)
```
```
This job has failed 4 runs in a row — worth a review.
```
**That alert was stale and the job was healthy.** `85edf6772bca` had completed on every run
from 2026-09-30 17:53 through 2026-10-05 07:00, and all three of its incidents were already
`resolved` as of **2026-09-30 17:55:13**. The notice came from the **2026-09-29 07:00** run
(incident `85edf6_7ab60a6b6d4a`, first seen 2026-09-29 07:00:45) and had been sitting
unread in the topic for six days.
It looked current because the *wording* matches the live 2026-09-28 managed-runtime
incident signature (`cron external worker exited before ownership acknowledgement`) — see
[[incidents/2026-09-28-hermes-managed-runtime-deps]].
**Rule: a failure notice in a topic is not evidence of a current failure.** Check the job's
`executions` row timestamp (and its `cron_incidents` state) before acting on a notice —
a resolved incident still leaves its old message behind.
## What was actually broken — seven jobs, four dead pins
The check prompted by the stale alert found the set the 2026-10-01 sweep had flagged, all
still pinned to upstreams that no longer work:
| Job | Name | Dead pin | Error |
|---|---|---|---|
| `1a8e90ea2271` | wiki-stocktake-maintenance | `ollama-cloud / qwen3.5:397b` | HTTP 410 — retired 2026-09-25 |
| `dca8482e4f76` | Daily Marketing Job Radar | `command-code / gpt-5.6-terra` | 403 `MODEL_NOT_IN_PLAN` |
| `fad5a86979d4` | Facebook Birthday Weekly Registry Sync | `nvidia / deepseek-v4-flash` | HTTP 400 — absent from live catalog |
| `0e753e9720fd` | Facebook Birthday Monthly Scrape | `nvidia / deepseek-v4-flash` | HTTP 400 — same pin |
| `06a0a1c8704f` | AI Newsletter Curation for Patrick | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — connection auth expired |
| `1b23c042e17d` | Enhanced Weekend Adventure Planner | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — same pin |
| `79f9bb41ecf5` | Weekly Review | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — same pin |
Four distinct fault classes, none self-recovering: retired model (410), plan-gated model
(403), catalog-missing model (400 ×2 jobs), provider-connection credential expiry (401 ×3 jobs).
The 401 pair is documented separately in
[[incidents/2026-10-01-cliproxy-connection-auth-expired]].
## Fix — repin to routing aliases
All seven moved to provider **`custom:omni-101`** (OmniRoute on the estate), using routing
aliases rather than another raw upstream model name:
| Alias | Jobs |
|---|---|
| `free-stack` | `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd` |
| `smart-route` | `dca8482e4f76` |
The root cause was **raw-pin brittleness** — a raw model name dies silently when the upstream
retires it, the catalog drops it, or a plan changes. An alias absorbs that. This is the same
lesson as the 2026-08-04 free-DeepSeek outage.
`free-stack` is in service again: it was on the broken-combo list as recently as 2026-08-14.
Its route was re-verified before being handed six jobs — do not assume the 2026-08 list
is still current.
## Verification
| Job | Evidence | Outcome |
|---|---|---|
| `79f9bb41ecf5` | manual run 2026-10-05 23:39 | succeeded; incident auto-resolved 23:39:12 |
| `06a0a1c8704f` | manual run 2026-10-05 23:45 | succeeded; incident auto-resolved 23:45:01 |
| `dca8482e4f76` | manual run 23:48 → completed 23:56 | delivered to `telegram:-1004321904721:1030` (message_id 9367); incident resolved 2026-10-06 00:05:07 |
Fleet-wide re-read after the repin: **30 jobs, 27 enabled / 3 disabled, `failure_streak` 0 on
every job** — down from seven jobs carrying streaks up to ×11.
Enabled fleet by pin after the fix: `smart-route` ×10, no-agent scripts ×8, `free-stack` ×6,
`deepseek-paid-smart` ×2, stale `custom:omni` ×1.
## Re-verification — 2026-10-29 03:0x AWST (stocktake)
This incident page had aged past the 21-day freshness threshold, so its claims were re-probed against live
state rather than left on trust.
| Claim | Live probe 2026-10-29 | Verdict |
|---|---|---|
| All 7 jobs repinned to `custom:omni-101` with routing aliases | `jobs.json`: `06a0a1c8704f` `free-stack`, `1b23c042e17d` `free-stack`, `79f9bb41ecf5` `free-stack`, `1a8e90ea2271` `free-stack`, `fad5a86979d4` `free-stack`, `0e753e9720fd` `free-stack`, `dca8482e4f76` `smart-route` | **Holds** |
| `free-stack` / `smart-route` are live aliases | `POST /v1/chat/completions` on CT600 → **HTTP 200** for both | **Holds** |
| `ollama-cloud/qwen3.5:397b` retired | **HTTP 410** — *"qwen3.5:397b was retired on September 25, 2026. Try glm-5.3-flash:cloud."* | **Holds** |
| `nvidia/deepseek-v4-flash` absent from live catalog | **HTTP 400** — *"Model 'deepseek-v4-flash' is not available in the active live catalog for provider 'nvidia'."* | **Holds** |
| `command-code/gpt-5.6-terra` plan-gated | **HTTP 401** — *"Invalid 'Authorization' header or token."* | **Pin still dead, but the error class changed** — the upstream now rejects the credential rather than the plan (403 → 401). The remediation verdict is unchanged; only the failure signature differs. |
| `cliprox/terra` connection never re-authenticated | **HTTP 200 on CT600**; **HTTP 401** (`No active credentials for provider: cliprox.`) on CT101 | **No longer holds** — see the SUPERSEDED marker in Residual above |
## Residual
- Four repinned jobs still carry the *pre-repin* `last_status: error` — `1b23c042e17d`,
`fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`. Their streaks are already 0; the status
clears on the next scheduled run. **Do not read those stale statuses as live failures.**
- `066bb5191280` still records `delivery_failed` (bare `telegram` target — separate open issue).
- ~~The `cliprox/terra` connection itself was never re-authenticated. That pin remains unusable.~~ **⚠ SUPERSEDED 2026-10-29 — this is no longer true.** A live probe of `cliprox/terra` against `custom:Omniroute:20128` (CT600) returned **HTTP 200**, so the connection has since been re-authenticated on CT600. The same pin on `custom:omni-101` (CT101) still returns **HTTP 401** (`No active credentials for provider: cliprox.`) — the nodes carry separate credentials. The repin itself was never reversed and remains correct. → [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
- The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can
run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day.
## Related
- [[systems/scheduled-tasks]]
- [[systems/model-providers]]
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
- [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]]