Files
agent-estate-wiki/incidents/2026-10-05-dead-model-pin-repin.md
T
Tony0410 cc95dcc54d stocktake: re-verify Tailscale table, correct falsified cliprox/terra claim, regenerate skills-index
Wiki maintenance (wiki-maintain.sh, exit 0) + live-vs-page drift audit.

Script:
- systems/skills-index.md regenerated from the live skill tree: 302 skills in 69
  categories; 45 archived skills under ~/.hermes/skills/.archive/ now excluded and
  noted. (303 SKILL.md files exist outside .archive/, but
  productivity/obsidian-git-sync/obsidian-git-sync is a nested alias duplicate,
  so 302 unique is correct.)
- Freshness report raw/stocktake/freshness-20261029.txt: 2 stale / 105 fresh /
  3 no-date. After the corrections below and a second script run: 0 stale /
  107 fresh / 3 no-date.
- Quartz rebuilt (110 pages -> 644 files) and deployed to public/.

Drift corrections (live-verified):
- infrastructure/tailscale.md: State column rebuilt from tailscale status --json,
  11 rows changed. Two were state transitions, not age drift:
  * sftpgo (CT299, pve) is back online - tailscale ping pong in 0s,
    systemctl is-active sftpgo -> active, tailscale ip -4 inside the guest.
  * m93p is online despite a stale peer record (Online: false in the JSON) -
    tailscale ping pong in 1ms, SSH over the tailnet returned up 11 days.
    Second occurrence of this pattern after 2026-10-26.
  The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged
  at 54 (53 live IPv4 peers + hermes); 76 peers total, 23 IPv6-only funnel
  ingress. No missing peers, no IP mismatches. Container identity re-confirmed
  from inside all 34 guests via pct exec tailscale ip -4 on both hypervisors;
  CT601 still absent from both.

- incidents/2026-10-01-cliproxy-connection-auth-expired.md: the residual claim
  "the connection credential itself was never re-authenticated" is falsified.
  Live POST /v1/chat/completions for cliprox/terra against custom:Omniroute:20128
  (CT600) returns HTTP 200 - the connection has since been re-authenticated. The
  same pin on custom:omni-101 (CT101) still returns HTTP 401, so the nodes carry
  separate credentials. Claim marked superseded, not deleted (SCHEMA rule 9); the
  remediation was never reversed and remains correct.

- incidents/2026-10-05-dead-model-pin-repin.md: every claim re-probed. All 7 jobs
  still on custom:omni-101 with routing aliases; both aliases HTTP 200;
  ollama-cloud/qwen3.5:397b still HTTP 410; nvidia/deepseek-v4-flash still
  HTTP 400; command-code/gpt-5.6-terra now HTTP 401 rather than the original 403
  - still dead, different failure signature.

- systems/scheduled-tasks.md: the Known-Issues entry repeating the
  "never re-authenticated" claim now carries the correction. Fleet figures
  re-checked against live jobs.json and remain accurate (29 jobs / 26 enabled /
  3 disabled, 5 non-clean last_status).

- log.md / index.md: stocktake entry appended and index last-updated refreshed.

Verified correct, no change: systems/homepage-dashboard (CT201 has 8084 and 8085
listening, no 8082, Homarr on 8102); CT201 Serve routes (/ -> 8085, /mgr -> 8084
on both :443 and :8443, Steel routes still gone); CT460 Serve routes (12 paths
on :443, plus :8443/webhook Funnel and :9443).

Noted for a human, not performed: vault mirror at /home/hermes/wiki/agent-estate-wiki
has drifted on both conventions (top-level 35 match / 3 stale / 72 missing;
wiki/ 41 match / 53 stale / 16 missing). A full mirror pass is larger than a
stocktake's mandate.
2026-10-29 03:42:54 +08:00

7.7 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route incident resolved 2026-10-06 2026-10-29 2026-10-29 high
cron
model-pin
omniroute
incident
resolved
live-jobs-json
live-omniroute-chat-completions-probe
cron-executions-db
cron-incidents-db
session-20260810_122140_d687279c

Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route

Window: failures 2026-10-01 → 2026-10-05; remediated 2026-10-05 23:16 → 2026-10-06 00:05 AWST Status: Resolved — all seven repinned, every failure_streak cleared to 0; three verified by live run Trigger: an interactive review on 2026-10-05 23:16 that began from a stale cron failure notice

Trigger: a six-day-old alert

The review started because Anthony replied to a cron failure notice for 85edf6772bca (Tech & AI Newsletter Digest (Analytical)) reading:

Restart-safe cron worker dispatch failed: cron external worker exited before ownership
acknowledgement (exit 1)
This job has failed 4 runs in a row — worth a review.

That alert was stale and the job was healthy. 85edf6772bca had completed on every run from 2026-09-30 17:53 through 2026-10-05 07:00, and all three of its incidents were already resolved as of 2026-09-30 17:55:13. The notice came from the 2026-09-29 07:00 run (incident 85edf6_7ab60a6b6d4a, first seen 2026-09-29 07:00:45) and had been sitting unread in the topic for six days.

It looked current because the wording matches the live 2026-09-28 managed-runtime incident signature (cron external worker exited before ownership acknowledgement) — see incidents/2026-09-28-hermes-managed-runtime-deps.

Rule: a failure notice in a topic is not evidence of a current failure. Check the job's executions row timestamp (and its cron_incidents state) before acting on a notice — a resolved incident still leaves its old message behind.

What was actually broken — seven jobs, four dead pins

The check prompted by the stale alert found the set the 2026-10-01 sweep had flagged, all still pinned to upstreams that no longer work:

Job Name Dead pin Error
1a8e90ea2271 wiki-stocktake-maintenance ollama-cloud / qwen3.5:397b HTTP 410 — retired 2026-09-25
dca8482e4f76 Daily Marketing Job Radar command-code / gpt-5.6-terra 403 MODEL_NOT_IN_PLAN
fad5a86979d4 Facebook Birthday Weekly Registry Sync nvidia / deepseek-v4-flash HTTP 400 — absent from live catalog
0e753e9720fd Facebook Birthday Monthly Scrape nvidia / deepseek-v4-flash HTTP 400 — same pin
06a0a1c8704f AI Newsletter Curation for Patrick custom:Omniroute:20128 / cliprox/terra HTTP 401 — connection auth expired
1b23c042e17d Enhanced Weekend Adventure Planner custom:Omniroute:20128 / cliprox/terra HTTP 401 — same pin
79f9bb41ecf5 Weekly Review custom:Omniroute:20128 / cliprox/terra HTTP 401 — same pin

Four distinct fault classes, none self-recovering: retired model (410), plan-gated model (403), catalog-missing model (400 ×2 jobs), provider-connection credential expiry (401 ×3 jobs). The 401 pair is documented separately in incidents/2026-10-01-cliproxy-connection-auth-expired.

Fix — repin to routing aliases

All seven moved to provider custom:omni-101 (OmniRoute on the estate), using routing aliases rather than another raw upstream model name:

Alias Jobs
free-stack 06a0a1c8704f, 1b23c042e17d, 79f9bb41ecf5, 1a8e90ea2271, fad5a86979d4, 0e753e9720fd
smart-route dca8482e4f76

The root cause was raw-pin brittleness — a raw model name dies silently when the upstream retires it, the catalog drops it, or a plan changes. An alias absorbs that. This is the same lesson as the 2026-08-04 free-DeepSeek outage.

free-stack is in service again: it was on the broken-combo list as recently as 2026-08-14. Its route was re-verified before being handed six jobs — do not assume the 2026-08 list is still current.

Verification

Job Evidence Outcome
79f9bb41ecf5 manual run 2026-10-05 23:39 succeeded; incident auto-resolved 23:39:12
06a0a1c8704f manual run 2026-10-05 23:45 succeeded; incident auto-resolved 23:45:01
dca8482e4f76 manual run 23:48 → completed 23:56 delivered to telegram:-1004321904721:1030 (message_id 9367); incident resolved 2026-10-06 00:05:07

Fleet-wide re-read after the repin: 30 jobs, 27 enabled / 3 disabled, failure_streak 0 on every job — down from seven jobs carrying streaks up to ×11.

Enabled fleet by pin after the fix: smart-route ×10, no-agent scripts ×8, free-stack ×6, deepseek-paid-smart ×2, stale custom:omni ×1.

Re-verification — 2026-10-29 03:0x AWST (stocktake)

This incident page had aged past the 21-day freshness threshold, so its claims were re-probed against live state rather than left on trust.

Claim Live probe 2026-10-29 Verdict
All 7 jobs repinned to custom:omni-101 with routing aliases jobs.json: 06a0a1c8704f free-stack, 1b23c042e17d free-stack, 79f9bb41ecf5 free-stack, 1a8e90ea2271 free-stack, fad5a86979d4 free-stack, 0e753e9720fd free-stack, dca8482e4f76 smart-route Holds
free-stack / smart-route are live aliases POST /v1/chat/completions on CT600 → HTTP 200 for both Holds
ollama-cloud/qwen3.5:397b retired HTTP 410 — "qwen3.5:397b was retired on September 25, 2026. Try glm-5.3-flash:cloud." Holds
nvidia/deepseek-v4-flash absent from live catalog HTTP 400 — "Model 'deepseek-v4-flash' is not available in the active live catalog for provider 'nvidia'." Holds
command-code/gpt-5.6-terra plan-gated HTTP 401 — "Invalid 'Authorization' header or token." Pin still dead, but the error class changed — the upstream now rejects the credential rather than the plan (403 → 401). The remediation verdict is unchanged; only the failure signature differs.
cliprox/terra connection never re-authenticated HTTP 200 on CT600; HTTP 401 (No active credentials for provider: cliprox.) on CT101 No longer holds — see the SUPERSEDED marker in Residual above

Residual

  • Four repinned jobs still carry the pre-repin last_status: error — 1b23c042e17d, fad5a86979d4, 0e753e9720fd, 1a8e90ea2271. Their streaks are already 0; the status clears on the next scheduled run. Do not read those stale statuses as live failures.
  • 066bb5191280 still records delivery_failed (bare telegram target — separate open issue).
  • The cliprox/terra connection itself was never re-authenticated. That pin remains unusable. ⚠ SUPERSEDED 2026-10-29 — this is no longer true. A live probe of cliprox/terra against custom:Omniroute:20128 (CT600) returned HTTP 200, so the connection has since been re-authenticated on CT600. The same pin on custom:omni-101 (CT101) still returns HTTP 401 (No active credentials for provider: cliprox.) — the nodes carry separate credentials. The repin itself was never reversed and remains correct. → incidents/2026-10-01-cliproxy-connection-auth-expired
  • The daily stocktake (1a8e90ea2271) had been blocked since 2026-09-25; with the repin it can run again at its 03:00 schedule, so wiki-maintain.sh no longer needs a manual run each day.