audit(2026-10-06): dead-model-pin repin resolved; failing set 7 -> 0

- new incident: incidents/2026-10-05-dead-model-pin-repin.md
- scheduled-tasks.md: Count block -> 0 failing, all streaks 0; 8 job rows updated;
  new 'Resolved 2026-10-05' subsection; 5 resolved bullets pruned from Open
- cliprox incident closed for cron (status: open -> resolved) + resolution section
- index.md: new incident linked, cron line + cliprox line updated
- current-state.md + log.md: 2026-10-06 entries
This commit is contained in:
Tony0410
2026-10-06 00:20:53 +08:00
parent 543e7782f8
commit 14b467e91b
6 changed files with 207 additions and 47 deletions
@@ -1,20 +1,20 @@
---
title: OmniRoute cliprox/terra Connection Auth Expiry
type: incident
status: open
status: resolved
created: 2026-10-02
updated: 2026-10-05
verified_on: 2026-10-05
updated: 2026-10-06
verified_on: 2026-10-06
confidence: high
tags: [cron, omniroute, provider, auth, incident]
tags: [cron, omniroute, provider, auth, incident, resolved]
sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
---
# OmniRoute `cliprox/terra` Connection Auth Expiry
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); still open
**Status:** Open — not self-recovering; **all 3 exposed jobs now confirmed broken**
**Impact:** 3 of 3 jobs on this pin have failed on it; no job on `cliprox/terra` has produced output since the credential lapsed
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); closed 2026-10-05 23:45 AWST
**Status:** **Resolved for cron** — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated
**Impact:** 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on `cliprox/terra` produced output while the credential was lapsed. Zero jobs remain on the pin.
## Symptoms
@@ -90,12 +90,29 @@ Corollary worth recording: a job can be recorded as **successfully delivered whi
having failed**, because the failure notice itself is a deliverable. Read the
execution `status`, not the delivery outcome, when judging whether a job worked.
## Fix
## Resolution — 2026-10-05 · closed for cron purposes
All three jobs were **repinned off `cliprox/terra`** on 2026-10-05 (23:39–23:45 AWST) as
part of the fleet-wide dead-model-pin sweep. `06a0a1c8704f` and `1b23c042e17d` moved to
`custom:omni-101` / `free-stack`; `79f9bb41ecf5` moved to `free-stack` and was verified by
a live manual run at 23:39. No enabled job references this pin any more, every
`failure_streak` in `jobs.json` is 0, and the incident is closed **for cron purposes**.
Note precisely what was and was not fixed: the underlying OmniRoute connection credential
for `cliprox/terra` was **never re-authenticated**. The jobs were moved off the pin
instead. Any future job that targets `cliprox/terra` — or any manual session that calls it
— will hit the same `HTTP 401 … connection(s) authentication expired` until someone
reconnects it in the OmniRoute dashboard. That is the one piece of this incident still
outstanding, and it is outside the cron fleet's blast radius.
See [[incidents/2026-10-05-dead-model-pin-repin]].
## Fix (as originally diagnosed)
Re-authenticate the OmniRoute connection for `cliprox/terra`. Do **not** repin these jobs
to another raw model as a first move — that treats a credential fault as a routing fault
and will produce a second, different outage. If the connection cannot be restored, repin
all three jobs together to a routing alias (`smart-route`).
all three jobs together to a routing alias (`smart-route`) — which is what was ultimately
done.
## Distinguish From
@@ -107,6 +124,7 @@ all three jobs together to a routing alias (`smart-route`).
## Related
- [[incidents/2026-10-05-dead-model-pin-repin]]
- [[systems/scheduled-tasks]]
- [[systems/model-providers]]
- [[systems/omniroute]]