audit(2026-10-06): dead-model-pin repin resolved; failing set 7 -> 0
- new incident: incidents/2026-10-05-dead-model-pin-repin.md - scheduled-tasks.md: Count block -> 0 failing, all streaks 0; 8 job rows updated; new 'Resolved 2026-10-05' subsection; 5 resolved bullets pruned from Open - cliprox incident closed for cron (status: open -> resolved) + resolution section - index.md: new incident linked, cron line + cliprox line updated - current-state.md + log.md: 2026-10-06 entries
This commit is contained in:
@@ -1,20 +1,20 @@
|
||||
---
|
||||
title: OmniRoute cliprox/terra Connection Auth Expiry
|
||||
type: incident
|
||||
status: open
|
||||
status: resolved
|
||||
created: 2026-10-02
|
||||
updated: 2026-10-05
|
||||
verified_on: 2026-10-05
|
||||
updated: 2026-10-06
|
||||
verified_on: 2026-10-06
|
||||
confidence: high
|
||||
tags: [cron, omniroute, provider, auth, incident]
|
||||
tags: [cron, omniroute, provider, auth, incident, resolved]
|
||||
sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
|
||||
---
|
||||
|
||||
# OmniRoute `cliprox/terra` Connection Auth Expiry
|
||||
|
||||
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); still open
|
||||
**Status:** Open — not self-recovering; **all 3 exposed jobs now confirmed broken**
|
||||
**Impact:** 3 of 3 jobs on this pin have failed on it; no job on `cliprox/terra` has produced output since the credential lapsed
|
||||
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); closed 2026-10-05 23:45 AWST
|
||||
**Status:** **Resolved for cron** — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated
|
||||
**Impact:** 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on `cliprox/terra` produced output while the credential was lapsed. Zero jobs remain on the pin.
|
||||
|
||||
## Symptoms
|
||||
|
||||
@@ -90,12 +90,29 @@ Corollary worth recording: a job can be recorded as **successfully delivered whi
|
||||
having failed**, because the failure notice itself is a deliverable. Read the
|
||||
execution `status`, not the delivery outcome, when judging whether a job worked.
|
||||
|
||||
## Fix
|
||||
## Resolution — 2026-10-05 · closed for cron purposes
|
||||
|
||||
All three jobs were **repinned off `cliprox/terra`** on 2026-10-05 (23:39–23:45 AWST) as
|
||||
part of the fleet-wide dead-model-pin sweep. `06a0a1c8704f` and `1b23c042e17d` moved to
|
||||
`custom:omni-101` / `free-stack`; `79f9bb41ecf5` moved to `free-stack` and was verified by
|
||||
a live manual run at 23:39. No enabled job references this pin any more, every
|
||||
`failure_streak` in `jobs.json` is 0, and the incident is closed **for cron purposes**.
|
||||
|
||||
Note precisely what was and was not fixed: the underlying OmniRoute connection credential
|
||||
for `cliprox/terra` was **never re-authenticated**. The jobs were moved off the pin
|
||||
instead. Any future job that targets `cliprox/terra` — or any manual session that calls it
|
||||
— will hit the same `HTTP 401 … connection(s) authentication expired` until someone
|
||||
reconnects it in the OmniRoute dashboard. That is the one piece of this incident still
|
||||
outstanding, and it is outside the cron fleet's blast radius.
|
||||
See [[incidents/2026-10-05-dead-model-pin-repin]].
|
||||
|
||||
## Fix (as originally diagnosed)
|
||||
|
||||
Re-authenticate the OmniRoute connection for `cliprox/terra`. Do **not** repin these jobs
|
||||
to another raw model as a first move — that treats a credential fault as a routing fault
|
||||
and will produce a second, different outage. If the connection cannot be restored, repin
|
||||
all three jobs together to a routing alias (`smart-route`).
|
||||
all three jobs together to a routing alias (`smart-route`) — which is what was ultimately
|
||||
done.
|
||||
|
||||
## Distinguish From
|
||||
|
||||
@@ -107,6 +124,7 @@ all three jobs together to a routing alias (`smart-route`).
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-05-dead-model-pin-repin]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/model-providers]]
|
||||
- [[systems/omniroute]]
|
||||
|
||||
Reference in New Issue
Block a user