- new incident: incidents/2026-10-05-dead-model-pin-repin.md - scheduled-tasks.md: Count block -> 0 failing, all streaks 0; 8 job rows updated; new 'Resolved 2026-10-05' subsection; 5 resolved bullets pruned from Open - cliprox incident closed for cron (status: open -> resolved) + resolution section - index.md: new incident linked, cron line + cliprox line updated - current-state.md + log.md: 2026-10-06 entries
5.9 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OmniRoute cliprox/terra Connection Auth Expiry | incident | resolved | 2026-10-02 | 2026-10-06 | 2026-10-06 | high |
|
|
OmniRoute cliprox/terra Connection Auth Expiry
Window: first failure 2026-10-01 10:00 AWST (job schedule 0 10 * * 4); closed 2026-10-05 23:45 AWST
Status: Resolved for cron — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated
Impact: 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on cliprox/terra produced output while the credential was lapsed. Zero jobs remain on the pin.
Symptoms
Job 06a0a1c8704f (AI Newsletter Curation for Patrick, Thu 10:00) aborted with:
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
Provider custom:Omniroute:20128, model cliprox/terra. The run's failure notice was
delivered to forum topic 346, but no newsletter digest was produced.
Root Cause
A connection credential at the provider, not a missing or plan-gated model and not a
routing error. The error names a single connection (All 1 connection(s)) whose
authorisation lapsed, so the model pin itself is still valid — the credential behind it
is not.
This matters because it is the one failure class from the 2026-10-01 sweep that cannot self-recover. A retired or plan-gated model stays broken until repinned; an expired connection stays broken until a human re-authenticates it in the dashboard.
Blast Radius
Three enabled jobs run cliprox/terra:
| Job | Name | Schedule | Delivery |
|---|---|---|---|
06a0a1c8704f |
AI Newsletter Curation for Patrick | 0 10 * * 4 |
telegram:-1004321904721:346 |
1b23c042e17d |
Enhanced Weekend Adventure Planner | 0 8 * * 5 |
telegram:-1004321904721:102 |
79f9bb41ecf5 |
Weekly Review | 0 18 * * 0 |
origin |
Only 06a0a1c8704f had run by 2026-10-02; the other two were exposed on their next
scheduled runs.
Second job confirmed — 2026-10-02 08:00
1b23c042e17d (Enhanced Weekend Adventure Planner, Fri 08:00) failed on schedule
with the identical error:
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
That is now 2 of 3 exposed jobs confirmed broken on this pin, which retires any
doubt about a one-job fluke. Its failure notice was delivered to topic 102, so the
run is recorded delivery_outcome: delivered even though no plan was produced.
79f9bb41ecf5 (Weekly Review, Sun 18:00) was the last untested one — its only
recorded run was 2026-09-27 18:00, four days before the connection lapsed, and it
succeeded. Its next scheduled run was 2026-10-04 18:00.
Third job confirmed — 2026-10-04 18:00 · pin settled 3/3
79f9bb41ecf5 ran on schedule on 2026-10-04 18:00 and failed with the identical
error again:
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
That settles it: 3 of 3 jobs on cliprox/terra have now failed on the same
credential fault. There is no remaining doubt about scope — it is not per-job, and it
will not self-recover. Failure streak for 79f9bb41ecf5 is 1; no review was produced.
The run's delivery_outcome is again delivered (the failure notice is the
deliverable), so the execution status is the only honest signal.
Corollary worth recording: a job can be recorded as successfully delivered while
having failed, because the failure notice itself is a deliverable. Read the
execution status, not the delivery outcome, when judging whether a job worked.
Resolution — 2026-10-05 · closed for cron purposes
All three jobs were repinned off cliprox/terra on 2026-10-05 (23:39–23:45 AWST) as
part of the fleet-wide dead-model-pin sweep. 06a0a1c8704f and 1b23c042e17d moved to
custom:omni-101 / free-stack; 79f9bb41ecf5 moved to free-stack and was verified by
a live manual run at 23:39. No enabled job references this pin any more, every
failure_streak in jobs.json is 0, and the incident is closed for cron purposes.
Note precisely what was and was not fixed: the underlying OmniRoute connection credential
for cliprox/terra was never re-authenticated. The jobs were moved off the pin
instead. Any future job that targets cliprox/terra — or any manual session that calls it
— will hit the same HTTP 401 … connection(s) authentication expired until someone
reconnects it in the OmniRoute dashboard. That is the one piece of this incident still
outstanding, and it is outside the cron fleet's blast radius.
See incidents/2026-10-05-dead-model-pin-repin.
Fix (as originally diagnosed)
Re-authenticate the OmniRoute connection for cliprox/terra. Do not repin these jobs
to another raw model as a first move — that treats a credential fault as a routing fault
and will produce a second, different outage. If the connection cannot be restored, repin
all three jobs together to a routing alias (smart-route) — which is what was ultimately
done.
Distinguish From
- Plan-gated model (403
MODEL_NOT_IN_PLAN, jobdca8482e4f76) — the upstream rejects the model by plan; no credential involved. - Retired model (HTTP 410, job
1a8e90ea2271) — the model no longer exists upstream. - Catalog-missing model (HTTP 400, jobs
fad5a86979d4/0e753e9720fd) — the model name is absent from the provider's live catalog.