Files
agent-estate-wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md
T

113 lines
4.6 KiB
Markdown

---
title: OmniRoute cliprox/terra Connection Auth Expiry
type: incident
status: open
created: 2026-10-02
updated: 2026-10-05
verified_on: 2026-10-05
confidence: high
tags: [cron, omniroute, provider, auth, incident]
sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
---
# OmniRoute `cliprox/terra` Connection Auth Expiry
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); still open
**Status:** Open — not self-recovering; **all 3 exposed jobs now confirmed broken**
**Impact:** 3 of 3 jobs on this pin have failed on it; no job on `cliprox/terra` has produced output since the credential lapsed
## Symptoms
Job `06a0a1c8704f` (**AI Newsletter Curation for Patrick**, Thu 10:00) aborted with:
```
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
```
Provider `custom:Omniroute:20128`, model `cliprox/terra`. The run's failure notice was
delivered to forum topic `346`, but no newsletter digest was produced.
## Root Cause
A **connection credential at the provider**, not a missing or plan-gated model and not a
routing error. The error names a single connection (`All 1 connection(s)`) whose
authorisation lapsed, so the model pin itself is still valid — the *credential behind it*
is not.
This matters because it is the one failure class from the 2026-10-01 sweep that **cannot
self-recover**. A retired or plan-gated model stays broken until repinned; an expired
connection stays broken until a human re-authenticates it in the dashboard.
## Blast Radius
Three enabled jobs run `cliprox/terra`:
| Job | Name | Schedule | Delivery |
|---|---|---|---|
| `06a0a1c8704f` | AI Newsletter Curation for Patrick | `0 10 * * 4` | `telegram:-1004321904721:346` |
| `1b23c042e17d` | Enhanced Weekend Adventure Planner | `0 8 * * 5` | `telegram:-1004321904721:102` |
| `79f9bb41ecf5` | Weekly Review | `0 18 * * 0` | `origin` |
Only `06a0a1c8704f` had run by 2026-10-02; the other two were exposed on their next
scheduled runs.
### Second job confirmed — 2026-10-02 08:00
`1b23c042e17d` (**Enhanced Weekend Adventure Planner**, Fri 08:00) failed on schedule
with the *identical* error:
```
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
```
That is now **2 of 3** exposed jobs confirmed broken on this pin, which retires any
doubt about a one-job fluke. Its failure notice was delivered to topic `102`, so the
run is recorded `delivery_outcome: delivered` even though no plan was produced.
`79f9bb41ecf5` (**Weekly Review**, Sun 18:00) was the last untested one — its only
recorded run was 2026-09-27 18:00, four days *before* the connection lapsed, and it
succeeded. Its next scheduled run was **2026-10-04 18:00**.
### Third job confirmed — 2026-10-04 18:00 · pin settled 3/3
`79f9bb41ecf5` ran on schedule on 2026-10-04 18:00 and failed with the *identical*
error again:
```
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
```
That settles it: **3 of 3** jobs on `cliprox/terra` have now failed on the same
credential fault. There is no remaining doubt about scope — it is not per-job, and it
will not self-recover. Failure streak for `79f9bb41ecf5` is 1; no review was produced.
The run's `delivery_outcome` is again `delivered` (the failure notice is the
deliverable), so the execution `status` is the only honest signal.
Corollary worth recording: a job can be recorded as **successfully delivered while
having failed**, because the failure notice itself is a deliverable. Read the
execution `status`, not the delivery outcome, when judging whether a job worked.
## Fix
Re-authenticate the OmniRoute connection for `cliprox/terra`. Do **not** repin these jobs
to another raw model as a first move — that treats a credential fault as a routing fault
and will produce a second, different outage. If the connection cannot be restored, repin
all three jobs together to a routing alias (`smart-route`).
## Distinguish From
- **Plan-gated model** (403 `MODEL_NOT_IN_PLAN`, job `dca8482e4f76`) — the upstream rejects
the model by plan; no credential involved.
- **Retired model** (HTTP 410, job `1a8e90ea2271`) — the model no longer exists upstream.
- **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model
name is absent from the provider's live catalog.
## Related
- [[systems/scheduled-tasks]]
- [[systems/model-providers]]
- [[systems/omniroute]]