Files
agent-estate-wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md
T
Tony0410 14b467e91b audit(2026-10-06): dead-model-pin repin resolved; failing set 7 -> 0
- new incident: incidents/2026-10-05-dead-model-pin-repin.md
- scheduled-tasks.md: Count block -> 0 failing, all streaks 0; 8 job rows updated;
  new 'Resolved 2026-10-05' subsection; 5 resolved bullets pruned from Open
- cliprox incident closed for cron (status: open -> resolved) + resolution section
- index.md: new incident linked, cron line + cliprox line updated
- current-state.md + log.md: 2026-10-06 entries
2026-10-06 00:20:53 +08:00

131 lines
5.9 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: OmniRoute cliprox/terra Connection Auth Expiry
type: incident
status: resolved
created: 2026-10-02
updated: 2026-10-06
verified_on: 2026-10-06
confidence: high
tags: [cron, omniroute, provider, auth, incident, resolved]
sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
---
# OmniRoute `cliprox/terra` Connection Auth Expiry
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); closed 2026-10-05 23:45 AWST
**Status:** **Resolved for cron** — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated
**Impact:** 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on `cliprox/terra` produced output while the credential was lapsed. Zero jobs remain on the pin.
## Symptoms
Job `06a0a1c8704f` (**AI Newsletter Curation for Patrick**, Thu 10:00) aborted with:
```
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
```
Provider `custom:Omniroute:20128`, model `cliprox/terra`. The run's failure notice was
delivered to forum topic `346`, but no newsletter digest was produced.
## Root Cause
A **connection credential at the provider**, not a missing or plan-gated model and not a
routing error. The error names a single connection (`All 1 connection(s)`) whose
authorisation lapsed, so the model pin itself is still valid — the *credential behind it*
is not.
This matters because it is the one failure class from the 2026-10-01 sweep that **cannot
self-recover**. A retired or plan-gated model stays broken until repinned; an expired
connection stays broken until a human re-authenticates it in the dashboard.
## Blast Radius
Three enabled jobs run `cliprox/terra`:
| Job | Name | Schedule | Delivery |
|---|---|---|---|
| `06a0a1c8704f` | AI Newsletter Curation for Patrick | `0 10 * * 4` | `telegram:-1004321904721:346` |
| `1b23c042e17d` | Enhanced Weekend Adventure Planner | `0 8 * * 5` | `telegram:-1004321904721:102` |
| `79f9bb41ecf5` | Weekly Review | `0 18 * * 0` | `origin` |
Only `06a0a1c8704f` had run by 2026-10-02; the other two were exposed on their next
scheduled runs.
### Second job confirmed — 2026-10-02 08:00
`1b23c042e17d` (**Enhanced Weekend Adventure Planner**, Fri 08:00) failed on schedule
with the *identical* error:
```
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
```
That is now **2 of 3** exposed jobs confirmed broken on this pin, which retires any
doubt about a one-job fluke. Its failure notice was delivered to topic `102`, so the
run is recorded `delivery_outcome: delivered` even though no plan was produced.
`79f9bb41ecf5` (**Weekly Review**, Sun 18:00) was the last untested one — its only
recorded run was 2026-09-27 18:00, four days *before* the connection lapsed, and it
succeeded. Its next scheduled run was **2026-10-04 18:00**.
### Third job confirmed — 2026-10-04 18:00 · pin settled 3/3
`79f9bb41ecf5` ran on schedule on 2026-10-04 18:00 and failed with the *identical*
error again:
```
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
```
That settles it: **3 of 3** jobs on `cliprox/terra` have now failed on the same
credential fault. There is no remaining doubt about scope — it is not per-job, and it
will not self-recover. Failure streak for `79f9bb41ecf5` is 1; no review was produced.
The run's `delivery_outcome` is again `delivered` (the failure notice is the
deliverable), so the execution `status` is the only honest signal.
Corollary worth recording: a job can be recorded as **successfully delivered while
having failed**, because the failure notice itself is a deliverable. Read the
execution `status`, not the delivery outcome, when judging whether a job worked.
## Resolution — 2026-10-05 · closed for cron purposes
All three jobs were **repinned off `cliprox/terra`** on 2026-10-05 (23:39–23:45 AWST) as
part of the fleet-wide dead-model-pin sweep. `06a0a1c8704f` and `1b23c042e17d` moved to
`custom:omni-101` / `free-stack`; `79f9bb41ecf5` moved to `free-stack` and was verified by
a live manual run at 23:39. No enabled job references this pin any more, every
`failure_streak` in `jobs.json` is 0, and the incident is closed **for cron purposes**.
Note precisely what was and was not fixed: the underlying OmniRoute connection credential
for `cliprox/terra` was **never re-authenticated**. The jobs were moved off the pin
instead. Any future job that targets `cliprox/terra` — or any manual session that calls it
— will hit the same `HTTP 401 … connection(s) authentication expired` until someone
reconnects it in the OmniRoute dashboard. That is the one piece of this incident still
outstanding, and it is outside the cron fleet's blast radius.
See [[incidents/2026-10-05-dead-model-pin-repin]].
## Fix (as originally diagnosed)
Re-authenticate the OmniRoute connection for `cliprox/terra`. Do **not** repin these jobs
to another raw model as a first move — that treats a credential fault as a routing fault
and will produce a second, different outage. If the connection cannot be restored, repin
all three jobs together to a routing alias (`smart-route`) — which is what was ultimately
done.
## Distinguish From
- **Plan-gated model** (403 `MODEL_NOT_IN_PLAN`, job `dca8482e4f76`) — the upstream rejects
the model by plan; no credential involved.
- **Retired model** (HTTP 410, job `1a8e90ea2271`) — the model no longer exists upstream.
- **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model
name is absent from the provider's live catalog.
## Related
- [[incidents/2026-10-05-dead-model-pin-repin]]
- [[systems/scheduled-tasks]]
- [[systems/model-providers]]
- [[systems/omniroute]]