Files
agent-estate-wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md
T

145 lines
7.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: OmniRoute cliprox/terra Connection Auth Expiry
type: incident
status: resolved
created: 2026-10-02
updated: 2026-11-20
verified_on: 2026-11-20
confidence: high
tags: [cron, omniroute, provider, auth, incident, resolved]
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
---
# OmniRoute `cliprox/terra` Connection Auth Expiry
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); closed 2026-10-05 23:45 AWST
**Status:** **Resolved for cron** — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated *as part of this incident* (⚠ **corrected 2026-10-29** — it has since been re-authenticated on CT600; see the Resolution section)
**Impact:** 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on `cliprox/terra` produced output while the credential was lapsed. Zero jobs remain on the pin.
## Symptoms
Job `06a0a1c8704f` (**AI Newsletter Curation for Patrick**, Thu 10:00) aborted with:
```
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
```
Provider `custom:Omniroute:20128`, model `cliprox/terra`. The run's failure notice was
delivered to forum topic `346`, but no newsletter digest was produced.
## Root Cause
A **connection credential at the provider**, not a missing or plan-gated model and not a
routing error. The error names a single connection (`All 1 connection(s)`) whose
authorisation lapsed, so the model pin itself is still valid — the *credential behind it*
is not.
This matters because it is the one failure class from the 2026-10-01 sweep that **cannot
self-recover**. A retired or plan-gated model stays broken until repinned; an expired
connection stays broken until a human re-authenticates it in the dashboard.
## Blast Radius
Three enabled jobs run `cliprox/terra`:
| Job | Name | Schedule | Delivery |
|---|---|---|---|
| `06a0a1c8704f` | AI Newsletter Curation for Patrick | `0 10 * * 4` | `telegram:-1004321904721:346` |
| `1b23c042e17d` | Enhanced Weekend Adventure Planner | `0 8 * * 5` | `telegram:-1004321904721:102` |
| `79f9bb41ecf5` | Weekly Review | `0 18 * * 0` | `origin` |
Only `06a0a1c8704f` had run by 2026-10-02; the other two were exposed on their next
scheduled runs.
### Second job confirmed — 2026-10-02 08:00
`1b23c042e17d` (**Enhanced Weekend Adventure Planner**, Fri 08:00) failed on schedule
with the *identical* error:
```
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
```
That is now **2 of 3** exposed jobs confirmed broken on this pin, which retires any
doubt about a one-job fluke. Its failure notice was delivered to topic `102`, so the
run is recorded `delivery_outcome: delivered` even though no plan was produced.
`79f9bb41ecf5` (**Weekly Review**, Sun 18:00) was the last untested one — its only
recorded run was 2026-09-27 18:00, four days *before* the connection lapsed, and it
succeeded. Its next scheduled run was **2026-10-04 18:00**.
### Third job confirmed — 2026-10-04 18:00 · pin settled 3/3
`79f9bb41ecf5` ran on schedule on 2026-10-04 18:00 and failed with the *identical*
error again:
```
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
```
That settles it: **3 of 3** jobs on `cliprox/terra` have now failed on the same
credential fault. There is no remaining doubt about scope — it is not per-job, and it
will not self-recover. Failure streak for `79f9bb41ecf5` is 1; no review was produced.
The run's `delivery_outcome` is again `delivered` (the failure notice is the
deliverable), so the execution `status` is the only honest signal.
Corollary worth recording: a job can be recorded as **successfully delivered while
having failed**, because the failure notice itself is a deliverable. Read the
execution `status`, not the delivery outcome, when judging whether a job worked.
## Resolution — 2026-10-05 · closed for cron purposes
All three jobs were **repinned off `cliprox/terra`** on 2026-10-05 (23:39–23:45 AWST) as
part of the fleet-wide dead-model-pin sweep. `06a0a1c8704f` and `1b23c042e17d` moved to
`custom:omni-101` / `free-stack`; `79f9bb41ecf5` moved to `free-stack` and was verified by
a live manual run at 23:39. No enabled job references this pin any more, every
`failure_streak` in `jobs.json` is 0, and the incident is closed **for cron purposes**.
Note precisely what was and was not fixed: the underlying OmniRoute connection credential
for `cliprox/terra` was **never re-authenticated** as part of this incident. The jobs were moved off the pin
instead.
> **⚠ CORRECTION 2026-10-29 — the "never re-authenticated" residual no longer holds.** A live probe of
> `cliprox/terra` against provider `custom:Omniroute:20128` (CT600, `http://omniroute:20128/v1`) returned
> **HTTP 200** with a completed completion — so the connection **has since been re-authenticated** on CT600,
> independently of this incident's remediation. The same model on the *other* OmniRoute node
> (`custom:omni-101`, CT101, `http://100.78.63.20:20128`) still returns **HTTP 401**
> (`No active credentials for provider: cliprox.`) — the two nodes carry different connection credentials.
> The original remediation (moving the three jobs off the pin) remains correct and was never reversed; what
> has changed is that the pin is **no longer unusable estate-wide**. Any future job that targets
> `cliprox/terra` **on CT600** will now succeed; the same pin **on CT101** will still hit
> `HTTP 401 … connection(s) authentication expired`. Zero cron jobs reference the pin either way.
> → [[incidents/2026-10-05-dead-model-pin-repin]]
## Fix (as originally diagnosed)
Re-authenticate the OmniRoute connection for `cliprox/terra`. Do **not** repin these jobs
to another raw model as a first move — that treats a credential fault as a routing fault
and will produce a second, different outage. If the connection cannot be restored, repin
all three jobs together to a routing alias (`smart-route`) — which is what was ultimately
done.
## Distinguish From
- **Plan-gated model** (403 `MODEL_NOT_IN_PLAN`, job `dca8482e4f76`) — the upstream rejects
the model by plan; no credential involved.
- **Retired model** (HTTP 410, job `1a8e90ea2271`) — the model no longer exists upstream.
- **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model
name is absent from the provider's live catalog.
> **Re-verified 2026-11-20 (stocktake).** The 2026-10-29 correction still holds, re-probed live:
> `cliprox/terra` on `custom:Omniroute:20128` (CT600) → **HTTP 200** (resolves to `…/terra`);
> on `custom:omni-101` (CT101) → **HTTP 401** `No active credentials for provider: cliprox.` —
> per-node credentials unchanged. `jobs.json`: **zero** jobs reference `cliprox` anywhere; all three
> originally-exposed jobs (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`) remain on
> `custom:omni-101` / `free-stack`, `failure_streak: 0`, `last_status: ok`. Resolution stands.
## Related
- [[incidents/2026-10-05-dead-model-pin-repin]]
- [[systems/scheduled-tasks]]
- [[systems/model-providers]]
- [[systems/omniroute]]