--- title: OmniRoute cliprox/terra Connection Auth Expiry type: incident status: resolved created: 2026-10-02 updated: 2026-10-29 verified_on: 2026-10-29 confidence: high tags: [cron, omniroute, provider, auth, incident, resolved] sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050] --- # OmniRoute `cliprox/terra` Connection Auth Expiry **Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); closed 2026-10-05 23:45 AWST **Status:** **Resolved for cron** — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated *as part of this incident* (⚠ **corrected 2026-10-29** — it has since been re-authenticated on CT600; see the Resolution section) **Impact:** 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on `cliprox/terra` produced output while the credential was lapsed. Zero jobs remain on the pin. ## Symptoms Job `06a0a1c8704f` (**AI Newsletter Curation for Patrick**, Thu 10:00) aborted with: ``` HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard ``` Provider `custom:Omniroute:20128`, model `cliprox/terra`. The run's failure notice was delivered to forum topic `346`, but no newsletter digest was produced. ## Root Cause A **connection credential at the provider**, not a missing or plan-gated model and not a routing error. The error names a single connection (`All 1 connection(s)`) whose authorisation lapsed, so the model pin itself is still valid — the *credential behind it* is not. This matters because it is the one failure class from the 2026-10-01 sweep that **cannot self-recover**. A retired or plan-gated model stays broken until repinned; an expired connection stays broken until a human re-authenticates it in the dashboard. ## Blast Radius Three enabled jobs run `cliprox/terra`: | Job | Name | Schedule | Delivery | |---|---|---|---| | `06a0a1c8704f` | AI Newsletter Curation for Patrick | `0 10 * * 4` | `telegram:-1004321904721:346` | | `1b23c042e17d` | Enhanced Weekend Adventure Planner | `0 8 * * 5` | `telegram:-1004321904721:102` | | `79f9bb41ecf5` | Weekly Review | `0 18 * * 0` | `origin` | Only `06a0a1c8704f` had run by 2026-10-02; the other two were exposed on their next scheduled runs. ### Second job confirmed — 2026-10-02 08:00 `1b23c042e17d` (**Enhanced Weekend Adventure Planner**, Fri 08:00) failed on schedule with the *identical* error: ``` HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard ``` That is now **2 of 3** exposed jobs confirmed broken on this pin, which retires any doubt about a one-job fluke. Its failure notice was delivered to topic `102`, so the run is recorded `delivery_outcome: delivered` even though no plan was produced. `79f9bb41ecf5` (**Weekly Review**, Sun 18:00) was the last untested one — its only recorded run was 2026-09-27 18:00, four days *before* the connection lapsed, and it succeeded. Its next scheduled run was **2026-10-04 18:00**. ### Third job confirmed — 2026-10-04 18:00 · pin settled 3/3 `79f9bb41ecf5` ran on schedule on 2026-10-04 18:00 and failed with the *identical* error again: ``` HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard ``` That settles it: **3 of 3** jobs on `cliprox/terra` have now failed on the same credential fault. There is no remaining doubt about scope — it is not per-job, and it will not self-recover. Failure streak for `79f9bb41ecf5` is 1; no review was produced. The run's `delivery_outcome` is again `delivered` (the failure notice is the deliverable), so the execution `status` is the only honest signal. Corollary worth recording: a job can be recorded as **successfully delivered while having failed**, because the failure notice itself is a deliverable. Read the execution `status`, not the delivery outcome, when judging whether a job worked. ## Resolution — 2026-10-05 · closed for cron purposes All three jobs were **repinned off `cliprox/terra`** on 2026-10-05 (23:39–23:45 AWST) as part of the fleet-wide dead-model-pin sweep. `06a0a1c8704f` and `1b23c042e17d` moved to `custom:omni-101` / `free-stack`; `79f9bb41ecf5` moved to `free-stack` and was verified by a live manual run at 23:39. No enabled job references this pin any more, every `failure_streak` in `jobs.json` is 0, and the incident is closed **for cron purposes**. Note precisely what was and was not fixed: the underlying OmniRoute connection credential for `cliprox/terra` was **never re-authenticated** as part of this incident. The jobs were moved off the pin instead. > **⚠ CORRECTION 2026-10-29 — the "never re-authenticated" residual no longer holds.** A live probe of > `cliprox/terra` against provider `custom:Omniroute:20128` (CT600, `http://omniroute:20128/v1`) returned > **HTTP 200** with a completed completion — so the connection **has since been re-authenticated** on CT600, > independently of this incident's remediation. The same model on the *other* OmniRoute node > (`custom:omni-101`, CT101, `http://100.78.63.20:20128`) still returns **HTTP 401** > (`No active credentials for provider: cliprox.`) — the two nodes carry different connection credentials. > The original remediation (moving the three jobs off the pin) remains correct and was never reversed; what > has changed is that the pin is **no longer unusable estate-wide**. Any future job that targets > `cliprox/terra` **on CT600** will now succeed; the same pin **on CT101** will still hit > `HTTP 401 … connection(s) authentication expired`. Zero cron jobs reference the pin either way. > → [[incidents/2026-10-05-dead-model-pin-repin]] ## Fix (as originally diagnosed) Re-authenticate the OmniRoute connection for `cliprox/terra`. Do **not** repin these jobs to another raw model as a first move — that treats a credential fault as a routing fault and will produce a second, different outage. If the connection cannot be restored, repin all three jobs together to a routing alias (`smart-route`) — which is what was ultimately done. ## Distinguish From - **Plan-gated model** (403 `MODEL_NOT_IN_PLAN`, job `dca8482e4f76`) — the upstream rejects the model by plan; no credential involved. - **Retired model** (HTTP 410, job `1a8e90ea2271`) — the model no longer exists upstream. - **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model name is absent from the provider's live catalog. ## Related - [[incidents/2026-10-05-dead-model-pin-repin]] - [[systems/scheduled-tasks]] - [[systems/model-providers]] - [[systems/omniroute]]