Files
agent-estate-wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md
T

4.6 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
OmniRoute cliprox/terra Connection Auth Expiry incident open 2026-10-02 2026-10-05 2026-10-05 high
cron
omniroute
provider
auth
incident
live-jobs-json
cron-executions-db
cron-session-cron_06a0a1c8704f_20261001_100040
cron-session-cron_79f9bb41ecf5_20261004_180050

OmniRoute cliprox/terra Connection Auth Expiry

Window: first failure 2026-10-01 10:00 AWST (job schedule 0 10 * * 4); still open Status: Open — not self-recovering; all 3 exposed jobs now confirmed broken Impact: 3 of 3 jobs on this pin have failed on it; no job on cliprox/terra has produced output since the credential lapsed

Symptoms

Job 06a0a1c8704f (AI Newsletter Curation for Patrick, Thu 10:00) aborted with:

HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard

Provider custom:Omniroute:20128, model cliprox/terra. The run's failure notice was delivered to forum topic 346, but no newsletter digest was produced.

Root Cause

A connection credential at the provider, not a missing or plan-gated model and not a routing error. The error names a single connection (All 1 connection(s)) whose authorisation lapsed, so the model pin itself is still valid — the credential behind it is not.

This matters because it is the one failure class from the 2026-10-01 sweep that cannot self-recover. A retired or plan-gated model stays broken until repinned; an expired connection stays broken until a human re-authenticates it in the dashboard.

Blast Radius

Three enabled jobs run cliprox/terra:

Job Name Schedule Delivery
06a0a1c8704f AI Newsletter Curation for Patrick 0 10 * * 4 telegram:-1004321904721:346
1b23c042e17d Enhanced Weekend Adventure Planner 0 8 * * 5 telegram:-1004321904721:102
79f9bb41ecf5 Weekly Review 0 18 * * 0 origin

Only 06a0a1c8704f had run by 2026-10-02; the other two were exposed on their next scheduled runs.

Second job confirmed — 2026-10-02 08:00

1b23c042e17d (Enhanced Weekend Adventure Planner, Fri 08:00) failed on schedule with the identical error:

HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard

That is now 2 of 3 exposed jobs confirmed broken on this pin, which retires any doubt about a one-job fluke. Its failure notice was delivered to topic 102, so the run is recorded delivery_outcome: delivered even though no plan was produced.

79f9bb41ecf5 (Weekly Review, Sun 18:00) was the last untested one — its only recorded run was 2026-09-27 18:00, four days before the connection lapsed, and it succeeded. Its next scheduled run was 2026-10-04 18:00.

Third job confirmed — 2026-10-04 18:00 · pin settled 3/3

79f9bb41ecf5 ran on schedule on 2026-10-04 18:00 and failed with the identical error again:

HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard

That settles it: 3 of 3 jobs on cliprox/terra have now failed on the same credential fault. There is no remaining doubt about scope — it is not per-job, and it will not self-recover. Failure streak for 79f9bb41ecf5 is 1; no review was produced. The run's delivery_outcome is again delivered (the failure notice is the deliverable), so the execution status is the only honest signal.

Corollary worth recording: a job can be recorded as successfully delivered while having failed, because the failure notice itself is a deliverable. Read the execution status, not the delivery outcome, when judging whether a job worked.

Fix

Re-authenticate the OmniRoute connection for cliprox/terra. Do not repin these jobs to another raw model as a first move — that treats a credential fault as a routing fault and will produce a second, different outage. If the connection cannot be restored, repin all three jobs together to a routing alias (smart-route).

Distinguish From

  • Plan-gated model (403 MODEL_NOT_IN_PLAN, job dca8482e4f76) — the upstream rejects the model by plan; no credential involved.
  • Retired model (HTTP 410, job 1a8e90ea2271) — the model no longer exists upstream.
  • Catalog-missing model (HTTP 400, jobs fad5a86979d4 / 0e753e9720fd) — the model name is absent from the provider's live catalog.