Files
agent-estate-wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md
T
Tony0410 14b467e91b audit(2026-10-06): dead-model-pin repin resolved; failing set 7 -> 0
- new incident: incidents/2026-10-05-dead-model-pin-repin.md
- scheduled-tasks.md: Count block -> 0 failing, all streaks 0; 8 job rows updated;
  new 'Resolved 2026-10-05' subsection; 5 resolved bullets pruned from Open
- cliprox incident closed for cron (status: open -> resolved) + resolution section
- index.md: new incident linked, cron line + cliprox line updated
- current-state.md + log.md: 2026-10-06 entries
2026-10-06 00:20:53 +08:00

5.9 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
OmniRoute cliprox/terra Connection Auth Expiry incident resolved 2026-10-02 2026-10-06 2026-10-06 high
cron
omniroute
provider
auth
incident
resolved
live-jobs-json
cron-executions-db
cron-session-cron_06a0a1c8704f_20261001_100040
cron-session-cron_79f9bb41ecf5_20261004_180050

OmniRoute cliprox/terra Connection Auth Expiry

Window: first failure 2026-10-01 10:00 AWST (job schedule 0 10 * * 4); closed 2026-10-05 23:45 AWST Status: Resolved for cron — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated Impact: 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on cliprox/terra produced output while the credential was lapsed. Zero jobs remain on the pin.

Symptoms

Job 06a0a1c8704f (AI Newsletter Curation for Patrick, Thu 10:00) aborted with:

HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard

Provider custom:Omniroute:20128, model cliprox/terra. The run's failure notice was delivered to forum topic 346, but no newsletter digest was produced.

Root Cause

A connection credential at the provider, not a missing or plan-gated model and not a routing error. The error names a single connection (All 1 connection(s)) whose authorisation lapsed, so the model pin itself is still valid — the credential behind it is not.

This matters because it is the one failure class from the 2026-10-01 sweep that cannot self-recover. A retired or plan-gated model stays broken until repinned; an expired connection stays broken until a human re-authenticates it in the dashboard.

Blast Radius

Three enabled jobs run cliprox/terra:

Job Name Schedule Delivery
06a0a1c8704f AI Newsletter Curation for Patrick 0 10 * * 4 telegram:-1004321904721:346
1b23c042e17d Enhanced Weekend Adventure Planner 0 8 * * 5 telegram:-1004321904721:102
79f9bb41ecf5 Weekly Review 0 18 * * 0 origin

Only 06a0a1c8704f had run by 2026-10-02; the other two were exposed on their next scheduled runs.

Second job confirmed — 2026-10-02 08:00

1b23c042e17d (Enhanced Weekend Adventure Planner, Fri 08:00) failed on schedule with the identical error:

HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard

That is now 2 of 3 exposed jobs confirmed broken on this pin, which retires any doubt about a one-job fluke. Its failure notice was delivered to topic 102, so the run is recorded delivery_outcome: delivered even though no plan was produced.

79f9bb41ecf5 (Weekly Review, Sun 18:00) was the last untested one — its only recorded run was 2026-09-27 18:00, four days before the connection lapsed, and it succeeded. Its next scheduled run was 2026-10-04 18:00.

Third job confirmed — 2026-10-04 18:00 · pin settled 3/3

79f9bb41ecf5 ran on schedule on 2026-10-04 18:00 and failed with the identical error again:

HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard

That settles it: 3 of 3 jobs on cliprox/terra have now failed on the same credential fault. There is no remaining doubt about scope — it is not per-job, and it will not self-recover. Failure streak for 79f9bb41ecf5 is 1; no review was produced. The run's delivery_outcome is again delivered (the failure notice is the deliverable), so the execution status is the only honest signal.

Corollary worth recording: a job can be recorded as successfully delivered while having failed, because the failure notice itself is a deliverable. Read the execution status, not the delivery outcome, when judging whether a job worked.

Resolution — 2026-10-05 · closed for cron purposes

All three jobs were repinned off cliprox/terra on 2026-10-05 (23:39–23:45 AWST) as part of the fleet-wide dead-model-pin sweep. 06a0a1c8704f and 1b23c042e17d moved to custom:omni-101 / free-stack; 79f9bb41ecf5 moved to free-stack and was verified by a live manual run at 23:39. No enabled job references this pin any more, every failure_streak in jobs.json is 0, and the incident is closed for cron purposes.

Note precisely what was and was not fixed: the underlying OmniRoute connection credential for cliprox/terra was never re-authenticated. The jobs were moved off the pin instead. Any future job that targets cliprox/terra — or any manual session that calls it — will hit the same HTTP 401 … connection(s) authentication expired until someone reconnects it in the OmniRoute dashboard. That is the one piece of this incident still outstanding, and it is outside the cron fleet's blast radius. See incidents/2026-10-05-dead-model-pin-repin.

Fix (as originally diagnosed)

Re-authenticate the OmniRoute connection for cliprox/terra. Do not repin these jobs to another raw model as a first move — that treats a credential fault as a routing fault and will produce a second, different outage. If the connection cannot be restored, repin all three jobs together to a routing alias (smart-route) — which is what was ultimately done.

Distinguish From

  • Plan-gated model (403 MODEL_NOT_IN_PLAN, job dca8482e4f76) — the upstream rejects the model by plan; no credential involved.
  • Retired model (HTTP 410, job 1a8e90ea2271) — the model no longer exists upstream.
  • Catalog-missing model (HTTP 400, jobs fad5a86979d4 / 0e753e9720fd) — the model name is absent from the provider's live catalog.