7.3 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OmniRoute cliprox/terra Connection Auth Expiry | incident | resolved | 2026-10-02 | 2026-11-20 | 2026-11-20 | high |
|
|
OmniRoute cliprox/terra Connection Auth Expiry
Window: first failure 2026-10-01 10:00 AWST (job schedule 0 10 * * 4); closed 2026-10-05 23:45 AWST
Status: Resolved for cron — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated as part of this incident (⚠ corrected 2026-10-29 — it has since been re-authenticated on CT600; see the Resolution section)
Impact: 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on cliprox/terra produced output while the credential was lapsed. Zero jobs remain on the pin.
Symptoms
Job 06a0a1c8704f (AI Newsletter Curation for Patrick, Thu 10:00) aborted with:
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
Provider custom:Omniroute:20128, model cliprox/terra. The run's failure notice was
delivered to forum topic 346, but no newsletter digest was produced.
Root Cause
A connection credential at the provider, not a missing or plan-gated model and not a
routing error. The error names a single connection (All 1 connection(s)) whose
authorisation lapsed, so the model pin itself is still valid — the credential behind it
is not.
This matters because it is the one failure class from the 2026-10-01 sweep that cannot self-recover. A retired or plan-gated model stays broken until repinned; an expired connection stays broken until a human re-authenticates it in the dashboard.
Blast Radius
Three enabled jobs run cliprox/terra:
| Job | Name | Schedule | Delivery |
|---|---|---|---|
06a0a1c8704f |
AI Newsletter Curation for Patrick | 0 10 * * 4 |
telegram:-1004321904721:346 |
1b23c042e17d |
Enhanced Weekend Adventure Planner | 0 8 * * 5 |
telegram:-1004321904721:102 |
79f9bb41ecf5 |
Weekly Review | 0 18 * * 0 |
origin |
Only 06a0a1c8704f had run by 2026-10-02; the other two were exposed on their next
scheduled runs.
Second job confirmed — 2026-10-02 08:00
1b23c042e17d (Enhanced Weekend Adventure Planner, Fri 08:00) failed on schedule
with the identical error:
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
That is now 2 of 3 exposed jobs confirmed broken on this pin, which retires any
doubt about a one-job fluke. Its failure notice was delivered to topic 102, so the
run is recorded delivery_outcome: delivered even though no plan was produced.
79f9bb41ecf5 (Weekly Review, Sun 18:00) was the last untested one — its only
recorded run was 2026-09-27 18:00, four days before the connection lapsed, and it
succeeded. Its next scheduled run was 2026-10-04 18:00.
Third job confirmed — 2026-10-04 18:00 · pin settled 3/3
79f9bb41ecf5 ran on schedule on 2026-10-04 18:00 and failed with the identical
error again:
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
That settles it: 3 of 3 jobs on cliprox/terra have now failed on the same
credential fault. There is no remaining doubt about scope — it is not per-job, and it
will not self-recover. Failure streak for 79f9bb41ecf5 is 1; no review was produced.
The run's delivery_outcome is again delivered (the failure notice is the
deliverable), so the execution status is the only honest signal.
Corollary worth recording: a job can be recorded as successfully delivered while
having failed, because the failure notice itself is a deliverable. Read the
execution status, not the delivery outcome, when judging whether a job worked.
Resolution — 2026-10-05 · closed for cron purposes
All three jobs were repinned off cliprox/terra on 2026-10-05 (23:39–23:45 AWST) as
part of the fleet-wide dead-model-pin sweep. 06a0a1c8704f and 1b23c042e17d moved to
custom:omni-101 / free-stack; 79f9bb41ecf5 moved to free-stack and was verified by
a live manual run at 23:39. No enabled job references this pin any more, every
failure_streak in jobs.json is 0, and the incident is closed for cron purposes.
Note precisely what was and was not fixed: the underlying OmniRoute connection credential
for cliprox/terra was never re-authenticated as part of this incident. The jobs were moved off the pin
instead.
⚠ CORRECTION 2026-10-29 — the "never re-authenticated" residual no longer holds. A live probe of
cliprox/terraagainst providercustom:Omniroute:20128(CT600,http://omniroute:20128/v1) returned HTTP 200 with a completed completion — so the connection has since been re-authenticated on CT600, independently of this incident's remediation. The same model on the other OmniRoute node (custom:omni-101, CT101,http://100.78.63.20:20128) still returns HTTP 401 (No active credentials for provider: cliprox.) — the two nodes carry different connection credentials. The original remediation (moving the three jobs off the pin) remains correct and was never reversed; what has changed is that the pin is no longer unusable estate-wide. Any future job that targetscliprox/terraon CT600 will now succeed; the same pin on CT101 will still hitHTTP 401 … connection(s) authentication expired. Zero cron jobs reference the pin either way. → incidents/2026-10-05-dead-model-pin-repin
Fix (as originally diagnosed)
Re-authenticate the OmniRoute connection for cliprox/terra. Do not repin these jobs
to another raw model as a first move — that treats a credential fault as a routing fault
and will produce a second, different outage. If the connection cannot be restored, repin
all three jobs together to a routing alias (smart-route) — which is what was ultimately
done.
Distinguish From
- Plan-gated model (403
MODEL_NOT_IN_PLAN, jobdca8482e4f76) — the upstream rejects the model by plan; no credential involved. - Retired model (HTTP 410, job
1a8e90ea2271) — the model no longer exists upstream. - Catalog-missing model (HTTP 400, jobs
fad5a86979d4/0e753e9720fd) — the model name is absent from the provider's live catalog.
Re-verified 2026-11-20 (stocktake). The 2026-10-29 correction still holds, re-probed live:
cliprox/terraoncustom:Omniroute:20128(CT600) → HTTP 200 (resolves to…/terra); oncustom:omni-101(CT101) → HTTP 401No active credentials for provider: cliprox.— per-node credentials unchanged.jobs.json: zero jobs referencecliproxanywhere; all three originally-exposed jobs (06a0a1c8704f,1b23c042e17d,79f9bb41ecf5) remain oncustom:omni-101/free-stack,failure_streak: 0,last_status: ok. Resolution stands.