Wiki maintenance (wiki-maintain.sh, exit 0) + live-vs-page drift audit.
Script:
- systems/skills-index.md regenerated from the live skill tree: 302 skills in 69
categories; 45 archived skills under ~/.hermes/skills/.archive/ now excluded and
noted. (303 SKILL.md files exist outside .archive/, but
productivity/obsidian-git-sync/obsidian-git-sync is a nested alias duplicate,
so 302 unique is correct.)
- Freshness report raw/stocktake/freshness-20261029.txt: 2 stale / 105 fresh /
3 no-date. After the corrections below and a second script run: 0 stale /
107 fresh / 3 no-date.
- Quartz rebuilt (110 pages -> 644 files) and deployed to public/.
Drift corrections (live-verified):
- infrastructure/tailscale.md: State column rebuilt from tailscale status --json,
11 rows changed. Two were state transitions, not age drift:
* sftpgo (CT299, pve) is back online - tailscale ping pong in 0s,
systemctl is-active sftpgo -> active, tailscale ip -4 inside the guest.
* m93p is online despite a stale peer record (Online: false in the JSON) -
tailscale ping pong in 1ms, SSH over the tailnet returned up 11 days.
Second occurrence of this pattern after 2026-10-26.
The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged
at 54 (53 live IPv4 peers + hermes); 76 peers total, 23 IPv6-only funnel
ingress. No missing peers, no IP mismatches. Container identity re-confirmed
from inside all 34 guests via pct exec tailscale ip -4 on both hypervisors;
CT601 still absent from both.
- incidents/2026-10-01-cliproxy-connection-auth-expired.md: the residual claim
"the connection credential itself was never re-authenticated" is falsified.
Live POST /v1/chat/completions for cliprox/terra against custom:Omniroute:20128
(CT600) returns HTTP 200 - the connection has since been re-authenticated. The
same pin on custom:omni-101 (CT101) still returns HTTP 401, so the nodes carry
separate credentials. Claim marked superseded, not deleted (SCHEMA rule 9); the
remediation was never reversed and remains correct.
- incidents/2026-10-05-dead-model-pin-repin.md: every claim re-probed. All 7 jobs
still on custom:omni-101 with routing aliases; both aliases HTTP 200;
ollama-cloud/qwen3.5:397b still HTTP 410; nvidia/deepseek-v4-flash still
HTTP 400; command-code/gpt-5.6-terra now HTTP 401 rather than the original 403
- still dead, different failure signature.
- systems/scheduled-tasks.md: the Known-Issues entry repeating the
"never re-authenticated" claim now carries the correction. Fleet figures
re-checked against live jobs.json and remain accurate (29 jobs / 26 enabled /
3 disabled, 5 non-clean last_status).
- log.md / index.md: stocktake entry appended and index last-updated refreshed.
Verified correct, no change: systems/homepage-dashboard (CT201 has 8084 and 8085
listening, no 8082, Homarr on 8102); CT201 Serve routes (/ -> 8085, /mgr -> 8084
on both :443 and :8443, Steel routes still gone); CT460 Serve routes (12 paths
on :443, plus :8443/webhook Funnel and :9443).
Noted for a human, not performed: vault mirror at /home/hermes/wiki/agent-estate-wiki
has drifted on both conventions (top-level 35 match / 3 stale / 72 missing;
wiki/ 41 match / 53 stale / 16 missing). A full mirror pass is larger than a
stocktake's mandate.
139 lines
6.7 KiB
Markdown
139 lines
6.7 KiB
Markdown
---
|
||
title: OmniRoute cliprox/terra Connection Auth Expiry
|
||
type: incident
|
||
status: resolved
|
||
created: 2026-10-02
|
||
updated: 2026-10-29
|
||
verified_on: 2026-10-29
|
||
confidence: high
|
||
tags: [cron, omniroute, provider, auth, incident, resolved]
|
||
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
|
||
---
|
||
|
||
# OmniRoute `cliprox/terra` Connection Auth Expiry
|
||
|
||
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); closed 2026-10-05 23:45 AWST
|
||
**Status:** **Resolved for cron** — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated *as part of this incident* (⚠ **corrected 2026-10-29** — it has since been re-authenticated on CT600; see the Resolution section)
|
||
**Impact:** 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on `cliprox/terra` produced output while the credential was lapsed. Zero jobs remain on the pin.
|
||
|
||
## Symptoms
|
||
|
||
Job `06a0a1c8704f` (**AI Newsletter Curation for Patrick**, Thu 10:00) aborted with:
|
||
|
||
```
|
||
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
|
||
— please reconnect in the dashboard
|
||
```
|
||
|
||
Provider `custom:Omniroute:20128`, model `cliprox/terra`. The run's failure notice was
|
||
delivered to forum topic `346`, but no newsletter digest was produced.
|
||
|
||
## Root Cause
|
||
|
||
A **connection credential at the provider**, not a missing or plan-gated model and not a
|
||
routing error. The error names a single connection (`All 1 connection(s)`) whose
|
||
authorisation lapsed, so the model pin itself is still valid — the *credential behind it*
|
||
is not.
|
||
|
||
This matters because it is the one failure class from the 2026-10-01 sweep that **cannot
|
||
self-recover**. A retired or plan-gated model stays broken until repinned; an expired
|
||
connection stays broken until a human re-authenticates it in the dashboard.
|
||
|
||
## Blast Radius
|
||
|
||
Three enabled jobs run `cliprox/terra`:
|
||
|
||
| Job | Name | Schedule | Delivery |
|
||
|---|---|---|---|
|
||
| `06a0a1c8704f` | AI Newsletter Curation for Patrick | `0 10 * * 4` | `telegram:-1004321904721:346` |
|
||
| `1b23c042e17d` | Enhanced Weekend Adventure Planner | `0 8 * * 5` | `telegram:-1004321904721:102` |
|
||
| `79f9bb41ecf5` | Weekly Review | `0 18 * * 0` | `origin` |
|
||
|
||
Only `06a0a1c8704f` had run by 2026-10-02; the other two were exposed on their next
|
||
scheduled runs.
|
||
|
||
### Second job confirmed — 2026-10-02 08:00
|
||
|
||
`1b23c042e17d` (**Enhanced Weekend Adventure Planner**, Fri 08:00) failed on schedule
|
||
with the *identical* error:
|
||
|
||
```
|
||
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
|
||
— please reconnect in the dashboard
|
||
```
|
||
|
||
That is now **2 of 3** exposed jobs confirmed broken on this pin, which retires any
|
||
doubt about a one-job fluke. Its failure notice was delivered to topic `102`, so the
|
||
run is recorded `delivery_outcome: delivered` even though no plan was produced.
|
||
|
||
`79f9bb41ecf5` (**Weekly Review**, Sun 18:00) was the last untested one — its only
|
||
recorded run was 2026-09-27 18:00, four days *before* the connection lapsed, and it
|
||
succeeded. Its next scheduled run was **2026-10-04 18:00**.
|
||
|
||
### Third job confirmed — 2026-10-04 18:00 · pin settled 3/3
|
||
|
||
`79f9bb41ecf5` ran on schedule on 2026-10-04 18:00 and failed with the *identical*
|
||
error again:
|
||
|
||
```
|
||
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
|
||
— please reconnect in the dashboard
|
||
```
|
||
|
||
That settles it: **3 of 3** jobs on `cliprox/terra` have now failed on the same
|
||
credential fault. There is no remaining doubt about scope — it is not per-job, and it
|
||
will not self-recover. Failure streak for `79f9bb41ecf5` is 1; no review was produced.
|
||
The run's `delivery_outcome` is again `delivered` (the failure notice is the
|
||
deliverable), so the execution `status` is the only honest signal.
|
||
|
||
Corollary worth recording: a job can be recorded as **successfully delivered while
|
||
having failed**, because the failure notice itself is a deliverable. Read the
|
||
execution `status`, not the delivery outcome, when judging whether a job worked.
|
||
|
||
## Resolution — 2026-10-05 · closed for cron purposes
|
||
|
||
All three jobs were **repinned off `cliprox/terra`** on 2026-10-05 (23:39–23:45 AWST) as
|
||
part of the fleet-wide dead-model-pin sweep. `06a0a1c8704f` and `1b23c042e17d` moved to
|
||
`custom:omni-101` / `free-stack`; `79f9bb41ecf5` moved to `free-stack` and was verified by
|
||
a live manual run at 23:39. No enabled job references this pin any more, every
|
||
`failure_streak` in `jobs.json` is 0, and the incident is closed **for cron purposes**.
|
||
|
||
Note precisely what was and was not fixed: the underlying OmniRoute connection credential
|
||
for `cliprox/terra` was **never re-authenticated** as part of this incident. The jobs were moved off the pin
|
||
instead.
|
||
|
||
> **⚠ CORRECTION 2026-10-29 — the "never re-authenticated" residual no longer holds.** A live probe of
|
||
> `cliprox/terra` against provider `custom:Omniroute:20128` (CT600, `http://omniroute:20128/v1`) returned
|
||
> **HTTP 200** with a completed completion — so the connection **has since been re-authenticated** on CT600,
|
||
> independently of this incident's remediation. The same model on the *other* OmniRoute node
|
||
> (`custom:omni-101`, CT101, `http://100.78.63.20:20128`) still returns **HTTP 401**
|
||
> (`No active credentials for provider: cliprox.`) — the two nodes carry different connection credentials.
|
||
> The original remediation (moving the three jobs off the pin) remains correct and was never reversed; what
|
||
> has changed is that the pin is **no longer unusable estate-wide**. Any future job that targets
|
||
> `cliprox/terra` **on CT600** will now succeed; the same pin **on CT101** will still hit
|
||
> `HTTP 401 … connection(s) authentication expired`. Zero cron jobs reference the pin either way.
|
||
> → [[incidents/2026-10-05-dead-model-pin-repin]]
|
||
|
||
## Fix (as originally diagnosed)
|
||
|
||
Re-authenticate the OmniRoute connection for `cliprox/terra`. Do **not** repin these jobs
|
||
to another raw model as a first move — that treats a credential fault as a routing fault
|
||
and will produce a second, different outage. If the connection cannot be restored, repin
|
||
all three jobs together to a routing alias (`smart-route`) — which is what was ultimately
|
||
done.
|
||
|
||
## Distinguish From
|
||
|
||
- **Plan-gated model** (403 `MODEL_NOT_IN_PLAN`, job `dca8482e4f76`) — the upstream rejects
|
||
the model by plan; no credential involved.
|
||
- **Retired model** (HTTP 410, job `1a8e90ea2271`) — the model no longer exists upstream.
|
||
- **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model
|
||
name is absent from the provider's live catalog.
|
||
|
||
## Related
|
||
|
||
- [[incidents/2026-10-05-dead-model-pin-repin]]
|
||
- [[systems/scheduled-tasks]]
|
||
- [[systems/model-providers]]
|
||
- [[systems/omniroute]]
|