Files
agent-estate-wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md
T
Tony0410 cc95dcc54d stocktake: re-verify Tailscale table, correct falsified cliprox/terra claim, regenerate skills-index
Wiki maintenance (wiki-maintain.sh, exit 0) + live-vs-page drift audit.

Script:
- systems/skills-index.md regenerated from the live skill tree: 302 skills in 69
  categories; 45 archived skills under ~/.hermes/skills/.archive/ now excluded and
  noted. (303 SKILL.md files exist outside .archive/, but
  productivity/obsidian-git-sync/obsidian-git-sync is a nested alias duplicate,
  so 302 unique is correct.)
- Freshness report raw/stocktake/freshness-20261029.txt: 2 stale / 105 fresh /
  3 no-date. After the corrections below and a second script run: 0 stale /
  107 fresh / 3 no-date.
- Quartz rebuilt (110 pages -> 644 files) and deployed to public/.

Drift corrections (live-verified):
- infrastructure/tailscale.md: State column rebuilt from tailscale status --json,
  11 rows changed. Two were state transitions, not age drift:
  * sftpgo (CT299, pve) is back online - tailscale ping pong in 0s,
    systemctl is-active sftpgo -> active, tailscale ip -4 inside the guest.
  * m93p is online despite a stale peer record (Online: false in the JSON) -
    tailscale ping pong in 1ms, SSH over the tailnet returned up 11 days.
    Second occurrence of this pattern after 2026-10-26.
  The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged
  at 54 (53 live IPv4 peers + hermes); 76 peers total, 23 IPv6-only funnel
  ingress. No missing peers, no IP mismatches. Container identity re-confirmed
  from inside all 34 guests via pct exec tailscale ip -4 on both hypervisors;
  CT601 still absent from both.

- incidents/2026-10-01-cliproxy-connection-auth-expired.md: the residual claim
  "the connection credential itself was never re-authenticated" is falsified.
  Live POST /v1/chat/completions for cliprox/terra against custom:Omniroute:20128
  (CT600) returns HTTP 200 - the connection has since been re-authenticated. The
  same pin on custom:omni-101 (CT101) still returns HTTP 401, so the nodes carry
  separate credentials. Claim marked superseded, not deleted (SCHEMA rule 9); the
  remediation was never reversed and remains correct.

- incidents/2026-10-05-dead-model-pin-repin.md: every claim re-probed. All 7 jobs
  still on custom:omni-101 with routing aliases; both aliases HTTP 200;
  ollama-cloud/qwen3.5:397b still HTTP 410; nvidia/deepseek-v4-flash still
  HTTP 400; command-code/gpt-5.6-terra now HTTP 401 rather than the original 403
  - still dead, different failure signature.

- systems/scheduled-tasks.md: the Known-Issues entry repeating the
  "never re-authenticated" claim now carries the correction. Fleet figures
  re-checked against live jobs.json and remain accurate (29 jobs / 26 enabled /
  3 disabled, 5 non-clean last_status).

- log.md / index.md: stocktake entry appended and index last-updated refreshed.

Verified correct, no change: systems/homepage-dashboard (CT201 has 8084 and 8085
listening, no 8082, Homarr on 8102); CT201 Serve routes (/ -> 8085, /mgr -> 8084
on both :443 and :8443, Steel routes still gone); CT460 Serve routes (12 paths
on :443, plus :8443/webhook Funnel and :9443).

Noted for a human, not performed: vault mirror at /home/hermes/wiki/agent-estate-wiki
has drifted on both conventions (top-level 35 match / 3 stale / 72 missing;
wiki/ 41 match / 53 stale / 16 missing). A full mirror pass is larger than a
stocktake's mandate.
2026-10-29 03:42:54 +08:00

6.7 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
OmniRoute cliprox/terra Connection Auth Expiry incident resolved 2026-10-02 2026-10-29 2026-10-29 high
cron
omniroute
provider
auth
incident
resolved
live-jobs-json
live-omniroute-chat-completions-probe
cron-executions-db
cron-session-cron_06a0a1c8704f_20261001_100040
cron-session-cron_79f9bb41ecf5_20261004_180050

OmniRoute cliprox/terra Connection Auth Expiry

Window: first failure 2026-10-01 10:00 AWST (job schedule 0 10 * * 4); closed 2026-10-05 23:45 AWST Status: Resolved for cron — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated as part of this incident (⚠ corrected 2026-10-29 — it has since been re-authenticated on CT600; see the Resolution section) Impact: 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on cliprox/terra produced output while the credential was lapsed. Zero jobs remain on the pin.

Symptoms

Job 06a0a1c8704f (AI Newsletter Curation for Patrick, Thu 10:00) aborted with:

HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard

Provider custom:Omniroute:20128, model cliprox/terra. The run's failure notice was delivered to forum topic 346, but no newsletter digest was produced.

Root Cause

A connection credential at the provider, not a missing or plan-gated model and not a routing error. The error names a single connection (All 1 connection(s)) whose authorisation lapsed, so the model pin itself is still valid — the credential behind it is not.

This matters because it is the one failure class from the 2026-10-01 sweep that cannot self-recover. A retired or plan-gated model stays broken until repinned; an expired connection stays broken until a human re-authenticates it in the dashboard.

Blast Radius

Three enabled jobs run cliprox/terra:

Job Name Schedule Delivery
06a0a1c8704f AI Newsletter Curation for Patrick 0 10 * * 4 telegram:-1004321904721:346
1b23c042e17d Enhanced Weekend Adventure Planner 0 8 * * 5 telegram:-1004321904721:102
79f9bb41ecf5 Weekly Review 0 18 * * 0 origin

Only 06a0a1c8704f had run by 2026-10-02; the other two were exposed on their next scheduled runs.

Second job confirmed — 2026-10-02 08:00

1b23c042e17d (Enhanced Weekend Adventure Planner, Fri 08:00) failed on schedule with the identical error:

HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard

That is now 2 of 3 exposed jobs confirmed broken on this pin, which retires any doubt about a one-job fluke. Its failure notice was delivered to topic 102, so the run is recorded delivery_outcome: delivered even though no plan was produced.

79f9bb41ecf5 (Weekly Review, Sun 18:00) was the last untested one — its only recorded run was 2026-09-27 18:00, four days before the connection lapsed, and it succeeded. Its next scheduled run was 2026-10-04 18:00.

Third job confirmed — 2026-10-04 18:00 · pin settled 3/3

79f9bb41ecf5 ran on schedule on 2026-10-04 18:00 and failed with the identical error again:

HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard

That settles it: 3 of 3 jobs on cliprox/terra have now failed on the same credential fault. There is no remaining doubt about scope — it is not per-job, and it will not self-recover. Failure streak for 79f9bb41ecf5 is 1; no review was produced. The run's delivery_outcome is again delivered (the failure notice is the deliverable), so the execution status is the only honest signal.

Corollary worth recording: a job can be recorded as successfully delivered while having failed, because the failure notice itself is a deliverable. Read the execution status, not the delivery outcome, when judging whether a job worked.

Resolution — 2026-10-05 · closed for cron purposes

All three jobs were repinned off cliprox/terra on 2026-10-05 (23:39–23:45 AWST) as part of the fleet-wide dead-model-pin sweep. 06a0a1c8704f and 1b23c042e17d moved to custom:omni-101 / free-stack; 79f9bb41ecf5 moved to free-stack and was verified by a live manual run at 23:39. No enabled job references this pin any more, every failure_streak in jobs.json is 0, and the incident is closed for cron purposes.

Note precisely what was and was not fixed: the underlying OmniRoute connection credential for cliprox/terra was never re-authenticated as part of this incident. The jobs were moved off the pin instead.

⚠ CORRECTION 2026-10-29 — the "never re-authenticated" residual no longer holds. A live probe of cliprox/terra against provider custom:Omniroute:20128 (CT600, http://omniroute:20128/v1) returned HTTP 200 with a completed completion — so the connection has since been re-authenticated on CT600, independently of this incident's remediation. The same model on the other OmniRoute node (custom:omni-101, CT101, http://100.78.63.20:20128) still returns HTTP 401 (No active credentials for provider: cliprox.) — the two nodes carry different connection credentials. The original remediation (moving the three jobs off the pin) remains correct and was never reversed; what has changed is that the pin is no longer unusable estate-wide. Any future job that targets cliprox/terra on CT600 will now succeed; the same pin on CT101 will still hit HTTP 401 … connection(s) authentication expired. Zero cron jobs reference the pin either way. → incidents/2026-10-05-dead-model-pin-repin

Fix (as originally diagnosed)

Re-authenticate the OmniRoute connection for cliprox/terra. Do not repin these jobs to another raw model as a first move — that treats a credential fault as a routing fault and will produce a second, different outage. If the connection cannot be restored, repin all three jobs together to a routing alias (smart-route) — which is what was ultimately done.

Distinguish From

  • Plan-gated model (403 MODEL_NOT_IN_PLAN, job dca8482e4f76) — the upstream rejects the model by plan; no credential involved.
  • Retired model (HTTP 410, job 1a8e90ea2271) — the model no longer exists upstream.
  • Catalog-missing model (HTTP 400, jobs fad5a86979d4 / 0e753e9720fd) — the model name is absent from the provider's live catalog.