stocktake: re-verify Tailscale table, correct falsified cliprox/terra claim, regenerate skills-index
Wiki maintenance (wiki-maintain.sh, exit 0) + live-vs-page drift audit.
Script:
- systems/skills-index.md regenerated from the live skill tree: 302 skills in 69
categories; 45 archived skills under ~/.hermes/skills/.archive/ now excluded and
noted. (303 SKILL.md files exist outside .archive/, but
productivity/obsidian-git-sync/obsidian-git-sync is a nested alias duplicate,
so 302 unique is correct.)
- Freshness report raw/stocktake/freshness-20261029.txt: 2 stale / 105 fresh /
3 no-date. After the corrections below and a second script run: 0 stale /
107 fresh / 3 no-date.
- Quartz rebuilt (110 pages -> 644 files) and deployed to public/.
Drift corrections (live-verified):
- infrastructure/tailscale.md: State column rebuilt from tailscale status --json,
11 rows changed. Two were state transitions, not age drift:
* sftpgo (CT299, pve) is back online - tailscale ping pong in 0s,
systemctl is-active sftpgo -> active, tailscale ip -4 inside the guest.
* m93p is online despite a stale peer record (Online: false in the JSON) -
tailscale ping pong in 1ms, SSH over the tailnet returned up 11 days.
Second occurrence of this pattern after 2026-10-26.
The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged
at 54 (53 live IPv4 peers + hermes); 76 peers total, 23 IPv6-only funnel
ingress. No missing peers, no IP mismatches. Container identity re-confirmed
from inside all 34 guests via pct exec tailscale ip -4 on both hypervisors;
CT601 still absent from both.
- incidents/2026-10-01-cliproxy-connection-auth-expired.md: the residual claim
"the connection credential itself was never re-authenticated" is falsified.
Live POST /v1/chat/completions for cliprox/terra against custom:Omniroute:20128
(CT600) returns HTTP 200 - the connection has since been re-authenticated. The
same pin on custom:omni-101 (CT101) still returns HTTP 401, so the nodes carry
separate credentials. Claim marked superseded, not deleted (SCHEMA rule 9); the
remediation was never reversed and remains correct.
- incidents/2026-10-05-dead-model-pin-repin.md: every claim re-probed. All 7 jobs
still on custom:omni-101 with routing aliases; both aliases HTTP 200;
ollama-cloud/qwen3.5:397b still HTTP 410; nvidia/deepseek-v4-flash still
HTTP 400; command-code/gpt-5.6-terra now HTTP 401 rather than the original 403
- still dead, different failure signature.
- systems/scheduled-tasks.md: the Known-Issues entry repeating the
"never re-authenticated" claim now carries the correction. Fleet figures
re-checked against live jobs.json and remain accurate (29 jobs / 26 enabled /
3 disabled, 5 non-clean last_status).
- log.md / index.md: stocktake entry appended and index last-updated refreshed.
Verified correct, no change: systems/homepage-dashboard (CT201 has 8084 and 8085
listening, no 8082, Homarr on 8102); CT201 Serve routes (/ -> 8085, /mgr -> 8084
on both :443 and :8443, Steel routes still gone); CT460 Serve routes (12 paths
on :443, plus :8443/webhook Funnel and :9443).
Noted for a human, not performed: vault mirror at /home/hermes/wiki/agent-estate-wiki
has drifted on both conventions (top-level 35 match / 3 stale / 72 missing;
wiki/ 41 match / 53 stale / 16 missing). A full mirror pass is larger than a
stocktake's mandate.
This commit is contained in:
@@ -3,17 +3,17 @@ title: OmniRoute cliprox/terra Connection Auth Expiry
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-02
|
||||
updated: 2026-10-06
|
||||
verified_on: 2026-10-06
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
confidence: high
|
||||
tags: [cron, omniroute, provider, auth, incident, resolved]
|
||||
sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
|
||||
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
|
||||
---
|
||||
|
||||
# OmniRoute `cliprox/terra` Connection Auth Expiry
|
||||
|
||||
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); closed 2026-10-05 23:45 AWST
|
||||
**Status:** **Resolved for cron** — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated
|
||||
**Status:** **Resolved for cron** — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated *as part of this incident* (⚠ **corrected 2026-10-29** — it has since been re-authenticated on CT600; see the Resolution section)
|
||||
**Impact:** 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on `cliprox/terra` produced output while the credential was lapsed. Zero jobs remain on the pin.
|
||||
|
||||
## Symptoms
|
||||
@@ -99,12 +99,20 @@ a live manual run at 23:39. No enabled job references this pin any more, every
|
||||
`failure_streak` in `jobs.json` is 0, and the incident is closed **for cron purposes**.
|
||||
|
||||
Note precisely what was and was not fixed: the underlying OmniRoute connection credential
|
||||
for `cliprox/terra` was **never re-authenticated**. The jobs were moved off the pin
|
||||
instead. Any future job that targets `cliprox/terra` — or any manual session that calls it
|
||||
— will hit the same `HTTP 401 … connection(s) authentication expired` until someone
|
||||
reconnects it in the OmniRoute dashboard. That is the one piece of this incident still
|
||||
outstanding, and it is outside the cron fleet's blast radius.
|
||||
See [[incidents/2026-10-05-dead-model-pin-repin]].
|
||||
for `cliprox/terra` was **never re-authenticated** as part of this incident. The jobs were moved off the pin
|
||||
instead.
|
||||
|
||||
> **⚠ CORRECTION 2026-10-29 — the "never re-authenticated" residual no longer holds.** A live probe of
|
||||
> `cliprox/terra` against provider `custom:Omniroute:20128` (CT600, `http://omniroute:20128/v1`) returned
|
||||
> **HTTP 200** with a completed completion — so the connection **has since been re-authenticated** on CT600,
|
||||
> independently of this incident's remediation. The same model on the *other* OmniRoute node
|
||||
> (`custom:omni-101`, CT101, `http://100.78.63.20:20128`) still returns **HTTP 401**
|
||||
> (`No active credentials for provider: cliprox.`) — the two nodes carry different connection credentials.
|
||||
> The original remediation (moving the three jobs off the pin) remains correct and was never reversed; what
|
||||
> has changed is that the pin is **no longer unusable estate-wide**. Any future job that targets
|
||||
> `cliprox/terra` **on CT600** will now succeed; the same pin **on CT101** will still hit
|
||||
> `HTTP 401 … connection(s) authentication expired`. Zero cron jobs reference the pin either way.
|
||||
> → [[incidents/2026-10-05-dead-model-pin-repin]]
|
||||
|
||||
## Fix (as originally diagnosed)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user