stocktake 2026-11-06 (pass 2): re-verify the one page sitting on the 21-day threshold
incidents/2026-10-14-omniroute-node-connection-refused-storm.md was at exactly 21d and
would have tripped the threshold on the next run. Its residual claim ("omniroute-1
answers again") was re-probed live and still holds: TCP connect to 100.78.63.20:20128
succeeds (no Errno 111), the Tailscale peer reads Online:true, and a keyed
POST /v1/chat/completions returns HTTP 200. Page re-verified (updated/verified_on ->
2026-11-06, review_after -> 2026-12-06) with the probe recorded on the page.
New trap recorded on the page and in systems/scheduled-tasks.md: on omniroute-1 the
RAW model id deepseek/deepseek-v4-flash returns HTTP 401 "No active credentials for
provider: deepseek", while the fleet's deepseek-paid-smart alias returns HTTP 200 and
resolves to that same model. The 401 is an alias-routing artefact, not an outage - probe
the alias the fleet pins, never the raw model id. (A first probe using the wrong
key_env also produced a 401; omni-101's key env is HERMES_CUSTOM_100_78_63_20_20128_API_KEY.)
Freshness re-run after the edits: 0 stale / 124 fresh / 3 no-date of 127 pages.
Quartz rebuilt and redeployed. Vault mirror synced to both conventions (sha-verified).
This commit is contained in:
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Node Outage — Connection-Refused Storm
|
||||
type: concept
|
||||
status: resolved
|
||||
created: 2026-10-16
|
||||
updated: 2026-10-16
|
||||
verified_on: 2026-10-16
|
||||
review_after: 2026-10-23
|
||||
updated: 2026-11-06
|
||||
verified_on: 2026-11-06
|
||||
review_after: 2026-12-06
|
||||
confidence: high
|
||||
tags: [cron, router, provider, infrastructure]
|
||||
sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, hermes-errors-log, live-probe]
|
||||
@@ -15,6 +15,18 @@ sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, herm
|
||||
|
||||
**Window:** 2026-10-14 08:30 → 2026-10-15 18:07 AWST (**~34 hours**)
|
||||
**Status:** **Resolved.** `omniroute-1` answers again; four scheduled jobs have since completed on it.
|
||||
|
||||
**Re-verified 2026-11-06 (stocktake) — the residual claim still holds, and the node is genuinely healthy.**
|
||||
Live probe of the incident's own endpoint: TCP **connect to `100.78.63.20:20128` succeeds** (no
|
||||
`Errno 111 Connection refused` — the incident's root cause has not recurred), the Tailscale peer
|
||||
`omniroute-1` reads `Online: true`, and a keyed `POST /v1/chat/completions` returns **HTTP 200**.
|
||||
`auto/best-chat` → `codestral-2508` (200), `auto/best-reasoning` → `deepseek-ai/deepseek-v4.1-flash` (200),
|
||||
and the fleet's `deepseek-paid-smart` pin → `deepseek/deepseek-v4-flash` (**200**) — confirming the
|
||||
[[systems/scheduled-tasks]] note that this pin dispatches on the `omniroute-1` node. ⚠ **One trap worth
|
||||
recording:** requesting the **raw** model id `deepseek/deepseek-v4-flash` (rather than the
|
||||
`deepseek-paid-smart` alias) returns **HTTP 401 `No active credentials for provider: deepseek`** — that is
|
||||
an alias-routing artefact on this node, **not** an outage and **not** the incident recurring. Probe the
|
||||
**alias** the fleet actually uses.
|
||||
**Impact:** **36 failed execution rows across 13 jobs** — 24 genuine model-call failures on 12 jobs, plus
|
||||
12 watchdog *symptom* reds. **Every model call attempted in the window failed** (see below). For scale:
|
||||
10-12 recorded `0 failed` and 10-13 recorded `1`.
|
||||
|
||||
Reference in New Issue
Block a user