stocktake 2026-11-06 (pass 2): re-verify the one page sitting on the 21-day threshold

incidents/2026-10-14-omniroute-node-connection-refused-storm.md was at exactly 21d and
would have tripped the threshold on the next run. Its residual claim ("omniroute-1
answers again") was re-probed live and still holds: TCP connect to 100.78.63.20:20128
succeeds (no Errno 111), the Tailscale peer reads Online:true, and a keyed
POST /v1/chat/completions returns HTTP 200. Page re-verified (updated/verified_on ->
2026-11-06, review_after -> 2026-12-06) with the probe recorded on the page.

New trap recorded on the page and in systems/scheduled-tasks.md: on omniroute-1 the
RAW model id deepseek/deepseek-v4-flash returns HTTP 401 "No active credentials for
provider: deepseek", while the fleet's deepseek-paid-smart alias returns HTTP 200 and
resolves to that same model. The 401 is an alias-routing artefact, not an outage - probe
the alias the fleet pins, never the raw model id. (A first probe using the wrong
key_env also produced a 401; omni-101's key env is HERMES_CUSTOM_100_78_63_20_20128_API_KEY.)

Freshness re-run after the edits: 0 stale / 124 fresh / 3 no-date of 127 pages.
Quartz rebuilt and redeployed. Vault mirror synced to both conventions (sha-verified).
This commit is contained in:
Tony0410
2026-11-06 04:07:12 +08:00
parent 88baf4fb5a
commit 2d5219b068
4 changed files with 35 additions and 5 deletions
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Node Outage — Connection-Refused Storm
type: concept type: concept
status: resolved status: resolved
created: 2026-10-16 created: 2026-10-16
updated: 2026-10-16 updated: 2026-11-06
verified_on: 2026-10-16 verified_on: 2026-11-06
review_after: 2026-10-23 review_after: 2026-12-06
confidence: high confidence: high
tags: [cron, router, provider, infrastructure] tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, hermes-errors-log, live-probe] sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, hermes-errors-log, live-probe]
@@ -15,6 +15,18 @@ sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, herm
**Window:** 2026-10-14 08:30 → 2026-10-15 18:07 AWST (**~34 hours**) **Window:** 2026-10-14 08:30 → 2026-10-15 18:07 AWST (**~34 hours**)
**Status:** **Resolved.** `omniroute-1` answers again; four scheduled jobs have since completed on it. **Status:** **Resolved.** `omniroute-1` answers again; four scheduled jobs have since completed on it.
**Re-verified 2026-11-06 (stocktake) — the residual claim still holds, and the node is genuinely healthy.**
Live probe of the incident's own endpoint: TCP **connect to `100.78.63.20:20128` succeeds** (no
`Errno 111 Connection refused` — the incident's root cause has not recurred), the Tailscale peer
`omniroute-1` reads `Online: true`, and a keyed `POST /v1/chat/completions` returns **HTTP 200**.
`auto/best-chat` → `codestral-2508` (200), `auto/best-reasoning` → `deepseek-ai/deepseek-v4.1-flash` (200),
and the fleet's `deepseek-paid-smart` pin → `deepseek/deepseek-v4-flash` (**200**) — confirming the
[[systems/scheduled-tasks]] note that this pin dispatches on the `omniroute-1` node. ⚠ **One trap worth
recording:** requesting the **raw** model id `deepseek/deepseek-v4-flash` (rather than the
`deepseek-paid-smart` alias) returns **HTTP 401 `No active credentials for provider: deepseek`** — that is
an alias-routing artefact on this node, **not** an outage and **not** the incident recurring. Probe the
**alias** the fleet actually uses.
**Impact:** **36 failed execution rows across 13 jobs** — 24 genuine model-call failures on 12 jobs, plus **Impact:** **36 failed execution rows across 13 jobs** — 24 genuine model-call failures on 12 jobs, plus
12 watchdog *symptom* reds. **Every model call attempted in the window failed** (see below). For scale: 12 watchdog *symptom* reds. **Every model call attempted in the window failed** (see below). For scale:
10-12 recorded `0 failed` and 10-13 recorded `1`. 10-12 recorded `0 failed` and 10-13 recorded `1`.
+13
View File
@@ -2171,3 +2171,16 @@ embedded credential was seen during verification and is not reproduced here ([RE
`deepseek-paid-smart`-on-`omniroute` dispatch fault and the `066bb5191280` delivery failure carried, not `deepseek-paid-smart`-on-`omniroute` dispatch fault and the `066bb5191280` delivery failure carried, not
re-diagnosed; the open secrets-at-rest incident not re-litigated. **No secrets, tokens or keys written or re-diagnosed; the open secrets-at-rest incident not re-litigated. **No secrets, tokens or keys written or
quoted.** quoted.**
- **Follow-up (04:0x AWST): the one page sitting exactly on the 21-day threshold was re-verified rather
than left to expire.** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` was at **21d** and
would have tripped the threshold tomorrow. Its residual claim ("`omniroute-1` answers again") was
re-probed **live** and still holds: TCP connect to `100.78.63.20:20128` succeeds (no `Errno 111`),
the Tailscale peer reads `Online: true`, and a keyed `POST /v1/chat/completions` returns **HTTP 200**.
Page re-verified (`updated`/`verified_on` -> 2026-11-06, `review_after` -> 2026-12-06) with the probe
recorded. ⚠ **New trap recorded on the page and in [[systems/scheduled-tasks]]:** on `omniroute-1` the
**raw** model id `deepseek/deepseek-v4-flash` returns **HTTP 401 `No active credentials for provider:
deepseek`**, while the fleet's **`deepseek-paid-smart` alias returns HTTP 200** and resolves to that same
model. The 401 is an **alias-routing artefact, not an outage** — probe the alias the fleet pins, never the
raw model id. (A first probe with the wrong `key_env` also produced a 401; the correct key for
`omni-101` is `HERMES_CUSTOM_100_78_63_20_20128_API_KEY`.)
+1 -1
View File
@@ -46,7 +46,7 @@ Threshold: >21 days since `updated`
- [ 8d] incidents/2026-10-05-dead-model-pin-repin.md - [ 8d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 13d] incidents/2026-10-09-cron-provider-timeout-storm.md - [ 13d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 5d] incidents/2026-10-09-memory-recall-relevance.md - [ 5d] incidents/2026-10-09-memory-recall-relevance.md
- [ 21d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md - [ 0d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 10d] incidents/2026-10-16-omniroute-503-resource-pressure.md - [ 10d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 18d] incidents/2026-10-18-degenerate-stream-canned-substitute.md - [ 18d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 10d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md - [ 10d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
+6 -1
View File
@@ -15,7 +15,12 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
## Count ## Count
- **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. - **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. -> [[runbooks/job-radar]] `CL_MODEL` caveat. - **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. ⚠ **Re-verified
2026-11-06 (stocktake): on `omniroute-1` (`100.78.63.20`) the raw model id `deepseek/deepseek-v4-flash`
returns **HTTP 401 `No active credentials for provider: deepseek`** while the fleet's **`deepseek-paid-smart`
alias returns HTTP 200** and resolves to `deepseek/deepseek-v4-flash`. That 401 is an **alias-routing
artefact, not an outage** — always probe the alias the fleet actually pins, never the raw model id.
→ [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] -> [[runbooks/job-radar]] `CL_MODEL` caveat.
- **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` + - **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` / `deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` /
1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and 1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and