From 2d5219b068f6bf09e19ae5d2df8a7431420a0a0f Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Fri, 6 Nov 2026 04:07:12 +0800 Subject: [PATCH] stocktake 2026-11-06 (pass 2): re-verify the one page sitting on the 21-day threshold incidents/2026-10-14-omniroute-node-connection-refused-storm.md was at exactly 21d and would have tripped the threshold on the next run. Its residual claim ("omniroute-1 answers again") was re-probed live and still holds: TCP connect to 100.78.63.20:20128 succeeds (no Errno 111), the Tailscale peer reads Online:true, and a keyed POST /v1/chat/completions returns HTTP 200. Page re-verified (updated/verified_on -> 2026-11-06, review_after -> 2026-12-06) with the probe recorded on the page. New trap recorded on the page and in systems/scheduled-tasks.md: on omniroute-1 the RAW model id deepseek/deepseek-v4-flash returns HTTP 401 "No active credentials for provider: deepseek", while the fleet's deepseek-paid-smart alias returns HTTP 200 and resolves to that same model. The 401 is an alias-routing artefact, not an outage - probe the alias the fleet pins, never the raw model id. (A first probe using the wrong key_env also produced a 401; omni-101's key env is HERMES_CUSTOM_100_78_63_20_20128_API_KEY.) Freshness re-run after the edits: 0 stale / 124 fresh / 3 no-date of 127 pages. Quartz rebuilt and redeployed. Vault mirror synced to both conventions (sha-verified). --- ...-omniroute-node-connection-refused-storm.md | 18 +++++++++++++++--- log.md | 13 +++++++++++++ raw/stocktake/freshness-20261106.txt | 2 +- systems/scheduled-tasks.md | 7 ++++++- 4 files changed, 35 insertions(+), 5 deletions(-) diff --git a/incidents/2026-10-14-omniroute-node-connection-refused-storm.md b/incidents/2026-10-14-omniroute-node-connection-refused-storm.md index beb1181..67fd0ce 100644 --- a/incidents/2026-10-14-omniroute-node-connection-refused-storm.md +++ b/incidents/2026-10-14-omniroute-node-connection-refused-storm.md @@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Node Outage — Connection-Refused Storm type: concept status: resolved created: 2026-10-16 -updated: 2026-10-16 -verified_on: 2026-10-16 -review_after: 2026-10-23 +updated: 2026-11-06 +verified_on: 2026-11-06 +review_after: 2026-12-06 confidence: high tags: [cron, router, provider, infrastructure] sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, hermes-errors-log, live-probe] @@ -15,6 +15,18 @@ sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, herm **Window:** 2026-10-14 08:30 → 2026-10-15 18:07 AWST (**~34 hours**) **Status:** **Resolved.** `omniroute-1` answers again; four scheduled jobs have since completed on it. + +**Re-verified 2026-11-06 (stocktake) — the residual claim still holds, and the node is genuinely healthy.** +Live probe of the incident's own endpoint: TCP **connect to `100.78.63.20:20128` succeeds** (no +`Errno 111 Connection refused` — the incident's root cause has not recurred), the Tailscale peer +`omniroute-1` reads `Online: true`, and a keyed `POST /v1/chat/completions` returns **HTTP 200**. +`auto/best-chat` → `codestral-2508` (200), `auto/best-reasoning` → `deepseek-ai/deepseek-v4.1-flash` (200), +and the fleet's `deepseek-paid-smart` pin → `deepseek/deepseek-v4-flash` (**200**) — confirming the +[[systems/scheduled-tasks]] note that this pin dispatches on the `omniroute-1` node. ⚠ **One trap worth +recording:** requesting the **raw** model id `deepseek/deepseek-v4-flash` (rather than the +`deepseek-paid-smart` alias) returns **HTTP 401 `No active credentials for provider: deepseek`** — that is +an alias-routing artefact on this node, **not** an outage and **not** the incident recurring. Probe the +**alias** the fleet actually uses. **Impact:** **36 failed execution rows across 13 jobs** — 24 genuine model-call failures on 12 jobs, plus 12 watchdog *symptom* reds. **Every model call attempted in the window failed** (see below). For scale: 10-12 recorded `0 failed` and 10-13 recorded `1`. diff --git a/log.md b/log.md index 1ef6285..c5b6e05 100644 --- a/log.md +++ b/log.md @@ -2171,3 +2171,16 @@ embedded credential was seen during verification and is not reproduced here ([RE `deepseek-paid-smart`-on-`omniroute` dispatch fault and the `066bb5191280` delivery failure carried, not re-diagnosed; the open secrets-at-rest incident not re-litigated. **No secrets, tokens or keys written or quoted.** + +- **Follow-up (04:0x AWST): the one page sitting exactly on the 21-day threshold was re-verified rather + than left to expire.** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` was at **21d** and + would have tripped the threshold tomorrow. Its residual claim ("`omniroute-1` answers again") was + re-probed **live** and still holds: TCP connect to `100.78.63.20:20128` succeeds (no `Errno 111`), + the Tailscale peer reads `Online: true`, and a keyed `POST /v1/chat/completions` returns **HTTP 200**. + Page re-verified (`updated`/`verified_on` -> 2026-11-06, `review_after` -> 2026-12-06) with the probe + recorded. ⚠ **New trap recorded on the page and in [[systems/scheduled-tasks]]:** on `omniroute-1` the + **raw** model id `deepseek/deepseek-v4-flash` returns **HTTP 401 `No active credentials for provider: + deepseek`**, while the fleet's **`deepseek-paid-smart` alias returns HTTP 200** and resolves to that same + model. The 401 is an **alias-routing artefact, not an outage** — probe the alias the fleet pins, never the + raw model id. (A first probe with the wrong `key_env` also produced a 401; the correct key for + `omni-101` is `HERMES_CUSTOM_100_78_63_20_20128_API_KEY`.) diff --git a/raw/stocktake/freshness-20261106.txt b/raw/stocktake/freshness-20261106.txt index 7db6a59..e34f2df 100644 --- a/raw/stocktake/freshness-20261106.txt +++ b/raw/stocktake/freshness-20261106.txt @@ -46,7 +46,7 @@ Threshold: >21 days since `updated` - [ 8d] incidents/2026-10-05-dead-model-pin-repin.md - [ 13d] incidents/2026-10-09-cron-provider-timeout-storm.md - [ 5d] incidents/2026-10-09-memory-recall-relevance.md -- [ 21d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 0d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md - [ 10d] incidents/2026-10-16-omniroute-503-resource-pressure.md - [ 18d] incidents/2026-10-18-degenerate-stream-canned-substitute.md - [ 10d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 3483a5d..47201cc 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -15,7 +15,12 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db ## Count - **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. -- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. -> [[runbooks/job-radar]] `CL_MODEL` caveat. +- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. ⚠ **Re-verified +2026-11-06 (stocktake): on `omniroute-1` (`100.78.63.20`) the raw model id `deepseek/deepseek-v4-flash` +returns **HTTP 401 `No active credentials for provider: deepseek`** while the fleet's **`deepseek-paid-smart` +alias returns HTTP 200** and resolves to `deepseek/deepseek-v4-flash`. That 401 is an **alias-routing +artefact, not an outage** — always probe the alias the fleet actually pins, never the raw model id. +→ [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] -> [[runbooks/job-radar]] `CL_MODEL` caveat. - **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and