audit corrections 2026-11-05: fleet count 29/26/3 (was 158/144/14), 11-04 closes 167 completed/2 failed/1 running, 11-03 re-settled 171/1, deepseek-paid-smart 503 on omniroute:20128 vs 200 on omniroute-1

This commit is contained in:
Tony0410
2026-11-05 01:38:41 +08:00
parent 3b5daccb4d
commit 20aa11b0c4
5 changed files with 49 additions and 8 deletions
+8 -3
View File
@@ -24,15 +24,20 @@ Updated: 2026-11-05
from Stage 1's candidate files**, re-verified every candidate live in Camofox, and wrote **5 roles (board from Stage 1's candidate files**, re-verified every candidate live in Camofox, and wrote **5 roles (board
479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at
16:08:59 - see [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. 16:08:59 - see [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]].
- **Fleet: 158 jobs (144 enabled / 14 disabled).** In-window ledger: **170 rows - 167 `completed` / 2 `failed` / - **Fleet: 29 jobs (26 enabled / 3 disabled).** In-window ledger: **170 rows - 167 `completed` / 2 `failed` /
1 `item_suppressed`**. **Non-clean `last_status` 2 -> 3**; **unclosed incident rows 1 -> 3** (all 1 `running`**. *(fleet and last row corrected 2026-11-05 01:3x - the first draft read 158 jobs / 144 / 14 and a
closing `item_suppressed` row; live `jobs.json` + `hermes cron list` give 29 / 26 / 3, and the 170th 11-04 row is
this audit, `running`. 11-03 has since settled to 171 `completed` / 1 `failed`.)* **Non-clean `last_status` 2 -> 3**; **unclosed incident rows 1 -> 3** (all
`state = alerted`, incl. the 11-03 `timeout` row `dca848_dfd4bea84f2a`). Deliveries on 11-04: **8 rows, 8 `state = alerted`, incl. the 11-03 `timeout` row `dca848_dfd4bea84f2a`). Deliveries on 11-04: **8 rows, 8
`delivered`** - the failure notices were delivered, not silent. `delivered`** - the failure notices were delivered, not silent.
- **The cover-letter route is only alive by override.** `regenerate_cover_letter.py` line 82 reads `CL_MODEL`, - **The cover-letter route is only alive by override.** `regenerate_cover_letter.py` line 82 reads `CL_MODEL`,
defaulting to **`deepseek-paid-smart`**, which returned **HTTP 402** during the 11-04 run; that process was defaulting to **`deepseek-paid-smart`**, which returned **HTTP 402** during the 11-04 run; that process was
pinned to **`gemini/gemini-3.5-flash`**. **No `CL_MODEL` exists in `~/.hermes/.env`**, so the override did not pinned to **`gemini/gemini-3.5-flash`**. **No `CL_MODEL` exists in `~/.hermes/.env`**, so the override did not
persist. Endpoint check 2026-11-05: `omniroute:20128` -> **401** and `100.78.63.20:20128` -> **401** (live, persist. Endpoint check 2026-11-05: `omniroute:20128` -> **401** and `100.78.63.20:20128` -> **401** (live,
key-gated), and the same model still serves other jobs - so re-test before calling the route dead. key-gated - that is auth, not health). **Keyed re-probe 2026-11-05 01:33:12 AWST:** a completion request with
`deepseek-paid-smart` returns **503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`) on `omniroute:20128`,
but **200** on `100.78.63.20:20128` (`omniroute-1`), which resolved the alias to **`deepseek/deepseek-v4-flash`**.
So the model is **undispatchable on the CT600 node only** - an override or the `omniroute-1` route is required.
- **CT299 froze a second time.** A manual `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and - **CT299 froze a second time.** A manual `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and
wedged; the guest was frozen until **~00:30 on 11-05** (~16 h). Live probe **2026-11-05 00:54:21 AWST**: wedged; the guest was frozen until **~00:30 on 11-05** (~16 h). Live probe **2026-11-05 00:54:21 AWST**:
`LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump processes; `LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump processes;
+5 -2
View File
@@ -11,9 +11,12 @@ tags: [index, infrastructure, systems, stream-failure, ct299]
# Anthony's Systems Wiki # Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure, > Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-11-05 (nightly audit - the 11-04 fleet settled 170 rows: 167 `completed` / 2 `failed` / 1 `item_suppressed`; non-clean `last_status` 2 -> 3; unclosed incident rows 1 -> 3; deliveries 8/8 `delivered`. Both 07:00 failures were one upstream stream fault - and the Job Radar's Stage 2 slot reconstructed the missing handoff and saved the day. CT299 froze a second time.) > model routing and automation. Last updated: 2026-11-05 (nightly audit - the 11-04 fleet settled 170 rows: 167 `completed` / 2 `failed` / 1 `running`; non-clean `last_status` 2 -> 3; unclosed incident rows 1 -> 3; deliveries 8/8 `delivered`. Both 07:00 failures were one upstream stream fault - and the Job Radar's Stage 2 slot reconstructed the missing handoff and saved the day. CT299 froze a second time.)
> >
> **2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze:** the **11-04 fleet settled 170 rows - 167 `completed` / 2 `failed` / 1 `item_suppressed`**, **non-clean `last_status` 2 -> 3**, **unclosed incident rows 1 -> 3**, **deliveries 8 rows / 8 `delivered`**. Both failures are the *same* upstream fault: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) died **36 seconds apart at 09:41-09:42**, each after **2 h 41 m**, on `RuntimeError: Upstream stream failed before completion.` - both pinned `custom:omni-101` / `smart-route`. **The day was not lost:** the 11:30 Stage 2 slot **reconstructed the missing handoff from Stage 1's own candidate files**, re-verified every candidate live, and still wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. The cover-letter route survived only on an override: `regenerate_cover_letter.py` defaults to `deepseek-paid-smart`, which threw **HTTP 402** mid-run, so the letter was pinned to `gemini/gemini-3.5-flash` via `CL_MODEL` - and that override **was never written to `.env`**, so it did not persist. And **CT299 was frozen twice, not once**: a second, **manual** `vzdump 299 ... --storage ssd0` wedged from **11-04 08:30:04** until **11-05 ~00:30** (~16 h); the live probe at 00:54:21 AWST shows `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. > **2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze:** the **11-04 fleet settled 170 rows - 167 `completed` / 2 `failed` / 1 `running`**, **non-clean `last_status` 2 -> 3**, **unclosed incident rows 1 -> 3**, **deliveries 8 rows / 8 `delivered`**. Both failures are the *same* upstream fault: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) died **36 seconds apart at 09:41-09:42**, each after **2 h 41 m**, on `RuntimeError: Upstream stream failed before completion.` - both pinned `custom:omni-101` / `smart-route`. **The day was not lost:** the 11:30 Stage 2 slot **reconstructed the missing handoff from Stage 1's own candidate files**, re-verified every candidate live, and still wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. The cover-letter route survived only on an override: `regenerate_cover_letter.py` defaults to `deepseek-paid-smart`, which threw **HTTP 402** mid-run, so the letter was pinned to `gemini/gemini-3.5-flash` via `CL_MODEL` - and that override **was never written to `.env`**, so it did not persist. And **CT299 was frozen twice, not once**: a second, **manual** `vzdump 299 ... --storage ssd0` wedged from **11-04 08:30:04** until **11-05 ~00:30** (~16 h); the live probe at 00:54:21 AWST shows `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. ⚠ **Corrected the same night:** the first draft of this entry read the fleet as **158 jobs (144 / 14)** and the
> 11-04 ledger as closing `1 item_suppressed`; live reads give **29 jobs (26 / 3)** and a final row of **1 `running`**
> (this audit). Keyed probe **2026-11-05 01:33**: `deepseek-paid-smart` is **undispatchable on `omniroute:20128`**
> (HTTP **503**, all targets skipped) but **serves normally on `omniroute-1`** (HTTP **200** -> `deepseek/deepseek-v4-flash`).
> **2026-11-04 (nightly audit) — the ledger raised its own red row:** the **11-03 fleet settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) — **unclosed incident rows 0 → 1**, non-clean `last_status` **1 → 2**, deliveries **7 rows / 7 `delivered`**. The single failure is the **Job Radar Stage 1** (`dca8482e4f76`), which ran **07:00:33 → 13:28:52 (6 h 28 m 18 s)** and died on **`RuntimeError: stream active timeout`** — a **new terminal signature**, not the iteration cap — after going **silent for 65 m 21 s**; its incident `dca848_dfd4bea84f2a` (`timeout`) is **`alerted` and open** and the notice **was delivered** (13:28:51.741724) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. That answers the 11-02 page's open question: **output is suppressed when the run ends `completed`, delivered when it ends `failed`** — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so **2026-11-02's and 2026-11-03's candidate sets are both unrecovered**. ⚠ **The 03:00 stocktake overran for a second day** (`1a8e90ea2271`, 03:00:30 → 10:41:13 = **7 h 40 m 43 s**, `suppressed`, nothing delivered, artefacts uncommitted again). **New page:** [[systems/hermes-hq-push]] (v0.9.27, enabled; manifest says it **replaces `dispatch-push`**, and no trace of the old plugin survives). > **2026-11-04 (nightly audit) — the ledger raised its own red row:** the **11-03 fleet settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) — **unclosed incident rows 0 → 1**, non-clean `last_status` **1 → 2**, deliveries **7 rows / 7 `delivered`**. The single failure is the **Job Radar Stage 1** (`dca8482e4f76`), which ran **07:00:33 → 13:28:52 (6 h 28 m 18 s)** and died on **`RuntimeError: stream active timeout`** — a **new terminal signature**, not the iteration cap — after going **silent for 65 m 21 s**; its incident `dca848_dfd4bea84f2a` (`timeout`) is **`alerted` and open** and the notice **was delivered** (13:28:51.741724) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. That answers the 11-02 page's open question: **output is suppressed when the run ends `completed`, delivered when it ends `failed`** — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so **2026-11-02's and 2026-11-03's candidate sets are both unrecovered**. ⚠ **The 03:00 stocktake overran for a second day** (`1a8e90ea2271`, 03:00:30 → 10:41:13 = **7 h 40 m 43 s**, `suppressed`, nothing delivered, artefacts uncommitted again). **New page:** [[systems/hermes-hq-push]] (v0.9.27, enabled; manifest says it **replaces `dispatch-push`**, and no trace of the old plugin survives).
> >
+30 -1
View File
@@ -1978,7 +1978,7 @@ embedded credential was seen during verification and is not reproduced here ([RE
00:54:21 AWST: `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump procs. 00:54:21 AWST: `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump procs.
- **Updated [[infrastructure/tailscale]]** - the sftpgo row now records recovery from *two* freezes. - **Updated [[infrastructure/tailscale]]** - the sftpgo row now records recovery from *two* freezes.
- **Updated [[systems/scheduled-tasks]]** - new ledger bullet: 11-04 settled **170 rows - 167 `completed` / - **Updated [[systems/scheduled-tasks]]** - new ledger bullet: 11-04 settled **170 rows - 167 `completed` /
2 `failed` / 1 `item_suppressed`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3**; 2 `failed` / 1 `running`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3**;
deliveries **8/8 `delivered`**; the two failures are one upstream signature; stocktake overran for the third deliveries **8/8 `delivered`**; the two failures are one upstream signature; stocktake overran for the third
day (5 h 41 m 46 s, `free-stack` pin). day (5 h 41 m 46 s, `free-stack` pin).
- **Updated [[runbooks/job-radar]]** - cover-letter model route: `regenerate_cover_letter.py` defaults to - **Updated [[runbooks/job-radar]]** - cover-letter model route: `regenerate_cover_letter.py` defaults to
@@ -1998,3 +1998,32 @@ embedded credential was seen during verification and is not reproduced here ([RE
decision reversed. **No secrets, passwords, tokens or keys written or quoted.** decision reversed. **No secrets, passwords, tokens or keys written or quoted.**
- *network-map.md re-verified (live probe 2026-11-05):* corrected the ct460 port table — **9377 is Camofox browser REST** (`/home/hermes/services/camofox-browser`), not Stratos; `8790` and `5174` are no longer listening; added `9120` (wiki serve), `9222` (Camofox CDP/WS), `7777` (OpenChamber), `6767` (Paseo), `8085` (AgentMail webhook), `9900` (gateway). Added the full Tailscale Serve table (12 routes on `:443`, `/` on `:9443` → 5173, Funnel `:8443` `/webhook` → 8085). Two routes are dead: `/agent-estate` → 8769 and `/synergy-prep` → 9123 have no listener. - *network-map.md re-verified (live probe 2026-11-05):* corrected the ct460 port table — **9377 is Camofox browser REST** (`/home/hermes/services/camofox-browser`), not Stratos; `8790` and `5174` are no longer listening; added `9120` (wiki serve), `9222` (Camofox CDP/WS), `7777` (OpenChamber), `6767` (Paseo), `8085` (AgentMail webhook), `9900` (gateway). Added the full Tailscale Serve table (12 routes on `:443`, `/` on `:9443` → 5173, Funnel `:8443` `/webhook` → 8085). Two routes are dead: `/agent-estate` → 8769 and `/synergy-prep` → 9123 have no listener.
## [2026-11-05] correction + live probe | fleet count, 11-04 ledger composition, `deepseek-paid-smart` dispatch
- **What was wrong.** The 2026-11-05 nightly-audit entry (written 00:5x AWST) recorded the fleet as **158 jobs
(144 enabled / 14 disabled)** and the 11-04 ledger as closing **1 `item_suppressed`**. Both were wrong. Live
re-reads - `~/.hermes/cron/jobs.json`, `hermes cron list`, and the in-flight job count - each give **29 jobs /
26 enabled / 3 disabled**; `executions.db` shows 11-04 at **170 rows - 167 `completed` / 2 `failed` / 1
`running`** (this audit), with **no `item_suppressed` status anywhere on 11-04**.
- **Where corrected:** `systems/scheduled-tasks.md` (Count bullet + correction clause),
`current-state.md` (11-05 block), `index.md` (11-05 ledger paragraph), and this entry. No other page carried
the bad number.
- **11-03 re-settle:** `171 completed / 1 failed` (172 rows) - its `running` row has since completed - so 11-03
and 11-04 are the only consecutive days with a `failed` row since 10-31.
- **Live probe - model dispatch, 2026-11-05 01:33:12 AWST (first-hand, keyed):**
`POST http://omniroute:20128/v1/chat/completions` with `deepseek-paid-smart` -> **HTTP 503**
`ALL_TARGETS_SKIPPED` (`poolSize: 5`, `attempted: 0`). The same request to
`http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, `model:
deepseek/deepseek-v4-flash`. Unkeyed `/v1/models` on both hosts returns **401** - auth-gating, not health. The
fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101`, i.e.
the `omniroute-1` node), which is why the audit still completes while a direct CT600 probe 503s.
- **Re-verified and matching (no change needed):** `cron_incidents` - 3 unclosed rows, all `state = alerted`;
`deliveries.db` - 8 rows on 11-04, **8/8 `delivered`**, including both `for_failure` notices (09:41:43,
09:42:20); non-clean `last_status` **3**; pin distribution **14 `smart-route` / 6 `free-stack` / 8 unpinned /
1 `deepseek-paid-smart`**.
- **Files touched this pass:** `systems/scheduled-tasks.md`, `current-state.md`, `runbooks/job-radar.md`
(`CL_MODEL` caveat now carries the keyed probe), `index.md`, `log.md`. Frontmatter `updated` / `verified_on`
bumped to 2026-11-05 where stale.
- **Not done / unchanged:** no job added, removed, disabled or re-pinned; no page removed or archived; no
decision reversed. **No secrets, passwords, tokens or keys written or quoted** - the API key was read from
`~/.hermes/.env` into the shell for the probe only.
+5 -1
View File
@@ -47,7 +47,11 @@ Cron jobs inherit the profile-level `fallback_providers` chain; Hermes does not
> process and confirm the letters rendered - do **not** ship a run without its cover letters. Note the same model > process and confirm the letters rendered - do **not** ship a run without its cover letters. Note the same model
> name still serves other cron jobs successfully (e.g. the 23:59 nightly audit), and the endpoint itself is live: > name still serves other cron jobs successfully (e.g. the 23:59 nightly audit), and the endpoint itself is live:
> `omniroute:20128/v1/models` -> **401** and `100.78.63.20:20128/v1/models` -> **401** on 2026-11-05 (key-gated, > `omniroute:20128/v1/models` -> **401** and `100.78.63.20:20128/v1/models` -> **401** on 2026-11-05 (key-gated,
> not down). Treat a 402 as **route/time-specific until re-tested**, not as proof the route is dead. > not down). Treat a 402 as **route/time-specific until re-tested**, not as proof the route is dead. **Keyed re-test 2026-11-05 01:33:12 AWST:** `deepseek-paid-smart` -> **503 `ALL_TARGETS_SKIPPED`**
> (`poolSize 5`, `attempted 0`) on `omniroute:20128`, but **200** on `100.78.63.20:20128` (`omniroute-1`, resolved to
> `deepseek/deepseek-v4-flash`). As of that probe the default `CL_MODEL` **cannot dispatch on the CT600 node**, so a
> `CL_MODEL` override (or an `omniroute-1` base URL) is required for cover-letter runs. An unkeyed `/v1/models` 401 is
> auth-gating, not health.
A successful radar run includes the cover-letter stage. A successful radar run includes the cover-letter stage.
+1 -1
View File
@@ -14,7 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **158 jobs (144 enabled / 14 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `item_suppressed`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. - **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. -> [[runbooks/job-radar]] `CL_MODEL` caveat.
- **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` + - **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` / `deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` /
1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and 1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and