audit corrections 2026-11-05: fleet count 29/26/3 (was 158/144/14), 11-04 closes 167 completed/2 failed/1 running, 11-03 re-settled 171/1, deepseek-paid-smart 503 on omniroute:20128 vs 200 on omniroute-1
This commit is contained in:
+8
-3
@@ -24,15 +24,20 @@ Updated: 2026-11-05
|
||||
from Stage 1's candidate files**, re-verified every candidate live in Camofox, and wrote **5 roles (board
|
||||
479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at
|
||||
16:08:59 - see [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]].
|
||||
- **Fleet: 158 jobs (144 enabled / 14 disabled).** In-window ledger: **170 rows - 167 `completed` / 2 `failed` /
|
||||
1 `item_suppressed`**. **Non-clean `last_status` 2 -> 3**; **unclosed incident rows 1 -> 3** (all
|
||||
- **Fleet: 29 jobs (26 enabled / 3 disabled).** In-window ledger: **170 rows - 167 `completed` / 2 `failed` /
|
||||
1 `running`**. *(fleet and last row corrected 2026-11-05 01:3x - the first draft read 158 jobs / 144 / 14 and a
|
||||
closing `item_suppressed` row; live `jobs.json` + `hermes cron list` give 29 / 26 / 3, and the 170th 11-04 row is
|
||||
this audit, `running`. 11-03 has since settled to 171 `completed` / 1 `failed`.)* **Non-clean `last_status` 2 -> 3**; **unclosed incident rows 1 -> 3** (all
|
||||
`state = alerted`, incl. the 11-03 `timeout` row `dca848_dfd4bea84f2a`). Deliveries on 11-04: **8 rows, 8
|
||||
`delivered`** - the failure notices were delivered, not silent.
|
||||
- **The cover-letter route is only alive by override.** `regenerate_cover_letter.py` line 82 reads `CL_MODEL`,
|
||||
defaulting to **`deepseek-paid-smart`**, which returned **HTTP 402** during the 11-04 run; that process was
|
||||
pinned to **`gemini/gemini-3.5-flash`**. **No `CL_MODEL` exists in `~/.hermes/.env`**, so the override did not
|
||||
persist. Endpoint check 2026-11-05: `omniroute:20128` -> **401** and `100.78.63.20:20128` -> **401** (live,
|
||||
key-gated), and the same model still serves other jobs - so re-test before calling the route dead.
|
||||
key-gated - that is auth, not health). **Keyed re-probe 2026-11-05 01:33:12 AWST:** a completion request with
|
||||
`deepseek-paid-smart` returns **503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`) on `omniroute:20128`,
|
||||
but **200** on `100.78.63.20:20128` (`omniroute-1`), which resolved the alias to **`deepseek/deepseek-v4-flash`**.
|
||||
So the model is **undispatchable on the CT600 node only** - an override or the `omniroute-1` route is required.
|
||||
- **CT299 froze a second time.** A manual `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and
|
||||
wedged; the guest was frozen until **~00:30 on 11-05** (~16 h). Live probe **2026-11-05 00:54:21 AWST**:
|
||||
`LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump processes;
|
||||
|
||||
Reference in New Issue
Block a user