wiki: nightly audit 2026-11-04 window — new incident (Job Radar + Newsletter Digest both killed by upstream stream failure; Stage 2 rebuilt handoff and delivered 5 roles + 1 letter); CT299 second freeze documented; CL_MODEL 402 caveat in job-radar runbook; scheduled-tasks/current-state/index/log updated

This commit is contained in:
hermes
2026-11-05 01:04:31 +08:00
parent 785fbc40b2
commit e4086ec18e
8 changed files with 361 additions and 35 deletions
+37
View File
@@ -1960,3 +1960,40 @@ audit committed them.
**Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no
decision reversed. **No secrets, passwords, tokens or keys written or quoted** — a Gitea remote URL with an
embedded credential was seen during verification and is not reproduced here ([REDACTED]).
## [2026-11-05] audit - window 2026-11-04
- **New page:** `incidents/2026-11-04-upstream-stream-failure-0700-cohort.md` - two 07:00 jobs
(`dca8482e4f76` **Daily Marketing Job Radar - Discovery**, `85edf6772bca` **Tech & AI Newsletter Digest
(Analytical)**) died **36 s apart** (09:41:44 / 09:42:20) on `RuntimeError: Upstream stream failed before
completion.` after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`; both incidents still open
(`state = alerted`). Records the recovery: Stage 2 reconstructed the missing handoff from Stage 1's own
candidate files, re-verified every candidate live, and wrote **5 roles (board 479 -> 484) + 1 canonical
cover letter** (`MISSING_COVER_LETTER_COUNT 0`), consuming the handoff at 16:08:59.
- **Updated `incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md`** - CT299 was frozen **twice**, not
once. Second freeze: a **manual** `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and wedged
(`unexpected status`) until **11-05 ~00:30**. Also records the earlier clean failure at **07:36:44**
(log `vzdump-lxc-299-2026_11_04-07_36_44.log` on the dump target) and three `push_file` tasks from CT460
at 07:17-07:33, plus the abandoned `/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299`. Live probe 2026-11-05
00:54:21 AWST: `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump procs.
- **Updated [[infrastructure/tailscale]]** - the sftpgo row now records recovery from *two* freezes.
- **Updated [[systems/scheduled-tasks]]** - new ledger bullet: 11-04 settled **170 rows - 167 `completed` /
2 `failed` / 1 `item_suppressed`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3**;
deliveries **8/8 `delivered`**; the two failures are one upstream signature; stocktake overran for the third
day (5 h 41 m 46 s, `free-stack` pin).
- **Updated [[runbooks/job-radar]]** - cover-letter model route: `regenerate_cover_letter.py` defaults to
`CL_MODEL=deepseek-paid-smart` (line 82); the 11-04 run pinned `gemini/gemini-3.5-flash` after **HTTP 402**.
`grep` shows **no `CL_MODEL` in `~/.hermes/.env`** - the override was process-local and did not persist, so
the next run inherits the default again. Endpoint probe 2026-11-05: `omniroute:20128/v1/models` -> **401**
and `100.78.63.20:20128/v1/models` -> **401** (live, key-gated) - so the route is reachable; the same model
serves other cron jobs successfully. Treat a 402 as route/time-specific until re-tested.
- **Updated [[current-state]]** - new "Recent Changes (2026-11-05)" block; the page had been sitting at the
2026-10-30 block across five audits.
- **Updated [[index]]** - new dated ledger paragraph, incidents list gains the new page, "Last updated" moved
2026-11-03 -> 2026-11-05.
- **Live verification this audit:** pve (100.96.100.82) SSH probe at 2026-11-05 00:54:21 AWST (lock, cgroup
freeze flags, guest ALIVE, sftpgo active, proc count, task list, dump dir); SFTPGo reached end-to-end from
CT460 on `100.88.223.70:8080`; `jobs.json`, `executions.db`, `cron_incidents`, `deliveries.db` read live.
- **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or re-pinned; no
decision reversed. **No secrets, passwords, tokens or keys written or quoted.**