wiki: nightly audit 2026-11-04 window — new incident (Job Radar + Newsletter Digest both killed by upstream stream failure; Stage 2 rebuilt handoff and delivered 5 roles + 1 letter); CT299 second freeze documented; CL_MODEL 402 caveat in job-radar runbook; scheduled-tasks/current-state/index/log updated

This commit is contained in:
hermes
2026-11-05 01:04:31 +08:00
parent 785fbc40b2
commit e4086ec18e
8 changed files with 361 additions and 35 deletions
+34 -4
View File
@@ -3,16 +3,46 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-10-31
verified_on: 2026-10-31
updated: 2026-11-05
verified_on: 2026-11-05
review_after: 2026-11-07
confidence: high
tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents]
---
# Current State
Updated: 2026-10-30
Updated: 2026-11-05
## Recent Changes (2026-11-05, nightly audit; live probes 2026-11-05 00:0x-00:5x AWST)
- **A dead route took out two 07:00 jobs at once - and the safety-net slot saved the day.** `dca8482e4f76`
(**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**)
both died on `RuntimeError: Upstream stream failed before completion.` - **36 seconds apart**, at **09:41:44**
and **09:42:20**, each after **2 h 41 m** on `custom:omni-101` / `smart-route`. Neither produced output; both
raised and alerted a `cron_incidents` row. The 11:30 Stage 2 slot then **reconstructed the missing handoff
from Stage 1's candidate files**, re-verified every candidate live in Camofox, and wrote **5 roles (board
479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at
16:08:59 - see [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]].
- **Fleet: 158 jobs (144 enabled / 14 disabled).** In-window ledger: **170 rows - 167 `completed` / 2 `failed` /
1 `item_suppressed`**. **Non-clean `last_status` 2 -> 3**; **unclosed incident rows 1 -> 3** (all
`state = alerted`, incl. the 11-03 `timeout` row `dca848_dfd4bea84f2a`). Deliveries on 11-04: **8 rows, 8
`delivered`** - the failure notices were delivered, not silent.
- **The cover-letter route is only alive by override.** `regenerate_cover_letter.py` line 82 reads `CL_MODEL`,
defaulting to **`deepseek-paid-smart`**, which returned **HTTP 402** during the 11-04 run; that process was
pinned to **`gemini/gemini-3.5-flash`**. **No `CL_MODEL` exists in `~/.hermes/.env`**, so the override did not
persist. Endpoint check 2026-11-05: `omniroute:20128` -> **401** and `100.78.63.20:20128` -> **401** (live,
key-gated), and the same model still serves other jobs - so re-test before calling the route dead.
- **CT299 froze a second time.** A manual `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and
wedged; the guest was frozen until **~00:30 on 11-05** (~16 h). Live probe **2026-11-05 00:54:21 AWST**:
`LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump processes;
SFTPGo reachable from CT460 at `100.88.223.70:8080`. Abandoned temp dir
`/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299` still on disk. ->
[[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]
- **wiki-stocktake-maintenance overran for the third day running** - `1a8e90ea2271` ran **03:00:31 -> 08:42:17
(5 h 41 m 46 s)**, hit the turn ceiling, reported incomplete, delivery `suppressed`. Pin:
`custom:omni-101` / **`free-stack`**.
## Recent Changes (2026-10-30, nightly audit; live probes 2026-10-30 00:0x AWST)