wiki: nightly audit 2026-11-04 window — new incident (Job Radar + Newsletter Digest both killed by upstream stream failure; Stage 2 rebuilt handoff and delivered 5 roles + 1 letter); CT299 second freeze documented; CL_MODEL 402 caveat in job-radar runbook; scheduled-tasks/current-state/index/log updated
This commit is contained in:
+34
-4
@@ -3,16 +3,46 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-31
|
||||
verified_on: 2026-10-31
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [system]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-30
|
||||
Updated: 2026-11-05
|
||||
|
||||
## Recent Changes (2026-11-05, nightly audit; live probes 2026-11-05 00:0x-00:5x AWST)
|
||||
|
||||
- **A dead route took out two 07:00 jobs at once - and the safety-net slot saved the day.** `dca8482e4f76`
|
||||
(**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**)
|
||||
both died on `RuntimeError: Upstream stream failed before completion.` - **36 seconds apart**, at **09:41:44**
|
||||
and **09:42:20**, each after **2 h 41 m** on `custom:omni-101` / `smart-route`. Neither produced output; both
|
||||
raised and alerted a `cron_incidents` row. The 11:30 Stage 2 slot then **reconstructed the missing handoff
|
||||
from Stage 1's candidate files**, re-verified every candidate live in Camofox, and wrote **5 roles (board
|
||||
479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at
|
||||
16:08:59 - see [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]].
|
||||
- **Fleet: 158 jobs (144 enabled / 14 disabled).** In-window ledger: **170 rows - 167 `completed` / 2 `failed` /
|
||||
1 `item_suppressed`**. **Non-clean `last_status` 2 -> 3**; **unclosed incident rows 1 -> 3** (all
|
||||
`state = alerted`, incl. the 11-03 `timeout` row `dca848_dfd4bea84f2a`). Deliveries on 11-04: **8 rows, 8
|
||||
`delivered`** - the failure notices were delivered, not silent.
|
||||
- **The cover-letter route is only alive by override.** `regenerate_cover_letter.py` line 82 reads `CL_MODEL`,
|
||||
defaulting to **`deepseek-paid-smart`**, which returned **HTTP 402** during the 11-04 run; that process was
|
||||
pinned to **`gemini/gemini-3.5-flash`**. **No `CL_MODEL` exists in `~/.hermes/.env`**, so the override did not
|
||||
persist. Endpoint check 2026-11-05: `omniroute:20128` -> **401** and `100.78.63.20:20128` -> **401** (live,
|
||||
key-gated), and the same model still serves other jobs - so re-test before calling the route dead.
|
||||
- **CT299 froze a second time.** A manual `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and
|
||||
wedged; the guest was frozen until **~00:30 on 11-05** (~16 h). Live probe **2026-11-05 00:54:21 AWST**:
|
||||
`LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump processes;
|
||||
SFTPGo reachable from CT460 at `100.88.223.70:8080`. Abandoned temp dir
|
||||
`/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299` still on disk. ->
|
||||
[[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]
|
||||
- **wiki-stocktake-maintenance overran for the third day running** - `1a8e90ea2271` ran **03:00:31 -> 08:42:17
|
||||
(5 h 41 m 46 s)**, hit the turn ceiling, reported incomplete, delivery `suppressed`. Pin:
|
||||
`custom:omni-101` / **`free-stack`**.
|
||||
|
||||
|
||||
## Recent Changes (2026-10-30, nightly audit; live probes 2026-10-30 00:0x AWST)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user