audit(2026-10-20): omni-101 503 burst 2 — incident back to active; scheduled-tasks rebuilt from jobs.json

- burst 2 (2026-10-19): 9 model calls across 8 jobs, 159/13 day
- burst-2 table names/schedules/aliases corrected from live jobs.json
- non-clean last_status 7 -> 12; unclosed incident rows 13 (query stated)
- 2026-10-17 tail corrected 162/5 -> 163/5
This commit is contained in:
Hermes Rhino
2026-10-20 00:12:39 +08:00
parent 93c55efc94
commit 95fd7d33f1
6 changed files with 195 additions and 40 deletions
+28
View File
@@ -912,3 +912,31 @@ with a working tree carrying uncommitted 10-17/10-18 changes, and it has **no re
pushed. **Not fixed** — a full mirror pass is a bigger call than an audit's mandate; flagged here.
No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe status codes only.
## [2026-10-20] audit | Nightly wiki audit — `omni-101` 503 burst 2 (2026-10-19)
- **Incident re-opened.** [[incidents/2026-10-16-omniroute-503-resource-pressure]] went back to
`status: active`. A second, identical `HTTP 503 … resource pressure` burst hit `omniroute-1` on 2026-10-19 —
**9 model calls across 8 jobs** (06:01, then 07:01 → 12:01) — against burst 1's 9 across 7
(2026-10-16 08:00 → 10-17 03:00). Whole day **159 completed / 13 failed**; 2026-10-18 was 168 / 1.
- **Facts corrected from the source of truth** (`~/.hermes/cron/jobs.json`, `executions.db`) instead of being
carried forward: the burst-2 table's job *names*, *schedules* and *model aliases* were wrong
(`85edf6772bca` is **Tech & AI Newsletter Digest (Analytical)**, not "Fringe Festival Plan"; `ea79a2107e36`
is the **Mnemosyne Health Monitor**, not a news digest; `ecc1fc13655a` is **FreshRSS Midday News Briefing**),
`08eb86f02d37` fails at **08:31** (not 08:01, it is a `30 8 * * *` job), and 2026-10-17 reads **163 / 5**,
not 162 / 5. **All eight failing jobs read `provider: custom:omni-101`, `model: smart-route`** — the burst-1
note's wider alias mix (`deepseek-paid-smart`, `free-IA`, `dark`) no longer matches `jobs.json` and is now
flagged in the page for confirmation rather than silently overwritten.
- **Incident ledger read properly for the first time.** Unclosed rows = **13** (12 `alerted` / 1 `detected`),
not the `7` the 10-19 entry recorded — the earlier figures used a narrower filter. A raw `closed_at IS NULL`
count reads **12** at 10-19 00:20 and **13** now; the entry states the query so future entries compare like
with like. 4 rows opened and 2 closed on 10-19, plus `ea79a2_747b27556fd0` closed 10-20 00:04:24.
- **Pages updated:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (burst-2 table rebuilt from
`jobs.json`, new findings, status back to active), `systems/scheduled-tasks.md` (new current-count bullet,
non-clean `last_status` rewritten 7 → 12, Known Issues addition, 8 job-table rows corrected),
`systems/omniroute.md` (new Recent Changes section), `current-state.md` (new Recent Changes section),
`index.md` (scheduled-tasks and incident summaries, "Last updated" → 2026-10-20), `log.md`.
- **Nothing removed or archived** — no page became obsolete and no one-way sync was run.
- **Fleet:** 28 jobs / 25 enabled / 3 disabled (unchanged). **15 sessions in the window, all `cron`** — no
interactive sessions, so there were no user-facing decisions to record.
- No secrets written; host names, IPs, job/incident IDs, ports and probe status codes only.