audit 2026-10-18: new incident page (omniroute-1 503 resource pressure, 9 calls / 7 jobs), fleet re-verified 28 jobs / 25 enabled / 8 non-clean, Mosquito Tracker removed, radar bot token dead
This commit is contained in:
+38
-3
@@ -3,8 +3,8 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-16
|
||||
verified_on: 2026-10-16
|
||||
updated: 2026-10-18
|
||||
verified_on: 2026-10-18
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [system]
|
||||
@@ -12,7 +12,42 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-16
|
||||
Updated: 2026-10-18
|
||||
|
||||
## Recent Changes (2026-10-18, nightly audit; live probes 2026-10-18 00:31-00:36 AWST)
|
||||
|
||||
- **The `omniroute-1` node is up — and this time it shed load instead of refusing connections.** Three days
|
||||
after the 34-hour TCP blackout, the same node produced a second, different failure mode: **9 model calls
|
||||
across 7 jobs died with `RuntimeError: HTTP 503: Service temporarily unavailable due to resource
|
||||
pressure.`** between **10-16 08:00:37 and 10-17 03:00:28** (~19 h), while `/api/health` answered
|
||||
`200 {"status":"ok"}` and `/v1/models` answered `401` before, during and after (re-probed 10-18 00:33).
|
||||
→ [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
- **It cleared itself and the fleet is green again.** `10-16 = 165 completed / 8 failed`, `10-17 = 162 / 5`
|
||||
(**all five before 03:01**), `10-18` so far `7 / 0` — **no failure row of any kind since 10-17 03:00**.
|
||||
Every job that failed re-ran clean on its next slot: `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17
|
||||
08:31, `ecc1fc13655a` and `ea79a2107e36` both 10-17 12:02, each of those auto-closing its own incident.
|
||||
- **Fleet count fell 29 → 28: the Mosquito Tracker was removed.** `1b31187397a5` is absent from
|
||||
`jobs.json` (file `updated_at 2026-10-18T00:36:17+08:00`); its last execution was **10-16 15:00**
|
||||
(`completed`, delivered). The three disabled jobs are unchanged — **25 enabled / 3 disabled**.
|
||||
- **Non-clean `last_status`: 12 → 8.** Two of the eight cannot self-heal before their next slot —
|
||||
`1b23c042e17d` sits at `error` (streak 2) until **Fri 10-23 08:00**; `0e519874d280`, `06a0a1c8704f`,
|
||||
`c8d52904e1d6`, `0e753e9720fd` and `066bb5191280` are unchanged. **Open incidents 12 → 8**
|
||||
(5 `alerted` / 3 `detected`); **9 opened and 6 closed** inside this window. → [[systems/scheduled-tasks]]
|
||||
- **The watchdog double-counts — 4 of the 13 failure rows are one sentinel.** `96eb95ce12f2` exited 1 on
|
||||
10-16 12:15 + 12:30 and 10-17 00:15 + 00:30, each row quoting the coverage monitor's own 503 error.
|
||||
Read failures per job, not per row. → [[systems/mnemosyne]]
|
||||
- **Second consecutive stocktake hole.** The 10-17 03:00 slot failed (503), so
|
||||
`raw/stocktake/freshness-20261017.txt` does not exist and `systems/skills-index.md` is stamped **10-16**
|
||||
(10-15 is missing too, from the storm — `RuntimeError: Connection error.`). The **10-18 03:00** slot
|
||||
closes it; nothing was regenerated by hand, because a manual run would race the scheduled one.
|
||||
- **Fifth consecutive window with no interactive session** — **30 sessions** across the two days, all
|
||||
`cron`, and **zero** non-cron sessions since 2026-10-15. Every finding here comes from live system
|
||||
state, not conversation.
|
||||
- **The GitHub Trending Radar is dead at the delivery end.** The Telegram bot token in
|
||||
`~/workspace/github-trending-radar/config.json` returns **HTTP 401** on `getMe` (probed 10-18 00:35);
|
||||
the job's GitHub PAT still answers **200**, so only the delivery credential is gone. The cron row stays
|
||||
green because `last_status` was set by the **10-12 09:00** run (`last_run_at 2026-10-12T09:00:12`).
|
||||
Next exercise: **Mon 2026-10-19 09:00**. **Not repaired** — a replacement bot token is Anthony's call.
|
||||
|
||||
## Recent Changes (2026-10-16, nightly audit; live probes 2026-10-16 00:0x–00:2x AWST)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user