stocktake(2026-11-15): skills-index regenerated (299/67); storm incident re-verified (all 12 rows resolved, evidence-window note); conduit memory-timeout fix LOST in plugin 0.168.x restructure — page → degraded with regression section; 0 stale after fixes
This commit is contained in:
@@ -3,9 +3,9 @@ title: Fleet-Wide Provider Timeout Storm — 2026-10-09
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-09
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-12-15
|
||||
confidence: high
|
||||
tags: [cron, omniroute, router, timeout, incident, resolved]
|
||||
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
|
||||
@@ -18,6 +18,15 @@ sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-2026100
|
||||
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
|
||||
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
|
||||
The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
|
||||
|
||||
> **Re-verified 2026-11-15** (3 a.m. stocktake): all 12 of the storm's `cron_incidents` rows still read
|
||||
> `resolved` — none reopened. `1b23c042e17d` (the weekly Friday job, last hold-out) is healthy —
|
||||
> `last_status: ok`, last run 2026-11-13 08:11 AWST. One incident is currently open fleet-wide
|
||||
> (`ea79a2_601037516abd`, Mnemosyne Health Monitor, first seen 2026-11-15 00:03) but it is the
|
||||
> capture-stall detection firing → [[incidents/2026-11-14-mnemosyne-capture-stall]], unrelated to
|
||||
> this storm. ⚠ Evidence window: `executions.db` retention now reaches back only to 2026-11-09, so
|
||||
> the 2026-10-09 execution counts below are **no longer re-derivable from the DB** — they stand on
|
||||
> the 2026-10-24 read (the incident rows themselves persist, which is what this check re-read).
|
||||
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
|
||||
|
||||
## What happened
|
||||
|
||||
Reference in New Issue
Block a user