🦉 wiki audit 2026-10-22: omni-101 pre-dispatch filter skip (3rd failure mode, new page), Hermes 0.22.3 update with snapshot skipped, Gitea backup backlog 76
- new: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md - systems/hermes-agent.md: version -> 0.22.3 (5b147ab036), 22:47-22:57 update, snapshot stage skipped, advisories unactioned - systems/scheduled-tasks.md: 2026-10-21 = 171 completed / 1 failed; non-clean stays 9 with 08eb86f02d37 out, dca8482e4f76 in - systems/omniroute.md: one pin, three signatures in eight days - incidents/2026-10-16-omniroute-503-resource-pressure.md: window update, stays monitoring - incidents/2026-08-08-gitea-config-backup-push-dead.md: 2026-10-21 run, backlog 75 -> 76, message_id 8449 - infrastructure/backups.md: 76 commits; pre_update_backup:false now exercised - current-state.md, index.md, log.md updated
This commit is contained in:
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
|
||||
type: incident
|
||||
status: monitoring
|
||||
created: 2026-10-18
|
||||
updated: 2026-10-21
|
||||
verified_on: 2026-10-21
|
||||
review_after: 2026-10-28
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
review_after: 2026-10-29
|
||||
confidence: high
|
||||
tags: [cron, router, provider, infrastructure]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
|
||||
@@ -220,6 +220,25 @@ clean running, not a fix).
|
||||
`nous` (`https://inference-api.nousresearch.com/v1`), `model.default: smart-route` →
|
||||
`meituan/longcat-2.5-preview:free`. Pinned jobs are unaffected; unpinned jobs are. → [[systems/current-profile]]
|
||||
|
||||
## Update 2026-10-21 (window 2026-10-20 → 10-21)
|
||||
|
||||
- **A third failure mode from the same pin — and it is not 503.** The only failed cron row on **2026-10-21**
|
||||
was `dca8482e4f76` (Daily Marketing Job Radar, `custom:omni-101` / `smart-route`), which died after
|
||||
**40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch
|
||||
filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]].
|
||||
- **The resource-pressure string did not appear at all on 2026-10-21** (zero ledger rows; last occurrence
|
||||
remains 2026-10-19 09:02), so the second burst stays closed and this page stays `status: monitoring`.
|
||||
- **One row closed, one opened.** `08eb86_472c7b609c80` (the Gmail Action Digest 503s, first seen 10-16 08:33)
|
||||
**resolved 2026-10-21 08:37:21** — the job's own 08:30 slot ran clean that morning (its 10-20 run was the
|
||||
orphan; no incident row was ever raised for it), which also cleared `08eb86f02d37` from the non-clean table.
|
||||
`dca848_014affeb6954` opened 07:40:42 and is **still `alerted`**. Unclosed rows **stay at 9**
|
||||
(8 `alerted` / 1 `detected`).
|
||||
- **Windows:** 2026-10-20 settled at **169 `completed` / 1 `unknown`** (the gateway-orphaned run; the 10-21
|
||||
audit's 168 / 0 was a mid-flight snapshot taken before the 23:59 slot landed). **2026-10-21: 171 `completed`
|
||||
/ 1 `failed`** (+ this audit).
|
||||
- **Still no failover.** Every job that has failed on this node since 2026-10-14 still reads
|
||||
`provider: custom:omni-101` with `model: smart-route` or `free-stack`, and none has a second pin.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node.
|
||||
|
||||
Reference in New Issue
Block a user