🦉 wiki audit 2026-10-21: gateway unclean-exit orphaned run (new), Gitea config-backup push dead since 2026-08-08 (new), model default → nous, 503 back to monitoring, canned-substitute trigger corrected

This commit is contained in:
Tony0410
2026-10-21 00:11:44 +08:00
parent fbd4f297eb
commit b662254569
10 changed files with 386 additions and 25 deletions
@@ -1,11 +1,11 @@
---
title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts
type: incident
status: active
status: monitoring
created: 2026-10-18
updated: 2026-10-20
verified_on: 2026-10-20
review_after: 2026-10-27
updated: 2026-10-21
verified_on: 2026-10-21
review_after: 2026-10-28
confidence: high
tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
@@ -13,10 +13,21 @@ sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-jso
# OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts (burst 1: 2026-10-16 → 10-17; burst 2: 2026-10-19)
**Status: ACTIVE — this is a recurrence, not a closed incident.** The failure mode came back **twice in four
**Status: `MONITORING` as of 2026-10-21 — not closed.** The failure mode came back **twice in four
days** (10-16 → 10-17 and 10-19) with an identical signature on an identical node, and the **exact same
9-model-call blast radius** each time. The node was healthy before, between and after both bursts.
> **2026-10-21 update — burst 2 is over; the page returns to `monitoring` on the 18–24 h rule it set itself.**
> **2026-10-20 ran 168 `completed` / 0 `failed`** (plus one `unknown` orphan — see
> [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]), the **first zero-failure day since 2026-10-12**,
> and **no execution row carries the resource-pressure string** (the last such failure remains 2026-10-19).
> **4 incident rows closed on 10-20** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`,
> `ecc1fc_21a72aa11cc4`), taking unclosed rows **13 → 9** (8 `alerted` / 1 `detected`). **Nothing was changed on
> the node** — the recovery is self-clearing, exactly as in burst 1, and the underlying exposure is untouched:
> the faults are still invisible to `/api/health`, still single-node, and a fresh burst would still take out
> most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing
> risk, not a resolved incident.
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
@@ -188,6 +199,27 @@ a second provider pin on a subset of jobs, or router-side failover between `omni
cost of leaving it open: **two of the last four days took double-digit failures from this one node**, and every
model-calling cron job still has exactly one route.
## Update 2026-10-21 (audit of the 2026-10-20 window): the burst did **not** resume
The 2026-10-19 burst was followed by **2026-10-20's 168 `completed` / 0 `failed`** — a clean day and the
first zero-failure day since 2026-10-12, with **zero** rows carrying the resource-pressure string and **zero
new incident rows** opened for the whole window. The 503 fault's last recorded failure is therefore
**2026-10-19**, and per [[systems/scheduled-tasks]] the page status returns to **`monitoring`** (18–24 h of
clean running, not a fix).
**What did *not* change, and matters more than the clean day:**
- **No failover was added.** 20 of 28 jobs still pin `custom:omni-101` (`smart-route` ×11, `free-stack` ×6,
`deepseek-paid-smart` ×1) or `custom:omni` ×2. One quiet day on a single-route topology is luck, not
margin, and this page has already been wrong once about the burst being over (it was downgraded 44 h
before burst 2).
- **The 8 jobs with no model pin no longer ride the router at all.** The global provider default moved to
`nous` direct (see below), so the fleet is now split across **two** upstream paths for the first time —
worth watching as a separate risk, since a Nous-side fault now has its own blast radius.
- **Global default changed between 2026-10-09 and 2026-10-20:** `provider: custom` →
`nous` (`https://inference-api.nousresearch.com/v1`), `model.default: smart-route` →
`meituan/longcat-2.5-preview:free`. Pinned jobs are unaffected; unpinned jobs are. → [[systems/current-profile]]
## Related
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node.