From 3919ec85576692a3ae7beba5e9f17e5cbb331ef5 Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Thu, 22 Oct 2026 00:16:39 +0800 Subject: [PATCH] =?UTF-8?q?=F0=9F=A6=89=20wiki=20audit=202026-10-22:=20omn?= =?UTF-8?q?i-101=20pre-dispatch=20filter=20skip=20(3rd=20failure=20mode,?= =?UTF-8?q?=20new=20page),=20Hermes=200.22.3=20update=20with=20snapshot=20?= =?UTF-8?q?skipped,=20Gitea=20backup=20backlog=2076?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - new: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md - systems/hermes-agent.md: version -> 0.22.3 (5b147ab036), 22:47-22:57 update, snapshot stage skipped, advisories unactioned - systems/scheduled-tasks.md: 2026-10-21 = 171 completed / 1 failed; non-clean stays 9 with 08eb86f02d37 out, dca8482e4f76 in - systems/omniroute.md: one pin, three signatures in eight days - incidents/2026-10-16-omniroute-503-resource-pressure.md: window update, stays monitoring - incidents/2026-08-08-gitea-config-backup-push-dead.md: 2026-10-21 run, backlog 75 -> 76, message_id 8449 - infrastructure/backups.md: 76 commits; pre_update_backup:false now exercised - current-state.md, index.md, log.md updated --- current-state.md | 38 ++++++++- ...026-08-08-gitea-config-backup-push-dead.md | 21 ++++- ...6-10-16-omniroute-503-resource-pressure.md | 25 +++++- ...0-21-omniroute-pre-dispatch-filter-skip.md | 79 +++++++++++++++++++ index.md | 5 +- infrastructure/backups.md | 12 ++- log.md | 36 +++++++++ systems/hermes-agent.md | 8 +- systems/omniroute.md | 18 ++++- systems/scheduled-tasks.md | 9 ++- 10 files changed, 225 insertions(+), 26 deletions(-) create mode 100644 incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md diff --git a/current-state.md b/current-state.md index ffe9f79..833b592 100644 --- a/current-state.md +++ b/current-state.md @@ -3,16 +3,46 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-21 -verified_on: 2026-10-21 -review_after: 2026-10-28 +updated: 2026-10-22 +verified_on: 2026-10-22 +review_after: 2026-10-29 confidence: high tags: [system] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] --- # Current State -Updated: 2026-10-21 +Updated: 2026-10-22 + +## Recent Changes (2026-10-22, nightly audit; live probes 2026-10-21 23:59 → 2026-10-22 00:20 AWST) + +- **Hermes Agent updated to `0.22.3`.** `hermes update` ran **2026-10-21 22:47:25 → 22:57:38 AWST** + (action `9eed3a99e799347c691ea52a72e122a1`), from pre-update tree `0.22.2` / `d923a5eb487a17c0…` to commit + `5b147ab036d6…` (detached, install stamp `displayVersion 0.22.3`, built 2026-10-21T14:57:38Z). Stages + `plan` / `apply` (git) / `deps` / `build` all succeeded, **but `snapshot` = `skipped`** + (`disabled by updates.pre_update_backup (mode: off)`). Gateway restarted via **systemd** — pid `868087`, + `phase: running`, `api_server: connected`, heartbeat 2026-10-22 00:13 AWST — and the dashboard was respawned + by argv on `:9119` (pid `869852`). Post-run advisories, **neither yet acted on**: rebuild the search index + (`hermes sessions optimize-storage`) and *Preparing pinned cua-driver (Computer Use)…* (no completion line in + the log). → [[systems/hermes-agent]] +- **The only cron failure of 2026-10-21 is a new failure mode.** Window: **171 `completed` / 1 `failed`** (+ this + audit). `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`) failed + at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by + pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string (zero such rows on 10-21; + last 2026-10-19 09:02) — while the sibling 07:00 job `85edf6772bca` completed normally. Incident + `dca848_014affeb6954` **`alerted`, unclosed**; the run produced no output (`cron_incomplete_no_output`, + 82 msgs / 47 tool calls). → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] +- **Incidents: 1 closed / 1 opened; unclosed stays 9** (8 `alerted` / 1 `detected`). `08eb86_472c7b609c80` + **resolved 2026-10-21 08:37:21** after the job's 08:30 slot ran clean, which also cleared `08eb86f02d37` from + the non-clean table. Non-clean `last_status` **stays 9** (7 `error` / 2 `delivery_failed`), one swap in/out. +- **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the orphaned `08eb86f02d37` run. The previous + audit's "168 / 0" was taken before the 23:59 slot landed. + → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] +- **Config backup: 76 commits now unpushed.** The 2026-10-21 11:00 run created commit `a8486a20` and the push + failed for the 75th consecutive night; Telegram notice `message_id 8449` was delivered, AgentMail failed. + → [[incidents/2026-08-08-gitea-config-backup-push-dead]] +- **12 sessions in the window, all `cron` — no interactive sessions.** Fleet unchanged at **28 jobs + (25 enabled / 3 disabled)**; **17 pin `custom:omni-101`**, **7 carry no pin**, 1 pins `custom:omni`. ## Recent Changes (2026-10-21, 03:00 stocktake; live probes 03:00-03:12 AWST) diff --git a/incidents/2026-08-08-gitea-config-backup-push-dead.md b/incidents/2026-08-08-gitea-config-backup-push-dead.md index b179a72..c1ea6c0 100644 --- a/incidents/2026-08-08-gitea-config-backup-push-dead.md +++ b/incidents/2026-08-08-gitea-config-backup-push-dead.md @@ -3,9 +3,9 @@ title: Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 Whi type: incident status: active created: 2026-10-21 -updated: 2026-10-21 -verified_on: 2026-10-21 -review_after: 2026-10-28 +updated: 2026-10-22 +verified_on: 2026-10-22 +review_after: 2026-10-29 confidence: high tags: [backup, gitea, cron, git, green-vs-actual] sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki] @@ -83,6 +83,21 @@ tail -20 ~/.hermes/config-backup/backup.log tail -3 ~/.hermes/config-backup/.backup_history ``` +## Update 2026-10-21 (nightly run — the backlog grew again) + +| Probe | Result | +|---|---| +| 2026-10-21 11:00 run | commit **`a8486a20`** created (`🦝 nightly backup: 2026-10-21 11:00:14 AWST`) — **push failed again**, `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address` | +| `git rev-list --count origin/main..main` | **76** (was 75) — the backlog grows by exactly one commit per night | +| Telegram notice | delivered: `🚨 BACKUP FAILED — 2026-10-21 11:00:14 AWST`, **`message_id 8449`**; the AgentMail step failed again | +| GitHub fallback | `fatal: 'github' does not appear to be a git repository`; its "pushing clean branch" recovery (`github-clean-1792551629`) also failed and was deleted | +| `.backup_history` | `2026-10-19 … FAIL`, `2026-10-20 … FAIL`, **`2026-10-21 … FAIL`** | +| cron row `e27da6c34bbb` | still `last_status: ok` — the green-vs-actual gap is now **74 days** wide | + +**Compounding factor found the same night:** the `hermes update` that ran at **2026-10-21 22:47 AWST** recorded +its own `snapshot` stage as **`skipped`** (`disabled by updates.pre_update_backup (mode: off)`), so for that +change as well the *only* off-box copy path was this dead push → [[infrastructure/backups]]. + ## Related - [[infrastructure/backups]] diff --git a/incidents/2026-10-16-omniroute-503-resource-pressure.md b/incidents/2026-10-16-omniroute-503-resource-pressure.md index f7939dd..b1c2a88 100644 --- a/incidents/2026-10-16-omniroute-503-resource-pressure.md +++ b/incidents/2026-10-16-omniroute-503-resource-pressure.md @@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu type: incident status: monitoring created: 2026-10-18 -updated: 2026-10-21 -verified_on: 2026-10-21 -review_after: 2026-10-28 +updated: 2026-10-22 +verified_on: 2026-10-22 +review_after: 2026-10-29 confidence: high tags: [cron, router, provider, infrastructure] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe] @@ -220,6 +220,25 @@ clean running, not a fix). `nous` (`https://inference-api.nousresearch.com/v1`), `model.default: smart-route` → `meituan/longcat-2.5-preview:free`. Pinned jobs are unaffected; unpinned jobs are. → [[systems/current-profile]] +## Update 2026-10-21 (window 2026-10-20 → 10-21) + +- **A third failure mode from the same pin — and it is not 503.** The only failed cron row on **2026-10-21** + was `dca8482e4f76` (Daily Marketing Job Radar, `custom:omni-101` / `smart-route`), which died after + **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch + filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. +- **The resource-pressure string did not appear at all on 2026-10-21** (zero ledger rows; last occurrence + remains 2026-10-19 09:02), so the second burst stays closed and this page stays `status: monitoring`. +- **One row closed, one opened.** `08eb86_472c7b609c80` (the Gmail Action Digest 503s, first seen 10-16 08:33) + **resolved 2026-10-21 08:37:21** — the job's own 08:30 slot ran clean that morning (its 10-20 run was the + orphan; no incident row was ever raised for it), which also cleared `08eb86f02d37` from the non-clean table. + `dca848_014affeb6954` opened 07:40:42 and is **still `alerted`**. Unclosed rows **stay at 9** + (8 `alerted` / 1 `detected`). +- **Windows:** 2026-10-20 settled at **169 `completed` / 1 `unknown`** (the gateway-orphaned run; the 10-21 + audit's 168 / 0 was a mid-flight snapshot taken before the 23:59 slot landed). **2026-10-21: 171 `completed` + / 1 `failed`** (+ this audit). +- **Still no failover.** Every job that has failed on this node since 2026-10-14 still reads + `provider: custom:omni-101` with `model: smart-route` or `free-stack`, and none has a second pin. + ## Related - [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node. diff --git a/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md b/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md new file mode 100644 index 0000000..ca50448 --- /dev/null +++ b/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md @@ -0,0 +1,79 @@ +--- +title: Daily Marketing Job Radar — "All Targets Skipped by Pre-Dispatch Filters" (2026-10-21) +type: incident +status: unresolved +created: 2026-10-22 +updated: 2026-10-22 +verified_on: 2026-10-22 +review_after: 2026-10-29 +confidence: high +tags: [cron, router, provider, job-search, failure-mode] +sources: [live-jobs-json, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-cron-output-file] +--- + +# Daily Marketing Job Radar — Router Pre-Dispatch Filter Skip (2026-10-21) + +## Headline + +The **Daily Marketing Job Radar** (`dca8482e4f76`) — the weekday **07:00** slot that scrapes +SEEK / LinkedIn / Indeed / Jora, verifies against direct adverts, dedupes into the Notion job DB and +delivers a Telegram summary — **failed on 2026-10-21 after 40 m 36 s** with a signature the wiki has not +recorded before: + +``` +RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s +``` + +This is the **third distinct failure mode from the same provider pin** (`provider: custom:omni-101`, +`model: smart-route`) after the connection-refused storm (2026-10-14/15) and the two +`HTTP 503 … resource pressure` bursts (2026-10-16/17, 2026-10-19) documented in +[[incidents/2026-10-16-omniroute-503-resource-pressure]]. **The 503 string does not appear in any +2026-10-21 row.** + +## Evidence (live, 2026-10-22 00:1x AWST) + +| Probe | Result | +|---|---| +| `jobs.json` `dca8482e4f76` | `last_status: error`, `failure_streak: 1`, `schedule: 0 7 * * 1-5`, `provider: custom:omni-101`, `model: smart-route`, `deliver: telegram:-1004321904721:1030` | +| executions row `cd1d1f6a64ed427b9de13f31086ffb78` | `status: failed` — claimed **07:00:00.99**, started **07:00:12.40**, last progress **07:40:01.33**, finished **07:40:48.54** (**40 m 36 s**), `delivery_outcome: delivered` | +| `cron_incidents` row `dca848_014affeb6954` | `state: alerted`, `failure_type: unknown`, `first_seen_at: 2026-10-21T07:40:42.61`, `alerted_at: 2026-10-21T07:40:48.50`, **`closed_at: null`**, `error` = the string above | +| session `cron_dca8482e4f76_20261021_070022` | **82 messages / 47 tool calls**, ended 07:40:42, `end_reason: cron_incomplete_no_output` | +| cron output file | `/home/hermes/.hermes/cron/output/dca8482e4f76/2026-10-21_07-40-42.md` — `# Cron Job: Daily Marketing Job Radar (FAILED)` | +| the same hour's sibling | `85edf6772bca` ran 07:00:16 → 07:03:44 `cron_complete` (4 msgs) — the node was serving other jobs at that hour | +| whole 2026-10-21 fleet | **171 `completed` / 1 `failed`** — this row is the only failure of the day | +| 503 resource-pressure rows, 2026-10-21 | **zero** (last occurrence remains 2026-10-19 09:02) | + +## What the error says, and what it does not + +- **Not a 503**, **not a connection error**, **not a repetition-loop cut** — those three strings are the + entire recorded failure vocabulary of this fleet, and none of them matches. +- The phrasing is the harness/router's own. The wiki has **no page documenting the pre-dispatch filter + chain** that emits it, so the mechanism is unverified. What the string does assert: every dispatch target + was filtered out **before** a model call was made, and the guard reset **19 m 20 s** into the run before the + job was failed. The run itself spent **40 m 36 s** alive (07:00:12 → 07:40:48). +- **Not caused by the 2026-10-21 `hermes update`.** That update ran 22:47 → 22:57 AWST — 15 h *after* this + 07:00 failure. → [[systems/hermes-agent]] +- **Not a delivery problem.** `delivery_outcome: delivered`: the failure notice reached topic `:1030`, so the + slot was visibly red in Telegram rather than silently blank. + +## Impact + +One weekday radar slot produced nothing: no new roles discovered, verified, de-duped or written to the Notion +job DB, and no candidate summary delivered to topic `:1030`. The job recovers on its next weekday slot +(2026-10-22 07:00) if the guard does not repeat. + +## Open questions / next actions (proposed, not applied — needs Anthony's go-ahead) + +1. **Find which component emits this string.** If it is a router-side guard with a fixed reset window, a + job-level retry after the reset would have recovered the slot; if it is a local filter chain, one filter is + probably mis-configured for this job's toolset. +2. **Capture router-side telemetry at failure time** on the `omni-101` node — three distinct failure modes + from one pin in eight days with no host telemetry captured for any of them. +3. **Give the radar a second provider pin.** It has no failover: one node's guard blanks the slot. + +## Related + +- [[incidents/2026-10-16-omniroute-503-resource-pressure]] +- [[systems/omniroute]] +- [[systems/scheduled-tasks]] +- [[systems/hermes-agent]] diff --git a/index.md b/index.md index 4c4f42e..ca53b9d 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-21 +> model routing and automation. Last updated: 2026-10-22 > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -34,10 +34,11 @@ ### Integrations - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP). - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. -- [[systems/scheduled-tasks]] — **As of 2026-10-21 00:0x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12.** The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** +- [[systems/scheduled-tasks]] — **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. The **2026-10-17** slot failed (`HTTP 503`), but the **10-18 03:00** slot closed the hole: `freshness-20261018.txt` written, index regenerated and committed (`dd53e5f`) — freshness **47 stale / 41 fresh / 7 no-date**. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13). ## Incidents +- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` **`alerted`, unclosed**; the only cron failure of the day (171 `completed` / 1 `failed`). **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]] - [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery. - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]] - [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**. diff --git a/infrastructure/backups.md b/infrastructure/backups.md index 923ec27..4223c3d 100644 --- a/infrastructure/backups.md +++ b/infrastructure/backups.md @@ -3,8 +3,8 @@ title: Backups type: infrastructure status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-10-22 +verified_on: 2026-10-22 confidence: medium tags: [infrastructure, backups] sources: [raw/configs/hermes-config-sanitized.txt] @@ -49,13 +49,17 @@ How Hermes's own configuration/state gets backed up, and what's missing. > tarball step in [[runbooks/backup-wiki]] has never been executed (the runbook's `mkdir -p` would > create it on first use). - `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net. + **Exercised on 2026-10-21:** the 22:47 `hermes update` (0.22.2 → 0.22.3) recorded its `snapshot` stage as + `skipped` with reason `disabled by updates.pre_update_backup (mode: off)`, so that update shipped with **no + local snapshot and no off-box copy** — the Gitea mirror below has been dead since 2026-08-08. + → [[systems/hermes-agent]] ## ⚠ Scheduled config backup — the Gitea mirror has been dead since 2026-08-08 `e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its execution row reads `completed`.** Last commit that reached Gitea: **2026-08-07 11:00:47**; the local repo is -**75 commits ahead of `origin/main`**. Root cause is a credential: `origin` is +**76 commits ahead of `origin/main`**. Root cause is a credential: `origin` is `https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — HTTPS, no userinfo, no credential helper — so the non-interactive push dies on `fatal: could not read Username … No such device or address` → `✗ Push to Gitea FAILED` → @@ -64,7 +68,7 @@ helper — so the non-interactive push dies on **Consequence for this page: everything above describes a topology that has not carried `~/.hermes` off the box since 2026-08-07.** The 2026-07-22 full zip and the wiki/authoring repos remain the only recent off-box copies. -Telegram failure notices *have* been firing daily (last: 2026-10-20 11:00, `message_id 8447`); the AgentMail +Telegram failure notices *have* been firing daily (last: 2026-10-21 11:00:14, `message_id 8449`); the AgentMail notification step failed. ## Relevant runbooks diff --git a/log.md b/log.md index 49a2274..4f1f7c5 100644 --- a/log.md +++ b/log.md @@ -1023,3 +1023,39 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st - **Fleet unchanged** — 28 jobs / 25 enabled. This job (`1a8e90ea2271`) is `active`, `last_status: ok`, `model: free-stack` / `provider: custom:omni-101`, next run 2026-10-22 03:00. - **No secrets written.** Only host names, IPs, container IDs, job IDs, port numbers and HTTP status codes. + +## [2026-10-21] audit | Nightly wiki audit — window 2026-10-21 00:00 → 23:59 AWST: Hermes updated to 0.22.3 with its snapshot skipped, and a third failure signature from the `omni-101` pin + +- **New page: [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]].** The **Daily Marketing Job Radar** + (`dca8482e4f76`, 07:00 weekday, `custom:omni-101` / `smart-route`) failed **2026-10-21 07:40:48 after + 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch + filters reset after 19m 20s` — a string this fleet has never produced before, and **not** the 503 + resource-pressure error (zero such rows exist for 10-21; last 2026-10-19 09:02). Session + `cron_dca8482e4f76_20261021_070022` wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls); + incident `dca848_014affeb6954` is **`alerted` and unclosed**; it was the day's only failure + (171 `completed` / 1 `failed`). +- **Infrastructure: Hermes Agent updated to `0.22.3`** — `hermes update` ran **22:47:25 → 22:57:38 AWST** + (action `9eed3a99e799347c691ea52a72e122a1`), pre `0.22.2` / `d923a5eb…` → commit `5b147ab036…` (detached), + stamp built 2026-10-21T14:57:38Z. **Stage `snapshot` = `skipped`** (`disabled by + updates.pre_update_backup (mode: off)`), so the update had no snapshot and the off-box copy path is still + dead. Gateway restarted via systemd (pid `868087`, `phase: running`, heartbeat 00:13 AWST), dashboard + respawned by argv (`:9119`, pid `869852`). Advisories printed, neither actioned: rebuild the search index + (`hermes sessions optimize-storage`) and *Preparing pinned cua-driver…* (no completion line in the log). + The intermediate `d923a5eb` rebuild was never recorded on the page, so the previously-noted + `v0.21.5+6169.gab86a20` (2026-10-04) was already stale — corrected. +- **Pages updated:** `systems/hermes-agent.md` (version/upstream bullets rewritten, frontmatter dated), + `systems/scheduled-tasks.md` (new current-count bullet; 10-21 bullet marked superseded), + `systems/omniroute.md` (new 2026-10-21 section: three signatures, one pin), + `incidents/2026-10-16-omniroute-503-resource-pressure.md` (window-close update; stays `monitoring`), + `incidents/2026-08-08-gitea-config-backup-push-dead.md` (2026-10-21 run: backlog 75 → **76**, + `message_id 8449`), `infrastructure/backups.md` (76 commits; `pre_update_backup: false` now has a + concrete exercised example), `current-state.md` (new dated section), `index.md` (new incident entry, + scheduled-tasks summary, date). +- **Ledger facts as recorded this window:** fleet **28 jobs / 25 enabled / 3 disabled**; non-clean + `last_status` **9** (7 `error` / 2 `delivery_failed`) with one swap — `08eb86f02d37` out (its 08:30 slot ran + clean and `08eb86_472c7b609c80` **resolved 08:37:21**), `dca8482e4f76` in; **1 incident opened / 1 closed, + unclosed stays 9** (8 `alerted` / 1 `detected`); **12 sessions, all `cron`, no interactive sessions**; + 2026-10-20 settled at **169 `completed` / 1 `unknown`** (the orphan) — the 10-21 audit's "168 / 0" was taken + before the 23:59 slot landed, and that correction is recorded in both `systems/scheduled-tasks.md` and + `current-state.md`. +- **No secrets written.** Host names, IPs, job IDs, incident IDs, ports, commit SHAs and HTTP status codes only. diff --git a/systems/hermes-agent.md b/systems/hermes-agent.md index 3b85bd8..fcf7c38 100644 --- a/systems/hermes-agent.md +++ b/systems/hermes-agent.md @@ -3,8 +3,8 @@ title: Hermes Agent type: system status: active created: 2026-07-22 -updated: 2026-10-11 -verified_on: 2026-10-11 +updated: 2026-10-22 +verified_on: 2026-10-22 confidence: high tags: [system, hermes, agent, llm] sources: [live-hermes-version, live-config, live-skill-inventory] @@ -16,10 +16,10 @@ sources: [live-hermes-version, live-config, live-skill-inventory] Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks. ## Current Status -- **Version:** v0.21.5+6169.gab86a20 (2026.9.24), upstream `ab86a20a` — verified live 2026-10-04. Same semver token as the earlier `v0.21.5+3837.gbfda74c` (`27062c34`), but a newer build: the install tree was rebuilt 2026-10-03 10:11 AWST. +- **Version:** `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`. - **Install method:** git - **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15) -- **Upstream status:** tree rebuilt 2026-10-03 from upstream `ab86a20a`; update availability not re-checked since. +- **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on. - **Install directory:** `/home/hermes/.hermes/hermes-agent` - **Managed runtimes:** 3 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments//venv` (was 10 — the 2026-10-03 rebuild pruned superseded environments and created two new ones at 09:59/10:00; the third survives from 2026-09-28). `hermes-runtime-deps-guard.sh` runs every 15 min via `mnemosyne-watchdog.py` and now asserts `dotenv` + `mnemosyne` in the installer python *and every managed venv*, **plus** `fastembed` + `FASTEMBED_CACHE_PATH` + the local model dir in the bootstrap-selected venv — see [[incidents/2026-09-28-hermes-managed-runtime-deps]] and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]]. - **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions diff --git a/systems/omniroute.md b/systems/omniroute.md index de0c246..d30b831 100644 --- a/systems/omniroute.md +++ b/systems/omniroute.md @@ -3,8 +3,8 @@ title: OmniRoute type: system status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-10-22 +verified_on: 2026-10-22 confidence: high tags: [routing, provider, model, inference] sources: [raw/configs/hermes-config-sanitized.txt] @@ -12,6 +12,20 @@ sources: [raw/configs/hermes-config-sanitized.txt] # OmniRoute +## Recent Changes (2026-10-21) + +- **Third failure mode on the same node, and it is not the 503 string.** `dca8482e4f76` (Daily Marketing Job + Radar) failed on 2026-10-21 at 07:40:48 after 40 m 36 s with + `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after + 19m 20s`. **Zero `HTTP 503 … resource pressure` rows exist for that day** (last: 2026-10-19 09:02) and the + sibling 07:00 job `85edf6772bca` completed normally, so the node was serving work at that hour. + → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] +- **One pin, three signatures in eight days:** connection-refused storm (10-14/15), resource-pressure 503s + (10-16/17 and 10-19), pre-dispatch filter skip (10-21) — all on `custom:omni-101`, all with + `model: smart-route`. No job in the fleet has a second provider pin. +- **Router-side telemetry is still missing for every one of them** — `omni-101` host logs have not been + captured during a failure, which is why none of the three has a confirmed root cause. + ## Recent Changes (2026-10-20) - **Recurring `503 … resource pressure` load-shedding on `omniroute-1` — second burst in four days.** On diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index cd57fef..1de2462 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-21 -verified_on: 2026-10-21 -review_after: 2026-10-28 +updated: 2026-10-22 +verified_on: 2026-10-22 +review_after: 2026-10-29 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] @@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **⚠ Current as of 2026-10-21 00:0x AWST (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]] +- **Current as of 2026-10-22 00:1x AWST (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]]. +- **⚠ Current as of 2026-10-21 00:0x AWST *(superseded by the 2026-10-22 entry above)* (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **Current as of 2026-10-20 00:09 AWST *(superseded by the 2026-10-21 entry above)* — the `omni-101` 503 fault is BACK, and it is now the fleet's defining risk.** Window 2026-10-19: **159 `completed` / 13 `failed`**, against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all carry the identical `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on `custom:omni-101`, in two clusters — **06:01** (one job) and **07:01 → 12:01** (seven more runs across seven jobs). The other **4 rows are derivative Mnemosyne Watchdog exits** (`96eb95ce12f2` at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the **second burst in four days** on the same node with the same blast radius, while `/api/health` read `200 {"status":"ok"}` throughout — so [[incidents/2026-10-16-omniroute-503-resource-pressure]] is back to **`status: active`** (it had been downgraded to `monitoring` 44 h earlier). **Fleet unchanged at 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` 7 → 12** (10 `error` + 2 `delivery_failed`); **unclosed incident rows 13** (12 `alerted` / 1 `detected`) — 4 opened and 2 closed on 10-19. Every `failure_streak` still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. **15 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]] - **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]