🦉 wiki audit 2026-10-22: omni-101 pre-dispatch filter skip (3rd failure mode, new page), Hermes 0.22.3 update with snapshot skipped, Gitea backup backlog 76

- new: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- systems/hermes-agent.md: version -> 0.22.3 (5b147ab036), 22:47-22:57 update, snapshot stage skipped, advisories unactioned
- systems/scheduled-tasks.md: 2026-10-21 = 171 completed / 1 failed; non-clean stays 9 with 08eb86f02d37 out, dca8482e4f76 in
- systems/omniroute.md: one pin, three signatures in eight days
- incidents/2026-10-16-omniroute-503-resource-pressure.md: window update, stays monitoring
- incidents/2026-08-08-gitea-config-backup-push-dead.md: 2026-10-21 run, backlog 75 -> 76, message_id 8449
- infrastructure/backups.md: 76 commits; pre_update_backup:false now exercised
- current-state.md, index.md, log.md updated
This commit is contained in:
Tony0410
2026-10-22 00:16:39 +08:00
parent c1d3b59260
commit 3919ec8557
10 changed files with 225 additions and 26 deletions
+34 -4
View File
@@ -3,16 +3,46 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-21 updated: 2026-10-22
verified_on: 2026-10-21 verified_on: 2026-10-22
review_after: 2026-10-28 review_after: 2026-10-29
confidence: high confidence: high
tags: [system] tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
--- ---
# Current State # Current State
Updated: 2026-10-21 Updated: 2026-10-22
## Recent Changes (2026-10-22, nightly audit; live probes 2026-10-21 23:59 → 2026-10-22 00:20 AWST)
- **Hermes Agent updated to `0.22.3`.** `hermes update` ran **2026-10-21 22:47:25 → 22:57:38 AWST**
(action `9eed3a99e799347c691ea52a72e122a1`), from pre-update tree `0.22.2` / `d923a5eb487a17c0…` to commit
`5b147ab036d6…` (detached, install stamp `displayVersion 0.22.3`, built 2026-10-21T14:57:38Z). Stages
`plan` / `apply` (git) / `deps` / `build` all succeeded, **but `snapshot` = `skipped`**
(`disabled by updates.pre_update_backup (mode: off)`). Gateway restarted via **systemd** — pid `868087`,
`phase: running`, `api_server: connected`, heartbeat 2026-10-22 00:13 AWST — and the dashboard was respawned
by argv on `:9119` (pid `869852`). Post-run advisories, **neither yet acted on**: rebuild the search index
(`hermes sessions optimize-storage`) and *Preparing pinned cua-driver (Computer Use)…* (no completion line in
the log). → [[systems/hermes-agent]]
- **The only cron failure of 2026-10-21 is a new failure mode.** Window: **171 `completed` / 1 `failed`** (+ this
audit). `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`) failed
at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by
pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string (zero such rows on 10-21;
last 2026-10-19 09:02) — while the sibling 07:00 job `85edf6772bca` completed normally. Incident
`dca848_014affeb6954` **`alerted`, unclosed**; the run produced no output (`cron_incomplete_no_output`,
82 msgs / 47 tool calls). → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
- **Incidents: 1 closed / 1 opened; unclosed stays 9** (8 `alerted` / 1 `detected`). `08eb86_472c7b609c80`
**resolved 2026-10-21 08:37:21** after the job's 08:30 slot ran clean, which also cleared `08eb86f02d37` from
the non-clean table. Non-clean `last_status` **stays 9** (7 `error` / 2 `delivery_failed`), one swap in/out.
- **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the orphaned `08eb86f02d37` run. The previous
audit's "168 / 0" was taken before the 23:59 slot landed.
→ [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]
- **Config backup: 76 commits now unpushed.** The 2026-10-21 11:00 run created commit `a8486a20` and the push
failed for the 75th consecutive night; Telegram notice `message_id 8449` was delivered, AgentMail failed.
→ [[incidents/2026-08-08-gitea-config-backup-push-dead]]
- **12 sessions in the window, all `cron` — no interactive sessions.** Fleet unchanged at **28 jobs
(25 enabled / 3 disabled)**; **17 pin `custom:omni-101`**, **7 carry no pin**, 1 pins `custom:omni`.
## Recent Changes (2026-10-21, 03:00 stocktake; live probes 03:00-03:12 AWST) ## Recent Changes (2026-10-21, 03:00 stocktake; live probes 03:00-03:12 AWST)
@@ -3,9 +3,9 @@ title: Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 Whi
type: incident type: incident
status: active status: active
created: 2026-10-21 created: 2026-10-21
updated: 2026-10-21 updated: 2026-10-22
verified_on: 2026-10-21 verified_on: 2026-10-22
review_after: 2026-10-28 review_after: 2026-10-29
confidence: high confidence: high
tags: [backup, gitea, cron, git, green-vs-actual] tags: [backup, gitea, cron, git, green-vs-actual]
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki] sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki]
@@ -83,6 +83,21 @@ tail -20 ~/.hermes/config-backup/backup.log
tail -3 ~/.hermes/config-backup/.backup_history tail -3 ~/.hermes/config-backup/.backup_history
``` ```
## Update 2026-10-21 (nightly run — the backlog grew again)
| Probe | Result |
|---|---|
| 2026-10-21 11:00 run | commit **`a8486a20`** created (`🦝 nightly backup: 2026-10-21 11:00:14 AWST`) — **push failed again**, `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address` |
| `git rev-list --count origin/main..main` | **76** (was 75) — the backlog grows by exactly one commit per night |
| Telegram notice | delivered: `🚨 BACKUP FAILED — 2026-10-21 11:00:14 AWST`, **`message_id 8449`**; the AgentMail step failed again |
| GitHub fallback | `fatal: 'github' does not appear to be a git repository`; its "pushing clean branch" recovery (`github-clean-1792551629`) also failed and was deleted |
| `.backup_history` | `2026-10-19 … FAIL`, `2026-10-20 … FAIL`, **`2026-10-21 … FAIL`** |
| cron row `e27da6c34bbb` | still `last_status: ok` — the green-vs-actual gap is now **74 days** wide |
**Compounding factor found the same night:** the `hermes update` that ran at **2026-10-21 22:47 AWST** recorded
its own `snapshot` stage as **`skipped`** (`disabled by updates.pre_update_backup (mode: off)`), so for that
change as well the *only* off-box copy path was this dead push → [[infrastructure/backups]].
## Related ## Related
- [[infrastructure/backups]] - [[infrastructure/backups]]
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
type: incident type: incident
status: monitoring status: monitoring
created: 2026-10-18 created: 2026-10-18
updated: 2026-10-21 updated: 2026-10-22
verified_on: 2026-10-21 verified_on: 2026-10-22
review_after: 2026-10-28 review_after: 2026-10-29
confidence: high confidence: high
tags: [cron, router, provider, infrastructure] tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
@@ -220,6 +220,25 @@ clean running, not a fix).
`nous` (`https://inference-api.nousresearch.com/v1`), `model.default: smart-route` → `nous` (`https://inference-api.nousresearch.com/v1`), `model.default: smart-route` →
`meituan/longcat-2.5-preview:free`. Pinned jobs are unaffected; unpinned jobs are. → [[systems/current-profile]] `meituan/longcat-2.5-preview:free`. Pinned jobs are unaffected; unpinned jobs are. → [[systems/current-profile]]
## Update 2026-10-21 (window 2026-10-20 → 10-21)
- **A third failure mode from the same pin — and it is not 503.** The only failed cron row on **2026-10-21**
was `dca8482e4f76` (Daily Marketing Job Radar, `custom:omni-101` / `smart-route`), which died after
**40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch
filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]].
- **The resource-pressure string did not appear at all on 2026-10-21** (zero ledger rows; last occurrence
remains 2026-10-19 09:02), so the second burst stays closed and this page stays `status: monitoring`.
- **One row closed, one opened.** `08eb86_472c7b609c80` (the Gmail Action Digest 503s, first seen 10-16 08:33)
**resolved 2026-10-21 08:37:21** — the job's own 08:30 slot ran clean that morning (its 10-20 run was the
orphan; no incident row was ever raised for it), which also cleared `08eb86f02d37` from the non-clean table.
`dca848_014affeb6954` opened 07:40:42 and is **still `alerted`**. Unclosed rows **stay at 9**
(8 `alerted` / 1 `detected`).
- **Windows:** 2026-10-20 settled at **169 `completed` / 1 `unknown`** (the gateway-orphaned run; the 10-21
audit's 168 / 0 was a mid-flight snapshot taken before the 23:59 slot landed). **2026-10-21: 171 `completed`
/ 1 `failed`** (+ this audit).
- **Still no failover.** Every job that has failed on this node since 2026-10-14 still reads
`provider: custom:omni-101` with `model: smart-route` or `free-stack`, and none has a second pin.
## Related ## Related
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node. - [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node.
@@ -0,0 +1,79 @@
---
title: Daily Marketing Job Radar — "All Targets Skipped by Pre-Dispatch Filters" (2026-10-21)
type: incident
status: unresolved
created: 2026-10-22
updated: 2026-10-22
verified_on: 2026-10-22
review_after: 2026-10-29
confidence: high
tags: [cron, router, provider, job-search, failure-mode]
sources: [live-jobs-json, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-cron-output-file]
---
# Daily Marketing Job Radar — Router Pre-Dispatch Filter Skip (2026-10-21)
## Headline
The **Daily Marketing Job Radar** (`dca8482e4f76`) — the weekday **07:00** slot that scrapes
SEEK / LinkedIn / Indeed / Jora, verifies against direct adverts, dedupes into the Notion job DB and
delivers a Telegram summary — **failed on 2026-10-21 after 40 m 36 s** with a signature the wiki has not
recorded before:
```
RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s
```
This is the **third distinct failure mode from the same provider pin** (`provider: custom:omni-101`,
`model: smart-route`) after the connection-refused storm (2026-10-14/15) and the two
`HTTP 503 … resource pressure` bursts (2026-10-16/17, 2026-10-19) documented in
[[incidents/2026-10-16-omniroute-503-resource-pressure]]. **The 503 string does not appear in any
2026-10-21 row.**
## Evidence (live, 2026-10-22 00:1x AWST)
| Probe | Result |
|---|---|
| `jobs.json` `dca8482e4f76` | `last_status: error`, `failure_streak: 1`, `schedule: 0 7 * * 1-5`, `provider: custom:omni-101`, `model: smart-route`, `deliver: telegram:-1004321904721:1030` |
| executions row `cd1d1f6a64ed427b9de13f31086ffb78` | `status: failed` — claimed **07:00:00.99**, started **07:00:12.40**, last progress **07:40:01.33**, finished **07:40:48.54** (**40 m 36 s**), `delivery_outcome: delivered` |
| `cron_incidents` row `dca848_014affeb6954` | `state: alerted`, `failure_type: unknown`, `first_seen_at: 2026-10-21T07:40:42.61`, `alerted_at: 2026-10-21T07:40:48.50`, **`closed_at: null`**, `error` = the string above |
| session `cron_dca8482e4f76_20261021_070022` | **82 messages / 47 tool calls**, ended 07:40:42, `end_reason: cron_incomplete_no_output` |
| cron output file | `/home/hermes/.hermes/cron/output/dca8482e4f76/2026-10-21_07-40-42.md` — `# Cron Job: Daily Marketing Job Radar (FAILED)` |
| the same hour's sibling | `85edf6772bca` ran 07:00:16 → 07:03:44 `cron_complete` (4 msgs) — the node was serving other jobs at that hour |
| whole 2026-10-21 fleet | **171 `completed` / 1 `failed`** — this row is the only failure of the day |
| 503 resource-pressure rows, 2026-10-21 | **zero** (last occurrence remains 2026-10-19 09:02) |
## What the error says, and what it does not
- **Not a 503**, **not a connection error**, **not a repetition-loop cut** — those three strings are the
entire recorded failure vocabulary of this fleet, and none of them matches.
- The phrasing is the harness/router's own. The wiki has **no page documenting the pre-dispatch filter
chain** that emits it, so the mechanism is unverified. What the string does assert: every dispatch target
was filtered out **before** a model call was made, and the guard reset **19 m 20 s** into the run before the
job was failed. The run itself spent **40 m 36 s** alive (07:00:12 → 07:40:48).
- **Not caused by the 2026-10-21 `hermes update`.** That update ran 22:47 → 22:57 AWST — 15 h *after* this
07:00 failure. → [[systems/hermes-agent]]
- **Not a delivery problem.** `delivery_outcome: delivered`: the failure notice reached topic `:1030`, so the
slot was visibly red in Telegram rather than silently blank.
## Impact
One weekday radar slot produced nothing: no new roles discovered, verified, de-duped or written to the Notion
job DB, and no candidate summary delivered to topic `:1030`. The job recovers on its next weekday slot
(2026-10-22 07:00) if the guard does not repeat.
## Open questions / next actions (proposed, not applied — needs Anthony's go-ahead)
1. **Find which component emits this string.** If it is a router-side guard with a fixed reset window, a
job-level retry after the reset would have recovered the slot; if it is a local filter chain, one filter is
probably mis-configured for this job's toolset.
2. **Capture router-side telemetry at failure time** on the `omni-101` node — three distinct failure modes
from one pin in eight days with no host telemetry captured for any of them.
3. **Give the radar a second provider pin.** It has no failover: one node's guard blanks the slot.
## Related
- [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- [[systems/omniroute]]
- [[systems/scheduled-tasks]]
- [[systems/hermes-agent]]
+3 -2
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki # Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure, > Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-21 > model routing and automation. Last updated: 2026-10-22
> >
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
@@ -34,10 +34,11 @@
### Integrations ### Integrations
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP). - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP).
- [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip.
- [[systems/scheduled-tasks]] — **As of 2026-10-21 00:0x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12.** The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/scheduled-tasks]] — **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. The **2026-10-17** slot failed (`HTTP 503`), but the **10-18 03:00** slot closed the hole: `freshness-20261018.txt` written, index regenerated and committed (`dd53e5f`) — freshness **47 stale / 41 fresh / 7 no-date**. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13). - [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. The **2026-10-17** slot failed (`HTTP 503`), but the **10-18 03:00** slot closed the hole: `freshness-20261018.txt` written, index regenerated and committed (`dd53e5f`) — freshness **47 stale / 41 fresh / 7 no-date**. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
## Incidents ## Incidents
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` **`alerted`, unclosed**; the only cron failure of the day (171 `completed` / 1 `failed`). **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery. - [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery.
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]] - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**. - [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**.
+8 -4
View File
@@ -3,8 +3,8 @@ title: Backups
type: infrastructure type: infrastructure
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-10-22
verified_on: 2026-10-20 verified_on: 2026-10-22
confidence: medium confidence: medium
tags: [infrastructure, backups] tags: [infrastructure, backups]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -49,13 +49,17 @@ How Hermes's own configuration/state gets backed up, and what's missing.
> tarball step in [[runbooks/backup-wiki]] has never been executed (the runbook's `mkdir -p` would > tarball step in [[runbooks/backup-wiki]] has never been executed (the runbook's `mkdir -p` would
> create it on first use). > create it on first use).
- `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net. - `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net.
**Exercised on 2026-10-21:** the 22:47 `hermes update` (0.22.2 → 0.22.3) recorded its `snapshot` stage as
`skipped` with reason `disabled by updates.pre_update_backup (mode: off)`, so that update shipped with **no
local snapshot and no off-box copy** — the Gitea mirror below has been dead since 2026-08-08.
→ [[systems/hermes-agent]]
## ⚠ Scheduled config backup — the Gitea mirror has been dead since 2026-08-08 ## ⚠ Scheduled config backup — the Gitea mirror has been dead since 2026-08-08
`e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits `e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits
locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its
execution row reads `completed`.** Last commit that reached Gitea: **2026-08-07 11:00:47**; the local repo is execution row reads `completed`.** Last commit that reached Gitea: **2026-08-07 11:00:47**; the local repo is
**75 commits ahead of `origin/main`**. Root cause is a credential: `origin` is **76 commits ahead of `origin/main`**. Root cause is a credential: `origin` is
`https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — HTTPS, no userinfo, no credential `https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — HTTPS, no userinfo, no credential
helper — so the non-interactive push dies on helper — so the non-interactive push dies on
`fatal: could not read Username … No such device or address` → `✗ Push to Gitea FAILED` → `fatal: could not read Username … No such device or address` → `✗ Push to Gitea FAILED` →
@@ -64,7 +68,7 @@ helper — so the non-interactive push dies on
**Consequence for this page: everything above describes a topology that has not carried `~/.hermes` off the box **Consequence for this page: everything above describes a topology that has not carried `~/.hermes` off the box
since 2026-08-07.** The 2026-07-22 full zip and the wiki/authoring repos remain the only recent off-box copies. since 2026-08-07.** The 2026-07-22 full zip and the wiki/authoring repos remain the only recent off-box copies.
Telegram failure notices *have* been firing daily (last: 2026-10-20 11:00, `message_id 8447`); the AgentMail Telegram failure notices *have* been firing daily (last: 2026-10-21 11:00:14, `message_id 8449`); the AgentMail
notification step failed. notification step failed.
## Relevant runbooks ## Relevant runbooks
+36
View File
@@ -1023,3 +1023,39 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st
- **Fleet unchanged** — 28 jobs / 25 enabled. This job (`1a8e90ea2271`) is `active`, `last_status: ok`, - **Fleet unchanged** — 28 jobs / 25 enabled. This job (`1a8e90ea2271`) is `active`, `last_status: ok`,
`model: free-stack` / `provider: custom:omni-101`, next run 2026-10-22 03:00. `model: free-stack` / `provider: custom:omni-101`, next run 2026-10-22 03:00.
- **No secrets written.** Only host names, IPs, container IDs, job IDs, port numbers and HTTP status codes. - **No secrets written.** Only host names, IPs, container IDs, job IDs, port numbers and HTTP status codes.
## [2026-10-21] audit | Nightly wiki audit — window 2026-10-21 00:00 → 23:59 AWST: Hermes updated to 0.22.3 with its snapshot skipped, and a third failure signature from the `omni-101` pin
- **New page: [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]].** The **Daily Marketing Job Radar**
(`dca8482e4f76`, 07:00 weekday, `custom:omni-101` / `smart-route`) failed **2026-10-21 07:40:48 after
40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch
filters reset after 19m 20s` — a string this fleet has never produced before, and **not** the 503
resource-pressure error (zero such rows exist for 10-21; last 2026-10-19 09:02). Session
`cron_dca8482e4f76_20261021_070022` wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls);
incident `dca848_014affeb6954` is **`alerted` and unclosed**; it was the day's only failure
(171 `completed` / 1 `failed`).
- **Infrastructure: Hermes Agent updated to `0.22.3`** — `hermes update` ran **22:47:25 → 22:57:38 AWST**
(action `9eed3a99e799347c691ea52a72e122a1`), pre `0.22.2` / `d923a5eb…` → commit `5b147ab036…` (detached),
stamp built 2026-10-21T14:57:38Z. **Stage `snapshot` = `skipped`** (`disabled by
updates.pre_update_backup (mode: off)`), so the update had no snapshot and the off-box copy path is still
dead. Gateway restarted via systemd (pid `868087`, `phase: running`, heartbeat 00:13 AWST), dashboard
respawned by argv (`:9119`, pid `869852`). Advisories printed, neither actioned: rebuild the search index
(`hermes sessions optimize-storage`) and *Preparing pinned cua-driver…* (no completion line in the log).
The intermediate `d923a5eb` rebuild was never recorded on the page, so the previously-noted
`v0.21.5+6169.gab86a20` (2026-10-04) was already stale — corrected.
- **Pages updated:** `systems/hermes-agent.md` (version/upstream bullets rewritten, frontmatter dated),
`systems/scheduled-tasks.md` (new current-count bullet; 10-21 bullet marked superseded),
`systems/omniroute.md` (new 2026-10-21 section: three signatures, one pin),
`incidents/2026-10-16-omniroute-503-resource-pressure.md` (window-close update; stays `monitoring`),
`incidents/2026-08-08-gitea-config-backup-push-dead.md` (2026-10-21 run: backlog 75 → **76**,
`message_id 8449`), `infrastructure/backups.md` (76 commits; `pre_update_backup: false` now has a
concrete exercised example), `current-state.md` (new dated section), `index.md` (new incident entry,
scheduled-tasks summary, date).
- **Ledger facts as recorded this window:** fleet **28 jobs / 25 enabled / 3 disabled**; non-clean
`last_status` **9** (7 `error` / 2 `delivery_failed`) with one swap — `08eb86f02d37` out (its 08:30 slot ran
clean and `08eb86_472c7b609c80` **resolved 08:37:21**), `dca8482e4f76` in; **1 incident opened / 1 closed,
unclosed stays 9** (8 `alerted` / 1 `detected`); **12 sessions, all `cron`, no interactive sessions**;
2026-10-20 settled at **169 `completed` / 1 `unknown`** (the orphan) — the 10-21 audit's "168 / 0" was taken
before the 23:59 slot landed, and that correction is recorded in both `systems/scheduled-tasks.md` and
`current-state.md`.
- **No secrets written.** Host names, IPs, job IDs, incident IDs, ports, commit SHAs and HTTP status codes only.
+4 -4
View File
@@ -3,8 +3,8 @@ title: Hermes Agent
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-11 updated: 2026-10-22
verified_on: 2026-10-11 verified_on: 2026-10-22
confidence: high confidence: high
tags: [system, hermes, agent, llm] tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory] sources: [live-hermes-version, live-config, live-skill-inventory]
@@ -16,10 +16,10 @@ sources: [live-hermes-version, live-config, live-skill-inventory]
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks. Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
## Current Status ## Current Status
- **Version:** v0.21.5+6169.gab86a20 (2026.9.24), upstream `ab86a20a` — verified live 2026-10-04. Same semver token as the earlier `v0.21.5+3837.gbfda74c` (`27062c34`), but a newer build: the install tree was rebuilt 2026-10-03 10:11 AWST. - **Version:** `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`.
- **Install method:** git - **Install method:** git
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15) - **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
- **Upstream status:** tree rebuilt 2026-10-03 from upstream `ab86a20a`; update availability not re-checked since. - **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on.
- **Install directory:** `/home/hermes/.hermes/hermes-agent` - **Install directory:** `/home/hermes/.hermes/hermes-agent`
- **Managed runtimes:** 3 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv` (was 10 — the 2026-10-03 rebuild pruned superseded environments and created two new ones at 09:59/10:00; the third survives from 2026-09-28). `hermes-runtime-deps-guard.sh` runs every 15 min via `mnemosyne-watchdog.py` and now asserts `dotenv` + `mnemosyne` in the installer python *and every managed venv*, **plus** `fastembed` + `FASTEMBED_CACHE_PATH` + the local model dir in the bootstrap-selected venv — see [[incidents/2026-09-28-hermes-managed-runtime-deps]] and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]]. - **Managed runtimes:** 3 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv` (was 10 — the 2026-10-03 rebuild pruned superseded environments and created two new ones at 09:59/10:00; the third survives from 2026-09-28). `hermes-runtime-deps-guard.sh` runs every 15 min via `mnemosyne-watchdog.py` and now asserts `dotenv` + `mnemosyne` in the installer python *and every managed venv*, **plus** `fastembed` + `FASTEMBED_CACHE_PATH` + the local model dir in the bootstrap-selected venv — see [[incidents/2026-09-28-hermes-managed-runtime-deps]] and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions - **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
+16 -2
View File
@@ -3,8 +3,8 @@ title: OmniRoute
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-10-22
verified_on: 2026-10-20 verified_on: 2026-10-22
confidence: high confidence: high
tags: [routing, provider, model, inference] tags: [routing, provider, model, inference]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -12,6 +12,20 @@ sources: [raw/configs/hermes-config-sanitized.txt]
# OmniRoute # OmniRoute
## Recent Changes (2026-10-21)
- **Third failure mode on the same node, and it is not the 503 string.** `dca8482e4f76` (Daily Marketing Job
Radar) failed on 2026-10-21 at 07:40:48 after 40 m 36 s with
`RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after
19m 20s`. **Zero `HTTP 503 … resource pressure` rows exist for that day** (last: 2026-10-19 09:02) and the
sibling 07:00 job `85edf6772bca` completed normally, so the node was serving work at that hour.
→ [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
- **One pin, three signatures in eight days:** connection-refused storm (10-14/15), resource-pressure 503s
(10-16/17 and 10-19), pre-dispatch filter skip (10-21) — all on `custom:omni-101`, all with
`model: smart-route`. No job in the fleet has a second provider pin.
- **Router-side telemetry is still missing for every one of them** — `omni-101` host logs have not been
captured during a failure, which is why none of the three has a confirmed root cause.
## Recent Changes (2026-10-20) ## Recent Changes (2026-10-20)
- **Recurring `503 … resource pressure` load-shedding on `omniroute-1` — second burst in four days.** On - **Recurring `503 … resource pressure` load-shedding on `omniroute-1` — second burst in four days.** On
+5 -4
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-21 updated: 2026-10-22
verified_on: 2026-10-21 verified_on: 2026-10-22
review_after: 2026-10-28 review_after: 2026-10-29
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db] sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **⚠ Current as of 2026-10-21 00:0x AWST (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **Current as of 2026-10-22 00:1x AWST (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]].
- **⚠ Current as of 2026-10-21 00:0x AWST *(superseded by the 2026-10-22 entry above)* (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **Current as of 2026-10-20 00:09 AWST *(superseded by the 2026-10-21 entry above)* — the `omni-101` 503 fault is BACK, and it is now the fleet's defining risk.** Window 2026-10-19: **159 `completed` / 13 `failed`**, against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all carry the identical `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on `custom:omni-101`, in two clusters — **06:01** (one job) and **07:01 → 12:01** (seven more runs across seven jobs). The other **4 rows are derivative Mnemosyne Watchdog exits** (`96eb95ce12f2` at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the **second burst in four days** on the same node with the same blast radius, while `/api/health` read `200 {"status":"ok"}` throughout — so [[incidents/2026-10-16-omniroute-503-resource-pressure]] is back to **`status: active`** (it had been downgraded to `monitoring` 44 h earlier). **Fleet unchanged at 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` 7 → 12** (10 `error` + 2 `delivery_failed`); **unclosed incident rows 13** (12 `alerted` / 1 `detected`) — 4 opened and 2 closed on 10-19. Every `failure_streak` still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. **15 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **Current as of 2026-10-20 00:09 AWST *(superseded by the 2026-10-21 entry above)* — the `omni-101` 503 fault is BACK, and it is now the fleet's defining risk.** Window 2026-10-19: **159 `completed` / 13 `failed`**, against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all carry the identical `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on `custom:omni-101`, in two clusters — **06:01** (one job) and **07:01 → 12:01** (seven more runs across seven jobs). The other **4 rows are derivative Mnemosyne Watchdog exits** (`96eb95ce12f2` at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the **second burst in four days** on the same node with the same blast radius, while `/api/health` read `200 {"status":"ok"}` throughout — so [[incidents/2026-10-16-omniroute-503-resource-pressure]] is back to **`status: active`** (it had been downgraded to `monitoring` 44 h earlier). **Fleet unchanged at 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` 7 → 12** (10 `error` + 2 `delivery_failed`); **unclosed incident rows 13** (12 `alerted` / 1 `detected`) — 4 opened and 2 closed on 10-19. Every `failure_streak` still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. **15 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]] - **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]