🦉 wiki audit 2026-10-22: omni-101 pre-dispatch filter skip (3rd failure mode, new page), Hermes 0.22.3 update with snapshot skipped, Gitea backup backlog 76

- new: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- systems/hermes-agent.md: version -> 0.22.3 (5b147ab036), 22:47-22:57 update, snapshot stage skipped, advisories unactioned
- systems/scheduled-tasks.md: 2026-10-21 = 171 completed / 1 failed; non-clean stays 9 with 08eb86f02d37 out, dca8482e4f76 in
- systems/omniroute.md: one pin, three signatures in eight days
- incidents/2026-10-16-omniroute-503-resource-pressure.md: window update, stays monitoring
- incidents/2026-08-08-gitea-config-backup-push-dead.md: 2026-10-21 run, backlog 75 -> 76, message_id 8449
- infrastructure/backups.md: 76 commits; pre_update_backup:false now exercised
- current-state.md, index.md, log.md updated
This commit is contained in:
Tony0410
2026-10-22 00:16:39 +08:00
parent c1d3b59260
commit 3919ec8557
10 changed files with 225 additions and 26 deletions
+4 -4
View File
@@ -3,8 +3,8 @@ title: Hermes Agent
type: system
status: active
created: 2026-07-22
updated: 2026-10-11
verified_on: 2026-10-11
updated: 2026-10-22
verified_on: 2026-10-22
confidence: high
tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory]
@@ -16,10 +16,10 @@ sources: [live-hermes-version, live-config, live-skill-inventory]
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
## Current Status
- **Version:** v0.21.5+6169.gab86a20 (2026.9.24), upstream `ab86a20a` — verified live 2026-10-04. Same semver token as the earlier `v0.21.5+3837.gbfda74c` (`27062c34`), but a newer build: the install tree was rebuilt 2026-10-03 10:11 AWST.
- **Version:** `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`.
- **Install method:** git
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
- **Upstream status:** tree rebuilt 2026-10-03 from upstream `ab86a20a`; update availability not re-checked since.
- **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on.
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
- **Managed runtimes:** 3 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv` (was 10 — the 2026-10-03 rebuild pruned superseded environments and created two new ones at 09:59/10:00; the third survives from 2026-09-28). `hermes-runtime-deps-guard.sh` runs every 15 min via `mnemosyne-watchdog.py` and now asserts `dotenv` + `mnemosyne` in the installer python *and every managed venv*, **plus** `fastembed` + `FASTEMBED_CACHE_PATH` + the local model dir in the bootstrap-selected venv — see [[incidents/2026-09-28-hermes-managed-runtime-deps]] and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
+16 -2
View File
@@ -3,8 +3,8 @@ title: OmniRoute
type: system
status: active
created: 2026-07-22
updated: 2026-10-20
verified_on: 2026-10-20
updated: 2026-10-22
verified_on: 2026-10-22
confidence: high
tags: [routing, provider, model, inference]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -12,6 +12,20 @@ sources: [raw/configs/hermes-config-sanitized.txt]
# OmniRoute
## Recent Changes (2026-10-21)
- **Third failure mode on the same node, and it is not the 503 string.** `dca8482e4f76` (Daily Marketing Job
Radar) failed on 2026-10-21 at 07:40:48 after 40 m 36 s with
`RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after
19m 20s`. **Zero `HTTP 503 … resource pressure` rows exist for that day** (last: 2026-10-19 09:02) and the
sibling 07:00 job `85edf6772bca` completed normally, so the node was serving work at that hour.
→ [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
- **One pin, three signatures in eight days:** connection-refused storm (10-14/15), resource-pressure 503s
(10-16/17 and 10-19), pre-dispatch filter skip (10-21) — all on `custom:omni-101`, all with
`model: smart-route`. No job in the fleet has a second provider pin.
- **Router-side telemetry is still missing for every one of them** — `omni-101` host logs have not been
captured during a failure, which is why none of the three has a confirmed root cause.
## Recent Changes (2026-10-20)
- **Recurring `503 … resource pressure` load-shedding on `omniroute-1` — second burst in four days.** On
+5 -4
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-21
verified_on: 2026-10-21
review_after: 2026-10-28
updated: 2026-10-22
verified_on: 2026-10-22
review_after: 2026-10-29
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **⚠ Current as of 2026-10-21 00:0x AWST (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **Current as of 2026-10-22 00:1x AWST (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]].
- **⚠ Current as of 2026-10-21 00:0x AWST *(superseded by the 2026-10-22 entry above)* (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **Current as of 2026-10-20 00:09 AWST *(superseded by the 2026-10-21 entry above)* — the `omni-101` 503 fault is BACK, and it is now the fleet's defining risk.** Window 2026-10-19: **159 `completed` / 13 `failed`**, against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all carry the identical `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on `custom:omni-101`, in two clusters — **06:01** (one job) and **07:01 → 12:01** (seven more runs across seven jobs). The other **4 rows are derivative Mnemosyne Watchdog exits** (`96eb95ce12f2` at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the **second burst in four days** on the same node with the same blast radius, while `/api/health` read `200 {"status":"ok"}` throughout — so [[incidents/2026-10-16-omniroute-503-resource-pressure]] is back to **`status: active`** (it had been downgraded to `monitoring` 44 h earlier). **Fleet unchanged at 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` 7 → 12** (10 `error` + 2 `delivery_failed`); **unclosed incident rows 13** (12 `alerted` / 1 `detected`) — 4 opened and 2 closed on 10-19. Every `failure_streak` still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. **15 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]