🦉 wiki audit 2026-10-22: omni-101 pre-dispatch filter skip (3rd failure mode, new page), Hermes 0.22.3 update with snapshot skipped, Gitea backup backlog 76

- new: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- systems/hermes-agent.md: version -> 0.22.3 (5b147ab036), 22:47-22:57 update, snapshot stage skipped, advisories unactioned
- systems/scheduled-tasks.md: 2026-10-21 = 171 completed / 1 failed; non-clean stays 9 with 08eb86f02d37 out, dca8482e4f76 in
- systems/omniroute.md: one pin, three signatures in eight days
- incidents/2026-10-16-omniroute-503-resource-pressure.md: window update, stays monitoring
- incidents/2026-08-08-gitea-config-backup-push-dead.md: 2026-10-21 run, backlog 75 -> 76, message_id 8449
- infrastructure/backups.md: 76 commits; pre_update_backup:false now exercised
- current-state.md, index.md, log.md updated
This commit is contained in:
Tony0410
2026-10-22 00:16:39 +08:00
parent c1d3b59260
commit 3919ec8557
10 changed files with 225 additions and 26 deletions
@@ -3,9 +3,9 @@ title: Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 Whi
type: incident
status: active
created: 2026-10-21
updated: 2026-10-21
verified_on: 2026-10-21
review_after: 2026-10-28
updated: 2026-10-22
verified_on: 2026-10-22
review_after: 2026-10-29
confidence: high
tags: [backup, gitea, cron, git, green-vs-actual]
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki]
@@ -83,6 +83,21 @@ tail -20 ~/.hermes/config-backup/backup.log
tail -3 ~/.hermes/config-backup/.backup_history
```
## Update 2026-10-21 (nightly run — the backlog grew again)
| Probe | Result |
|---|---|
| 2026-10-21 11:00 run | commit **`a8486a20`** created (`🦝 nightly backup: 2026-10-21 11:00:14 AWST`) — **push failed again**, `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address` |
| `git rev-list --count origin/main..main` | **76** (was 75) — the backlog grows by exactly one commit per night |
| Telegram notice | delivered: `🚨 BACKUP FAILED — 2026-10-21 11:00:14 AWST`, **`message_id 8449`**; the AgentMail step failed again |
| GitHub fallback | `fatal: 'github' does not appear to be a git repository`; its "pushing clean branch" recovery (`github-clean-1792551629`) also failed and was deleted |
| `.backup_history` | `2026-10-19 … FAIL`, `2026-10-20 … FAIL`, **`2026-10-21 … FAIL`** |
| cron row `e27da6c34bbb` | still `last_status: ok` — the green-vs-actual gap is now **74 days** wide |
**Compounding factor found the same night:** the `hermes update` that ran at **2026-10-21 22:47 AWST** recorded
its own `snapshot` stage as **`skipped`** (`disabled by updates.pre_update_backup (mode: off)`), so for that
change as well the *only* off-box copy path was this dead push → [[infrastructure/backups]].
## Related
- [[infrastructure/backups]]
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
type: incident
status: monitoring
created: 2026-10-18
updated: 2026-10-21
verified_on: 2026-10-21
review_after: 2026-10-28
updated: 2026-10-22
verified_on: 2026-10-22
review_after: 2026-10-29
confidence: high
tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
@@ -220,6 +220,25 @@ clean running, not a fix).
`nous` (`https://inference-api.nousresearch.com/v1`), `model.default: smart-route` →
`meituan/longcat-2.5-preview:free`. Pinned jobs are unaffected; unpinned jobs are. → [[systems/current-profile]]
## Update 2026-10-21 (window 2026-10-20 → 10-21)
- **A third failure mode from the same pin — and it is not 503.** The only failed cron row on **2026-10-21**
was `dca8482e4f76` (Daily Marketing Job Radar, `custom:omni-101` / `smart-route`), which died after
**40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch
filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]].
- **The resource-pressure string did not appear at all on 2026-10-21** (zero ledger rows; last occurrence
remains 2026-10-19 09:02), so the second burst stays closed and this page stays `status: monitoring`.
- **One row closed, one opened.** `08eb86_472c7b609c80` (the Gmail Action Digest 503s, first seen 10-16 08:33)
**resolved 2026-10-21 08:37:21** — the job's own 08:30 slot ran clean that morning (its 10-20 run was the
orphan; no incident row was ever raised for it), which also cleared `08eb86f02d37` from the non-clean table.
`dca848_014affeb6954` opened 07:40:42 and is **still `alerted`**. Unclosed rows **stay at 9**
(8 `alerted` / 1 `detected`).
- **Windows:** 2026-10-20 settled at **169 `completed` / 1 `unknown`** (the gateway-orphaned run; the 10-21
audit's 168 / 0 was a mid-flight snapshot taken before the 23:59 slot landed). **2026-10-21: 171 `completed`
/ 1 `failed`** (+ this audit).
- **Still no failover.** Every job that has failed on this node since 2026-10-14 still reads
`provider: custom:omni-101` with `model: smart-route` or `free-stack`, and none has a second pin.
## Related
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node.
@@ -0,0 +1,79 @@
---
title: Daily Marketing Job Radar — "All Targets Skipped by Pre-Dispatch Filters" (2026-10-21)
type: incident
status: unresolved
created: 2026-10-22
updated: 2026-10-22
verified_on: 2026-10-22
review_after: 2026-10-29
confidence: high
tags: [cron, router, provider, job-search, failure-mode]
sources: [live-jobs-json, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-cron-output-file]
---
# Daily Marketing Job Radar — Router Pre-Dispatch Filter Skip (2026-10-21)
## Headline
The **Daily Marketing Job Radar** (`dca8482e4f76`) — the weekday **07:00** slot that scrapes
SEEK / LinkedIn / Indeed / Jora, verifies against direct adverts, dedupes into the Notion job DB and
delivers a Telegram summary — **failed on 2026-10-21 after 40 m 36 s** with a signature the wiki has not
recorded before:
```
RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s
```
This is the **third distinct failure mode from the same provider pin** (`provider: custom:omni-101`,
`model: smart-route`) after the connection-refused storm (2026-10-14/15) and the two
`HTTP 503 … resource pressure` bursts (2026-10-16/17, 2026-10-19) documented in
[[incidents/2026-10-16-omniroute-503-resource-pressure]]. **The 503 string does not appear in any
2026-10-21 row.**
## Evidence (live, 2026-10-22 00:1x AWST)
| Probe | Result |
|---|---|
| `jobs.json` `dca8482e4f76` | `last_status: error`, `failure_streak: 1`, `schedule: 0 7 * * 1-5`, `provider: custom:omni-101`, `model: smart-route`, `deliver: telegram:-1004321904721:1030` |
| executions row `cd1d1f6a64ed427b9de13f31086ffb78` | `status: failed` — claimed **07:00:00.99**, started **07:00:12.40**, last progress **07:40:01.33**, finished **07:40:48.54** (**40 m 36 s**), `delivery_outcome: delivered` |
| `cron_incidents` row `dca848_014affeb6954` | `state: alerted`, `failure_type: unknown`, `first_seen_at: 2026-10-21T07:40:42.61`, `alerted_at: 2026-10-21T07:40:48.50`, **`closed_at: null`**, `error` = the string above |
| session `cron_dca8482e4f76_20261021_070022` | **82 messages / 47 tool calls**, ended 07:40:42, `end_reason: cron_incomplete_no_output` |
| cron output file | `/home/hermes/.hermes/cron/output/dca8482e4f76/2026-10-21_07-40-42.md` — `# Cron Job: Daily Marketing Job Radar (FAILED)` |
| the same hour's sibling | `85edf6772bca` ran 07:00:16 → 07:03:44 `cron_complete` (4 msgs) — the node was serving other jobs at that hour |
| whole 2026-10-21 fleet | **171 `completed` / 1 `failed`** — this row is the only failure of the day |
| 503 resource-pressure rows, 2026-10-21 | **zero** (last occurrence remains 2026-10-19 09:02) |
## What the error says, and what it does not
- **Not a 503**, **not a connection error**, **not a repetition-loop cut** — those three strings are the
entire recorded failure vocabulary of this fleet, and none of them matches.
- The phrasing is the harness/router's own. The wiki has **no page documenting the pre-dispatch filter
chain** that emits it, so the mechanism is unverified. What the string does assert: every dispatch target
was filtered out **before** a model call was made, and the guard reset **19 m 20 s** into the run before the
job was failed. The run itself spent **40 m 36 s** alive (07:00:12 → 07:40:48).
- **Not caused by the 2026-10-21 `hermes update`.** That update ran 22:47 → 22:57 AWST — 15 h *after* this
07:00 failure. → [[systems/hermes-agent]]
- **Not a delivery problem.** `delivery_outcome: delivered`: the failure notice reached topic `:1030`, so the
slot was visibly red in Telegram rather than silently blank.
## Impact
One weekday radar slot produced nothing: no new roles discovered, verified, de-duped or written to the Notion
job DB, and no candidate summary delivered to topic `:1030`. The job recovers on its next weekday slot
(2026-10-22 07:00) if the guard does not repeat.
## Open questions / next actions (proposed, not applied — needs Anthony's go-ahead)
1. **Find which component emits this string.** If it is a router-side guard with a fixed reset window, a
job-level retry after the reset would have recovered the slot; if it is a local filter chain, one filter is
probably mis-configured for this job's toolset.
2. **Capture router-side telemetry at failure time** on the `omni-101` node — three distinct failure modes
from one pin in eight days with no host telemetry captured for any of them.
3. **Give the radar a second provider pin.** It has no failover: one node's guard blanks the slot.
## Related
- [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- [[systems/omniroute]]
- [[systems/scheduled-tasks]]
- [[systems/hermes-agent]]