Files
agent-estate-wiki/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
T
Tony0410 3919ec8557 🦉 wiki audit 2026-10-22: omni-101 pre-dispatch filter skip (3rd failure mode, new page), Hermes 0.22.3 update with snapshot skipped, Gitea backup backlog 76
- new: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- systems/hermes-agent.md: version -> 0.22.3 (5b147ab036), 22:47-22:57 update, snapshot stage skipped, advisories unactioned
- systems/scheduled-tasks.md: 2026-10-21 = 171 completed / 1 failed; non-clean stays 9 with 08eb86f02d37 out, dca8482e4f76 in
- systems/omniroute.md: one pin, three signatures in eight days
- incidents/2026-10-16-omniroute-503-resource-pressure.md: window update, stays monitoring
- incidents/2026-08-08-gitea-config-backup-push-dead.md: 2026-10-21 run, backlog 75 -> 76, message_id 8449
- infrastructure/backups.md: 76 commits; pre_update_backup:false now exercised
- current-state.md, index.md, log.md updated
2026-10-22 00:16:39 +08:00

4.6 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Daily Marketing Job Radar — "All Targets Skipped by Pre-Dispatch Filters" (2026-10-21) incident unresolved 2026-10-22 2026-10-22 2026-10-22 2026-10-29 high
cron
router
provider
job-search
failure-mode
live-jobs-json
live-sqlite-cron-executions
live-sqlite-cron-incidents
live-sqlite-sessions
live-cron-output-file

Daily Marketing Job Radar — Router Pre-Dispatch Filter Skip (2026-10-21)

Headline

The Daily Marketing Job Radar (dca8482e4f76) — the weekday 07:00 slot that scrapes SEEK / LinkedIn / Indeed / Jora, verifies against direct adverts, dedupes into the Notion job DB and delivers a Telegram summary — failed on 2026-10-21 after 40 m 36 s with a signature the wiki has not recorded before:

RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s

This is the third distinct failure mode from the same provider pin (provider: custom:omni-101, model: smart-route) after the connection-refused storm (2026-10-14/15) and the two HTTP 503 … resource pressure bursts (2026-10-16/17, 2026-10-19) documented in incidents/2026-10-16-omniroute-503-resource-pressure. The 503 string does not appear in any 2026-10-21 row.

Evidence (live, 2026-10-22 00:1x AWST)

Probe Result
jobs.json dca8482e4f76 last_status: error, failure_streak: 1, schedule: 0 7 * * 1-5, provider: custom:omni-101, model: smart-route, deliver: telegram:-1004321904721:1030
executions row cd1d1f6a64ed427b9de13f31086ffb78 status: failed — claimed 07:00:00.99, started 07:00:12.40, last progress 07:40:01.33, finished 07:40:48.54 (40 m 36 s), delivery_outcome: delivered
cron_incidents row dca848_014affeb6954 state: alerted, failure_type: unknown, first_seen_at: 2026-10-21T07:40:42.61, alerted_at: 2026-10-21T07:40:48.50, closed_at: null, error = the string above
session cron_dca8482e4f76_20261021_070022 82 messages / 47 tool calls, ended 07:40:42, end_reason: cron_incomplete_no_output
cron output file /home/hermes/.hermes/cron/output/dca8482e4f76/2026-10-21_07-40-42.md — # Cron Job: Daily Marketing Job Radar (FAILED)
the same hour's sibling 85edf6772bca ran 07:00:16 → 07:03:44 cron_complete (4 msgs) — the node was serving other jobs at that hour
whole 2026-10-21 fleet 171 completed / 1 failed — this row is the only failure of the day
503 resource-pressure rows, 2026-10-21 zero (last occurrence remains 2026-10-19 09:02)

What the error says, and what it does not

  • Not a 503, not a connection error, not a repetition-loop cut — those three strings are the entire recorded failure vocabulary of this fleet, and none of them matches.
  • The phrasing is the harness/router's own. The wiki has no page documenting the pre-dispatch filter chain that emits it, so the mechanism is unverified. What the string does assert: every dispatch target was filtered out before a model call was made, and the guard reset 19 m 20 s into the run before the job was failed. The run itself spent 40 m 36 s alive (07:00:12 → 07:40:48).
  • Not caused by the 2026-10-21 hermes update. That update ran 22:47 → 22:57 AWST — 15 h after this 07:00 failure. → systems/hermes-agent
  • Not a delivery problem. delivery_outcome: delivered: the failure notice reached topic :1030, so the slot was visibly red in Telegram rather than silently blank.

Impact

One weekday radar slot produced nothing: no new roles discovered, verified, de-duped or written to the Notion job DB, and no candidate summary delivered to topic :1030. The job recovers on its next weekday slot (2026-10-22 07:00) if the guard does not repeat.

Open questions / next actions (proposed, not applied — needs Anthony's go-ahead)

  1. Find which component emits this string. If it is a router-side guard with a fixed reset window, a job-level retry after the reset would have recovered the slot; if it is a local filter chain, one filter is probably mis-configured for this job's toolset.
  2. Capture router-side telemetry at failure time on the omni-101 node — three distinct failure modes from one pin in eight days with no host telemetry captured for any of them.
  3. Give the radar a second provider pin. It has no failover: one node's guard blanks the slot.