Files
agent-estate-wiki/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
Hermes ab3652aa20 Stocktake 2026-11-12: tailscale live-vs-page drift corrected; sftpgo incident re-verified; pre-dispatch-filter signature found recurring on a second job/pin
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json.
  mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping
  (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online.
  omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout).
  9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge
  (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table
  unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed.
- incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer
  Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared.
- incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on
  cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved
  2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded.
- Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits),
  work-outlook (himalaya still login not supported), omniroute-free-deepseek
  (good-free/free-stack both HTTP 200, non-deepseek resolution).
- Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md.
- wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
2026-11-12 03:19:12 +08:00

7.2 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Daily Marketing Job Radar — "All Targets Skipped by Pre-Dispatch Filters" (2026-10-21) incident unresolved 2026-10-22 2026-11-12 2026-11-12 2026-11-26 high
cron
router
provider
job-search
failure-mode
live-jobs-json
live-sqlite-cron-executions
live-sqlite-cron-incidents
live-sqlite-sessions
live-cron-output-file

Daily Marketing Job Radar — Router Pre-Dispatch Filter Skip (2026-10-21)

Headline

The Daily Marketing Job Radar (dca8482e4f76) — the weekday 07:00 slot that scrapes SEEK / LinkedIn / Indeed / Jora, verifies against direct adverts, dedupes into the Notion job DB and delivers a Telegram summary — failed on 2026-10-21 after 40 m 36 s with a signature the wiki has not recorded before:

RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s

This is the third distinct failure mode from the same provider pin (provider: custom:omni-101, model: smart-route) after the connection-refused storm (2026-10-14/15) and the two HTTP 503 … resource pressure bursts (2026-10-16/17, 2026-10-19) documented in incidents/2026-10-16-omniroute-503-resource-pressure. The 503 string does not appear in any 2026-10-21 row.

Evidence (live, 2026-10-22 00:1x AWST)

Probe Result
jobs.json dca8482e4f76 last_status: error, failure_streak: 1, schedule: 0 7 * * 1-5, provider: custom:omni-101, model: smart-route, deliver: telegram:-1004321904721:1030
executions row cd1d1f6a64ed427b9de13f31086ffb78 status: failed — claimed 07:00:00.99, started 07:00:12.40, last progress 07:40:01.33, finished 07:40:48.54 (40 m 36 s), delivery_outcome: delivered
cron_incidents row dca848_014affeb6954 state: alerted, failure_type: unknown, first_seen_at: 2026-10-21T07:40:42.61, alerted_at: 2026-10-21T07:40:48.50, closed_at: null, error = the string above
session cron_dca8482e4f76_20261021_070022 82 messages / 47 tool calls, ended 07:40:42, end_reason: cron_incomplete_no_output
cron output file /home/hermes/.hermes/cron/output/dca8482e4f76/2026-10-21_07-40-42.md — # Cron Job: Daily Marketing Job Radar (FAILED)
the same hour's sibling 85edf6772bca ran 07:00:16 → 07:03:44 cron_complete (4 msgs) — the node was serving other jobs at that hour
whole 2026-10-21 fleet 171 completed / 1 failed — this row is the only failure of the day
503 resource-pressure rows, 2026-10-21 zero (last occurrence remains 2026-10-19 09:02)

What the error says, and what it does not

  • Not a 503, not a connection error, not a repetition-loop cut — those three strings are the entire recorded failure vocabulary of this fleet, and none of them matches.
  • The phrasing is the harness/router's own. The wiki has no page documenting the pre-dispatch filter chain that emits it, so the mechanism is unverified. What the string does assert: every dispatch target was filtered out before a model call was made, and the guard reset 19 m 20 s into the run before the job was failed. The run itself spent 40 m 36 s alive (07:00:12 → 07:40:48).
  • Not caused by the 2026-10-21 hermes update. That update ran 22:47 → 22:57 AWST — 15 h after this 07:00 failure. → systems/hermes-agent
  • Not a delivery problem. delivery_outcome: delivered: the failure notice reached topic :1030, so the slot was visibly red in Telegram rather than silently blank.

Impact

One weekday radar slot produced nothing: no new roles discovered, verified, de-duped or written to the Notion job DB, and no candidate summary delivered to topic :1030. The job recovers on its next weekday slot (2026-10-22 07:00) if the guard does not repeat.

RECURRENCE — 2026-11-10, on a different job and a different pin (verified live 2026-11-12)

The signature is not unique to the Job Radar, and it has now recurred on a second job — which changes the "one job's guard" hypothesis of the original open questions into a fleet-level failure mode.

Probe Result
executions.db (retention floor 2026-11-06T05:09:30+08:00, so this is an in-window read and not an all-time sweep) exactly one row carries the string: cbfe434a3b5e (Wiki Audit — Nightly), 2026-11-10, status: failed, error RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters…
the original 2026-10-21 Job Radar row outside the retention floor, so it cannot be re-read from the DB today; its evidence above stands on the 2026-10-22 capture. The per-job answer is unchanged in jobs.json: dca8482e4f76 reads ok today
cron_incidents row cbfe43_ed01845310d8 failure_type: rate_limit, first_seen_at 2026-11-11T00:12, closed_at: 2026-11-12T00:25 → resolved (it was still detected/open at the 2026-11-11 audit)
dca8482e4f76 (Job Radar, the original subject) 4 consecutive clean slots — 2026-11-06, 11-09, 11-10, 11-11 all completed. The radar itself has not repeated the fault
the recurring job's pin cbfe434a3b5e runs custom:omni-101 / free-stack — not the smart-route pin the original failure used

What this adds to the original diagnosis: the string fires on at least two different models and two different jobs, so open question 1 ("if it is a local filter chain, one filter is probably mis-configured for this job's toolset") now looks less likely than the router-side-guard reading — a guard that blanks every target. The prefix differs between the two occurrences (HTTP 429 in 2026-11, bare Service temporarily unavailable in 2026-10), which is the same "error class changes, verdict holds" pattern already recorded for the 401/403 cliproxy case → incidents/2026-10-01-cliproxy-connection-auth-expired. The 2026-11 instance was credits-driven (credits 100%, guard reset after 6 h 24 m) — a quota exhaustion, not a config fault, which also matches the fleet's paid-pin posture in systems/scheduled-tasks (this stocktake itself runs deepseek-paid-smart, the only paid pin in the fleet). Open question 3 (give the radar a second provider pin — it has no failover) therefore now applies to the whole free-tier cohort, not just the radar, and remains unapplied pending Anthony's go-ahead.

Open questions / next actions (proposed, not applied — needs Anthony's go-ahead)

  1. Find which component emits this string. If it is a router-side guard with a fixed reset window, a job-level retry after the reset would have recovered the slot; if it is a local filter chain, one filter is probably mis-configured for this job's toolset.
  2. Capture router-side telemetry at failure time on the omni-101 node — three distinct failure modes from one pin in eight days with no host telemetry captured for any of them.
  3. Give the radar a second provider pin. It has no failover: one node's guard blanks the slot.