--- title: Daily Marketing Job Radar — "All Targets Skipped by Pre-Dispatch Filters" (2026-10-21) type: incident status: unresolved created: 2026-10-22 updated: 2026-11-12 verified_on: 2026-11-12 review_after: 2026-11-26 confidence: high tags: [cron, router, provider, job-search, failure-mode] sources: [live-jobs-json, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-cron-output-file] --- # Daily Marketing Job Radar — Router Pre-Dispatch Filter Skip (2026-10-21) ## Headline The **Daily Marketing Job Radar** (`dca8482e4f76`) — the weekday **07:00** slot that scrapes SEEK / LinkedIn / Indeed / Jora, verifies against direct adverts, dedupes into the Notion job DB and delivers a Telegram summary — **failed on 2026-10-21 after 40 m 36 s** with a signature the wiki has not recorded before: ``` RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s ``` This is the **third distinct failure mode from the same provider pin** (`provider: custom:omni-101`, `model: smart-route`) after the connection-refused storm (2026-10-14/15) and the two `HTTP 503 … resource pressure` bursts (2026-10-16/17, 2026-10-19) documented in [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **The 503 string does not appear in any 2026-10-21 row.** ## Evidence (live, 2026-10-22 00:1x AWST) | Probe | Result | |---|---| | `jobs.json` `dca8482e4f76` | `last_status: error`, `failure_streak: 1`, `schedule: 0 7 * * 1-5`, `provider: custom:omni-101`, `model: smart-route`, `deliver: telegram:-1004321904721:1030` | | executions row `cd1d1f6a64ed427b9de13f31086ffb78` | `status: failed` — claimed **07:00:00.99**, started **07:00:12.40**, last progress **07:40:01.33**, finished **07:40:48.54** (**40 m 36 s**), `delivery_outcome: delivered` | | `cron_incidents` row `dca848_014affeb6954` | `state: alerted`, `failure_type: unknown`, `first_seen_at: 2026-10-21T07:40:42.61`, `alerted_at: 2026-10-21T07:40:48.50`, **`closed_at: null`**, `error` = the string above | | session `cron_dca8482e4f76_20261021_070022` | **82 messages / 47 tool calls**, ended 07:40:42, `end_reason: cron_incomplete_no_output` | | cron output file | `/home/hermes/.hermes/cron/output/dca8482e4f76/2026-10-21_07-40-42.md` — `# Cron Job: Daily Marketing Job Radar (FAILED)` | | the same hour's sibling | `85edf6772bca` ran 07:00:16 → 07:03:44 `cron_complete` (4 msgs) — the node was serving other jobs at that hour | | whole 2026-10-21 fleet | **171 `completed` / 1 `failed`** — this row is the only failure of the day | | 503 resource-pressure rows, 2026-10-21 | **zero** (last occurrence remains 2026-10-19 09:02) | ## What the error says, and what it does not - **Not a 503**, **not a connection error**, **not a repetition-loop cut** — those three strings are the entire recorded failure vocabulary of this fleet, and none of them matches. - The phrasing is the harness/router's own. The wiki has **no page documenting the pre-dispatch filter chain** that emits it, so the mechanism is unverified. What the string does assert: every dispatch target was filtered out **before** a model call was made, and the guard reset **19 m 20 s** into the run before the job was failed. The run itself spent **40 m 36 s** alive (07:00:12 → 07:40:48). - **Not caused by the 2026-10-21 `hermes update`.** That update ran 22:47 → 22:57 AWST — 15 h *after* this 07:00 failure. → [[systems/hermes-agent]] - **Not a delivery problem.** `delivery_outcome: delivered`: the failure notice reached topic `:1030`, so the slot was visibly red in Telegram rather than silently blank. ## Impact One weekday radar slot produced nothing: no new roles discovered, verified, de-duped or written to the Notion job DB, and no candidate summary delivered to topic `:1030`. The job recovers on its next weekday slot (2026-10-22 07:00) if the guard does not repeat. ## RECURRENCE — 2026-11-10, on a different job and a different pin (verified live 2026-11-12) The signature is **not unique to the Job Radar**, and it has now recurred on a second job — which changes the "one job's guard" hypothesis of the original open questions into a fleet-level failure mode. | Probe | Result | |---|---| | `executions.db` (retention floor **2026-11-06T05:09:30+08:00**, so this is an in-window read and **not** an all-time sweep) | exactly **one** row carries the string: **`cbfe434a3b5e` (Wiki Audit — Nightly), 2026-11-10, `status: failed`**, error `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters…` | | the original 2026-10-21 Job Radar row | **outside the retention floor**, so it cannot be re-read from the DB today; its evidence above stands on the 2026-10-22 capture. The per-job answer is unchanged in `jobs.json`: `dca8482e4f76` reads `ok` today | | `cron_incidents` row `cbfe43_ed01845310d8` | `failure_type: rate_limit`, `first_seen_at 2026-11-11T00:12`, **`closed_at: 2026-11-12T00:25` → `resolved`** (it was still `detected`/open at the 2026-11-11 audit) | | `dca8482e4f76` (Job Radar, the original subject) | **4 consecutive clean slots** — 2026-11-06, 11-09, 11-10, 11-11 all `completed`. The radar itself has **not** repeated the fault | | the recurring job's pin | `cbfe434a3b5e` runs `custom:omni-101` / **`free-stack`** — **not** the `smart-route` pin the original failure used | **What this adds to the original diagnosis:** the string fires on at least **two different models** and **two different jobs**, so open question 1 ("if it is a local filter chain, one filter is probably mis-configured for this job's toolset") now looks **less likely** than the router-side-guard reading — a guard that blanks *every* target. The **prefix differs** between the two occurrences (`HTTP 429` in 2026-11, bare `Service temporarily unavailable` in 2026-10), which is the same "error class changes, verdict holds" pattern already recorded for the 401/403 cliproxy case → [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. The 2026-11 instance was **credits-driven** (`credits 100%`, guard reset after 6 h 24 m) — a quota exhaustion, not a config fault, which also matches the fleet's paid-pin posture in [[systems/scheduled-tasks]] (this stocktake itself runs `deepseek-paid-smart`, the only paid pin in the fleet). Open question 3 (**give the radar a second provider pin — it has no failover**) therefore now applies to the whole free-tier cohort, not just the radar, and remains unapplied pending Anthony's go-ahead. ## Open questions / next actions (proposed, not applied — needs Anthony's go-ahead) 1. **Find which component emits this string.** If it is a router-side guard with a fixed reset window, a job-level retry after the reset would have recovered the slot; if it is a local filter chain, one filter is probably mis-configured for this job's toolset. 2. **Capture router-side telemetry at failure time** on the `omni-101` node — three distinct failure modes from one pin in eight days with no host telemetry captured for any of them. 3. **Give the radar a second provider pin.** It has no failover: one node's guard blanks the slot. ## Related - [[incidents/2026-10-16-omniroute-503-resource-pressure]] - [[systems/omniroute]] - [[systems/scheduled-tasks]] - [[systems/hermes-agent]]