audit: 2026-10-28 nightly — radar budget overrun now a recorded failure (new incident page); radar split into two cron jobs; tailscale peer table re-verified; TTS provider corrected to the Mac mini speech server
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76 (Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and cover letters never ran (Notion DB held 451 entries, 0 added). Incident row dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified). Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5, deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5, topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3). Also: the two incident rows the 2026-10-26 planned restart raised auto-closed 00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of 108) — committed here and index.md corrected; tailscale.md peer table rebuilt from tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained the voice pipeline section. New: incidents/2026-10-27-job-radar-budget-exhausted.md Nothing removed or archived. No secrets.
This commit is contained in:
+49
-8
@@ -3,10 +3,10 @@ title: Daily Marketing Job Radar Operations and Recovery
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-27
|
||||
updated: 2026-10-27
|
||||
updated: 2026-10-28
|
||||
review_after: 2026-11-10
|
||||
verified_on: 2026-10-27
|
||||
version_applies_to: Hermes Agent cron dca8482e4f76
|
||||
verified_on: 2026-10-28
|
||||
version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2)
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, dashboard, runbook]
|
||||
sources: [live-cron-output, live-notion-api, live-dashboard-api, hermes-docs]
|
||||
@@ -22,11 +22,14 @@ Run a verified Western Australian marketing-job search, deduplicate against the
|
||||
|
||||
| Setting | Value |
|
||||
|---|---|
|
||||
| Cron job | `dca8482e4f76` — Daily Marketing Job Radar |
|
||||
| Schedule | `0 7 * * 1-5` (07:00 AWST weekdays) |
|
||||
| Delivery | Telegram group `-1004321904721`, topic `1030` |
|
||||
| Cron job — stage 1 | `dca8482e4f76` — Daily Marketing Job Radar - Discovery (Discovery → Verify → Dedup → Score; no Notion write, no cover letters) |
|
||||
| Schedule — stage 1 | `0 7 * * 1-5` (07:00 AWST weekdays) |
|
||||
| Delivery — stage 1 | `local` (dashboard/CLI); failures → Telegram group `-1004321904721`, topic `1030` |
|
||||
| Cron job — stage 2 | `5fc292ff0f3b` — Daily Marketing Job Radar - Write and Letters (created 2026-10-27 17:34:15) |
|
||||
| Schedule — stage 2 | `30 11,14 * * 1-5` (11:30 and 14:30 AWST weekdays) |
|
||||
| Delivery — stage 2 | Telegram group `-1004321904721`, topic `1030` |
|
||||
| Skill | `job-search-automation` |
|
||||
| Primary model | `custom:omni-101` / `gpt-5.6-terra` (upstream `command-code`) — ⚠ 403 `MODEL_NOT_IN_PLAN` since 2026-09-26 |
|
||||
| Primary model (both stages) | `custom:omni-101` / `smart-route` (live 2026-10-28) — the 2026-09-26 `gpt-5.6-terra` 403 is historical |
|
||||
| First fallback | `opencode-go / mimo-v2.5` |
|
||||
| Dashboard | `https://hermes.kangaroo-eel.ts.net/jobs/` |
|
||||
| Dashboard backend | `127.0.0.1:9099` |
|
||||
@@ -279,6 +282,17 @@ produced a visibly different document — Anthony rejected exactly that mismatch
|
||||
- All five artefacts (`.txt .json .html .docx .pdf`) must still exist and be nonempty, and the HTML must
|
||||
contain no unresolved `{{...}}` tokens — the HTML is now the source of truth the PDF follows.
|
||||
|
||||
### 2026-10-27 07:00 slot — the same overrun, now recorded as `failed` (and the trigger for the split)
|
||||
|
||||
The 07:00 slot ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**) and closed **`failed`** on delivery
|
||||
`delivered`. The report it sent opens *"Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call
|
||||
budget exhausted before Notion write stage)"*: discovery, verification, dedup and classification finished;
|
||||
**Notion writes, cover letters and the final audit did not run**. Notion DB
|
||||
`0f90ba2b-8b10-4d02-a62c-6f692d5b1168` held **451 entries** with **0** added for 2026-10-27 at run time.
|
||||
Incident row `dca848_e93a538ade6b` opened 11:03:53 (`alerted` 11:03:59) and was **still open** at the
|
||||
2026-10-28 audit. The split above is the response — see
|
||||
[[incidents/2026-10-27-job-radar-budget-exhausted]] for the full ledger.
|
||||
|
||||
### 2026-10-26 07:00 slot — `RUN INCOMPLETE`, and why the backlog was finished by hand
|
||||
|
||||
The 07:00 run started on schedule and closed **`completed` at 09:10**, but the report it delivered read
|
||||
@@ -287,11 +301,38 @@ the **Notion writes and the entire cover-letter stage never executed**. The same
|
||||
interactively (rows written for the outstanding strong matches, cover letters generated and verified).
|
||||
|
||||
- **A `completed` execution row is not a completed radar run.** Read the delivered text for the
|
||||
`RUN INCOMPLETE` marker before treating a slot as done.
|
||||
`RUN INCOMPLETE` marker before treating a slot as done. A *failed* row with a delivered report is the honest
|
||||
version of the same fault — the 2026-10-27 slot is the first time it reached the incident ledger.
|
||||
- Budget the browser-heavy stages (one tab in flight, close before opening the next — see the
|
||||
2026-10-12 hazard below) and keep headroom for the Notion + cover-letter writes, which sit last in the
|
||||
pipeline.
|
||||
|
||||
## Two-stage split (changed 2026-10-27)
|
||||
|
||||
The single-job radar was split after two consecutive budget overruns (2026-10-26 `completed` but
|
||||
`RUN INCOMPLETE`, 2026-10-27 closed `failed` — see
|
||||
[[incidents/2026-10-27-job-radar-budget-exhausted]]). Discovery is the stage that eats the tool budget, so it
|
||||
now owns a job of its own and the writes own a second one.
|
||||
|
||||
| Stage | Job id | Schedule | Writes | Delivers |
|
||||
|---|---|---|---|---|
|
||||
| 1 — Discovery | `dca8482e4f76` (existing id) | `0 7 * * 1-5` | a handoff file only | `local`; failures → topic 1030 |
|
||||
| 2 — Write and Letters | `5fc292ff0f3b` (new) | `30 11,14 * * 1-5` | Notion entries, cover letters, final audit, report | Telegram topic 1030 |
|
||||
|
||||
- **Stage 1** does `DISCOVERY → VERIFY → DEDUP → SCORE`, writes the handoff file, and triggers stage 2. It must
|
||||
**not** write to Notion and must **not** generate cover letters.
|
||||
- **Stage 2** lands the handoff: Notion entries, cover letters, the missing-letter audit and the report. Its
|
||||
prompt is not a copy of stage 1's — it reports what it wrote, not what it found.
|
||||
- Stage 1 keeps the `custom:omni-101` / `smart-route` pin. Both jobs still inherit the profile-level
|
||||
`fallback_providers` chain; Hermes stores no per-job fallback list.
|
||||
- First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both
|
||||
`completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle had not yet run at the
|
||||
2026-10-28 audit.**
|
||||
|
||||
**Recovery if stage 2 lands nothing:** check that stage 1 actually wrote its handoff file before running stage 2
|
||||
by hand (`hermes cron run 5fc292ff0f3b`). A stage-2 run with no handoff is a no-op, not a failed search — do not
|
||||
re-run stage 1's discovery just to satisfy it.
|
||||
|
||||
## Browser-layer hazard inside the run (2026-10-12)
|
||||
|
||||
This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove
|
||||
|
||||
Reference in New Issue
Block a user