--- title: Daily Marketing Job Radar Operations and Recovery type: runbook status: active created: 2026-07-27 updated: 2026-10-30 review_after: 2026-11-13 verified_on: 2026-10-30 version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2) confidence: high tags: [automation, cron, jobs, notion, dashboard, runbook] sources: [live-cron-output, live-notion-api, live-dashboard-api, hermes-docs] --- # Daily Marketing Job Radar Operations and Recovery ## Purpose Run a verified Western Australian marketing-job search, deduplicate against the complete Notion board, create only fresh direct adverts, score new roles against Anthony's resume, automatically produce cover letters for strong matches, and expose the results through the Jobs Radar dashboard. ## Production configuration | Setting | Value | |---|---| | Cron job — stage 1 | `dca8482e4f76` — Daily Marketing Job Radar - Discovery (Discovery → Verify → Dedup → Score; no Notion write, no cover letters) | | Schedule — stage 1 | `0 7 * * 1-5` (07:00 AWST weekdays) | | Delivery — stage 1 | `local` (dashboard/CLI); failures → Telegram group `-1004321904721`, topic `1030` | | Cron job — stage 2 | `5fc292ff0f3b` — Daily Marketing Job Radar - Write and Letters (created 2026-10-27 17:34:15) | | Schedule — stage 2 | `30 11,14 * * 1-5` (11:30 and 14:30 AWST weekdays) | | Delivery — stage 2 | Telegram group `-1004321904721`, topic `1030` | | Skill | `job-search-automation` | | Primary model (both stages) | `custom:omni-101` / `smart-route` (live 2026-10-28) — the 2026-09-26 `gpt-5.6-terra` 403 is historical | | First fallback | `opencode-go / mimo-v2.5` | | Dashboard | `https://hermes.kangaroo-eel.ts.net/jobs/` | | Dashboard backend | `127.0.0.1:9099` | | Notion database | `0f90ba2b-8b10-4d02-a62c-6f692d5b1168` | Cron jobs inherit the profile-level `fallback_providers` chain; Hermes does not currently store a separate fallback chain in each cron record. MiMo v2.5 is first in the global chain, so Terra fails over to it before the older fallbacks. ## Cover letters are non-negotiable A successful radar run includes the cover-letter stage. - Audit **every Strong Match added today**, including pages created by an interrupted earlier run. - A valid result requires a canonical rendered `.html` URL in Notion, not plain text and not the dashboard root. - Production template files: - `/home/hermes/.hermes/dashboard/templates/cover-letter.html` - `/home/hermes/.hermes/dashboard/templates/Anthony_Martin_Cover_Letter_Template.docx` - The old upload-cache location is not authoritative and may be evicted. - Required nonempty artifacts: `.txt`, `.json`, `.html`, `.docx`, `.pdf`. - The HTML must contain zero unresolved `{{...}}` tokens and return HTTP 200. - Retry generation once when an artifact, Notion link or HTTP check fails. - A clean run requires `MISSING_COVER_LETTER_COUNT=0`. Any remaining missing letter must be reported prominently with its Notion page ID; it must never be silently omitted. ### Live audit — 27 July 2026 - Same-day strong matches: **1** - Verified canonical HTML cover letters: **1** - Missing cover letters: **0** - The Market Creations Agency strong match has all five artifact formats, zero unresolved template tokens and a live HTTP 200 HTML link. ## Required production pipeline 1. Obtain the live AWST date and calculate a seven-day freshness cutoff. 2. Search SEEK, LinkedIn, Indeed, WA Government Jobs and direct employer sites. 3. Open and verify every individual advert. Search snippets are discovery evidence only. 4. Reject stale, expired, cached, interstate, non-marketing, salary-guide, search/category and aggregator pages. 5. Reject **all Pacific Energy roles**. Anthony was made redundant there in July 2026; old Pacific Energy adverts must never be presented as opportunities. 6. Accept only direct individual advert URLs: - SEEK `/job/` - LinkedIn `/jobs/view/...` - Indeed `/viewjob?jk=` - direct employer career pages - individual WA Government role pages 7. A WA Government slug page is valid without `AdvertID` in its URL when the page proves the exact role and agency, WA location, unique job/pool reference, future closing date and active **Apply Now** function. 8. Query the complete Notion database with live pagination. Deduplicate by exact URL, normalized company plus role, and materially similar company-role combinations. 9. Create only verified nonduplicates and require returned Notion page IDs. 10. Refresh `POST http://127.0.0.1:9099/api/refresh`, then confirm every page and direct URL through `/api/jobs`. 11. Score newly created roles against `/home/hermes/.hermes/radar-ref/resumes/current_resume_from_website.txt` using the documented 100-point rubric. Strong Match starts at 70. 12. For each new strong match, run: ```bash /home/hermes/.hermes/hermes-agent/venv/bin/python \ /home/hermes/.hermes/scripts/regenerate_cover_letter.py \ \ "Generate the initial tailored cover letter using the verified job description. Preserve Anthony's factual employment history and use the canonical teal Montserrat template." ``` 13. Verify nonempty `.txt`, `.json`, `.html`, `.docx` and `.pdf` artifacts; no unresolved `{{...}}` tokens; and a live `.html` Notion Cover Letter URL. 14. Report only facts produced by tools in that run. Never estimate counts or fabricate Notion writes. ## Dashboard behaviour Source: `/home/hermes/.hermes/dashboard/index.html` Backend: `/home/hermes/.hermes/scripts/job_dashboard_server.py` ### Layout (top to bottom) 1. **Hero header** — RADAR logo, name, dateline 2. **Market Intel panel** — expandable salary/location/source stats from `/api/market-intel` 3. **KPI strip** — clickable filter cards: Total Jobs, Backlog, Applied, Phone Screen, Interviews, Offers, Closing Soon 4. **Filter bar** — text search, view toggles (table/cards), Refresh button 5. **Stats strip** — This Week, Stale 14d+, Strong Match, Past Close, Applied Rate 6. **Pipeline Funnel** — clickable per-status breakdown with bar chart 7. **Job table/cards** — sortable columns, inline status pills, cover letter links, edit actions ### Column sorting Click any column header (Status, Company, Role, Match, Salary, Age, Closes, Updated) to sort ascending; click again for descending. Arrow indicator shows current direction. ### Status management - **Bulk update**: select jobs via checkboxes → pick status from dropdown → Apply. Sends `{ids, updates: {status}}` to `POST /api/jobs/bulk-update`. - **Single edit**: click ✎ → modal with Status, Priority, Role, Salary, Closing Date, URL, Notes, Cover Letter, Attachments. Saves to Notion via `POST /api/jobs//update`. - **Funnel/status filters**: clicking a status in the funnel or KPI strip filters the table to that status. ### Rich text notes - Write/Preview editor in the edit modal (monospace textarea + markdown preview) - Supports: bold `**`, italic `*`, links `[text](url)`, bullet lists `- item` - Syncs to Notion `Notes / JD Summary` rich_text property on every save ### Document attachments - Attach any URL (documents, links, files) to a job entry - Stored as **bookmark blocks** on the Notion page body (not a property) - Lazy-loaded when edit modal opens: `GET /api/jobs//attachments` - Add: `POST /api/jobs//attachments` with `{name, url}` - Delete: `POST /api/jobs//attachments/delete` with `{block_id}` - Visible in Notion when you open the page ### Cover letter integration - Regenerate button (🔄) on each row calls `POST /api/jobs//cover-letter/regenerate` - CL URL stored in Notion `Cover Letter` property and rendered as clickable link in table - All 5 artifact formats: TXT, JSON, HTML, DOCX, PDF ### Market Intel - Expandable panel at top of page - Data from `GET /api/market-intel`: total tracked, salary stats (avg/median/min/max), counts - Lazy-loaded on expand; also refreshes on ↺ Refresh ### API endpoints | Endpoint | Method | Purpose | |---|---|---| | `/api/jobs` | GET | All jobs (cached, paginated from Notion) | | `/api/stats` | GET | Aggregate statistics | | `/api/refresh` | POST | Force Notion cache refresh | | `/api/market-intel` | GET | Salary/location/source analytics | | `/api/jobs//update` | POST | Update single job properties | | `/api/jobs//attachments` | GET | Fetch page block attachments | | `/api/jobs//attachments` | POST | Add bookmark attachment to page | | `/api/jobs//attachments/delete` | POST | Remove block from page | | `/api/jobs//cover-letter/regenerate` | POST | Regenerate cover letter | | `/api/jobs//match` | GET | Match analysis for a job | | `/api/jobs/bulk-update` | POST | Update multiple jobs at once | | `/api/jobs/create` | POST | Create new job in Notion | - Company names and job titles with a valid `URL` render as blue external links and open the direct advert in a new tab. - URL values are protocol-validated; only `http:` and `https:` are rendered as links. - KPI/stat cards are one-click filters with an active highlight: - Total Jobs - Backlog - Applied - Interviews - Offers - Closing Soon - This Week - Stale - Strong Matches - Past Closing - Application Rate - Funnel status rows are also one-click exact-status filters. - Existing status editing remains through the job edit modal and bulk-action controls, which write back to Notion through `/api/update`. - The frontend filter/link test harness is `/tmp/radar_dashboard_test.js` (temporary; recreate if absent). ## 27 July 2026 incident ### Symptoms - The morning cron returned old and irrelevant listings, including the stale Pacific Energy Marketing Specialist advert associated with Anthony's redundancy. - Aggregator mirrors, generic SEEK salary pages and category pages were treated as job adverts. - Counts and quality signals were presented without a demonstrated live Notion query/write trail. - The dashboard had valid URLs for almost every record, but the frontend never rendered `job.url`; users could not open the actual adverts. - The earlier prompt explicitly narrowed the task so far that it accidentally prohibited the documented strong-match cover-letter stage. ### Recovery - Archived all **16** records created by the contaminated morning run: 16 successful, zero failures. - Replaced the cron prompt with deterministic freshness, source, verification, Notion, scoring, exclusion and cover-letter gates. - Pinned the cron primary to `gpt-5.6-terra`. - Added and directly probed `opencode-go / mimo-v2.5`; it returned `MIMO_FALLBACK_OK`, then was inserted first in the inherited fallback chain. - Repaired `regenerate_cover_letter.py`: - absolute Hermes paths rather than `~` expansion - one correct command dispatcher rather than two conflicting `__main__` blocks - Repaired dashboard direct links and one-click quick filters. - The corrected Terra run created four verified fresh direct LinkedIn roles. One scored 75 and produced all five cover-letter artifact formats. - A follow-up run queried 166 live Notion records across two pages, verified three current direct adverts, and correctly skipped all three as duplicates. No Pacific Energy records were added. ### Final verification - Dashboard API: 166 records - Records created on 27 July after cleanup: 4 - Pacific Energy records created on 27 July: 0 - Records with direct URLs: 164 of 166 - Strong-match cover letter artifact sizes were nonzero for all five formats - Unresolved template tokens: 0 - Dashboard, API and generated cover-letter URLs returned HTTP 200 - Synthetic dashboard tests passed all quick-filter categories, unsafe-URL rejection and rendered job-link checks ## Tailscale route recovery found during verification The dashboard's local service was healthy while the tailnet URL failed because the CT460 `:443` Serve table had been cleared. Running the original atomic helper restored Serve but also erased the public `:8443` Funnel route. `/home/hermes/.hermes/scripts/tailscale-serve-apply.sh` now atomically restores both: - tailnet-only `:443`: `/`, `/jobs`, `/desktop`, `/vnc-camofox`, `/evening-grid.html` - public `:8443`: `/webhook` → `http://127.0.0.1:8085/webhook` Important: `tailscale serve reset` clears Funnel state too. Never restore only the 443 routes and assume 8443 survived. ## Verification commands ```bash # Cron configuration hermes cron list # Fallback order hermes fallback list # Dashboard/API curl -I https://hermes.kangaroo-eel.ts.net/jobs/ curl -sS https://hermes.kangaroo-eel.ts.net/jobs/api/jobs | jq '.jobs | length' # Local service ss -ltnp 'sport = :9099' curl -sS http://127.0.0.1:9099/api/jobs | jq '.jobs | length' # Route table tailscale serve status # Atomic route rebuild /home/hermes/.hermes/scripts/tailscale-serve-apply.sh --dry-run /home/hermes/.hermes/scripts/tailscale-serve-apply.sh ``` ## Backups from the incident - `/home/hermes/workspace/index.html.radar-links-backup-20260727` - `/home/hermes/workspace/jobs.json.radar-prompt-backup-20260727` - `/home/hermes/workspace/config.yaml.pre-mimo-fallback-20260727` ## Model failure mode — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 30) Every run of the radar failed with: ``` [command-code/gpt-5.6-terra] [403]: MODEL_NOT_IN_PLAN: GPT-5.6 Terra available in Pro and above plans ``` **Cause:** the job is pinned to `gpt-5.6-terra` served through the `command-code` upstream. That upstream's plan no longer includes Terra, so the request is rejected before inference runs. Failure streak reached 5 by 2026-09-30 (last attempt 17:59 AWST). **Diagnosis:** ```bash hermes cron list # job dca8482e4f76 — check last_status + failure_streak hermes cron run dca8482e4f76 # reproduce and capture the 403 ``` **Recovery:** repin the job to a model the upstream still serves (e.g. an `opencode-go` or `smart-route` model), or keep Terra but route it through a provider whose plan includes it. Then re-run once and confirm `execution_success: true`. Do **not** allow a silent fallback — the radar must surface a model failure rather than emit an empty "no jobs found" digest. ## Cover-letter PDF pipeline — printed from the dashboard HTML (changed 2026-10-26) The PDF is now produced from the **canonical rendered HTML**, so a download matches the letter on screen. Previously ReportLab (`build_cover_letter_pdf.py`) laid the letter out independently of the template and produced a visibly different document — Anthony rejected exactly that mismatch. - **Renderer:** `~/.hermes/scripts/build_cover_letter_pdf_from_html.py` (created 2026-10-26 10:37) drives **Chromium `--print-to-pdf`** over the canonical HTML and writes the PDF into `~/.hermes/radar-ref/drafts/`. **Do not go back to ReportLab.** - **Model:** the cover-letter generation default moved from the `cover-letter-writer` combo to **`deepseek-paid-smart`** on 2026-10-26 10:47 (`regenerate_cover_letter.py`, env override `CL_MODEL`). - **Dashboard timeout:** `job_dashboard_server.py` must allow **`timeout=600`** on the regenerate subprocess (set 2026-10-26 10:48, up from 120 s). One run is an LLM call + DOCX assembly + a Chromium print; it can exceed two minutes. The dashboard server was restarted to pick the change up. - All five artefacts (`.txt .json .html .docx .pdf`) must still exist and be nonempty, and the HTML must contain no unresolved `{{...}}` tokens — the HTML is now the source of truth the PDF follows. ### 2026-10-27 07:00 slot — the same overrun, now recorded as `failed` (and the trigger for the split) The 07:00 slot ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**) and closed **`failed`** on delivery `delivered`. The report it sent opens *"Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery, verification, dedup and classification finished; **Notion writes, cover letters and the final audit did not run**. Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168` held **451 entries** with **0** added for 2026-10-27 at run time. Incident row `dca848_e93a538ade6b` opened 11:03:53 (`alerted` 11:03:59) and was **still open** at the 2026-10-28 audit. The split above is the response — see [[incidents/2026-10-27-job-radar-budget-exhausted]] for the full ledger. ### 2026-10-26 07:00 slot — `RUN INCOMPLETE`, and why the backlog was finished by hand The 07:00 run started on schedule and closed **`completed` at 09:10**, but the report it delivered read **“RUN INCOMPLETE — tool-call budget exhausted”**: discovery and scoring consumed the run's tool budget, so the **Notion writes and the entire cover-letter stage never executed**. The same-day backlog was then run interactively (rows written for the outstanding strong matches, cover letters generated and verified). - **A `completed` execution row is not a completed radar run.** Read the delivered text for the `RUN INCOMPLETE` marker before treating a slot as done. A *failed* row with a delivered report is the honest version of the same fault — the 2026-10-27 slot is the first time it reached the incident ledger. - Budget the browser-heavy stages (one tab in flight, close before opening the next — see the 2026-10-12 hazard below) and keep headroom for the Notion + cover-letter writes, which sit last in the pipeline. ## Two-stage split (changed 2026-10-27) The single-job radar was split after two consecutive budget overruns (2026-10-26 `completed` but `RUN INCOMPLETE`, 2026-10-27 closed `failed` — see [[incidents/2026-10-27-job-radar-budget-exhausted]]). Discovery is the stage that eats the tool budget, so it now owns a job of its own and the writes own a second one. | Stage | Job id | Schedule | Writes | Delivers | |---|---|---|---|---| | 1 — Discovery | `dca8482e4f76` (existing id) | `0 7 * * 1-5` | a handoff file only | `local`; failures → topic 1030 | | 2 — Write and Letters | `5fc292ff0f3b` (new) | `30 11,14 * * 1-5` | Notion entries, cover letters, final audit, report | Telegram topic 1030 | - **Stage 1** does `DISCOVERY → VERIFY → DEDUP → SCORE`, writes the handoff file, and triggers stage 2. It must **not** write to Notion and must **not** generate cover letters. - **Stage 2** lands the handoff: Notion entries, cover letters, the missing-letter audit and the report. Its prompt is not a copy of stage 1's — it reports what it wrote, not what it found. - Stage 1 keeps the `custom:omni-101` / `smart-route` pin. Both jobs still inherit the profile-level `fallback_providers` chain; Hermes stores no per-job fallback list. - First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both `completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle ran on 2026-10-28 and landed nothing** — stage 1 never wrote its handoff (see the 2026-10-28 slot below). ✅ **The second real cycle (2026-10-29) completed end to end** — handoff written, consumed and delivered; see the 2026-10-29 slot below. **Recovery if stage 2 lands nothing:** check that stage 1 actually wrote its handoff file before running stage 2 by hand (`hermes cron run 5fc292ff0f3b`). A stage-2 run with no handoff is a no-op, not a failed search — do not re-run stage 1's discovery just to satisfy it. ### 2026-10-28 07:00 slot — stage 1 hit the iteration cap, so stage 2 had nothing to land Stage 1 (`dca8482e4f76`) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**) and closed **`failed`** on *"hit the agent iteration cap before the handoff could be written"* — its **second** consecutive failure (`failure_streak: 2`). No handoff file was written: `~/.hermes/radar-ref/handoff/` held only `2026-10-27.json` at the 2026-10-29 audit. Both stage-2 slots then behaved exactly as designed — **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed`, both `delivery_outcome: suppressed`. **A stage-2 run with no handoff is a silent no-op, not a failure**, which is the recovery rule below holding in production for the first time. Read with 10-27, the picture is: *the split stops the writes from being lost, but it does not make discovery finish.* Two runs, two different ceilings — 10-27 stopped at the **tool-call budget** after 4 h 2 m (writes not started), 10-28 stopped at the **agent iteration cap** after 3 h 32 m, earlier still (no handoff at all). The discovery stage is the thing to bound, not the wall clock. Full ledger: [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. ### 2026-10-29 07:00 slot — the split finally finished: handoff written, consumed, and delivered Stage 1 (`dca8482e4f76`) ran **07:00:59 → 10:16:12 AWST** (**3 h 15 m 13 s**) and closed **`completed`** — the first post-split run to finish *and* write its handoff. `~/.hermes/radar-ref/handoff/2026-10-29.json` (24.8 KB) records `run_date 2026-10-29T08:23:51+08:00`, **170 discovered / 33 duplicates avoided / 37 verified**, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Both of the ledger rows the previous two days opened (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) **closed `resolved` at 10:16:12**, and this run opened no new row. Stage 2 (`5fc292ff0f3b`) fired at **10:13:46**, consumed the handoff at **10:40:59**, closed **10:44:48 `completed`** and **delivered to topic 1030 in the same second**. Its `stage2_result`: **20 pages written / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`** — the Notion board went **455 → 475**. The 11:30 and 14:30 slots then no-oped exactly as designed (**10 m 46 s** and **1 m 51 s**, handoff already consumed). **What this does and does not settle:** - The split's core promise holds in production — the writes can no longer be lost to a discovery overrun, and the first full cycle landed 20 pages and 2 cover letters with **zero missing**. - It does **not** mean discovery is fixed. 3 h 15 m is still long, and the 10-27 **tool-call budget** and 10-28 **iteration cap** remain reachable. Treat a `completed` stage-1 row with **no handoff file** as the same fault, whatever the execution status says. - Stage 1 no longer delivers (verified live: `deliver: local`); its failure route to topic 1030 is unchanged. ## Browser-layer hazard inside the run (2026-10-12) This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove `camofox-browser.service` to its memory ceiling — `2.5G`, swap fully consumed (`available: 0B`), 383 tasks — after which REST `/tabs` and page evals hung in a way indistinguishable from ordinary tool timeouts. The run recovered by restarting the service at **07:23:50** and finishing **one tab at a time with an immediate close**, and it delivered normally. Operational implications for this runbook: - A **green cron row is not proof the browser layer was healthy** — check `systemctl --user status camofox-browser.service` (peak / swap / task count) after any long run. - Extraction must be bounded: one tab in flight, close before opening the next. Multi-tab pre-fetch is what pushed it over. - Limits live in `~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf` (`MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G`, `OOMPolicy=kill`). Raise them only if the host has real headroom — full detail in [[systems/browser-backend]]. ## Related - [[systems/scheduled-tasks]] - [[systems/browser-backend]] - [[infrastructure/tailscale]] - [[systems/model-providers]]