Files
agent-estate-wiki/runbooks/job-radar.md
T

25 KiB

title, type, status, created, updated, review_after, verified_on, version_applies_to, confidence, tags, sources
title type status created updated review_after verified_on version_applies_to confidence tags sources
Daily Marketing Job Radar Operations and Recovery runbook active 2026-07-27 2026-11-05 2026-11-13 2026-11-05 Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2) high
automation
cron
jobs
notion
dashboard
runbook
cover-letter
model-route
live-cron-output
live-notion-api
live-dashboard-api
hermes-docs
live-script-inspection
live-endpoint-probe

Daily Marketing Job Radar Operations and Recovery

Purpose

Run a verified Western Australian marketing-job search, deduplicate against the complete Notion board, create only fresh direct adverts, score new roles against Anthony's resume, automatically produce cover letters for strong matches, and expose the results through the Jobs Radar dashboard.

Production configuration

Setting Value
Cron job — stage 1 dca8482e4f76 — Daily Marketing Job Radar - Discovery (Discovery → Verify → Dedup → Score; no Notion write, no cover letters)
Schedule — stage 1 0 7 * * 1-5 (07:00 AWST weekdays)
Delivery — stage 1 local (dashboard/CLI); failures → Telegram group -1004321904721, topic 1030
Cron job — stage 2 5fc292ff0f3b — Daily Marketing Job Radar - Write and Letters (created 2026-10-27 17:34:15)
Schedule — stage 2 30 11,14 * * 1-5 (11:30 and 14:30 AWST weekdays)
Delivery — stage 2 Telegram group -1004321904721, topic 1030
Skill job-search-automation
Primary model (both stages) custom:omni-101 / smart-route (live 2026-10-28) — the 2026-09-26 gpt-5.6-terra 403 is historical
First fallback opencode-go / mimo-v2.5
Dashboard https://hermes.kangaroo-eel.ts.net/jobs/
Dashboard backend 127.0.0.1:9099
Notion database 0f90ba2b-8b10-4d02-a62c-6f692d5b1168

Cron jobs inherit the profile-level fallback_providers chain; Hermes does not currently store a separate fallback chain in each cron record. MiMo v2.5 is first in the global chain, so Terra fails over to it before the older fallbacks.

Cover letters are non-negotiable

Model route caveat - verified 2026-11-05. regenerate_cover_letter.py (line 82) reads os.environ.get("CL_MODEL", "deepseek-paid-smart"). On the 2026-11-04 run that default returned HTTP 402 (billing), and the run pinned gemini/gemini-3.5-flash via CL_MODEL to get the strong-match letter out. grep finds no CL_MODEL in ~/.hermes/.env - the override existed only inside that process, so the next run inherits the default again. If a run logs 402/503 while generating letters, re-pin CL_MODEL for the process and confirm the letters rendered - do not ship a run without its cover letters. Note the same model name still serves other cron jobs successfully (e.g. the 23:59 nightly audit), and the endpoint itself is live: omniroute:20128/v1/models -> 401 and 100.78.63.20:20128/v1/models -> 401 on 2026-11-05 (key-gated, not down). Treat a 402 as route/time-specific until re-tested, not as proof the route is dead. Keyed re-test 2026-11-05 01:33:12 AWST: deepseek-paid-smart -> 503 ALL_TARGETS_SKIPPED (poolSize 5, attempted 0) on omniroute:20128, but 200 on 100.78.63.20:20128 (omniroute-1, resolved to deepseek/deepseek-v4-flash). As of that probe the default CL_MODEL cannot dispatch on the CT600 node, so a CL_MODEL override (or an omniroute-1 base URL) is required for cover-letter runs. An unkeyed /v1/models 401 is auth-gating, not health.

A successful radar run includes the cover-letter stage.

  • Audit every Strong Match added today, including pages created by an interrupted earlier run.
  • A valid result requires a canonical rendered .html URL in Notion, not plain text and not the dashboard root.
  • Production template files:
    • /home/hermes/.hermes/dashboard/templates/cover-letter.html
    • /home/hermes/.hermes/dashboard/templates/Anthony_Martin_Cover_Letter_Template.docx
  • The old upload-cache location is not authoritative and may be evicted.
  • Required nonempty artifacts: .txt, .json, .html, .docx, .pdf.
  • The HTML must contain zero unresolved {{...}} tokens and return HTTP 200.
  • Retry generation once when an artifact, Notion link or HTTP check fails.
  • A clean run requires MISSING_COVER_LETTER_COUNT=0. Any remaining missing letter must be reported prominently with its Notion page ID; it must never be silently omitted.

Live audit — 27 July 2026

  • Same-day strong matches: 1
  • Verified canonical HTML cover letters: 1
  • Missing cover letters: 0
  • The Market Creations Agency strong match has all five artifact formats, zero unresolved template tokens and a live HTTP 200 HTML link.

Required production pipeline

  1. Obtain the live AWST date and calculate a seven-day freshness cutoff.
  2. Search SEEK, LinkedIn, Indeed, WA Government Jobs and direct employer sites.
  3. Open and verify every individual advert. Search snippets are discovery evidence only.
  4. Reject stale, expired, cached, interstate, non-marketing, salary-guide, search/category and aggregator pages.
  5. Reject all Pacific Energy roles. Anthony was made redundant there in July 2026; old Pacific Energy adverts must never be presented as opportunities.
  6. Accept only direct individual advert URLs:
    • SEEK /job/<numeric-id>
    • LinkedIn /jobs/view/...<numeric-id>
    • Indeed /viewjob?jk=<real-id>
    • direct employer career pages
    • individual WA Government role pages
  7. A WA Government slug page is valid without AdvertID in its URL when the page proves the exact role and agency, WA location, unique job/pool reference, future closing date and active Apply Now function.
  8. Query the complete Notion database with live pagination. Deduplicate by exact URL, normalized company plus role, and materially similar company-role combinations.
  9. Create only verified nonduplicates and require returned Notion page IDs.
  10. Refresh POST http://127.0.0.1:9099/api/refresh, then confirm every page and direct URL through /api/jobs.
  11. Score newly created roles against /home/hermes/.hermes/radar-ref/resumes/current_resume_from_website.txt using the documented 100-point rubric. Strong Match starts at 70.
  12. For each new strong match, run:
/home/hermes/.hermes/hermes-agent/venv/bin/python \
  /home/hermes/.hermes/scripts/regenerate_cover_letter.py \
  <NOTION_PAGE_ID> \
  "Generate the initial tailored cover letter using the verified job description. Preserve Anthony's factual employment history and use the canonical teal Montserrat template."
  1. Verify nonempty .txt, .json, .html, .docx and .pdf artifacts; no unresolved {{...}} tokens; and a live .html Notion Cover Letter URL.
  2. Report only facts produced by tools in that run. Never estimate counts or fabricate Notion writes.

Dashboard behaviour

Source: /home/hermes/.hermes/dashboard/index.html
Backend: /home/hermes/.hermes/scripts/job_dashboard_server.py

Layout (top to bottom)

  1. Hero header — RADAR logo, name, dateline
  2. Market Intel panel — expandable salary/location/source stats from /api/market-intel
  3. KPI strip — clickable filter cards: Total Jobs, Backlog, Applied, Phone Screen, Interviews, Offers, Closing Soon
  4. Filter bar — text search, view toggles (table/cards), Refresh button
  5. Stats strip — This Week, Stale 14d+, Strong Match, Past Close, Applied Rate
  6. Pipeline Funnel — clickable per-status breakdown with bar chart
  7. Job table/cards — sortable columns, inline status pills, cover letter links, edit actions

Column sorting

Click any column header (Status, Company, Role, Match, Salary, Age, Closes, Updated) to sort ascending; click again for descending. Arrow indicator shows current direction.

Status management

  • Bulk update: select jobs via checkboxes → pick status from dropdown → Apply. Sends {ids, updates: {status}} to POST /api/jobs/bulk-update.
  • Single edit: click ✎ → modal with Status, Priority, Role, Salary, Closing Date, URL, Notes, Cover Letter, Attachments. Saves to Notion via POST /api/jobs/<id>/update.
  • Funnel/status filters: clicking a status in the funnel or KPI strip filters the table to that status.

Rich text notes

  • Write/Preview editor in the edit modal (monospace textarea + markdown preview)
  • Supports: bold **, italic *, links [text](url), bullet lists - item
  • Syncs to Notion Notes / JD Summary rich_text property on every save

Document attachments

  • Attach any URL (documents, links, files) to a job entry
  • Stored as bookmark blocks on the Notion page body (not a property)
  • Lazy-loaded when edit modal opens: GET /api/jobs/<id>/attachments
  • Add: POST /api/jobs/<id>/attachments with {name, url}
  • Delete: POST /api/jobs/<id>/attachments/delete with {block_id}
  • Visible in Notion when you open the page

Cover letter integration

  • Regenerate button (🔄) on each row calls POST /api/jobs/<id>/cover-letter/regenerate
  • CL URL stored in Notion Cover Letter property and rendered as clickable link in table
  • All 5 artifact formats: TXT, JSON, HTML, DOCX, PDF

Market Intel

  • Expandable panel at top of page
  • Data from GET /api/market-intel: total tracked, salary stats (avg/median/min/max), counts
  • Lazy-loaded on expand; also refreshes on ↺ Refresh

API endpoints

Endpoint Method Purpose
/api/jobs GET All jobs (cached, paginated from Notion)
/api/stats GET Aggregate statistics
/api/refresh POST Force Notion cache refresh
/api/market-intel GET Salary/location/source analytics
/api/jobs/<id>/update POST Update single job properties
/api/jobs/<id>/attachments GET Fetch page block attachments
/api/jobs/<id>/attachments POST Add bookmark attachment to page
/api/jobs/<id>/attachments/delete POST Remove block from page
/api/jobs/<id>/cover-letter/regenerate POST Regenerate cover letter
/api/jobs/<id>/match GET Match analysis for a job
/api/jobs/bulk-update POST Update multiple jobs at once
/api/jobs/create POST Create new job in Notion
  • Company names and job titles with a valid URL render as blue external links and open the direct advert in a new tab.
  • URL values are protocol-validated; only http: and https: are rendered as links.
  • KPI/stat cards are one-click filters with an active highlight:
    • Total Jobs
    • Backlog
    • Applied
    • Interviews
    • Offers
    • Closing Soon
    • This Week
    • Stale
    • Strong Matches
    • Past Closing
    • Application Rate
  • Funnel status rows are also one-click exact-status filters.
  • Existing status editing remains through the job edit modal and bulk-action controls, which write back to Notion through /api/update.
  • The frontend filter/link test harness is /tmp/radar_dashboard_test.js (temporary; recreate if absent).

27 July 2026 incident

Symptoms

  • The morning cron returned old and irrelevant listings, including the stale Pacific Energy Marketing Specialist advert associated with Anthony's redundancy.
  • Aggregator mirrors, generic SEEK salary pages and category pages were treated as job adverts.
  • Counts and quality signals were presented without a demonstrated live Notion query/write trail.
  • The dashboard had valid URLs for almost every record, but the frontend never rendered job.url; users could not open the actual adverts.
  • The earlier prompt explicitly narrowed the task so far that it accidentally prohibited the documented strong-match cover-letter stage.

Recovery

  • Archived all 16 records created by the contaminated morning run: 16 successful, zero failures.
  • Replaced the cron prompt with deterministic freshness, source, verification, Notion, scoring, exclusion and cover-letter gates.
  • Pinned the cron primary to gpt-5.6-terra.
  • Added and directly probed opencode-go / mimo-v2.5; it returned MIMO_FALLBACK_OK, then was inserted first in the inherited fallback chain.
  • Repaired regenerate_cover_letter.py:
    • absolute Hermes paths rather than ~ expansion
    • one correct command dispatcher rather than two conflicting __main__ blocks
  • Repaired dashboard direct links and one-click quick filters.
  • The corrected Terra run created four verified fresh direct LinkedIn roles. One scored 75 and produced all five cover-letter artifact formats.
  • A follow-up run queried 166 live Notion records across two pages, verified three current direct adverts, and correctly skipped all three as duplicates. No Pacific Energy records were added.

Final verification

  • Dashboard API: 166 records
  • Records created on 27 July after cleanup: 4
  • Pacific Energy records created on 27 July: 0
  • Records with direct URLs: 164 of 166
  • Strong-match cover letter artifact sizes were nonzero for all five formats
  • Unresolved template tokens: 0
  • Dashboard, API and generated cover-letter URLs returned HTTP 200
  • Synthetic dashboard tests passed all quick-filter categories, unsafe-URL rejection and rendered job-link checks

Tailscale route recovery found during verification

The dashboard's local service was healthy while the tailnet URL failed because the CT460 :443 Serve table had been cleared. Running the original atomic helper restored Serve but also erased the public :8443 Funnel route.

/home/hermes/.hermes/scripts/tailscale-serve-apply.sh now atomically restores both:

  • tailnet-only :443: /, /jobs, /desktop, /vnc-camofox, /evening-grid.html
  • public :8443: /webhook → http://127.0.0.1:8085/webhook

Important: tailscale serve reset clears Funnel state too. Never restore only the 443 routes and assume 8443 survived.

Verification commands

# Cron configuration
hermes cron list

# Fallback order
hermes fallback list

# Dashboard/API
curl -I https://hermes.kangaroo-eel.ts.net/jobs/
curl -sS https://hermes.kangaroo-eel.ts.net/jobs/api/jobs | jq '.jobs | length'

# Local service
ss -ltnp 'sport = :9099'
curl -sS http://127.0.0.1:9099/api/jobs | jq '.jobs | length'

# Route table
tailscale serve status

# Atomic route rebuild
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh --dry-run
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh

Backups from the incident

  • /home/hermes/workspace/index.html.radar-links-backup-20260727
  • /home/hermes/workspace/jobs.json.radar-prompt-backup-20260727
  • /home/hermes/workspace/config.yaml.pre-mimo-fallback-20260727

Model failure mode — 403 MODEL_NOT_IN_PLAN (2026-09-26 → 30)

Every run of the radar failed with:

[command-code/gpt-5.6-terra] [403]: MODEL_NOT_IN_PLAN: GPT-5.6 Terra available in Pro and above plans

Cause: the job is pinned to gpt-5.6-terra served through the command-code upstream. That upstream's plan no longer includes Terra, so the request is rejected before inference runs. Failure streak reached 5 by 2026-09-30 (last attempt 17:59 AWST).

Diagnosis:

hermes cron list                 # job dca8482e4f76 — check last_status + failure_streak
hermes cron run dca8482e4f76     # reproduce and capture the 403

Recovery: repin the job to a model the upstream still serves (e.g. an opencode-go or smart-route model), or keep Terra but route it through a provider whose plan includes it. Then re-run once and confirm execution_success: true.

Do not allow a silent fallback — the radar must surface a model failure rather than emit an empty "no jobs found" digest.

Cover-letter PDF pipeline — printed from the dashboard HTML (changed 2026-10-26)

The PDF is now produced from the canonical rendered HTML, so a download matches the letter on screen. Previously ReportLab (build_cover_letter_pdf.py) laid the letter out independently of the template and produced a visibly different document — Anthony rejected exactly that mismatch.

  • Renderer: ~/.hermes/scripts/build_cover_letter_pdf_from_html.py (created 2026-10-26 10:37) drives Chromium --print-to-pdf over the canonical HTML and writes the PDF into ~/.hermes/radar-ref/drafts/. Do not go back to ReportLab.
  • Model: the cover-letter generation default moved from the cover-letter-writer combo to deepseek-paid-smart on 2026-10-26 10:47 (regenerate_cover_letter.py, env override CL_MODEL).
  • Dashboard timeout: job_dashboard_server.py must allow timeout=600 on the regenerate subprocess (set 2026-10-26 10:48, up from 120 s). One run is an LLM call + DOCX assembly + a Chromium print; it can exceed two minutes. The dashboard server was restarted to pick the change up.
  • All five artefacts (.txt .json .html .docx .pdf) must still exist and be nonempty, and the HTML must contain no unresolved {{...}} tokens — the HTML is now the source of truth the PDF follows.

2026-10-27 07:00 slot — the same overrun, now recorded as failed (and the trigger for the split)

The 07:00 slot ran 07:01:08 → 11:03:54 (4 h 2 m 46 s) and closed failed on delivery delivered. The report it sent opens "Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call budget exhausted before Notion write stage)": discovery, verification, dedup and classification finished; Notion writes, cover letters and the final audit did not run. Notion DB 0f90ba2b-8b10-4d02-a62c-6f692d5b1168 held 451 entries with 0 added for 2026-10-27 at run time. Incident row dca848_e93a538ade6b opened 11:03:53 (alerted 11:03:59) and was still open at the 2026-10-28 audit. The split above is the response — see incidents/2026-10-27-job-radar-budget-exhausted for the full ledger.

2026-10-26 07:00 slot — RUN INCOMPLETE, and why the backlog was finished by hand

The 07:00 run started on schedule and closed completed at 09:10, but the report it delivered read “RUN INCOMPLETE — tool-call budget exhausted”: discovery and scoring consumed the run's tool budget, so the Notion writes and the entire cover-letter stage never executed. The same-day backlog was then run interactively (rows written for the outstanding strong matches, cover letters generated and verified).

  • A completed execution row is not a completed radar run. Read the delivered text for the RUN INCOMPLETE marker before treating a slot as done. A failed row with a delivered report is the honest version of the same fault — the 2026-10-27 slot is the first time it reached the incident ledger.
  • Budget the browser-heavy stages (one tab in flight, close before opening the next — see the 2026-10-12 hazard below) and keep headroom for the Notion + cover-letter writes, which sit last in the pipeline.

Two-stage split (changed 2026-10-27)

The single-job radar was split after two consecutive budget overruns (2026-10-26 completed but RUN INCOMPLETE, 2026-10-27 closed failed — see incidents/2026-10-27-job-radar-budget-exhausted). Discovery is the stage that eats the tool budget, so it now owns a job of its own and the writes own a second one.

Stage Job id Schedule Writes Delivers
1 — Discovery dca8482e4f76 (existing id) 0 7 * * 1-5 a handoff file only local; failures → topic 1030
2 — Write and Letters 5fc292ff0f3b (new) 30 11,14 * * 1-5 Notion entries, cover letters, final audit, report Telegram topic 1030
  • Stage 1 does DISCOVERY → VERIFY → DEDUP → SCORE, writes the handoff file, and triggers stage 2. It must not write to Notion and must not generate cover letters.
  • Stage 2 lands the handoff: Notion entries, cover letters, the missing-letter audit and the report. Its prompt is not a copy of stage 1's — it reports what it wrote, not what it found.
  • Stage 1 keeps the custom:omni-101 / smart-route pin. Both jobs still inherit the profile-level fallback_providers chain; Hermes stores no per-job fallback list.
  • First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both completed with delivery suppressed. The first real 07:00 → 11:30 cycle ran on 2026-10-28 and landed nothing — stage 1 never wrote its handoff (see the 2026-10-28 slot below). ✅ The second real cycle (2026-10-29) completed end to end — handoff written, consumed and delivered; see the 2026-10-29 slot below.

Recovery if stage 2 lands nothing: check that stage 1 actually wrote its handoff file before running stage 2 by hand (hermes cron run 5fc292ff0f3b). A stage-2 run with no handoff is a no-op, not a failed search — do not re-run stage 1's discovery just to satisfy it.

2026-10-28 07:00 slot — stage 1 hit the iteration cap, so stage 2 had nothing to land

Stage 1 (dca8482e4f76) ran 07:00:32 → 10:33:18 (3 h 32 m 46 s) and closed failed on "hit the agent iteration cap before the handoff could be written" — its second consecutive failure (failure_streak: 2). No handoff file was written: ~/.hermes/radar-ref/handoff/ held only 2026-10-27.json at the 2026-10-29 audit.

Both stage-2 slots then behaved exactly as designed — 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43, both completed, both delivery_outcome: suppressed. A stage-2 run with no handoff is a silent no-op, not a failure, which is the recovery rule below holding in production for the first time.

Read with 10-27, the picture is: the split stops the writes from being lost, but it does not make discovery finish. Two runs, two different ceilings — 10-27 stopped at the tool-call budget after 4 h 2 m (writes not started), 10-28 stopped at the agent iteration cap after 3 h 32 m, earlier still (no handoff at all). The discovery stage is the thing to bound, not the wall clock.

Full ledger: incidents/2026-10-28-job-radar-stage1-iteration-cap.

2026-10-29 07:00 slot — the split finally finished: handoff written, consumed, and delivered

Stage 1 (dca8482e4f76) ran 07:00:59 → 10:16:12 AWST (3 h 15 m 13 s) and closed completed — the first post-split run to finish and write its handoff. ~/.hermes/radar-ref/handoff/2026-10-29.json (24.8 KB) records run_date 2026-10-29T08:23:51+08:00, 170 discovered / 33 duplicates avoided / 37 verified, sources seek ok, linkedin ok, indeed blocked, wa_gov ok. Both of the ledger rows the previous two days opened (dca848_e93a538ade6b, dca848_0a2f03b5952b) closed resolved at 10:16:12, and this run opened no new row.

Stage 2 (5fc292ff0f3b) fired at 10:13:46, consumed the handoff at 10:40:59, closed 10:44:48 completed and delivered to topic 1030 in the same second. Its stage2_result: 20 pages written / 20 page_ids / 2 cover letters / 2 same-day strong matches / verified_cover_letter_count 2 / missing_cover_letter_count 0 — the Notion board went 455 → 475. The 11:30 and 14:30 slots then no-oped exactly as designed (10 m 46 s and 1 m 51 s, handoff already consumed).

What this does and does not settle:

  • The split's core promise holds in production — the writes can no longer be lost to a discovery overrun, and the first full cycle landed 20 pages and 2 cover letters with zero missing.
  • It does not mean discovery is fixed. 3 h 15 m is still long, and the 10-27 tool-call budget and 10-28 iteration cap remain reachable. Treat a completed stage-1 row with no handoff file as the same fault, whatever the execution status says.
  • Stage 1 no longer delivers (verified live: deliver: local); its failure route to topic 1030 is unchanged.

Browser-layer hazard inside the run (2026-10-12)

This job is the most browser-heavy cron on the estate, and on 2026-10-12 (07:01 → 07:33) it drove camofox-browser.service to its memory ceiling — 2.5G, swap fully consumed (available: 0B), 383 tasks — after which REST /tabs and page evals hung in a way indistinguishable from ordinary tool timeouts. The run recovered by restarting the service at 07:23:50 and finishing one tab at a time with an immediate close, and it delivered normally.

Operational implications for this runbook:

  • A green cron row is not proof the browser layer was healthy — check systemctl --user status camofox-browser.service (peak / swap / task count) after any long run.
  • Extraction must be bounded: one tab in flight, close before opening the next. Multi-tab pre-fetch is what pushed it over.
  • Limits live in ~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf (MemoryHigh=2G, MemoryMax=3G, MemorySwapMax=1G, OOMPolicy=kill). Raise them only if the host has real headroom — full detail in systems/browser-backend.