25 KiB
title, type, status, created, updated, review_after, verified_on, version_applies_to, confidence, tags, sources
| title | type | status | created | updated | review_after | verified_on | version_applies_to | confidence | tags | sources | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Daily Marketing Job Radar Operations and Recovery | runbook | active | 2026-07-27 | 2026-11-05 | 2026-11-13 | 2026-11-05 | Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2) | high |
|
|
Daily Marketing Job Radar Operations and Recovery
Purpose
Run a verified Western Australian marketing-job search, deduplicate against the complete Notion board, create only fresh direct adverts, score new roles against Anthony's resume, automatically produce cover letters for strong matches, and expose the results through the Jobs Radar dashboard.
Production configuration
| Setting | Value |
|---|---|
| Cron job — stage 1 | dca8482e4f76 — Daily Marketing Job Radar - Discovery (Discovery → Verify → Dedup → Score; no Notion write, no cover letters) |
| Schedule — stage 1 | 0 7 * * 1-5 (07:00 AWST weekdays) |
| Delivery — stage 1 | local (dashboard/CLI); failures → Telegram group -1004321904721, topic 1030 |
| Cron job — stage 2 | 5fc292ff0f3b — Daily Marketing Job Radar - Write and Letters (created 2026-10-27 17:34:15) |
| Schedule — stage 2 | 30 11,14 * * 1-5 (11:30 and 14:30 AWST weekdays) |
| Delivery — stage 2 | Telegram group -1004321904721, topic 1030 |
| Skill | job-search-automation |
| Primary model (both stages) | custom:omni-101 / smart-route (live 2026-10-28) — the 2026-09-26 gpt-5.6-terra 403 is historical |
| First fallback | opencode-go / mimo-v2.5 |
| Dashboard | https://hermes.kangaroo-eel.ts.net/jobs/ |
| Dashboard backend | 127.0.0.1:9099 |
| Notion database | 0f90ba2b-8b10-4d02-a62c-6f692d5b1168 |
Cron jobs inherit the profile-level fallback_providers chain; Hermes does not currently store a separate fallback chain in each cron record. MiMo v2.5 is first in the global chain, so Terra fails over to it before the older fallbacks.
Cover letters are non-negotiable
Model route caveat - verified 2026-11-05.
regenerate_cover_letter.py(line 82) readsos.environ.get("CL_MODEL", "deepseek-paid-smart"). On the 2026-11-04 run that default returned HTTP 402 (billing), and the run pinnedgemini/gemini-3.5-flashviaCL_MODELto get the strong-match letter out.grepfinds noCL_MODELin~/.hermes/.env- the override existed only inside that process, so the next run inherits the default again. If a run logs 402/503 while generating letters, re-pinCL_MODELfor the process and confirm the letters rendered - do not ship a run without its cover letters. Note the same model name still serves other cron jobs successfully (e.g. the 23:59 nightly audit), and the endpoint itself is live:omniroute:20128/v1/models-> 401 and100.78.63.20:20128/v1/models-> 401 on 2026-11-05 (key-gated, not down). Treat a 402 as route/time-specific until re-tested, not as proof the route is dead. Keyed re-test 2026-11-05 01:33:12 AWST:deepseek-paid-smart-> 503ALL_TARGETS_SKIPPED(poolSize 5,attempted 0) onomniroute:20128, but 200 on100.78.63.20:20128(omniroute-1, resolved todeepseek/deepseek-v4-flash). As of that probe the defaultCL_MODELcannot dispatch on the CT600 node, so aCL_MODELoverride (or anomniroute-1base URL) is required for cover-letter runs. An unkeyed/v1/models401 is auth-gating, not health.
A successful radar run includes the cover-letter stage.
- Audit every Strong Match added today, including pages created by an interrupted earlier run.
- A valid result requires a canonical rendered
.htmlURL in Notion, not plain text and not the dashboard root. - Production template files:
/home/hermes/.hermes/dashboard/templates/cover-letter.html/home/hermes/.hermes/dashboard/templates/Anthony_Martin_Cover_Letter_Template.docx
- The old upload-cache location is not authoritative and may be evicted.
- Required nonempty artifacts:
.txt,.json,.html,.docx,.pdf. - The HTML must contain zero unresolved
{{...}}tokens and return HTTP 200. - Retry generation once when an artifact, Notion link or HTTP check fails.
- A clean run requires
MISSING_COVER_LETTER_COUNT=0. Any remaining missing letter must be reported prominently with its Notion page ID; it must never be silently omitted.
Live audit — 27 July 2026
- Same-day strong matches: 1
- Verified canonical HTML cover letters: 1
- Missing cover letters: 0
- The Market Creations Agency strong match has all five artifact formats, zero unresolved template tokens and a live HTTP 200 HTML link.
Required production pipeline
- Obtain the live AWST date and calculate a seven-day freshness cutoff.
- Search SEEK, LinkedIn, Indeed, WA Government Jobs and direct employer sites.
- Open and verify every individual advert. Search snippets are discovery evidence only.
- Reject stale, expired, cached, interstate, non-marketing, salary-guide, search/category and aggregator pages.
- Reject all Pacific Energy roles. Anthony was made redundant there in July 2026; old Pacific Energy adverts must never be presented as opportunities.
- Accept only direct individual advert URLs:
- SEEK
/job/<numeric-id> - LinkedIn
/jobs/view/...<numeric-id> - Indeed
/viewjob?jk=<real-id> - direct employer career pages
- individual WA Government role pages
- SEEK
- A WA Government slug page is valid without
AdvertIDin its URL when the page proves the exact role and agency, WA location, unique job/pool reference, future closing date and active Apply Now function. - Query the complete Notion database with live pagination. Deduplicate by exact URL, normalized company plus role, and materially similar company-role combinations.
- Create only verified nonduplicates and require returned Notion page IDs.
- Refresh
POST http://127.0.0.1:9099/api/refresh, then confirm every page and direct URL through/api/jobs. - Score newly created roles against
/home/hermes/.hermes/radar-ref/resumes/current_resume_from_website.txtusing the documented 100-point rubric. Strong Match starts at 70. - For each new strong match, run:
/home/hermes/.hermes/hermes-agent/venv/bin/python \
/home/hermes/.hermes/scripts/regenerate_cover_letter.py \
<NOTION_PAGE_ID> \
"Generate the initial tailored cover letter using the verified job description. Preserve Anthony's factual employment history and use the canonical teal Montserrat template."
- Verify nonempty
.txt,.json,.html,.docxand.pdfartifacts; no unresolved{{...}}tokens; and a live.htmlNotion Cover Letter URL. - Report only facts produced by tools in that run. Never estimate counts or fabricate Notion writes.
Dashboard behaviour
Source: /home/hermes/.hermes/dashboard/index.html
Backend: /home/hermes/.hermes/scripts/job_dashboard_server.py
Layout (top to bottom)
- Hero header — RADAR logo, name, dateline
- Market Intel panel — expandable salary/location/source stats from
/api/market-intel - KPI strip — clickable filter cards: Total Jobs, Backlog, Applied, Phone Screen, Interviews, Offers, Closing Soon
- Filter bar — text search, view toggles (table/cards), Refresh button
- Stats strip — This Week, Stale 14d+, Strong Match, Past Close, Applied Rate
- Pipeline Funnel — clickable per-status breakdown with bar chart
- Job table/cards — sortable columns, inline status pills, cover letter links, edit actions
Column sorting
Click any column header (Status, Company, Role, Match, Salary, Age, Closes, Updated) to sort ascending; click again for descending. Arrow indicator shows current direction.
Status management
- Bulk update: select jobs via checkboxes → pick status from dropdown → Apply. Sends
{ids, updates: {status}}toPOST /api/jobs/bulk-update. - Single edit: click ✎ → modal with Status, Priority, Role, Salary, Closing Date, URL, Notes, Cover Letter, Attachments. Saves to Notion via
POST /api/jobs/<id>/update. - Funnel/status filters: clicking a status in the funnel or KPI strip filters the table to that status.
Rich text notes
- Write/Preview editor in the edit modal (monospace textarea + markdown preview)
- Supports: bold
**, italic*, links[text](url), bullet lists- item - Syncs to Notion
Notes / JD Summaryrich_text property on every save
Document attachments
- Attach any URL (documents, links, files) to a job entry
- Stored as bookmark blocks on the Notion page body (not a property)
- Lazy-loaded when edit modal opens:
GET /api/jobs/<id>/attachments - Add:
POST /api/jobs/<id>/attachmentswith{name, url} - Delete:
POST /api/jobs/<id>/attachments/deletewith{block_id} - Visible in Notion when you open the page
Cover letter integration
- Regenerate button (🔄) on each row calls
POST /api/jobs/<id>/cover-letter/regenerate - CL URL stored in Notion
Cover Letterproperty and rendered as clickable link in table - All 5 artifact formats: TXT, JSON, HTML, DOCX, PDF
Market Intel
- Expandable panel at top of page
- Data from
GET /api/market-intel: total tracked, salary stats (avg/median/min/max), counts - Lazy-loaded on expand; also refreshes on ↺ Refresh
API endpoints
| Endpoint | Method | Purpose |
|---|---|---|
/api/jobs |
GET | All jobs (cached, paginated from Notion) |
/api/stats |
GET | Aggregate statistics |
/api/refresh |
POST | Force Notion cache refresh |
/api/market-intel |
GET | Salary/location/source analytics |
/api/jobs/<id>/update |
POST | Update single job properties |
/api/jobs/<id>/attachments |
GET | Fetch page block attachments |
/api/jobs/<id>/attachments |
POST | Add bookmark attachment to page |
/api/jobs/<id>/attachments/delete |
POST | Remove block from page |
/api/jobs/<id>/cover-letter/regenerate |
POST | Regenerate cover letter |
/api/jobs/<id>/match |
GET | Match analysis for a job |
/api/jobs/bulk-update |
POST | Update multiple jobs at once |
/api/jobs/create |
POST | Create new job in Notion |
- Company names and job titles with a valid
URLrender as blue external links and open the direct advert in a new tab. - URL values are protocol-validated; only
http:andhttps:are rendered as links. - KPI/stat cards are one-click filters with an active highlight:
- Total Jobs
- Backlog
- Applied
- Interviews
- Offers
- Closing Soon
- This Week
- Stale
- Strong Matches
- Past Closing
- Application Rate
- Funnel status rows are also one-click exact-status filters.
- Existing status editing remains through the job edit modal and bulk-action controls, which write back to Notion through
/api/update. - The frontend filter/link test harness is
/tmp/radar_dashboard_test.js(temporary; recreate if absent).
27 July 2026 incident
Symptoms
- The morning cron returned old and irrelevant listings, including the stale Pacific Energy Marketing Specialist advert associated with Anthony's redundancy.
- Aggregator mirrors, generic SEEK salary pages and category pages were treated as job adverts.
- Counts and quality signals were presented without a demonstrated live Notion query/write trail.
- The dashboard had valid URLs for almost every record, but the frontend never rendered
job.url; users could not open the actual adverts. - The earlier prompt explicitly narrowed the task so far that it accidentally prohibited the documented strong-match cover-letter stage.
Recovery
- Archived all 16 records created by the contaminated morning run: 16 successful, zero failures.
- Replaced the cron prompt with deterministic freshness, source, verification, Notion, scoring, exclusion and cover-letter gates.
- Pinned the cron primary to
gpt-5.6-terra. - Added and directly probed
opencode-go / mimo-v2.5; it returnedMIMO_FALLBACK_OK, then was inserted first in the inherited fallback chain. - Repaired
regenerate_cover_letter.py:- absolute Hermes paths rather than
~expansion - one correct command dispatcher rather than two conflicting
__main__blocks
- absolute Hermes paths rather than
- Repaired dashboard direct links and one-click quick filters.
- The corrected Terra run created four verified fresh direct LinkedIn roles. One scored 75 and produced all five cover-letter artifact formats.
- A follow-up run queried 166 live Notion records across two pages, verified three current direct adverts, and correctly skipped all three as duplicates. No Pacific Energy records were added.
Final verification
- Dashboard API: 166 records
- Records created on 27 July after cleanup: 4
- Pacific Energy records created on 27 July: 0
- Records with direct URLs: 164 of 166
- Strong-match cover letter artifact sizes were nonzero for all five formats
- Unresolved template tokens: 0
- Dashboard, API and generated cover-letter URLs returned HTTP 200
- Synthetic dashboard tests passed all quick-filter categories, unsafe-URL rejection and rendered job-link checks
Tailscale route recovery found during verification
The dashboard's local service was healthy while the tailnet URL failed because the CT460 :443 Serve table had been cleared. Running the original atomic helper restored Serve but also erased the public :8443 Funnel route.
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh now atomically restores both:
- tailnet-only
:443:/,/jobs,/desktop,/vnc-camofox,/evening-grid.html - public
:8443:/webhook→http://127.0.0.1:8085/webhook
Important: tailscale serve reset clears Funnel state too. Never restore only the 443 routes and assume 8443 survived.
Verification commands
# Cron configuration
hermes cron list
# Fallback order
hermes fallback list
# Dashboard/API
curl -I https://hermes.kangaroo-eel.ts.net/jobs/
curl -sS https://hermes.kangaroo-eel.ts.net/jobs/api/jobs | jq '.jobs | length'
# Local service
ss -ltnp 'sport = :9099'
curl -sS http://127.0.0.1:9099/api/jobs | jq '.jobs | length'
# Route table
tailscale serve status
# Atomic route rebuild
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh --dry-run
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh
Backups from the incident
/home/hermes/workspace/index.html.radar-links-backup-20260727/home/hermes/workspace/jobs.json.radar-prompt-backup-20260727/home/hermes/workspace/config.yaml.pre-mimo-fallback-20260727
Model failure mode — 403 MODEL_NOT_IN_PLAN (2026-09-26 → 30)
Every run of the radar failed with:
[command-code/gpt-5.6-terra] [403]: MODEL_NOT_IN_PLAN: GPT-5.6 Terra available in Pro and above plans
Cause: the job is pinned to gpt-5.6-terra served through the command-code upstream. That upstream's plan no longer includes Terra, so the request is rejected before inference runs. Failure streak reached 5 by 2026-09-30 (last attempt 17:59 AWST).
Diagnosis:
hermes cron list # job dca8482e4f76 — check last_status + failure_streak
hermes cron run dca8482e4f76 # reproduce and capture the 403
Recovery: repin the job to a model the upstream still serves (e.g. an opencode-go or smart-route model), or keep Terra but route it through a provider whose plan includes it. Then re-run once and confirm execution_success: true.
Do not allow a silent fallback — the radar must surface a model failure rather than emit an empty "no jobs found" digest.
Cover-letter PDF pipeline — printed from the dashboard HTML (changed 2026-10-26)
The PDF is now produced from the canonical rendered HTML, so a download matches the letter on screen.
Previously ReportLab (build_cover_letter_pdf.py) laid the letter out independently of the template and
produced a visibly different document — Anthony rejected exactly that mismatch.
- Renderer:
~/.hermes/scripts/build_cover_letter_pdf_from_html.py(created 2026-10-26 10:37) drives Chromium--print-to-pdfover the canonical HTML and writes the PDF into~/.hermes/radar-ref/drafts/. Do not go back to ReportLab. - Model: the cover-letter generation default moved from the
cover-letter-writercombo todeepseek-paid-smarton 2026-10-26 10:47 (regenerate_cover_letter.py, env overrideCL_MODEL). - Dashboard timeout:
job_dashboard_server.pymust allowtimeout=600on the regenerate subprocess (set 2026-10-26 10:48, up from 120 s). One run is an LLM call + DOCX assembly + a Chromium print; it can exceed two minutes. The dashboard server was restarted to pick the change up. - All five artefacts (
.txt .json .html .docx .pdf) must still exist and be nonempty, and the HTML must contain no unresolved{{...}}tokens — the HTML is now the source of truth the PDF follows.
2026-10-27 07:00 slot — the same overrun, now recorded as failed (and the trigger for the split)
The 07:00 slot ran 07:01:08 → 11:03:54 (4 h 2 m 46 s) and closed failed on delivery
delivered. The report it sent opens "Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call
budget exhausted before Notion write stage)": discovery, verification, dedup and classification finished;
Notion writes, cover letters and the final audit did not run. Notion DB
0f90ba2b-8b10-4d02-a62c-6f692d5b1168 held 451 entries with 0 added for 2026-10-27 at run time.
Incident row dca848_e93a538ade6b opened 11:03:53 (alerted 11:03:59) and was still open at the
2026-10-28 audit. The split above is the response — see
incidents/2026-10-27-job-radar-budget-exhausted for the full ledger.
2026-10-26 07:00 slot — RUN INCOMPLETE, and why the backlog was finished by hand
The 07:00 run started on schedule and closed completed at 09:10, but the report it delivered read
“RUN INCOMPLETE — tool-call budget exhausted”: discovery and scoring consumed the run's tool budget, so
the Notion writes and the entire cover-letter stage never executed. The same-day backlog was then run
interactively (rows written for the outstanding strong matches, cover letters generated and verified).
- A
completedexecution row is not a completed radar run. Read the delivered text for theRUN INCOMPLETEmarker before treating a slot as done. A failed row with a delivered report is the honest version of the same fault — the 2026-10-27 slot is the first time it reached the incident ledger. - Budget the browser-heavy stages (one tab in flight, close before opening the next — see the 2026-10-12 hazard below) and keep headroom for the Notion + cover-letter writes, which sit last in the pipeline.
Two-stage split (changed 2026-10-27)
The single-job radar was split after two consecutive budget overruns (2026-10-26 completed but
RUN INCOMPLETE, 2026-10-27 closed failed — see
incidents/2026-10-27-job-radar-budget-exhausted). Discovery is the stage that eats the tool budget, so it
now owns a job of its own and the writes own a second one.
| Stage | Job id | Schedule | Writes | Delivers |
|---|---|---|---|---|
| 1 — Discovery | dca8482e4f76 (existing id) |
0 7 * * 1-5 |
a handoff file only | local; failures → topic 1030 |
| 2 — Write and Letters | 5fc292ff0f3b (new) |
30 11,14 * * 1-5 |
Notion entries, cover letters, final audit, report | Telegram topic 1030 |
- Stage 1 does
DISCOVERY → VERIFY → DEDUP → SCORE, writes the handoff file, and triggers stage 2. It must not write to Notion and must not generate cover letters. - Stage 2 lands the handoff: Notion entries, cover letters, the missing-letter audit and the report. Its prompt is not a copy of stage 1's — it reports what it wrote, not what it found.
- Stage 1 keeps the
custom:omni-101/smart-routepin. Both jobs still inherit the profile-levelfallback_providerschain; Hermes stores no per-job fallback list. - First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both
completedwith deliverysuppressed. The first real 07:00 → 11:30 cycle ran on 2026-10-28 and landed nothing — stage 1 never wrote its handoff (see the 2026-10-28 slot below). ✅ The second real cycle (2026-10-29) completed end to end — handoff written, consumed and delivered; see the 2026-10-29 slot below.
Recovery if stage 2 lands nothing: check that stage 1 actually wrote its handoff file before running stage 2
by hand (hermes cron run 5fc292ff0f3b). A stage-2 run with no handoff is a no-op, not a failed search — do not
re-run stage 1's discovery just to satisfy it.
2026-10-28 07:00 slot — stage 1 hit the iteration cap, so stage 2 had nothing to land
Stage 1 (dca8482e4f76) ran 07:00:32 → 10:33:18 (3 h 32 m 46 s) and closed failed on
"hit the agent iteration cap before the handoff could be written" — its second consecutive failure
(failure_streak: 2). No handoff file was written: ~/.hermes/radar-ref/handoff/ held only 2026-10-27.json
at the 2026-10-29 audit.
Both stage-2 slots then behaved exactly as designed — 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43, both
completed, both delivery_outcome: suppressed. A stage-2 run with no handoff is a silent no-op, not a
failure, which is the recovery rule below holding in production for the first time.
Read with 10-27, the picture is: the split stops the writes from being lost, but it does not make discovery finish. Two runs, two different ceilings — 10-27 stopped at the tool-call budget after 4 h 2 m (writes not started), 10-28 stopped at the agent iteration cap after 3 h 32 m, earlier still (no handoff at all). The discovery stage is the thing to bound, not the wall clock.
Full ledger: incidents/2026-10-28-job-radar-stage1-iteration-cap.
2026-10-29 07:00 slot — the split finally finished: handoff written, consumed, and delivered
Stage 1 (dca8482e4f76) ran 07:00:59 → 10:16:12 AWST (3 h 15 m 13 s) and closed completed — the first
post-split run to finish and write its handoff. ~/.hermes/radar-ref/handoff/2026-10-29.json (24.8 KB) records
run_date 2026-10-29T08:23:51+08:00, 170 discovered / 33 duplicates avoided / 37 verified, sources
seek ok, linkedin ok, indeed blocked, wa_gov ok. Both of the ledger rows the previous two days opened
(dca848_e93a538ade6b, dca848_0a2f03b5952b) closed resolved at 10:16:12, and this run opened no new row.
Stage 2 (5fc292ff0f3b) fired at 10:13:46, consumed the handoff at 10:40:59, closed 10:44:48 completed
and delivered to topic 1030 in the same second. Its stage2_result: 20 pages written / 20 page_ids /
2 cover letters / 2 same-day strong matches / verified_cover_letter_count 2 / missing_cover_letter_count 0 —
the Notion board went 455 → 475. The 11:30 and 14:30 slots then no-oped exactly as designed
(10 m 46 s and 1 m 51 s, handoff already consumed).
What this does and does not settle:
- The split's core promise holds in production — the writes can no longer be lost to a discovery overrun, and the first full cycle landed 20 pages and 2 cover letters with zero missing.
- It does not mean discovery is fixed. 3 h 15 m is still long, and the 10-27 tool-call budget and 10-28
iteration cap remain reachable. Treat a
completedstage-1 row with no handoff file as the same fault, whatever the execution status says. - Stage 1 no longer delivers (verified live:
deliver: local); its failure route to topic 1030 is unchanged.
Browser-layer hazard inside the run (2026-10-12)
This job is the most browser-heavy cron on the estate, and on 2026-10-12 (07:01 → 07:33) it drove
camofox-browser.service to its memory ceiling — 2.5G, swap fully consumed (available: 0B), 383 tasks —
after which REST /tabs and page evals hung in a way indistinguishable from ordinary tool timeouts. The
run recovered by restarting the service at 07:23:50 and finishing one tab at a time with an immediate
close, and it delivered normally.
Operational implications for this runbook:
- A green cron row is not proof the browser layer was healthy — check
systemctl --user status camofox-browser.service(peak / swap / task count) after any long run. - Extraction must be bounded: one tab in flight, close before opening the next. Multi-tab pre-fetch is what pushed it over.
- Limits live in
~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf(MemoryHigh=2G,MemoryMax=3G,MemorySwapMax=1G,OOMPolicy=kill). Raise them only if the host has real headroom — full detail in systems/browser-backend.