wiki audit 2026-11-07: Job Radar concurrent-fire claim discard (new incident page), scheduled-tasks re-verified section, index+log+runbook updates, freshness 2026-11-07
This commit is contained in:
+28
-3
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-27
|
||||
updated: 2026-11-05
|
||||
review_after: 2026-11-13
|
||||
verified_on: 2026-11-05
|
||||
updated: 2026-11-07
|
||||
review_after: 2026-11-14
|
||||
verified_on: 2026-11-07
|
||||
version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2)
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, dashboard, runbook, cover-letter, model-route]
|
||||
@@ -409,6 +409,31 @@ Operational implications for this runbook:
|
||||
(`MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G`, `OOMPolicy=kill`). Raise them only if the host has
|
||||
real headroom — full detail in [[systems/browser-backend]].
|
||||
|
||||
## Concurrency, and what the stale-discard guard actually protects (2026-11-06)
|
||||
|
||||
**Two live runs of `5fc292ff0f3b` can coexist — and the later one loses.** On 2026-11-06 a `direct`-source
|
||||
fire (**08:27:09**, *not* the scheduler's `builtin` dispatch) ran **4 h 59 m 13 s** while the scheduler's own
|
||||
**11:30:32** slot ran concurrently (**1 h 56 m 12 s**). When the direct fire settled the claim and delivered
|
||||
at **13:26:22**, the 11:30 run was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on:
|
||||
|
||||
```
|
||||
Fire claim ownership lost; stale result was discarded.
|
||||
```
|
||||
|
||||
- **Read this string as *claim loss*, not *restart*.** Every earlier occurrence came from a restart or reload
|
||||
orphaning a live run ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]],
|
||||
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]). On 11-06 **no restart occurred** — the
|
||||
claim was lost to another fire of the same job. **Check `executions.db` for a concurrent row before
|
||||
blaming a restart**, and note `executions.db` holds exactly one row whose `error` contains `fire claim`.
|
||||
- **A discarded run is silent.** It leaves **no `.md` output file** in `cron/output/5fc292ff0f3b/` and raises
|
||||
**no `cron_incidents` row**, so the only trace is a red `executions.db` row — nothing announces it.
|
||||
- **The day is still recoverable.** The next slot runs a normal cycle (12 m - 1 h); on 11-06 the 14:30 slot
|
||||
took **12 m 27 s** and landed `MISSING_COVER_LETTER_COUNT 0`, *Run status: COMPLETE*.
|
||||
- **Suspected trigger to check first:** a Stage-1 handover or a manual fire starting while a slot is already
|
||||
in flight (timing on 11-06 was onset ~2 min before Stage 1's 08:29:21 handover close; the trigger's origin
|
||||
was not directly verified). Deliberately re-firing the job while a slot is live is the way to reproduce.
|
||||
- Full write-up: [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
|
||||
Reference in New Issue
Block a user