wiki audit 2026-11-07: Job Radar concurrent-fire claim discard (new incident page), scheduled-tasks re-verified section, index+log+runbook updates, freshness 2026-11-07

This commit is contained in:
hermes
2026-11-07 00:48:09 +08:00
parent 2d5219b068
commit b820993ecc
8 changed files with 442 additions and 17 deletions
+28 -3
View File
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery
type: runbook
status: active
created: 2026-07-27
updated: 2026-11-05
review_after: 2026-11-13
verified_on: 2026-11-05
updated: 2026-11-07
review_after: 2026-11-14
verified_on: 2026-11-07
version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2)
confidence: high
tags: [automation, cron, jobs, notion, dashboard, runbook, cover-letter, model-route]
@@ -409,6 +409,31 @@ Operational implications for this runbook:
(`MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G`, `OOMPolicy=kill`). Raise them only if the host has
real headroom — full detail in [[systems/browser-backend]].
## Concurrency, and what the stale-discard guard actually protects (2026-11-06)
**Two live runs of `5fc292ff0f3b` can coexist — and the later one loses.** On 2026-11-06 a `direct`-source
fire (**08:27:09**, *not* the scheduler's `builtin` dispatch) ran **4 h 59 m 13 s** while the scheduler's own
**11:30:32** slot ran concurrently (**1 h 56 m 12 s**). When the direct fire settled the claim and delivered
at **13:26:22**, the 11:30 run was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on:
```
Fire claim ownership lost; stale result was discarded.
```
- **Read this string as *claim loss*, not *restart*.** Every earlier occurrence came from a restart or reload
orphaning a live run ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]],
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]). On 11-06 **no restart occurred** — the
claim was lost to another fire of the same job. **Check `executions.db` for a concurrent row before
blaming a restart**, and note `executions.db` holds exactly one row whose `error` contains `fire claim`.
- **A discarded run is silent.** It leaves **no `.md` output file** in `cron/output/5fc292ff0f3b/` and raises
**no `cron_incidents` row**, so the only trace is a red `executions.db` row — nothing announces it.
- **The day is still recoverable.** The next slot runs a normal cycle (12 m - 1 h); on 11-06 the 14:30 slot
took **12 m 27 s** and landed `MISSING_COVER_LETTER_COUNT 0`, *Run status: COMPLETE*.
- **Suspected trigger to check first:** a Stage-1 handover or a manual fire starting while a slot is already
in flight (timing on 11-06 was onset ~2 min before Stage 1's 08:29:21 handover close; the trigger's origin
was not directly verified). Deliberately re-firing the job while a slot is live is the way to reproduce.
- Full write-up: [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]].
## Related
- [[systems/scheduled-tasks]]