wiki audit 2026-11-07: Job Radar concurrent-fire claim discard (new incident page), scheduled-tasks re-verified section, index+log+runbook updates, freshness 2026-11-07
This commit is contained in:
@@ -2184,3 +2184,45 @@ embedded credential was seen during verification and is not reproduced here ([RE
|
||||
model. The 401 is an **alias-routing artefact, not an outage** — probe the alias the fleet pins, never the
|
||||
raw model id. (A first probe with the wrong `key_env` also produced a 401; the correct key for
|
||||
`omni-101` is `HERMES_CUSTOM_100_78_63_20_20128_API_KEY`.)
|
||||
|
||||
## [2026-11-07] audit - window 2026-11-06
|
||||
|
||||
**Headline: a red row with no alert.** 2026-11-06 settled **172 execution rows across 16 distinct jobs - 170
|
||||
`completed` / 1 `failed` / 1 `running`** (this audit). The single failure is a **new, non-restart** signature:
|
||||
the Job Radar's **11:30 slot** (`5fc292ff0f3b`, row `95347fa5…`, 11:30:32 -> 13:26:44) was **discarded as
|
||||
stale** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job
|
||||
(08:27:09 -> 13:26:22, 4 h 59 m 13 s)** was still holding the claim and delivered first. Every previous
|
||||
occurrence of this string came from a restart/reload orphaning a live run; **this one had no restart** - the
|
||||
claim was lost to another fire of the same job. The failed row raised **no `cron_incidents` row** (newest
|
||||
ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted**. New page ->
|
||||
[[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]].
|
||||
|
||||
- **Ledger/day:** deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; fleet **29 jobs (26 enabled / 3
|
||||
disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`), the
|
||||
**fifth consecutive audit at the same count**, nothing added/retired/disabled/repinned, fleet `updated_at`
|
||||
**2026-11-07T00:39:46+08:00**. Non-clean `last_status` unchanged at **1** (`066bb5191280`,
|
||||
`delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0**; the one open page remains
|
||||
[[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (**untouched in-window** - no
|
||||
rotation, deletion or scrub).
|
||||
- **Job Radar recovered the day in full:** the 14:30 slot ran a normal **12 m 27 s** cycle and delivered at
|
||||
14:42:59 - `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`,
|
||||
cache total **494**, 10 today-dated rows, 20 non-empty artifacts, *Run status: COMPLETE*.
|
||||
- **The stocktake's overrun streak fully unwound:** `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m
|
||||
03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Stage 1 (`dca8482e4f76`) ran
|
||||
clean for a **second consecutive day** (07:00:44 -> 08:29:21, 1 h 28 m 37 s, `suppressed`), so the 11-04
|
||||
Open entry about its two lost candidate sets is **cleared in substance** - the 11-03 ledger row closed
|
||||
`resolved` at 2026-11-05T12:36:17 and both runs since have been clean.
|
||||
- **The last audit's open question answered itself:** the Mnemosyne Health Monitor's 11-05 12:00:45 slot
|
||||
(still in flight 12 h+ with no delivery at the last audit) **completed at 2026-11-06T00:14:39 after 12 h
|
||||
13 m 54 s and delivered**; its 11-06 12:00 slot then took **7 h 47 m 56 s** and delivered too. Ballooned on
|
||||
consecutive days, never lost, no incident row - **watching, not open.**
|
||||
- **Sessions:** **15 cron sessions started in-window (495 messages / 273 tool calls)**, 16 sessions with
|
||||
activity (incl. one `desktop` session, 146 messages). Busiest: `wiki-stocktake-maintenance` 157, Job Radar
|
||||
Discovery 92, `Enhanced Weekend Adventure Planner` 82, Job Radar Write and Letters (14:30) 28, Mnemosyne
|
||||
Health Monitor 27.
|
||||
- **Wiki maintenance:** `wiki-maintain.sh` run **after** this audit's writes (freshness report
|
||||
`raw/stocktake/freshness-20261107.txt`); skills-index regenerated; Quartz rebuilt. Pages touched:
|
||||
[[systems/scheduled-tasks]] (+frontmatter, +Count bullet, +`Re-verified 2026-11-07` section),
|
||||
[[current-state]] (+frontmatter, +Recent Changes block), this index (+frontmatter, +summary, +incident
|
||||
bullet), [[runbooks/job-radar]] (+concurrency subsection), this log, and the new incident page. Vault
|
||||
mirror `/home/hermes/wiki` updated (both conventions).
|
||||
|
||||
Reference in New Issue
Block a user