wiki audit 2026-11-07: Job Radar concurrent-fire claim discard (new incident page), scheduled-tasks re-verified section, index+log+runbook updates, freshness 2026-11-07

This commit is contained in:
hermes
2026-11-07 00:48:09 +08:00
parent 2d5219b068
commit b820993ecc
8 changed files with 442 additions and 17 deletions
+105 -3
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-11-06
verified_on: 2026-11-06
review_after: 2026-11-07
updated: 2026-11-07
verified_on: 2026-11-07
review_after: 2026-11-08
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
@@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
# Scheduled Tasks
## Count
- **Current as of 2026-11-07 (window 2026-11-06; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-07 00:3x-00:5x AWST):** **29 jobs (26 enabled / 3 disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; count and split unchanged across five consecutive audits, fleet `updated_at` **2026-11-07T00:39:46+08:00**). **11-06 settled: 172 rows, 16 distinct jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:31). The single `failed` row is a **new, non-restart failure signature**: the Job Radar's **11:30 Stage-2 slot** (`95347fa5…`) was **discarded as stale** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job, started 08:27:09, was still holding the claim** - that fire completed **13:26:22** and delivered, and the ledger records **no gateway restart** in the window -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; the radar still landed the day in full at 14:30 (`MISSING_COVER_LETTER_COUNT 0`, 4 same-day strong rows, 4 cover letters, 20 non-empty artifacts, *Run status: COMPLETE*). Fleet non-clean `last_status` unchanged at **1** (`066bb5191280`, `delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0** - no ledger row has `last_seen_at` on or after 2026-11-06 (newest activity remains the 2026-11-05T12:36:17 closures), so **the discarded run raised no incident row and no alert**; [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] remains the one open page. The stocktake's overrun streak **fully unwound**: `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Nothing added, retired, disabled or repinned.
- **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. ⚠ **Re-verified
2026-11-06 (stocktake): on `omniroute-1` (`100.78.63.20`) the raw model id `deepseek/deepseek-v4-flash`
@@ -304,6 +305,10 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
(`failed` + incident `dca848_dfd4bea84f2a` + a delivered notice) →
[[incidents/2026-11-03-job-radar-stream-active-timeout]]. **2026-11-02's and 2026-11-03's candidate sets are
both unrecovered.**
✅ **CLEARED in substance 2026-11-07:** the 11-03 ledger row closed `resolved` at **2026-11-05T12:36:17**, and
`dca8482e4f76` has run clean twice since - **5 h 35 m 35 s** on 11-05 and **1 h 28 m 37 s** on 11-06, both
`completed` / `suppressed`. The two unrecovered candidate sets stay lost; nothing in the ledger is red ->
[[systems/scheduled-tasks]].
- **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):**
- **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed
`status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent
@@ -1053,3 +1058,100 @@ no-date of 127 pages** - the new incident page included. The three no-date pages
timestamps), and the vault mirror (`/home/hermes/wiki`, both conventions).
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
decision reversed.
## Re-verified 2026-11-07 (window 2026-11-06, AWST; live reads 2026-11-07 00:3x-00:5x)
### The day
2026-11-06 closed **172 execution rows across 16 distinct jobs: 170 `completed` / 1 `failed` / 1
`running`** (this audit, 23:59:31 - so the running row is not lost work). Deliveries **10 rows / 10
`delivered` / 0 `for_failure`** (00:14:38, 07:06:34, 07:30:36, 08:29:16, 08:35:53, 12:20:23, 13:26:22,
14:42:58, 18:45:37, 19:48:39). Fleet **29 jobs (26 enabled / 3 disabled)** - unchanged for a fifth
consecutive audit; non-clean `last_status` still **1** (`066bb5191280`, `delivery_failed`, streak 0).
Unclosed incident rows **0 -> 0**: no `cron_incidents` row carries `last_seen_at >= 2026-11-06` and none
is outside `closed`/`resolved`, so the window's one `failed` row **raised no row in the ledger and no
alert** (see below).
The day ends a one-day zero-`failed` streak: 11-05 was the first clean ledger day since 11-02, and
11-06 breaks it again - but by a **new mechanism** with no restart and no upstream provider involvement.
### The window's only failure: a concurrent fire discarded the Job Radar's 11:30 slot
Full page: [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. In short:
- Stage 1 (`dca8482e4f76`) ran **07:00:44 -> 08:29:21 (1 h 28 m 37 s)**, `completed`, `suppressed` - its
second consecutive clean run.
- A **`direct`** fire of `5fc292ff0f3b` (Write and Letters) started **08:27:09** (`source: direct`, not the
scheduler's `builtin`) and ran **4 h 59 m 13 s**, completing and delivering at **13:26:22**.
- The scheduler's **own 11:30:32 slot** had started and was running concurrently; at **13:26:38** it logged
*"fire claim ownership lost; interrupting stale run"* and closed **13:26:44 `failed`** with
`Fire claim ownership lost; stale result was discarded.`
- The **14:30:32** slot then ran a normal cycle (**12 m 27 s**, delivered **14:42:59**) and the day landed in
full: `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache
total **494**, 10 today-dated rows, 20 artifacts present and non-empty, *Run status: COMPLETE*.
- **New signature meaning:** every earlier `lost its durable fire claim` case came from a restart/reload
orphaning a live run ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]],
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]). This one had **no restart** - the
claim was lost to another fire of the same job. `executions.db` holds **exactly one** row whose `error`
contains `fire claim`: this one.
### Long runs in-window (>90 m; normal cycles are 12 m - 1 h)
| Job | Slot | Duration | Outcome |
|---|---|---|---|
| `5fc292ff0f3b` (direct fire) | 08:27:09 -> 13:26:22 | 4 h 59 m 13 s | `completed`, delivered 13:26:22 |
| `5fc292ff0f3b` (11:30 slot) | 11:30:32 -> 13:26:44 | 1 h 56 m 12 s | **`failed`** - discarded as stale |
| `ea79a2107e36` (Mnemosyne Health Monitor) | 12:00:43 -> 19:48:39 | 7 h 47 m 56 s | `completed`, delivered 19:48:39 |
| `dca8482e4f76` (Stage 1) | 07:00:44 -> 08:29:21 | 1 h 28 m 37 s | `completed`, `suppressed` |
| `1a8e90ea2271` (stocktake) | 03:00:32 -> 04:11:35 | 1 h 11 m 03 s | `completed`, `suppressed` |
| `5fc292ff0f3b` (14:30 slot) | 14:30:32 -> 14:42:59 | 12 m 27 s | `completed`, delivered |
**The stocktake's overrun is over.** `1a8e90ea2271` ran **1 h 11 m 03 s** - the four-day streak reads
**7 h 40 m -> 5 h 41 m -> 3 h 02 m -> 1 h 11 m** for 11-03 -> 11-06, and it committed normally.
**The previous audit's open question is closed:** the Mnemosyne Health Monitor's 11-05 12:00:45 slot, which
was still in flight 12 h+ at the last audit with no delivery, **completed at 2026-11-06T00:14:39 after
12 h 13 m 54 s and delivered** (00:14:38). Its next 12:00 slot (11-06) then took **7 h 47 m 56 s** and also
delivered. Ballooned but never lost, and no incident row was ever raised for either.
### Sessions
**15 cron sessions started in the 11-06 window (495 messages / 273 tool calls)**, with **16** sessions
carrying activity (one `desktop` session, 146 messages). Busiest by message count:
`wiki-stocktake-maintenance` 157, `Daily Marketing Job Radar - Discovery` 92,
`Enhanced Weekend Adventure Planner` 82, `Daily Marketing Job Radar - Write and Letters` (14:30) 28,
`Mnemosyne Health Monitor` 27.
### Still open
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **still OPEN, untouched.** No
rotation, deletion or scrub happened in-window; not re-litigated by this audit.
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **NEW, OPEN.** The claim-discard
failure mode above. Candidate fixes are listed on the page; none applied.
- **Watching, not open:** the Mnemosyne Health Monitor's 12:00 slot has now ballooned on consecutive days
(12 h 13 m 54 s on 11-05, 7 h 47 m 56 s on 11-06) and still delivers. No incident row; no delivery loss.
### Freshness
`wiki-maintain.sh` (full run: skills-index regenerated -> freshness audit -> Quartz rebuild + serve restart)
completed at **2026-11-07T00:45:32+08:00**, run **after** this audit's writes:
- **Skills index:** `systems/skills-index.md` regenerated - **302 skills / 68 categories** (unchanged; the
wrapper de-dup fix holds).
- **Freshness: `0 stale (>=21d), 125 fresh, 3 no-date of 128 pages`** ->
`raw/stocktake/freshness-20261107.txt`. Nothing crossed the 21-day threshold. The page count is **+3** on
the 2026-11-05 stocktake's 125 (122 fresh / 3 no-date), consistent with the new pages the last two audits
and this one added - this audit contributes
`incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md`.
- **Site:** Quartz parsed **128** Markdown files and emitted **769** files (1 m 02 s); deployed to
`public/` and `hermes-wiki-serve.service` restarted. `public/` is generated output - excluded from the
freshness count by design.
### Files touched by this audit
- **New:** `incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md`,
`raw/stocktake/freshness-20261107.txt`.
- **Updated:** this page (+frontmatter, +Count bullet, +this section), `current-state.md`, `index.md`,
`log.md`, `runbooks/job-radar.md`, and the vault mirror (`/home/hermes/wiki`, both conventions).
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
decision reversed.
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system
status: active
created: 2026-07-22
updated: 2026-11-06
verified_on: 2026-11-06
updated: 2026-11-07
verified_on: 2026-11-07
confidence: high
tags: [skills, hermes]
sources: []