diff --git a/current-state.md b/current-state.md index afd23db..13fd9e8 100644 --- a/current-state.md +++ b/current-state.md @@ -3,16 +3,42 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-11-06 -verified_on: 2026-11-06 -review_after: 2026-11-08 +updated: 2026-11-07 +verified_on: 2026-11-07 +review_after: 2026-11-09 confidence: high tags: [system] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db] --- # Current State -Updated: 2026-11-05 +Updated: 2026-11-07 + +## Recent Changes (2026-11-07, nightly audit; live probes 2026-11-07 00:3x-00:5x AWST) + +- **A red row with no alert: the Job Radar's 11:30 slot was discarded as stale while a concurrent fire + held the claim.** 2026-11-06 settled **172 rows across 16 jobs - 170 `completed` / 1 `failed` / 1 + `running`** (this audit). The failure is `5fc292ff0f3b`'s **11:30:32 slot**, interrupted at **13:26:38** + and closed **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - because a **`direct` + fire of the same job (08:27:09 -> 13:26:22, 4 h 59 m 13 s)** was still holding the claim and delivered + first. **Every previous occurrence of this string came from a restart orphaning a run; this one had no + restart.** The failed row raised **no `cron_incidents` row** (newest ledger activity is still the + 2026-11-05T12:36:17 closures), so **nothing alerted**. The day still landed complete via the 14:30 slot + (12 m 27 s, `MISSING_COVER_LETTER_COUNT 0`) -> + [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. +- **The stocktake's overrun is fully unwound.** `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)** + against 7 h 40 m / 5 h 41 m / 3 h 02 m on 11-03/11-04/11-05, and committed normally. +- **Stage 1 ran clean for a second day** (`dca8482e4f76`, 07:00:44 -> 08:29:21, 1 h 28 m 37 s, `completed`, + `suppressed`), and the **11-04 Open entry about its two lost candidate sets is cleared in substance** - + its 11-03 ledger row closed `resolved` at 2026-11-05T12:36:17 and both runs since have been clean. +- **Ledger clean, deliveries clean:** **unclosed incident rows 0 -> 0**; deliveries **10 rows / 10 + `delivered` / 0 `for_failure`**; fleet **29 jobs (26 enabled / 3 disabled)** - fifth consecutive audit at + the same count, non-clean `last_status` still **1** (`066bb5191280`, `delivery_failed`, streak 0) -> + [[systems/scheduled-tasks]]. +- **The previous audit's open question closed itself:** the Mnemosyne Health Monitor's 11-05 12:00:45 slot + (still in flight 12 h+ with no delivery at the last audit) **completed at 2026-11-06T00:14:39 after + 12 h 13 m 54 s and delivered**; its 11-06 12:00 slot then took **7 h 47 m 56 s** and delivered too. The + **one still-open page is unchanged:** [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. ## Recent Changes (2026-11-06, nightly audit; live probes 2026-11-06 00:1x-00:4x AWST) diff --git a/incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md b/incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md new file mode 100644 index 0000000..4d996bf --- /dev/null +++ b/incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md @@ -0,0 +1,86 @@ +--- +title: Job Radar — a concurrent fire discarded the 11:30 slot's result +type: incident +status: open +created: 2026-11-07 +updated: 2026-11-07 +verified_on: 2026-11-07 +review_after: 2026-11-14 +confidence: high +tags: [cron, job-radar, fire-claim, concurrency, scheduler] +sources: [live-executions-db, live-deliveries-db, live-agent-log, live-cron-incidents, cron-output] +--- + +# Job Radar — a concurrent fire discarded the 11:30 slot's result (2026-11-06) + +**Summary.** On **2026-11-06** the *Daily Marketing Job Radar — Write and Letters* (`5fc292ff0f3b`, +schedule `30 11,14 * * 1-5`) had **two live runs at the same time**. A `direct`-source fire started at +**08:27:09** and ran **4 h 59 m 13 s**, completing and delivering its report at **13:26:22**. The +scheduler's own **11:30:32** slot had already started and was still running; when the direct fire settled +the claim, the 11:30 run was judged **stale**, interrupted at **13:26:38**, and closed **`failed`** at +**13:26:44** with a **new** error string: + +``` +Fire claim ownership lost; stale result was discarded. +``` + +This is the **first occurrence of a claim-loss failure with no gateway restart involved**, and it raised +**no `cron_incidents` row** — the fleet's only red row for the day is invisible to the alerting ledger. + +## Observed sequence (live `executions.db`, `deliveries.db`, `agent.log.1`) + +| Time (AWST) | Event | Evidence | +|---|---|---| +| 03:00:32 -> 04:11:35 | wiki stocktake (`1a8e90ea2271`) completes in 1 h 11 m 03 s — overrun streak over | executions row | +| 07:00:44 -> 08:29:21 | Stage 1 (`dca8482e4f76`) completes (1 h 28 m 37 s, `suppressed`), second clean run running | executions row | +| **08:27:09 -> 13:26:22** | **`direct` fire of `5fc292ff0f3b`** — 4 h 59 m 13 s, `completed`, **delivered** | row `ad1fe896…`; log 08:27:13 *"Running job 'Daily Marketing Job Radar - Write and Letters'"* | +| 11:30:32 -> 13:26:44 | scheduler's **own 11:30 slot** fires and runs **concurrently** for 1 h 56 m 12 s | row `95347fa5…` | +| 13:26:22 | direct fire finishes and delivers its report | delivery row 13:26:22 | +| **13:26:38** | *"Job '5fc292ff0f3b': fire claim ownership lost; interrupting stale run"* | `agent.log.1` WARNING | +| **13:26:43** | *"Job '…' failed: RuntimeError: Cron job '…' lost its durable fire claim"* | `agent.log.1` ERROR | +| **13:26:44** | 11:30 row closes **`failed`** — `Fire claim ownership lost; stale result was discarded.` | executions row | +| 14:30:32 -> 14:42:59 | 14:30 slot runs a normal cycle (**12 m 27 s**) and delivers a complete report | row `e7462cf1…` | + +**No `.md` output file** exists for the discarded 11:30 run — on 11-06 that job produced only +`2026-11-06_13-26-22.md` (the direct fire) and `2026-11-06_14-42-58.md` (the 14:30 slot). + +## What is *not* affected + +The day still landed in full. The 14:30 report (read from `cron/output/5fc292ff0f3b/2026-11-06_14-42-58.md`) +shows `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache +total **494**, 10 job rows dated 11-06, 20 artifacts present and non-empty, *Run status: COMPLETE*. + +## Why it matters + +1. **New cause for a known string.** Every earlier `lost its durable fire claim` case came from a + **restart or reload orphaning a live run** ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]], + [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]), where the text was *"lost its durable + fire claim ownership"*. Here **no restart occurred** — the claim was lost to *another fire of the same + job*. The string is a **generic claim-loss signal, not a restart indicator**; and `executions.db` + currently holds **exactly one** row whose `error` contains `fire claim`: this one. +2. **The failure produced no alert.** No `cron_incidents` row has `last_seen_at` on or after 2026-11-06 — + the ledger's newest activity remains the 2026-11-05T12:36:17 closures. A `failed` execution with no + ledger row means **no notification path fired**; the run only surfaces as a red row in `executions.db`. +3. **~2 hours of a scheduled slot's work was discarded, and the *later, scheduler-owned* run lost** to an + earlier out-of-band run. The stale-discard rule did what it was written to do (protect against stale + results) — but the loser was the scheduler's own slot, so the guard silently ate planned work. +4. **The winner was out-of-band.** Its `source` is `direct`, not the scheduler's `builtin`, and it started + ~2 minutes before Stage 1's handover run closed (08:29:21). The timing is consistent with + [[runbooks/job-radar]]'s Stage-1 handover trigger; the trigger's exact origin was **not directly verified** + in this audit. + +## Open questions / candidate fixes (none applied) + +- Should the scheduler **skip or suppress** a slot when a live claim for the same job already exists, + instead of launching a second concurrent run? +- Should an **out-of-band `direct` fire** (e.g. Stage 1's handover) mark the next scheduled slot consumed, + the way a consumed handoff file silences it? +- Should a **discarded-stale run raise an incident row** at all? Today the fault is silent by construction — + a `failed` execution with no ledger row and no alert. + +## Related + +- [[systems/scheduled-tasks]] +- [[runbooks/job-radar]] +- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] +- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] diff --git a/index.md b/index.md index 1bc795d..dfd0f6c 100644 --- a/index.md +++ b/index.md @@ -3,19 +3,21 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-06 -verified_on: 2026-11-06 +updated: 2026-11-07 +verified_on: 2026-11-07 confidence: high -tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident] +tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar] sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] --- # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-11-06. +> model routing and automation. Last updated: 2026-11-07. > > **2026-11-05 (stocktake) -** **the `systems/skills-index.md` flip-flop is fixed at the root**: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> [[incidents/2026-11-05-skills-index-generator-divergence]]. Docker inventory corrected against live probes on all five hosts - ~~the `worldmonitor-seeder-scheduled` container recorded yesterday **does not exist on any host**~~ **⚠ CORRECTED 2026-11-06: false - the seeders are transient systemd-timer-driven containers, absent only between runs, not removed.** Tailscale node table re-verified programmatically - 52 rows, 52 live peers, ~~every state matching~~ **⚠ CORRECTED 2026-11-06: `anthonys-mac-mini` has since come back online.** Freshness **0 stale / 122 fresh / 3 no-date of 125**.) > +> **2026-11-07 (nightly audit - window 2026-11-06) - a red row with no alert:** 2026-11-06 settled **172 execution rows across 16 jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit). The single failure is a **new, non-restart signature**: the Job Radar's **11:30 slot** (`5fc292ff0f3b`, 11:30:32 -> 13:26:44) was **discarded as stale** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job (08:27:09 -> 13:26:22, 4 h 59 m 13 s)** was still holding the claim and delivered first. **Every earlier occurrence of this string came from a restart/reload orphaning a run; this one had no restart** -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. It raised **no `cron_incidents` row** (newest ledger activity remains the 2026-11-05T12:36:17 closures) and **nothing alerted**; the failed run left no output file. The day still landed in full via the 14:30 slot (**12 m 27 s**, `SAME_DAY_STRONG_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache **494**, *Run status: COMPLETE*). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; fleet **29 jobs (26 / 3)** - **fifth consecutive audit at the same count**, nothing added/retired/disabled/repinned, non-clean `last_status` still **1** (`066bb5191280`). **The stocktake's overrun fully unwound** (`1a8e90ea2271`, 03:00:32 -> 04:11:35 = **1 h 11 m 03 s** against 7 h 40 m / 5 h 41 m / 3 h 02 m on 11-03 -> 11-05) and **Stage 1 ran clean for a second day** (`dca8482e4f76`, 1 h 28 m 37 s, `suppressed`), clearing the 11-04 Open entry in substance. The last audit's open question closed itself: the **Mnemosyne Health Monitor**'s 11-05 12:00:45 slot (12 h+ in flight, undelivered at the last audit) **completed at 2026-11-06T00:14:39 after 12 h 13 m 54 s and delivered**; its 11-06 slot took **7 h 47 m 56 s** and delivered too - ballooned on consecutive days, never lost, **watching, not open**. The one open page remains [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (**untouched in-window**). +> > **2026-11-06 (nightly audit - window 2026-11-05) - a clean ledger day and an empty incident ledger:** 2026-11-05 settled **175 rows - 173 `completed` / 2 `running` / 0 `failed`** across 18 jobs, the **first zero-`failed` day since 2026-11-02**. The only two `running` rows are the **Mnemosyne Health Monitor** (12:00:45 slot, still in flight **12 h+** at audit time) and this audit itself. **Unclosed incident rows 3 -> 0** - the *Newsletter Digest* row closed at **07:07:42** and both *Job Radar - Discovery* rows at **12:36:17**, when `dca8482e4f76` finally completed after **5 h 35 m 35 s**; that closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, `delivery_failed`, streak 0); deliveries **10 rows / 10 `delivered`**; the 03:00 stocktake overran **3 h 01 m 58 s** (down from 7 h 40 m and 5 h 41 m) and **committed `2f3fc99` at 05:55**. **Fleet unchanged at 29 jobs (26 / 3)** - a fourth audit at the same count. ⚠ **New and open:** live `.env` values (`TELEGRAM_BOT_TOKEN`, `MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit verbatim in **17 rows of `mnemosyne.db`** (`memories`, `episodic_memory`, `working_memory`, `gists`, `memoria_facts.context_snippet` and the FTS mirrors) - reproduced in full by each of the **7 uncompressed rotation snapshots** - and all four values sit verbatim in the **1.7 GB session store `state.db`** too; `CAMOFOX_API_KEY` sits verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and therefore in `cron/deliveries.db` -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. > > **2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze:** the **11-04 fleet settled 170 rows - 167 `completed` / 2 `failed` / 1 `running`**, **non-clean `last_status` 2 -> 3**, **unclosed incident rows 1 -> 3**, **deliveries 8 rows / 8 `delivered`**. Both failures are the *same* upstream fault: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) died **36 seconds apart at 09:41-09:42**, each after **2 h 41 m**, on `RuntimeError: Upstream stream failed before completion.` - both pinned `custom:omni-101` / `smart-route`. **The day was not lost:** the 11:30 Stage 2 slot **reconstructed the missing handoff from Stage 1's own candidate files**, re-verified every candidate live, and still wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. The cover-letter route survived only on an override: `regenerate_cover_letter.py` defaults to `deepseek-paid-smart`, which threw **HTTP 402** mid-run, so the letter was pinned to `gemini/gemini-3.5-flash` via `CL_MODEL` - and that override **was never written to `.env`**, so it did not persist. And **CT299 was frozen twice, not once**: a second, **manual** `vzdump 299 ... --storage ssd0` wedged from **11-04 08:30:04** until **11-05 ~00:30** (~16 h); the live probe at 00:54:21 AWST shows `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. ⚠ **Corrected the same night:** the first draft of this entry read the fleet as **158 jobs (144 / 14)** and the @@ -60,9 +62,10 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. - [[systems/hermes-hq-push]] — **Hermes HQ iPhone app plugin** (standalone, **v0.9.27**, `kind: standalone`, author *Hermes HQ*; tree mtime **2026-11-03 11:06**). `hermes plugins list` → **enabled** (state `user`); `config.yaml` `plugins.enabled` line **7556**. Host side of the Hermes HQ app: Apple push (finished turns, lock-screen approvals, "waiting for a password/login/code" alerts, **live Lock Screen cards** sealed to the phone), phone **device access** (contacts, location, reminders, alarms, photos, Health, motion, music), the desktop-app **Hermes HQ theme**, the app's **Terminal page**, fast **Sessions list** on a busy host, cross-posting answers back to Telegram/Discord, a **restart guard**, and **Grok Live** voice-token minting (`XAI_API_KEY` or subscription sign-in; bearer never reaches the phone). Tools `device_contacts_search` / `device_request_result`; 7 hooks. Its manifest states it **replaces `dispatch-push`** — and the replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory remains. **Different stack** from the Cadu/Vory bridge on [[systems/push-notifications]]. ⚠ Its `pre_tool_call` guard also **blocks any tool call whose arguments contain the literal text of a launchd one-off-restart command** — describe it, never quote it. - [[systems/scheduled-tasks]] — **As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and `ea79a2107e36` is ENABLED again (`last_status: ok`, streak 0) after its 00:01:08 slot closed `completed` at 10:02:48; deliveries 8/8 `delivered`.** **Prior window — as of 2026-10-30 03:3x:** 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** -- [[systems/skills-index]] — Installed skills inventory (**302 skills** (**68 categories**) as of the **2026-11-05** stocktake; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (46 archived skills) — the earlier 346/347 counts included it. Freshness at the 2026-11-05 run: **0 stale / 122 fresh / 3 no-date of 125 pages** (`freshness-20261105.txt`). Note: `SKILL.md` files on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so **302 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. +- [[systems/skills-index]] — Installed skills inventory (**302 skills** (**68 categories**) as of the **2026-11-05** stocktake; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (46 archived skills) — the earlier 346/347 counts included it. Freshness at the **2026-11-07** run: **0 stale / 125 fresh / 3 no-date of 128 pages** (`freshness-20261107.txt`). Note: `SKILL.md` files on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so **302 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents +- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **OPEN.** A `direct`-source fire of `5fc292ff0f3b` (08:27:09 -> 13:26:22, 4 h 59 m 13 s, delivered) settled the claim while the scheduler's **own 11:30:32 slot** was still running, so that slot was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - a **new string variant** and the first claim-loss failure **with no gateway restart involved**. It raised **no `cron_incidents` row**, so a `failed` execution with no alert was the day's only red row; the discarded run left no output file. The 14:30 slot recovered the day in a normal **12 m 27 s** cycle. Candidate fixes (slot suppression when a claim is live, consuming the next slot for out-of-band fires, raising an incident row on discard) are listed on the page and **none was applied**. - [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **OPEN.** Live `.env` values were found stored verbatim in **17 rows of `mnemosyne.db`** (`memories` 4, `episodic_memory` 3, `working_memory` 1, `gists` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the **7 uncompressed rotation snapshots**, and all four values also sit verbatim in the **1.7 GB session store `state.db`**, which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Nothing rotated, deleted or scrubbed; no value recorded. - [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter). - [[incidents/2026-11-05-skills-index-generator-divergence]] — **RESOLVED — the skills index flip-flop had two writers.** The repo `wiki-maintain.sh` (302 skills / 68 categories) and the stale `~/.hermes/scripts/wiki-maintain.sh` copy (349 / 26, because it used `Path.rglob()` and counted the parked `.archive/`) were **different scripts**, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with **no error raised by either**. The stale copy is now a **delegating wrapper**; the repo script gained `--no-build` so the wrapper honours the old interface without a second generator. diff --git a/log.md b/log.md index c5b6e05..b1dc1d9 100644 --- a/log.md +++ b/log.md @@ -2184,3 +2184,45 @@ embedded credential was seen during verification and is not reproduced here ([RE model. The 401 is an **alias-routing artefact, not an outage** — probe the alias the fleet pins, never the raw model id. (A first probe with the wrong `key_env` also produced a 401; the correct key for `omni-101` is `HERMES_CUSTOM_100_78_63_20_20128_API_KEY`.) + +## [2026-11-07] audit - window 2026-11-06 + +**Headline: a red row with no alert.** 2026-11-06 settled **172 execution rows across 16 distinct jobs - 170 +`completed` / 1 `failed` / 1 `running`** (this audit). The single failure is a **new, non-restart** signature: +the Job Radar's **11:30 slot** (`5fc292ff0f3b`, row `95347fa5…`, 11:30:32 -> 13:26:44) was **discarded as +stale** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job +(08:27:09 -> 13:26:22, 4 h 59 m 13 s)** was still holding the claim and delivered first. Every previous +occurrence of this string came from a restart/reload orphaning a live run; **this one had no restart** - the +claim was lost to another fire of the same job. The failed row raised **no `cron_incidents` row** (newest +ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted**. New page -> +[[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. + +- **Ledger/day:** deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; fleet **29 jobs (26 enabled / 3 + disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`), the + **fifth consecutive audit at the same count**, nothing added/retired/disabled/repinned, fleet `updated_at` + **2026-11-07T00:39:46+08:00**. Non-clean `last_status` unchanged at **1** (`066bb5191280`, + `delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0**; the one open page remains + [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (**untouched in-window** - no + rotation, deletion or scrub). +- **Job Radar recovered the day in full:** the 14:30 slot ran a normal **12 m 27 s** cycle and delivered at + 14:42:59 - `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, + cache total **494**, 10 today-dated rows, 20 non-empty artifacts, *Run status: COMPLETE*. +- **The stocktake's overrun streak fully unwound:** `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m + 03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Stage 1 (`dca8482e4f76`) ran + clean for a **second consecutive day** (07:00:44 -> 08:29:21, 1 h 28 m 37 s, `suppressed`), so the 11-04 + Open entry about its two lost candidate sets is **cleared in substance** - the 11-03 ledger row closed + `resolved` at 2026-11-05T12:36:17 and both runs since have been clean. +- **The last audit's open question answered itself:** the Mnemosyne Health Monitor's 11-05 12:00:45 slot + (still in flight 12 h+ with no delivery at the last audit) **completed at 2026-11-06T00:14:39 after 12 h + 13 m 54 s and delivered**; its 11-06 12:00 slot then took **7 h 47 m 56 s** and delivered too. Ballooned on + consecutive days, never lost, no incident row - **watching, not open.** +- **Sessions:** **15 cron sessions started in-window (495 messages / 273 tool calls)**, 16 sessions with + activity (incl. one `desktop` session, 146 messages). Busiest: `wiki-stocktake-maintenance` 157, Job Radar + Discovery 92, `Enhanced Weekend Adventure Planner` 82, Job Radar Write and Letters (14:30) 28, Mnemosyne + Health Monitor 27. +- **Wiki maintenance:** `wiki-maintain.sh` run **after** this audit's writes (freshness report + `raw/stocktake/freshness-20261107.txt`); skills-index regenerated; Quartz rebuilt. Pages touched: + [[systems/scheduled-tasks]] (+frontmatter, +Count bullet, +`Re-verified 2026-11-07` section), + [[current-state]] (+frontmatter, +Recent Changes block), this index (+frontmatter, +summary, +incident + bullet), [[runbooks/job-radar]] (+concurrency subsection), this log, and the new incident page. Vault + mirror `/home/hermes/wiki` updated (both conventions). diff --git a/raw/stocktake/freshness-20261107.txt b/raw/stocktake/freshness-20261107.txt new file mode 100644 index 0000000..6cd20fd --- /dev/null +++ b/raw/stocktake/freshness-20261107.txt @@ -0,0 +1,141 @@ +# Wiki freshness audit — 2026-11-07 +Threshold: >21 days since `updated` + +## STALE (0) + + +## FRESH (125) + +- [ 18d] CONTRIBUTING.md +- [ 12d] agents/hermes-chaos.md +- [ 12d] agents/hermes-production.md +- [ 18d] agents/index.md +- [ 9d] agents/nanobot.md +- [ 18d] agents/open-claw.md +- [ 3d] agents/qwenpaw.md +- [ 18d] agents/zeroclaw.md +- [ 9d] comparisons/camoufox-vs-chrome.md +- [ 18d] comparisons/hermes-vs-nanobot.md +- [ 18d] comparisons/memory-backends.md +- [ 18d] comparisons/model-routers.md +- [ 9d] comparisons/steel-vs-camoufox.md +- [ 18d] concepts/agent-memory.md +- [ 9d] concepts/browser-automation.md +- [ 18d] concepts/model-routing.md +- [ 18d] concepts/persistent-shells.md +- [ 18d] concepts/tool-calling.md +- [ 0d] current-state.md +- [ 9d] decisions/index.md +- [ 18d] decisions/keep-openrouter-as-fallback.md +- [ 18d] decisions/persistent-shell-enabled.md +- [ 9d] decisions/rollback-steel-to-camofox.md +- [ 18d] experiments/index.md +- [ 14d] incidents/2026-07-29-mnemosyne-relay.md +- [ 18d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 14d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 16d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 2d] incidents/2026-08-15-skills-index-drift.md +- [ 3d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 13d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 18d] incidents/2026-08-29-gmail-read-state.md +- [ 17d] incidents/2026-09-02-sftpgo-outage.md +- [ 3d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 9d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 12d] incidents/2026-10-03-ct450-hermes-update.md +- [ 12d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 9d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 14d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 6d] incidents/2026-10-09-memory-recall-relevance.md +- [ 1d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 11d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 19d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 11d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 16d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 13d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 12d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 12d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 12d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 10d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 8d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 8d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md +- [ 8d] incidents/2026-10-29-conduit-voice-silence.md +- [ 7d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +- [ 8d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md +- [ 7d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 6d] incidents/2026-10-31-asana-access-failures.md +- [ 6d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md +- [ 6d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md +- [ 5d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md +- [ 4d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md +- [ 3d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md +- [ 1d] incidents/2026-11-03-job-radar-stream-active-timeout.md +- [ 1d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md +- [ 2d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md +- [ 1d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md +- [ 2d] incidents/2026-11-05-skills-index-generator-divergence.md +- [ 0d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md +- [ 0d] index.md +- [ 3d] infrastructure/backups.md +- [ 18d] infrastructure/credentials-map.md +- [ 1d] infrastructure/docker-services.md +- [ 18d] infrastructure/domains-and-tunnels.md +- [ 2d] infrastructure/hosts.md +- [ 1d] infrastructure/network-map.md +- [ 6d] infrastructure/proxmox.md +- [ 1d] infrastructure/tailscale.md +- [ 3d] queries/retained-research-answers.md +- [ 13d] reports/communities-wa-social-media-audit-2026.md +- [ 6d] reports/uber-eats/2026-07.md +- [ 6d] reports/uber-eats/2026-08.md +- [ 6d] reports/uber-eats/2026-09.md +- [ 6d] reports/uber-eats/2026-10.md +- [ 6d] reports/uber-eats/index.md +- [ 18d] runbooks/backup-wiki.md +- [ 18d] runbooks/diagnose-docker.md +- [ 18d] runbooks/gateway-resource-resilience.md +- [ 0d] runbooks/job-radar.md +- [ 18d] runbooks/multiplexer-setup.md +- [ 12d] runbooks/provider-health.md +- [ 9d] runbooks/recover-docker-service.md +- [ 18d] runbooks/restart-browser.md +- [ 18d] runbooks/restart-hermes.md +- [ 18d] runbooks/restore-openrouter.md +- [ 18d] runbooks/test-openrouter.md +- [ 18d] runbooks/troubleshoot-tool-loop.md +- [ 12d] runbooks/update-hermes-safely.md +- [ 4d] systems/browser-backend.md +- [ 18d] systems/cloudflare-tunnel.md +- [ 6d] systems/conduit.md +- [ 12d] systems/current-profile.md +- [ 18d] systems/guanaco.md +- [ 18d] systems/headless-server.md +- [ 8d] systems/hermes-agent.md +- [ 3d] systems/hermes-hq-push.md +- [ 18d] systems/hermes-ui.md +- [ 3d] systems/home-assistant.md +- [ 8d] systems/homepage-dashboard.md +- [ 3d] systems/ignis.md +- [ 12d] systems/memory-backend.md +- [ 5d] systems/messaging-integrations.md +- [ 1d] systems/mnemosyne.md +- [ 12d] systems/model-providers.md +- [ 3d] systems/nanobot.md +- [ 18d] systems/nextcloud.md +- [ 18d] systems/obsidian.md +- [ 6d] systems/omniroute.md +- [ 18d] systems/opencode-go.md +- [ 18d] systems/openrouter.md +- [ 3d] systems/paseo.md +- [ 19d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 18d] systems/terminal-backend.md +- [ 18d] systems/tool-search.md + +## NO updated DATE (3) + +- SCHEMA.md +- experiments/active/example.md +- log.md + +TOTAL: 0 stale, 125 fresh, 3 no-date of 128 pages \ No newline at end of file diff --git a/runbooks/job-radar.md b/runbooks/job-radar.md index 332c75c..c3e4786 100644 --- a/runbooks/job-radar.md +++ b/runbooks/job-radar.md @@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery type: runbook status: active created: 2026-07-27 -updated: 2026-11-05 -review_after: 2026-11-13 -verified_on: 2026-11-05 +updated: 2026-11-07 +review_after: 2026-11-14 +verified_on: 2026-11-07 version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2) confidence: high tags: [automation, cron, jobs, notion, dashboard, runbook, cover-letter, model-route] @@ -409,6 +409,31 @@ Operational implications for this runbook: (`MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G`, `OOMPolicy=kill`). Raise them only if the host has real headroom — full detail in [[systems/browser-backend]]. +## Concurrency, and what the stale-discard guard actually protects (2026-11-06) + +**Two live runs of `5fc292ff0f3b` can coexist — and the later one loses.** On 2026-11-06 a `direct`-source +fire (**08:27:09**, *not* the scheduler's `builtin` dispatch) ran **4 h 59 m 13 s** while the scheduler's own +**11:30:32** slot ran concurrently (**1 h 56 m 12 s**). When the direct fire settled the claim and delivered +at **13:26:22**, the 11:30 run was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on: + +``` +Fire claim ownership lost; stale result was discarded. +``` + +- **Read this string as *claim loss*, not *restart*.** Every earlier occurrence came from a restart or reload + orphaning a live run ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]], + [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]). On 11-06 **no restart occurred** — the + claim was lost to another fire of the same job. **Check `executions.db` for a concurrent row before + blaming a restart**, and note `executions.db` holds exactly one row whose `error` contains `fire claim`. +- **A discarded run is silent.** It leaves **no `.md` output file** in `cron/output/5fc292ff0f3b/` and raises + **no `cron_incidents` row**, so the only trace is a red `executions.db` row — nothing announces it. +- **The day is still recoverable.** The next slot runs a normal cycle (12 m - 1 h); on 11-06 the 14:30 slot + took **12 m 27 s** and landed `MISSING_COVER_LETTER_COUNT 0`, *Run status: COMPLETE*. +- **Suspected trigger to check first:** a Stage-1 handover or a manual fire starting while a slot is already + in flight (timing on 11-06 was onset ~2 min before Stage 1's 08:29:21 handover close; the trigger's origin + was not directly verified). Deliberately re-firing the job while a slot is live is the way to reproduce. +- Full write-up: [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. + ## Related - [[systems/scheduled-tasks]] diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 47201cc..9009fe9 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-11-06 -verified_on: 2026-11-06 -review_after: 2026-11-07 +updated: 2026-11-07 +verified_on: 2026-11-07 +review_after: 2026-11-08 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db] @@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db # Scheduled Tasks ## Count +- **Current as of 2026-11-07 (window 2026-11-06; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-07 00:3x-00:5x AWST):** **29 jobs (26 enabled / 3 disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; count and split unchanged across five consecutive audits, fleet `updated_at` **2026-11-07T00:39:46+08:00**). **11-06 settled: 172 rows, 16 distinct jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:31). The single `failed` row is a **new, non-restart failure signature**: the Job Radar's **11:30 Stage-2 slot** (`95347fa5…`) was **discarded as stale** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job, started 08:27:09, was still holding the claim** - that fire completed **13:26:22** and delivered, and the ledger records **no gateway restart** in the window -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; the radar still landed the day in full at 14:30 (`MISSING_COVER_LETTER_COUNT 0`, 4 same-day strong rows, 4 cover letters, 20 non-empty artifacts, *Run status: COMPLETE*). Fleet non-clean `last_status` unchanged at **1** (`066bb5191280`, `delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0** - no ledger row has `last_seen_at` on or after 2026-11-06 (newest activity remains the 2026-11-05T12:36:17 closures), so **the discarded run raised no incident row and no alert**; [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] remains the one open page. The stocktake's overrun streak **fully unwound**: `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Nothing added, retired, disabled or repinned. - **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. - **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. ⚠ **Re-verified 2026-11-06 (stocktake): on `omniroute-1` (`100.78.63.20`) the raw model id `deepseek/deepseek-v4-flash` @@ -304,6 +305,10 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne (`failed` + incident `dca848_dfd4bea84f2a` + a delivered notice) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. **2026-11-02's and 2026-11-03's candidate sets are both unrecovered.** + ✅ **CLEARED in substance 2026-11-07:** the 11-03 ledger row closed `resolved` at **2026-11-05T12:36:17**, and + `dca8482e4f76` has run clean twice since - **5 h 35 m 35 s** on 11-05 and **1 h 28 m 37 s** on 11-06, both + `completed` / `suppressed`. The two unrecovered candidate sets stay lost; nothing in the ledger is red -> + [[systems/scheduled-tasks]]. - **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):** - **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed `status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent @@ -1053,3 +1058,100 @@ no-date of 127 pages** - the new incident page included. The three no-date pages timestamps), and the vault mirror (`/home/hermes/wiki`, both conventions). - **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no decision reversed. + +## Re-verified 2026-11-07 (window 2026-11-06, AWST; live reads 2026-11-07 00:3x-00:5x) + +### The day + +2026-11-06 closed **172 execution rows across 16 distinct jobs: 170 `completed` / 1 `failed` / 1 +`running`** (this audit, 23:59:31 - so the running row is not lost work). Deliveries **10 rows / 10 +`delivered` / 0 `for_failure`** (00:14:38, 07:06:34, 07:30:36, 08:29:16, 08:35:53, 12:20:23, 13:26:22, +14:42:58, 18:45:37, 19:48:39). Fleet **29 jobs (26 enabled / 3 disabled)** - unchanged for a fifth +consecutive audit; non-clean `last_status` still **1** (`066bb5191280`, `delivery_failed`, streak 0). +Unclosed incident rows **0 -> 0**: no `cron_incidents` row carries `last_seen_at >= 2026-11-06` and none +is outside `closed`/`resolved`, so the window's one `failed` row **raised no row in the ledger and no +alert** (see below). + +The day ends a one-day zero-`failed` streak: 11-05 was the first clean ledger day since 11-02, and +11-06 breaks it again - but by a **new mechanism** with no restart and no upstream provider involvement. + +### The window's only failure: a concurrent fire discarded the Job Radar's 11:30 slot + +Full page: [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. In short: + +- Stage 1 (`dca8482e4f76`) ran **07:00:44 -> 08:29:21 (1 h 28 m 37 s)**, `completed`, `suppressed` - its + second consecutive clean run. +- A **`direct`** fire of `5fc292ff0f3b` (Write and Letters) started **08:27:09** (`source: direct`, not the + scheduler's `builtin`) and ran **4 h 59 m 13 s**, completing and delivering at **13:26:22**. +- The scheduler's **own 11:30:32 slot** had started and was running concurrently; at **13:26:38** it logged + *"fire claim ownership lost; interrupting stale run"* and closed **13:26:44 `failed`** with + `Fire claim ownership lost; stale result was discarded.` +- The **14:30:32** slot then ran a normal cycle (**12 m 27 s**, delivered **14:42:59**) and the day landed in + full: `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache + total **494**, 10 today-dated rows, 20 artifacts present and non-empty, *Run status: COMPLETE*. +- **New signature meaning:** every earlier `lost its durable fire claim` case came from a restart/reload + orphaning a live run ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]], + [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]). This one had **no restart** - the + claim was lost to another fire of the same job. `executions.db` holds **exactly one** row whose `error` + contains `fire claim`: this one. + +### Long runs in-window (>90 m; normal cycles are 12 m - 1 h) + +| Job | Slot | Duration | Outcome | +|---|---|---|---| +| `5fc292ff0f3b` (direct fire) | 08:27:09 -> 13:26:22 | 4 h 59 m 13 s | `completed`, delivered 13:26:22 | +| `5fc292ff0f3b` (11:30 slot) | 11:30:32 -> 13:26:44 | 1 h 56 m 12 s | **`failed`** - discarded as stale | +| `ea79a2107e36` (Mnemosyne Health Monitor) | 12:00:43 -> 19:48:39 | 7 h 47 m 56 s | `completed`, delivered 19:48:39 | +| `dca8482e4f76` (Stage 1) | 07:00:44 -> 08:29:21 | 1 h 28 m 37 s | `completed`, `suppressed` | +| `1a8e90ea2271` (stocktake) | 03:00:32 -> 04:11:35 | 1 h 11 m 03 s | `completed`, `suppressed` | +| `5fc292ff0f3b` (14:30 slot) | 14:30:32 -> 14:42:59 | 12 m 27 s | `completed`, delivered | + +**The stocktake's overrun is over.** `1a8e90ea2271` ran **1 h 11 m 03 s** - the four-day streak reads +**7 h 40 m -> 5 h 41 m -> 3 h 02 m -> 1 h 11 m** for 11-03 -> 11-06, and it committed normally. + +**The previous audit's open question is closed:** the Mnemosyne Health Monitor's 11-05 12:00:45 slot, which +was still in flight 12 h+ at the last audit with no delivery, **completed at 2026-11-06T00:14:39 after +12 h 13 m 54 s and delivered** (00:14:38). Its next 12:00 slot (11-06) then took **7 h 47 m 56 s** and also +delivered. Ballooned but never lost, and no incident row was ever raised for either. + +### Sessions + +**15 cron sessions started in the 11-06 window (495 messages / 273 tool calls)**, with **16** sessions +carrying activity (one `desktop` session, 146 messages). Busiest by message count: +`wiki-stocktake-maintenance` 157, `Daily Marketing Job Radar - Discovery` 92, +`Enhanced Weekend Adventure Planner` 82, `Daily Marketing Job Radar - Write and Letters` (14:30) 28, +`Mnemosyne Health Monitor` 27. + +### Still open + +- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **still OPEN, untouched.** No + rotation, deletion or scrub happened in-window; not re-litigated by this audit. +- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **NEW, OPEN.** The claim-discard + failure mode above. Candidate fixes are listed on the page; none applied. +- **Watching, not open:** the Mnemosyne Health Monitor's 12:00 slot has now ballooned on consecutive days + (12 h 13 m 54 s on 11-05, 7 h 47 m 56 s on 11-06) and still delivers. No incident row; no delivery loss. + +### Freshness + +`wiki-maintain.sh` (full run: skills-index regenerated -> freshness audit -> Quartz rebuild + serve restart) +completed at **2026-11-07T00:45:32+08:00**, run **after** this audit's writes: + +- **Skills index:** `systems/skills-index.md` regenerated - **302 skills / 68 categories** (unchanged; the + wrapper de-dup fix holds). +- **Freshness: `0 stale (>=21d), 125 fresh, 3 no-date of 128 pages`** -> + `raw/stocktake/freshness-20261107.txt`. Nothing crossed the 21-day threshold. The page count is **+3** on + the 2026-11-05 stocktake's 125 (122 fresh / 3 no-date), consistent with the new pages the last two audits + and this one added - this audit contributes + `incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md`. +- **Site:** Quartz parsed **128** Markdown files and emitted **769** files (1 m 02 s); deployed to + `public/` and `hermes-wiki-serve.service` restarted. `public/` is generated output - excluded from the + freshness count by design. + +### Files touched by this audit + +- **New:** `incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md`, + `raw/stocktake/freshness-20261107.txt`. +- **Updated:** this page (+frontmatter, +Count bullet, +this section), `current-state.md`, `index.md`, + `log.md`, `runbooks/job-radar.md`, and the vault mirror (`/home/hermes/wiki`, both conventions). +- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no + decision reversed. diff --git a/systems/skills-index.md b/systems/skills-index.md index c2c8b1e..ef14a0c 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-11-06 -verified_on: 2026-11-06 +updated: 2026-11-07 +verified_on: 2026-11-07 confidence: high tags: [skills, hermes] sources: []