wiki audit 2026-11-07: Job Radar concurrent-fire claim discard (new incident page), scheduled-tasks re-verified section, index+log+runbook updates, freshness 2026-11-07

This commit is contained in:
hermes
2026-11-07 00:48:09 +08:00
parent 2d5219b068
commit b820993ecc
8 changed files with 442 additions and 17 deletions
+30 -4
View File
@@ -3,16 +3,42 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-06 updated: 2026-11-07
verified_on: 2026-11-06 verified_on: 2026-11-07
review_after: 2026-11-08 review_after: 2026-11-09
confidence: high confidence: high
tags: [system] tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
--- ---
# Current State # Current State
Updated: 2026-11-05 Updated: 2026-11-07
## Recent Changes (2026-11-07, nightly audit; live probes 2026-11-07 00:3x-00:5x AWST)
- **A red row with no alert: the Job Radar's 11:30 slot was discarded as stale while a concurrent fire
held the claim.** 2026-11-06 settled **172 rows across 16 jobs - 170 `completed` / 1 `failed` / 1
`running`** (this audit). The failure is `5fc292ff0f3b`'s **11:30:32 slot**, interrupted at **13:26:38**
and closed **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - because a **`direct`
fire of the same job (08:27:09 -> 13:26:22, 4 h 59 m 13 s)** was still holding the claim and delivered
first. **Every previous occurrence of this string came from a restart orphaning a run; this one had no
restart.** The failed row raised **no `cron_incidents` row** (newest ledger activity is still the
2026-11-05T12:36:17 closures), so **nothing alerted**. The day still landed complete via the 14:30 slot
(12 m 27 s, `MISSING_COVER_LETTER_COUNT 0`) ->
[[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]].
- **The stocktake's overrun is fully unwound.** `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)**
against 7 h 40 m / 5 h 41 m / 3 h 02 m on 11-03/11-04/11-05, and committed normally.
- **Stage 1 ran clean for a second day** (`dca8482e4f76`, 07:00:44 -> 08:29:21, 1 h 28 m 37 s, `completed`,
`suppressed`), and the **11-04 Open entry about its two lost candidate sets is cleared in substance** -
its 11-03 ledger row closed `resolved` at 2026-11-05T12:36:17 and both runs since have been clean.
- **Ledger clean, deliveries clean:** **unclosed incident rows 0 -> 0**; deliveries **10 rows / 10
`delivered` / 0 `for_failure`**; fleet **29 jobs (26 enabled / 3 disabled)** - fifth consecutive audit at
the same count, non-clean `last_status` still **1** (`066bb5191280`, `delivery_failed`, streak 0) ->
[[systems/scheduled-tasks]].
- **The previous audit's open question closed itself:** the Mnemosyne Health Monitor's 11-05 12:00:45 slot
(still in flight 12 h+ with no delivery at the last audit) **completed at 2026-11-06T00:14:39 after
12 h 13 m 54 s and delivered**; its 11-06 12:00 slot then took **7 h 47 m 56 s** and delivered too. The
**one still-open page is unchanged:** [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
## Recent Changes (2026-11-06, nightly audit; live probes 2026-11-06 00:1x-00:4x AWST) ## Recent Changes (2026-11-06, nightly audit; live probes 2026-11-06 00:1x-00:4x AWST)
@@ -0,0 +1,86 @@
---
title: Job Radar — a concurrent fire discarded the 11:30 slot's result
type: incident
status: open
created: 2026-11-07
updated: 2026-11-07
verified_on: 2026-11-07
review_after: 2026-11-14
confidence: high
tags: [cron, job-radar, fire-claim, concurrency, scheduler]
sources: [live-executions-db, live-deliveries-db, live-agent-log, live-cron-incidents, cron-output]
---
# Job Radar — a concurrent fire discarded the 11:30 slot's result (2026-11-06)
**Summary.** On **2026-11-06** the *Daily Marketing Job Radar — Write and Letters* (`5fc292ff0f3b`,
schedule `30 11,14 * * 1-5`) had **two live runs at the same time**. A `direct`-source fire started at
**08:27:09** and ran **4 h 59 m 13 s**, completing and delivering its report at **13:26:22**. The
scheduler's own **11:30:32** slot had already started and was still running; when the direct fire settled
the claim, the 11:30 run was judged **stale**, interrupted at **13:26:38**, and closed **`failed`** at
**13:26:44** with a **new** error string:
```
Fire claim ownership lost; stale result was discarded.
```
This is the **first occurrence of a claim-loss failure with no gateway restart involved**, and it raised
**no `cron_incidents` row** — the fleet's only red row for the day is invisible to the alerting ledger.
## Observed sequence (live `executions.db`, `deliveries.db`, `agent.log.1`)
| Time (AWST) | Event | Evidence |
|---|---|---|
| 03:00:32 -> 04:11:35 | wiki stocktake (`1a8e90ea2271`) completes in 1 h 11 m 03 s — overrun streak over | executions row |
| 07:00:44 -> 08:29:21 | Stage 1 (`dca8482e4f76`) completes (1 h 28 m 37 s, `suppressed`), second clean run running | executions row |
| **08:27:09 -> 13:26:22** | **`direct` fire of `5fc292ff0f3b`** — 4 h 59 m 13 s, `completed`, **delivered** | row `ad1fe896…`; log 08:27:13 *"Running job 'Daily Marketing Job Radar - Write and Letters'"* |
| 11:30:32 -> 13:26:44 | scheduler's **own 11:30 slot** fires and runs **concurrently** for 1 h 56 m 12 s | row `95347fa5…` |
| 13:26:22 | direct fire finishes and delivers its report | delivery row 13:26:22 |
| **13:26:38** | *"Job '5fc292ff0f3b': fire claim ownership lost; interrupting stale run"* | `agent.log.1` WARNING |
| **13:26:43** | *"Job '…' failed: RuntimeError: Cron job '…' lost its durable fire claim"* | `agent.log.1` ERROR |
| **13:26:44** | 11:30 row closes **`failed`** — `Fire claim ownership lost; stale result was discarded.` | executions row |
| 14:30:32 -> 14:42:59 | 14:30 slot runs a normal cycle (**12 m 27 s**) and delivers a complete report | row `e7462cf1…` |
**No `.md` output file** exists for the discarded 11:30 run — on 11-06 that job produced only
`2026-11-06_13-26-22.md` (the direct fire) and `2026-11-06_14-42-58.md` (the 14:30 slot).
## What is *not* affected
The day still landed in full. The 14:30 report (read from `cron/output/5fc292ff0f3b/2026-11-06_14-42-58.md`)
shows `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache
total **494**, 10 job rows dated 11-06, 20 artifacts present and non-empty, *Run status: COMPLETE*.
## Why it matters
1. **New cause for a known string.** Every earlier `lost its durable fire claim` case came from a
**restart or reload orphaning a live run** ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]],
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]), where the text was *"lost its durable
fire claim ownership"*. Here **no restart occurred** — the claim was lost to *another fire of the same
job*. The string is a **generic claim-loss signal, not a restart indicator**; and `executions.db`
currently holds **exactly one** row whose `error` contains `fire claim`: this one.
2. **The failure produced no alert.** No `cron_incidents` row has `last_seen_at` on or after 2026-11-06 —
the ledger's newest activity remains the 2026-11-05T12:36:17 closures. A `failed` execution with no
ledger row means **no notification path fired**; the run only surfaces as a red row in `executions.db`.
3. **~2 hours of a scheduled slot's work was discarded, and the *later, scheduler-owned* run lost** to an
earlier out-of-band run. The stale-discard rule did what it was written to do (protect against stale
results) — but the loser was the scheduler's own slot, so the guard silently ate planned work.
4. **The winner was out-of-band.** Its `source` is `direct`, not the scheduler's `builtin`, and it started
~2 minutes before Stage 1's handover run closed (08:29:21). The timing is consistent with
[[runbooks/job-radar]]'s Stage-1 handover trigger; the trigger's exact origin was **not directly verified**
in this audit.
## Open questions / candidate fixes (none applied)
- Should the scheduler **skip or suppress** a slot when a live claim for the same job already exists,
instead of launching a second concurrent run?
- Should an **out-of-band `direct` fire** (e.g. Stage 1's handover) mark the next scheduled slot consumed,
the way a consumed handoff file silences it?
- Should a **discarded-stale run raise an incident row** at all? Today the fault is silent by construction —
a `failed` execution with no ledger row and no alert.
## Related
- [[systems/scheduled-tasks]]
- [[runbooks/job-radar]]
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]
+8 -5
View File
File diff suppressed because one or more lines are too long
+42
View File
@@ -2184,3 +2184,45 @@ embedded credential was seen during verification and is not reproduced here ([RE
model. The 401 is an **alias-routing artefact, not an outage** — probe the alias the fleet pins, never the model. The 401 is an **alias-routing artefact, not an outage** — probe the alias the fleet pins, never the
raw model id. (A first probe with the wrong `key_env` also produced a 401; the correct key for raw model id. (A first probe with the wrong `key_env` also produced a 401; the correct key for
`omni-101` is `HERMES_CUSTOM_100_78_63_20_20128_API_KEY`.) `omni-101` is `HERMES_CUSTOM_100_78_63_20_20128_API_KEY`.)
## [2026-11-07] audit - window 2026-11-06
**Headline: a red row with no alert.** 2026-11-06 settled **172 execution rows across 16 distinct jobs - 170
`completed` / 1 `failed` / 1 `running`** (this audit). The single failure is a **new, non-restart** signature:
the Job Radar's **11:30 slot** (`5fc292ff0f3b`, row `95347fa5…`, 11:30:32 -> 13:26:44) was **discarded as
stale** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job
(08:27:09 -> 13:26:22, 4 h 59 m 13 s)** was still holding the claim and delivered first. Every previous
occurrence of this string came from a restart/reload orphaning a live run; **this one had no restart** - the
claim was lost to another fire of the same job. The failed row raised **no `cron_incidents` row** (newest
ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted**. New page ->
[[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]].
- **Ledger/day:** deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; fleet **29 jobs (26 enabled / 3
disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`), the
**fifth consecutive audit at the same count**, nothing added/retired/disabled/repinned, fleet `updated_at`
**2026-11-07T00:39:46+08:00**. Non-clean `last_status` unchanged at **1** (`066bb5191280`,
`delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0**; the one open page remains
[[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (**untouched in-window** - no
rotation, deletion or scrub).
- **Job Radar recovered the day in full:** the 14:30 slot ran a normal **12 m 27 s** cycle and delivered at
14:42:59 - `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`,
cache total **494**, 10 today-dated rows, 20 non-empty artifacts, *Run status: COMPLETE*.
- **The stocktake's overrun streak fully unwound:** `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m
03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Stage 1 (`dca8482e4f76`) ran
clean for a **second consecutive day** (07:00:44 -> 08:29:21, 1 h 28 m 37 s, `suppressed`), so the 11-04
Open entry about its two lost candidate sets is **cleared in substance** - the 11-03 ledger row closed
`resolved` at 2026-11-05T12:36:17 and both runs since have been clean.
- **The last audit's open question answered itself:** the Mnemosyne Health Monitor's 11-05 12:00:45 slot
(still in flight 12 h+ with no delivery at the last audit) **completed at 2026-11-06T00:14:39 after 12 h
13 m 54 s and delivered**; its 11-06 12:00 slot then took **7 h 47 m 56 s** and delivered too. Ballooned on
consecutive days, never lost, no incident row - **watching, not open.**
- **Sessions:** **15 cron sessions started in-window (495 messages / 273 tool calls)**, 16 sessions with
activity (incl. one `desktop` session, 146 messages). Busiest: `wiki-stocktake-maintenance` 157, Job Radar
Discovery 92, `Enhanced Weekend Adventure Planner` 82, Job Radar Write and Letters (14:30) 28, Mnemosyne
Health Monitor 27.
- **Wiki maintenance:** `wiki-maintain.sh` run **after** this audit's writes (freshness report
`raw/stocktake/freshness-20261107.txt`); skills-index regenerated; Quartz rebuilt. Pages touched:
[[systems/scheduled-tasks]] (+frontmatter, +Count bullet, +`Re-verified 2026-11-07` section),
[[current-state]] (+frontmatter, +Recent Changes block), this index (+frontmatter, +summary, +incident
bullet), [[runbooks/job-radar]] (+concurrency subsection), this log, and the new incident page. Vault
mirror `/home/hermes/wiki` updated (both conventions).
+141
View File
@@ -0,0 +1,141 @@
# Wiki freshness audit — 2026-11-07
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (125)
- [ 18d] CONTRIBUTING.md
- [ 12d] agents/hermes-chaos.md
- [ 12d] agents/hermes-production.md
- [ 18d] agents/index.md
- [ 9d] agents/nanobot.md
- [ 18d] agents/open-claw.md
- [ 3d] agents/qwenpaw.md
- [ 18d] agents/zeroclaw.md
- [ 9d] comparisons/camoufox-vs-chrome.md
- [ 18d] comparisons/hermes-vs-nanobot.md
- [ 18d] comparisons/memory-backends.md
- [ 18d] comparisons/model-routers.md
- [ 9d] comparisons/steel-vs-camoufox.md
- [ 18d] concepts/agent-memory.md
- [ 9d] concepts/browser-automation.md
- [ 18d] concepts/model-routing.md
- [ 18d] concepts/persistent-shells.md
- [ 18d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 9d] decisions/index.md
- [ 18d] decisions/keep-openrouter-as-fallback.md
- [ 18d] decisions/persistent-shell-enabled.md
- [ 9d] decisions/rollback-steel-to-camofox.md
- [ 18d] experiments/index.md
- [ 14d] incidents/2026-07-29-mnemosyne-relay.md
- [ 18d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 14d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 16d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 2d] incidents/2026-08-15-skills-index-drift.md
- [ 3d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 13d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 18d] incidents/2026-08-29-gmail-read-state.md
- [ 17d] incidents/2026-09-02-sftpgo-outage.md
- [ 3d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 9d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 12d] incidents/2026-10-03-ct450-hermes-update.md
- [ 12d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 9d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 14d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 6d] incidents/2026-10-09-memory-recall-relevance.md
- [ 1d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 11d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 19d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 11d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 16d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 13d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 12d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 12d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 12d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 10d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 8d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 8d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 8d] incidents/2026-10-29-conduit-voice-silence.md
- [ 7d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 8d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 7d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 6d] incidents/2026-10-31-asana-access-failures.md
- [ 6d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 6d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 5d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 4d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 3d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 1d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 1d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 2d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 1d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 2d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 0d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
- [ 0d] index.md
- [ 3d] infrastructure/backups.md
- [ 18d] infrastructure/credentials-map.md
- [ 1d] infrastructure/docker-services.md
- [ 18d] infrastructure/domains-and-tunnels.md
- [ 2d] infrastructure/hosts.md
- [ 1d] infrastructure/network-map.md
- [ 6d] infrastructure/proxmox.md
- [ 1d] infrastructure/tailscale.md
- [ 3d] queries/retained-research-answers.md
- [ 13d] reports/communities-wa-social-media-audit-2026.md
- [ 6d] reports/uber-eats/2026-07.md
- [ 6d] reports/uber-eats/2026-08.md
- [ 6d] reports/uber-eats/2026-09.md
- [ 6d] reports/uber-eats/2026-10.md
- [ 6d] reports/uber-eats/index.md
- [ 18d] runbooks/backup-wiki.md
- [ 18d] runbooks/diagnose-docker.md
- [ 18d] runbooks/gateway-resource-resilience.md
- [ 0d] runbooks/job-radar.md
- [ 18d] runbooks/multiplexer-setup.md
- [ 12d] runbooks/provider-health.md
- [ 9d] runbooks/recover-docker-service.md
- [ 18d] runbooks/restart-browser.md
- [ 18d] runbooks/restart-hermes.md
- [ 18d] runbooks/restore-openrouter.md
- [ 18d] runbooks/test-openrouter.md
- [ 18d] runbooks/troubleshoot-tool-loop.md
- [ 12d] runbooks/update-hermes-safely.md
- [ 4d] systems/browser-backend.md
- [ 18d] systems/cloudflare-tunnel.md
- [ 6d] systems/conduit.md
- [ 12d] systems/current-profile.md
- [ 18d] systems/guanaco.md
- [ 18d] systems/headless-server.md
- [ 8d] systems/hermes-agent.md
- [ 3d] systems/hermes-hq-push.md
- [ 18d] systems/hermes-ui.md
- [ 3d] systems/home-assistant.md
- [ 8d] systems/homepage-dashboard.md
- [ 3d] systems/ignis.md
- [ 12d] systems/memory-backend.md
- [ 5d] systems/messaging-integrations.md
- [ 1d] systems/mnemosyne.md
- [ 12d] systems/model-providers.md
- [ 3d] systems/nanobot.md
- [ 18d] systems/nextcloud.md
- [ 18d] systems/obsidian.md
- [ 6d] systems/omniroute.md
- [ 18d] systems/opencode-go.md
- [ 18d] systems/openrouter.md
- [ 3d] systems/paseo.md
- [ 19d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 18d] systems/terminal-backend.md
- [ 18d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 125 fresh, 3 no-date of 128 pages
+28 -3
View File
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery
type: runbook type: runbook
status: active status: active
created: 2026-07-27 created: 2026-07-27
updated: 2026-11-05 updated: 2026-11-07
review_after: 2026-11-13 review_after: 2026-11-14
verified_on: 2026-11-05 verified_on: 2026-11-07
version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2) version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2)
confidence: high confidence: high
tags: [automation, cron, jobs, notion, dashboard, runbook, cover-letter, model-route] tags: [automation, cron, jobs, notion, dashboard, runbook, cover-letter, model-route]
@@ -409,6 +409,31 @@ Operational implications for this runbook:
(`MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G`, `OOMPolicy=kill`). Raise them only if the host has (`MemoryHigh=2G`, `MemoryMax=3G`, `MemorySwapMax=1G`, `OOMPolicy=kill`). Raise them only if the host has
real headroom — full detail in [[systems/browser-backend]]. real headroom — full detail in [[systems/browser-backend]].
## Concurrency, and what the stale-discard guard actually protects (2026-11-06)
**Two live runs of `5fc292ff0f3b` can coexist — and the later one loses.** On 2026-11-06 a `direct`-source
fire (**08:27:09**, *not* the scheduler's `builtin` dispatch) ran **4 h 59 m 13 s** while the scheduler's own
**11:30:32** slot ran concurrently (**1 h 56 m 12 s**). When the direct fire settled the claim and delivered
at **13:26:22**, the 11:30 run was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on:
```
Fire claim ownership lost; stale result was discarded.
```
- **Read this string as *claim loss*, not *restart*.** Every earlier occurrence came from a restart or reload
orphaning a live run ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]],
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]). On 11-06 **no restart occurred** — the
claim was lost to another fire of the same job. **Check `executions.db` for a concurrent row before
blaming a restart**, and note `executions.db` holds exactly one row whose `error` contains `fire claim`.
- **A discarded run is silent.** It leaves **no `.md` output file** in `cron/output/5fc292ff0f3b/` and raises
**no `cron_incidents` row**, so the only trace is a red `executions.db` row — nothing announces it.
- **The day is still recoverable.** The next slot runs a normal cycle (12 m - 1 h); on 11-06 the 14:30 slot
took **12 m 27 s** and landed `MISSING_COVER_LETTER_COUNT 0`, *Run status: COMPLETE*.
- **Suspected trigger to check first:** a Stage-1 handover or a manual fire starting while a slot is already
in flight (timing on 11-06 was onset ~2 min before Stage 1's 08:29:21 handover close; the trigger's origin
was not directly verified). Deliberately re-firing the job while a slot is live is the way to reproduce.
- Full write-up: [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]].
## Related ## Related
- [[systems/scheduled-tasks]] - [[systems/scheduled-tasks]]
+105 -3
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-06 updated: 2026-11-07
verified_on: 2026-11-06 verified_on: 2026-11-07
review_after: 2026-11-07 review_after: 2026-11-08
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db] sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
@@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **Current as of 2026-11-07 (window 2026-11-06; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-07 00:3x-00:5x AWST):** **29 jobs (26 enabled / 3 disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; count and split unchanged across five consecutive audits, fleet `updated_at` **2026-11-07T00:39:46+08:00**). **11-06 settled: 172 rows, 16 distinct jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:31). The single `failed` row is a **new, non-restart failure signature**: the Job Radar's **11:30 Stage-2 slot** (`95347fa5…`) was **discarded as stale** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job, started 08:27:09, was still holding the claim** - that fire completed **13:26:22** and delivered, and the ledger records **no gateway restart** in the window -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; the radar still landed the day in full at 14:30 (`MISSING_COVER_LETTER_COUNT 0`, 4 same-day strong rows, 4 cover letters, 20 non-empty artifacts, *Run status: COMPLETE*). Fleet non-clean `last_status` unchanged at **1** (`066bb5191280`, `delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0** - no ledger row has `last_seen_at` on or after 2026-11-06 (newest activity remains the 2026-11-05T12:36:17 closures), so **the discarded run raised no incident row and no alert**; [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] remains the one open page. The stocktake's overrun streak **fully unwound**: `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Nothing added, retired, disabled or repinned.
- **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. - **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. ⚠ **Re-verified - **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. ⚠ **Re-verified
2026-11-06 (stocktake): on `omniroute-1` (`100.78.63.20`) the raw model id `deepseek/deepseek-v4-flash` 2026-11-06 (stocktake): on `omniroute-1` (`100.78.63.20`) the raw model id `deepseek/deepseek-v4-flash`
@@ -304,6 +305,10 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
(`failed` + incident `dca848_dfd4bea84f2a` + a delivered notice) → (`failed` + incident `dca848_dfd4bea84f2a` + a delivered notice) →
[[incidents/2026-11-03-job-radar-stream-active-timeout]]. **2026-11-02's and 2026-11-03's candidate sets are [[incidents/2026-11-03-job-radar-stream-active-timeout]]. **2026-11-02's and 2026-11-03's candidate sets are
both unrecovered.** both unrecovered.**
✅ **CLEARED in substance 2026-11-07:** the 11-03 ledger row closed `resolved` at **2026-11-05T12:36:17**, and
`dca8482e4f76` has run clean twice since - **5 h 35 m 35 s** on 11-05 and **1 h 28 m 37 s** on 11-06, both
`completed` / `suppressed`. The two unrecovered candidate sets stay lost; nothing in the ledger is red ->
[[systems/scheduled-tasks]].
- **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):** - **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):**
- **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed - **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed
`status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent `status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent
@@ -1053,3 +1058,100 @@ no-date of 127 pages** - the new incident page included. The three no-date pages
timestamps), and the vault mirror (`/home/hermes/wiki`, both conventions). timestamps), and the vault mirror (`/home/hermes/wiki`, both conventions).
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no - **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
decision reversed. decision reversed.
## Re-verified 2026-11-07 (window 2026-11-06, AWST; live reads 2026-11-07 00:3x-00:5x)
### The day
2026-11-06 closed **172 execution rows across 16 distinct jobs: 170 `completed` / 1 `failed` / 1
`running`** (this audit, 23:59:31 - so the running row is not lost work). Deliveries **10 rows / 10
`delivered` / 0 `for_failure`** (00:14:38, 07:06:34, 07:30:36, 08:29:16, 08:35:53, 12:20:23, 13:26:22,
14:42:58, 18:45:37, 19:48:39). Fleet **29 jobs (26 enabled / 3 disabled)** - unchanged for a fifth
consecutive audit; non-clean `last_status` still **1** (`066bb5191280`, `delivery_failed`, streak 0).
Unclosed incident rows **0 -> 0**: no `cron_incidents` row carries `last_seen_at >= 2026-11-06` and none
is outside `closed`/`resolved`, so the window's one `failed` row **raised no row in the ledger and no
alert** (see below).
The day ends a one-day zero-`failed` streak: 11-05 was the first clean ledger day since 11-02, and
11-06 breaks it again - but by a **new mechanism** with no restart and no upstream provider involvement.
### The window's only failure: a concurrent fire discarded the Job Radar's 11:30 slot
Full page: [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. In short:
- Stage 1 (`dca8482e4f76`) ran **07:00:44 -> 08:29:21 (1 h 28 m 37 s)**, `completed`, `suppressed` - its
second consecutive clean run.
- A **`direct`** fire of `5fc292ff0f3b` (Write and Letters) started **08:27:09** (`source: direct`, not the
scheduler's `builtin`) and ran **4 h 59 m 13 s**, completing and delivering at **13:26:22**.
- The scheduler's **own 11:30:32 slot** had started and was running concurrently; at **13:26:38** it logged
*"fire claim ownership lost; interrupting stale run"* and closed **13:26:44 `failed`** with
`Fire claim ownership lost; stale result was discarded.`
- The **14:30:32** slot then ran a normal cycle (**12 m 27 s**, delivered **14:42:59**) and the day landed in
full: `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache
total **494**, 10 today-dated rows, 20 artifacts present and non-empty, *Run status: COMPLETE*.
- **New signature meaning:** every earlier `lost its durable fire claim` case came from a restart/reload
orphaning a live run ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]],
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]). This one had **no restart** - the
claim was lost to another fire of the same job. `executions.db` holds **exactly one** row whose `error`
contains `fire claim`: this one.
### Long runs in-window (>90 m; normal cycles are 12 m - 1 h)
| Job | Slot | Duration | Outcome |
|---|---|---|---|
| `5fc292ff0f3b` (direct fire) | 08:27:09 -> 13:26:22 | 4 h 59 m 13 s | `completed`, delivered 13:26:22 |
| `5fc292ff0f3b` (11:30 slot) | 11:30:32 -> 13:26:44 | 1 h 56 m 12 s | **`failed`** - discarded as stale |
| `ea79a2107e36` (Mnemosyne Health Monitor) | 12:00:43 -> 19:48:39 | 7 h 47 m 56 s | `completed`, delivered 19:48:39 |
| `dca8482e4f76` (Stage 1) | 07:00:44 -> 08:29:21 | 1 h 28 m 37 s | `completed`, `suppressed` |
| `1a8e90ea2271` (stocktake) | 03:00:32 -> 04:11:35 | 1 h 11 m 03 s | `completed`, `suppressed` |
| `5fc292ff0f3b` (14:30 slot) | 14:30:32 -> 14:42:59 | 12 m 27 s | `completed`, delivered |
**The stocktake's overrun is over.** `1a8e90ea2271` ran **1 h 11 m 03 s** - the four-day streak reads
**7 h 40 m -> 5 h 41 m -> 3 h 02 m -> 1 h 11 m** for 11-03 -> 11-06, and it committed normally.
**The previous audit's open question is closed:** the Mnemosyne Health Monitor's 11-05 12:00:45 slot, which
was still in flight 12 h+ at the last audit with no delivery, **completed at 2026-11-06T00:14:39 after
12 h 13 m 54 s and delivered** (00:14:38). Its next 12:00 slot (11-06) then took **7 h 47 m 56 s** and also
delivered. Ballooned but never lost, and no incident row was ever raised for either.
### Sessions
**15 cron sessions started in the 11-06 window (495 messages / 273 tool calls)**, with **16** sessions
carrying activity (one `desktop` session, 146 messages). Busiest by message count:
`wiki-stocktake-maintenance` 157, `Daily Marketing Job Radar - Discovery` 92,
`Enhanced Weekend Adventure Planner` 82, `Daily Marketing Job Radar - Write and Letters` (14:30) 28,
`Mnemosyne Health Monitor` 27.
### Still open
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **still OPEN, untouched.** No
rotation, deletion or scrub happened in-window; not re-litigated by this audit.
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **NEW, OPEN.** The claim-discard
failure mode above. Candidate fixes are listed on the page; none applied.
- **Watching, not open:** the Mnemosyne Health Monitor's 12:00 slot has now ballooned on consecutive days
(12 h 13 m 54 s on 11-05, 7 h 47 m 56 s on 11-06) and still delivers. No incident row; no delivery loss.
### Freshness
`wiki-maintain.sh` (full run: skills-index regenerated -> freshness audit -> Quartz rebuild + serve restart)
completed at **2026-11-07T00:45:32+08:00**, run **after** this audit's writes:
- **Skills index:** `systems/skills-index.md` regenerated - **302 skills / 68 categories** (unchanged; the
wrapper de-dup fix holds).
- **Freshness: `0 stale (>=21d), 125 fresh, 3 no-date of 128 pages`** ->
`raw/stocktake/freshness-20261107.txt`. Nothing crossed the 21-day threshold. The page count is **+3** on
the 2026-11-05 stocktake's 125 (122 fresh / 3 no-date), consistent with the new pages the last two audits
and this one added - this audit contributes
`incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md`.
- **Site:** Quartz parsed **128** Markdown files and emitted **769** files (1 m 02 s); deployed to
`public/` and `hermes-wiki-serve.service` restarted. `public/` is generated output - excluded from the
freshness count by design.
### Files touched by this audit
- **New:** `incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md`,
`raw/stocktake/freshness-20261107.txt`.
- **Updated:** this page (+frontmatter, +Count bullet, +this section), `current-state.md`, `index.md`,
`log.md`, `runbooks/job-radar.md`, and the vault mirror (`/home/hermes/wiki`, both conventions).
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
decision reversed.
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-06 updated: 2026-11-07
verified_on: 2026-11-06 verified_on: 2026-11-07
confidence: high confidence: high
tags: [skills, hermes] tags: [skills, hermes]
sources: [] sources: []