audit 2026-11-06 (window 2026-11-05): incident ledger emptied 3 -> 0, first zero-failed day since 11-02, both 11-04 casualties recovered; NEW OPEN incident - live .env values verbatim in 17 mnemosyne rows, the 1.7 GB session store and two enabled Facebook Birthday prompts; freshness 0 stale / 124 fresh / 3 no-date of 127
This commit is contained in:
@@ -3,17 +3,18 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
review_after: 2026-11-06
|
||||
updated: 2026-11-06
|
||||
verified_on: 2026-11-06
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
||||
---
|
||||
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
|
||||
- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. -> [[runbooks/job-radar]] `CL_MODEL` caveat.
|
||||
- **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
`deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` /
|
||||
@@ -963,3 +964,87 @@ unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69
|
||||
whether the five-compaction churn contributes; whether a stream timeout on a file-deliverable job should
|
||||
**retry automatically**; `radar-ref/handoff/` retention (newest file 10-30); why Stage 2 still has no
|
||||
handoff to wait on; and why the 03:00 stocktake takes **7 h 40 m** and commits nothing.
|
||||
|
||||
## Re-verified 2026-11-06 (window 2026-11-05, AWST; live reads 2026-11-06 00:1x-00:4x)
|
||||
|
||||
### The day was a recovery day, not a repeat
|
||||
|
||||
- **Both 11-04 casualties ran clean.** `85edf6772bca` (*Tech & AI Newsletter Digest (Analytical)*, streak 1)
|
||||
completed its 07:00 run in **7 m 01 s** (07:00:42 -> 07:07:43); its incident row closed at **07:07:42**.
|
||||
`dca8482e4f76` (*Daily Marketing Job Radar - Discovery*, streak 2, dead twice on 11-04) also completed its
|
||||
07:00 run - but took **5 h 35 m 35 s** (07:00:42 -> 12:36:18), and that duration is what held its two
|
||||
incident rows open until **12:36:17**. Neither job re-raised `RuntimeError: Upstream stream failed before
|
||||
completion.`
|
||||
- **No `unknown`, `timeout` or `error` row exists anywhere in the 11-05 ledger** - the first such day since
|
||||
11-02.
|
||||
- **Deliveries: 10 rows, all `delivered`, none flagged `for_failure`** - 07:02:11, 07:07:42, 07:30:39,
|
||||
08:06:00, 08:35:30, 10:41:20, 12:15:25, 18:45:49, 19:25:00, 23:27:09 AWST.
|
||||
|
||||
### Long runs (completed, not failures)
|
||||
|
||||
| Job | Slot | Duration | Against its norm |
|
||||
|---|---|---|---|
|
||||
| `ea79a2107e36` Mnemosyne Health Monitor | 00:00 | **7 h 01 m 28 s** (00:00:43 -> 07:02:11) | normal 17-31 m |
|
||||
| `1a8e90ea2271` wiki-stocktake-maintenance | 03:00 | **3 h 01 m 58 s** (03:00:32 -> 06:02:30) | 11-02 7 h 40 m, 11-03 5 h 41 m |
|
||||
| `dca8482e4f76` Job Radar - Discovery | 07:00 | **5 h 35 m 35 s** | recovered from 2 failures |
|
||||
| `5fc292ff0f3b` Job Radar - Stage 2 | 14:30 | **8 h 56 m 17 s** (14:30:53 -> 23:27:10) | delivered 23:27:09 |
|
||||
| `5fc292ff0f3b` Job Radar - Stage 2 | 11:30 | 10 m 33 s | normal |
|
||||
| `85edf6772bca` Newsletter Digest | 07:00 | 7 m 01 s | normal |
|
||||
|
||||
### The 03:00 stocktake overran a third day - and committed this time
|
||||
|
||||
`1a8e90ea2271` ran **03:00:32 -> 06:02:30 (3 h 01 m 58 s)**, so the streak now reads
|
||||
**7 h 40 m -> 5 h 41 m -> 3 h 02 m** (improving, still ~6x norm), and it **did commit** its artefacts
|
||||
(`2f3fc99`, 05:55) - against the 11-04 state where it "reported incomplete and committed nothing". The vault
|
||||
repo `/home/hermes/wiki` picked that up at 05:58 (`ee00b5c`). The nightly audit did **not** have to commit the
|
||||
skills-index on the stocktake's behalf this pass.
|
||||
|
||||
### Sessions in the window
|
||||
|
||||
**26 sessions had activity touching 2026-11-05** - 21 `cron`, 4 `subagent`, 1 `desktop`. The only long-form
|
||||
one is the morning **Mnemosyne Health Monitor** run (`local-b433c51e...`, `cron`, 17 messages, 07:02-07:50):
|
||||
it requested tool approvals and received denials. No configuration change came out of it, and it is
|
||||
consistent with that job's 7 h 01 m ballooning above.
|
||||
|
||||
### Still open (2026-11-06 - carries its predecessor forward)
|
||||
|
||||
Everything on the 2026-11-04 list stands unchanged and not re-probed this window (single-node no-failover
|
||||
topology; the GitHub Trending Radar delivery token; Camofox containment under multi-tab load; the Conduit
|
||||
health store with no retention; the unreconciled admission files; the unidentified `-15` signal source; the
|
||||
30 s worker-ack window vs cold-start; whether a stream-silence timeout should retry a file-deliverable job;
|
||||
`radar-ref/handoff/` retention). **New this window:**
|
||||
|
||||
- **Live secrets sit verbatim in Mnemosyne rows and in two cron prompts.** Three `.env` values (`TELEGRAM_BOT_TOKEN`,
|
||||
`MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) appear in `memories`, `episodic_memory`,
|
||||
`working_memory`, `gists` and `memoria_facts.context_snippet` plus the FTS mirrors - **17 rows**, 4 + 6 + 7 per
|
||||
key - and each of the **7 uncompressed rotation snapshots** reproduces the whole DB; all four values also sit
|
||||
verbatim in the **1.7 GB session store `state.db`**, the likely capture origin. The Camofox key is embedded in
|
||||
the prompts of `0e753e9720fd` and `fad5a86979d4` (both **enabled** Facebook Birthday jobs) and therefore in
|
||||
`cron/deliveries.db` - the only `config-backup/` DB carrying any tested value. **Found, not remediated** ->
|
||||
[[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
|
||||
- **`ea79a2107e36` ballooned twice out of two runs** (7 h 01 m from the 00:00 slot, then still running 12 h
|
||||
after its 12:00:45 slot) - against a 17-31 m normal. Cause unexplained; the 12:00 run had delivered nothing
|
||||
at audit time.
|
||||
- **`066bb5191280`** (Uber Eats Monthly Spend Report) is now the fleet's **only** non-clean `last_status`
|
||||
(`delivery_failed`, streak 0) - carried, not re-diagnosed this window.
|
||||
- **`deepseek-paid-smart` on `omniroute` (CT600) - carried, not re-probed this window.** 11-05 read: HTTP
|
||||
**503** `ALL_TARGETS_SKIPPED` on `omniroute:20128` vs HTTP **200** on `omniroute-1`
|
||||
(100.78.63.20:20128 -> `deepseek/deepseek-v4-flash`). This audit remains the fleet's only
|
||||
`deepseek-paid-smart` pin (provider `custom:omni-101`), which is why it still completes.
|
||||
|
||||
### Freshness
|
||||
|
||||
`raw/stocktake/freshness-20261106.txt` (re-run after this audit's writes): **0 stale (>=90d), 124 fresh, 3
|
||||
no-date of 127 pages** - the new incident page included. The three no-date pages are unchanged (`SCHEMA.md`,
|
||||
`experiments/active/example.md`, `log.md`).
|
||||
|
||||
### Files touched by this audit
|
||||
|
||||
- **New:** `incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md`,
|
||||
`raw/stocktake/freshness-20261106.txt`.
|
||||
- **Updated:** this page (+frontmatter, +Count bullet, +this section), `current-state.md`, `index.md`,
|
||||
`log.md`, `incidents/2026-11-03-job-radar-stream-active-timeout.md`,
|
||||
`incidents/2026-11-04-upstream-stream-failure-0700-cohort.md` (both -> `status: resolved` with closure
|
||||
timestamps), and the vault mirror (`/home/hermes/wiki`, both conventions).
|
||||
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
|
||||
decision reversed.
|
||||
|
||||
Reference in New Issue
Block a user