audit 2026-11-06 (window 2026-11-05): incident ledger emptied 3 -> 0, first zero-failed day since 11-02, both 11-04 casualties recovered; NEW OPEN incident - live .env values verbatim in 17 mnemosyne rows, the 1.7 GB session store and two enabled Facebook Birthday prompts; freshness 0 stale / 124 fresh / 3 no-date of 127

This commit is contained in:
Tony0410
2026-11-06 00:29:52 +08:00
parent 2f3fc9952b
commit eba8062735
9 changed files with 460 additions and 22 deletions
+27 -4
View File
@@ -3,17 +3,40 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-05 updated: 2026-11-06
verified_on: 2026-11-05 verified_on: 2026-11-06
review_after: 2026-11-07 review_after: 2026-11-08
confidence: high confidence: high
tags: [system] tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
--- ---
# Current State # Current State
Updated: 2026-11-05 Updated: 2026-11-05
## Recent Changes (2026-11-06, nightly audit; live probes 2026-11-06 00:1x-00:4x AWST)
- **A clean ledger day and an empty incident ledger.** 2026-11-05 settled **175 rows - 173 `completed` / 2
`running` / 0 `failed`** across 18 jobs: the **first zero-`failed` day since 2026-11-02**. **Unclosed
incident rows 3 -> 0** - the *Tech & AI Newsletter Digest* row closed at **07:07:42** and both *Job Radar -
Discovery* rows at **12:36:17**, when that job finally completed after **5 h 35 m 35 s**. Fleet non-clean
`last_status` **3 -> 1** (only `066bb5191280`, `delivery_failed`). Deliveries **10 rows / 10 `delivered`**
-> [[systems/scheduled-tasks]].
- **The stocktake's three-day overrun streak broke, and it committed.** `1a8e90ea2271` ran **03:00:32 ->
06:02:30 (3 h 01 m 58 s)** - **7 h 40 m -> 5 h 41 m -> 3 h 02 m** over three days - and **committed
`2f3fc99` at 05:55**, against its 11-04 state of "reported incomplete, committed nothing".
- **Long runs, not failures:** the **Mnemosyne Health Monitor**'s 00:00 slot took **7 h 01 m 28 s** (normal
17-31 m) and its **12:00:45 slot was still running 12 h later with no delivery**; **Stage 2**'s 14:30 slot
took **8 h 56 m 17 s** and delivered at 23:27:09.
- **New open incident - live secrets stored verbatim.** Three `.env` values (`TELEGRAM_BOT_TOKEN`,
`MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit in plaintext in Mnemosyne rows - including
the FTS mirrors) - 4 + 6 + 7 = **17 rows** - reproduced in full by each of the **7 uncompressed rotation
snapshots**; all four values also sit verbatim in the **1.7 GB session store `state.db`**, the likely capture
origin. `CAMOFOX_API_KEY` sits verbatim in the prompts of two **enabled** Facebook Birthday jobs
(`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Found, not remediated; no value is recorded
anywhere -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
- **Fleet unchanged:** **29 jobs (26 enabled / 3 disabled)**, a fourth audit at the same count; no job added,
retired, disabled or repinned.
## Recent Changes (2026-11-05, stocktake; live probes 2026-11-05 05:1x-05:3x AWST) ## Recent Changes (2026-11-05, stocktake; live probes 2026-11-05 05:1x-05:3x AWST)
- **The `systems/skills-index.md` flip-flop is FIXED at the root - it had two writers, not one bad writer.** - **The `systems/skills-index.md` flip-flop is FIXED at the root - it had two writers, not one bad writer.**
@@ -1,10 +1,10 @@
--- ---
title: Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff title: Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
type: incident type: incident
status: open status: resolved
created: 2026-11-04 created: 2026-11-04
updated: 2026-11-04 updated: 2026-11-06
verified_on: 2026-11-04 verified_on: 2026-11-06
review_after: 2026-11-11 review_after: 2026-11-11
confidence: high confidence: high
tags: [cron, job-radar, stream-timeout, silent-failure, notion, context-compaction] tags: [cron, job-radar, stream-timeout, silent-failure, notion, context-compaction]
@@ -109,6 +109,18 @@ the pattern, not because the link is proven.
5. Should a stream timeout on a *file-deliverable* job trigger an automatic retry instead of leaving the 5. Should a stream timeout on a *file-deliverable* job trigger an automatic retry instead of leaving the
day's discovery to the next slot? day's discovery to the next slot?
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
- The last open row raised here, `dca848_dfd4bea84f2a` (`timeout`, opened **2026-11-03 13:28:51**), **closed on
2026-11-05 at 12:36:17** - together with its 11-04 successor row `dca848_3b1bb1e9b003`.
- The job behind it, `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), **completed** its 2026-11-05
07:00 run in **5 h 35 m 35 s** (07:00:42 -> 12:36:18), with no repeat of
`RuntimeError: Upstream stream failed before completion.`
- **Status -> `resolved`.** The underlying questions (whether the 65 m 21 s stream silence is the watchdog
threshold or a silently dead socket; whether a stream timeout on a file-deliverable job should retry
automatically) remain open in [[systems/scheduled-tasks]] - they are no longer tracked as an incident
against this window.
## Related ## Related
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] - [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]
@@ -1,10 +1,10 @@
--- ---
title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)" title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)"
type: incident type: incident
status: open status: resolved
created: 2026-11-05 created: 2026-11-05
updated: 2026-11-05 updated: 2026-11-06
verified_on: 2026-11-05 verified_on: 2026-11-06
confidence: high confidence: high
tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident] tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident]
sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json] sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json]
@@ -113,6 +113,18 @@ until re-tested** rather than as proof the route is dead. See [[runbooks/job-rad
route incident - and whether the 11-03 row should be closed. route incident - and whether the 11-03 row should be closed.
4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]]. 4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]].
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
- **Both affected jobs ran clean on 2026-11-05.** `85edf6772bca` (*Tech & AI Newsletter Digest (Analytical)*)
completed its 07:00 run in **7 m 01 s** (07:00:42 -> 07:07:43), its incident row closing at **07:07:42**;
`dca8482e4f76` (*Daily Marketing Job Radar - Discovery*) completed in **5 h 35 m 35 s** and its two rows
closed at **12:36:17**. **Fleet-wide unclosed incident rows: 3 -> 0** - the ledger was empty for the first
time since 2026-11-02, and 2026-11-05 was the first zero-`failed` ledger day since then too.
- The Stage 2 salvage documented above therefore stands as the whole recovery - no follow-up failure arrived.
- The `CL_MODEL` / HTTP 402 cover-letter caveat is **unchanged and still needs a decision**: the
`gemini/gemini-3.5-flash` pin was never written to `.env`, so it did not persist.
- **Status -> `resolved`.** A recurrence should be opened as a new incident with its own window.
## Related ## Related
- [[incidents/2026-11-03-job-radar-stream-active-timeout]] - [[incidents/2026-11-03-job-radar-stream-active-timeout]]
@@ -0,0 +1,113 @@
---
title: Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05)
type: incident
status: open
created: 2026-11-06
updated: 2026-11-06
verified_on: 2026-11-06
review_after: 2026-11-13
confidence: high
tags: [security, credentials, mnemosyne, memory, sessions, cron, incident]
sources: [live-env-scan, live-sqlite-mnemosyne, live-jobs-json, live-state-db, live-deliveries-db, live-config-backup]
---
# Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05)
**Status: OPEN - found by the nightly audit (2026-11-05 window), not remediated.** No key was rotated, no
memory row deleted and no cron prompt scrubbed. This page records *where* the values live and how many rows
carry them; **no credential value appears on it** ([[SCHEMA]] rule 1 - record where secrets live, never their
values).
## What was found
Three exposures, all verified by direct read on **2026-11-06 00:22-00:50 AWST**.
### 1. Live `.env` values as plaintext rows in the memory store
**17 rows** of `~/.hermes/mnemosyne/data/mnemosyne.db` carry values from `~/.hermes/.env`. Method: every
TEXT/BLOB column of every table, plus the FTS mirrors, scanned for exact substring membership.
| Key (name only) | Rows | Columns carrying it |
|---|---|---|
| `TELEGRAM_BOT_TOKEN` | 4 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1 |
| `MATTERMOST_TOKEN` | 6 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x2, `memoria_facts.context_snippet` x1, `fts_episodes.content` x1 |
| `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` | 7 | `memories.content` x2, `working_memory.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1, `fts_working.content` x1 |
- **Every hit is in `mnemosyne.db`** (27.7 MB, live). The `episodic_memory.db` and `memoria_facts.db` files
beside it are **0-byte legacy stubs** (created 2026-10-27) - the tables with those names live *inside*
`mnemosyne.db`. Nothing was found in the cross-agent shared surface
(`~/.hermes/data/shared/mnemosyne.db`).
- **The rotation snapshots carry them too.** Seven uncompressed snapshots under
`~/.hermes/mnemosyne/backups/` (`mnemosyne-pre-orphan-cleanup.db`, `-20261004_030035`, `-20261011_030046`,
`-20261018_030021`, `-20261025_030108`, `-20261101_030033`, `-20261105_140314`) contain all three values in
full; the 2026-09-27 snapshot predates them. The `.db.gz` snapshots are **compressed and were not
substring-tested** - assume the same until proven otherwise.
### 2. One live key embedded verbatim in two cron prompts
The `CAMOFOX_API_KEY` value appears in the `prompt` text of two **enabled** jobs in
`~/.hermes/cron/jobs.json`:
- `0e753e9720fd` - *Facebook Birthday Monthly Scrape* (`0 9 1 * *`)
- `fad5a86979d4` - *Facebook Birthday Weekly Registry Sync* (`0 3 * * 1`)
Consequence: the same value is inside delivery records in `~/.hermes/cron/deliveries.db`. Of the **10 `.db`
files under `~/.hermes/config-backup/`** (six of them 0-byte), `cron/deliveries.db` is the **only** one that
carries any tested value; `cron/executions.db`, `cron/notepad.db` and `data/shared/mnemosyne.db` carry none.
### 3. Session transcripts hold all four values
`~/.hermes/state.db` (1.7 GB, the session store) contains all four values. Measured as 4 MB chunks
containing the value: **`CAMOFOX_API_KEY` 178, `MATTERMOST_TOKEN` 64,
`HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` 61, `TELEGRAM_BOT_TOKEN` 22**. This is the **most likely origin of
exposure 1** - the memory-capture path writes from session text, so a session in which a key was printed
becomes a memory row - and it is itself a plaintext at-rest store that deleting memory rows does not clear.
## Why it matters
- Mnemosyne rows are **recall material**: they are pulled into a session's context by semantic or FTS match,
so a secret in a row is a secret in a prompt - echoable into any cron delivery (Telegram, Mattermost,
email), any subagent transcript, or the cross-agent shared surface.
- It turns every **backup** into a credential copy. Deleting the rows does not recall the copies;
**rotation is the only remedy that fully closes it**.
- `jobs.json` is the fleet's source of truth and is exported and examined routinely; a prompt-embedded key
is exposed to anything that reads job definitions.
- `state.db` is the fleet's largest artifact (1.7 GB) and is copied into snapshots and repair backups.
## Origin (inferred - not established)
Almost certainly the **memory-capture path storing verbatim session text** in which a key was printed by a
tool call or a `.env` read. Not verified: no capture-time or delivery-time redaction filter is known to
exist, and the affected rows carry no marker distinguishing them from ordinary content.
## Remediation options (proposed, none executed)
1. **Rotate** the three Mnemosyne-exposed values and the Camofox key - touches live services; needs a
maintenance window and explicit approval.
2. **Redact or forget the affected rows** (`mnemosyne forget <id>` per row), then **re-take the rotation
snapshots** so the superseded values leave the backup set.
3. **Move the two Facebook Birthday jobs off prompt-embedded keys** - read the key from the environment or
config at run time.
4. **Add capture-time and delivery-time redaction** so `.env` values cannot enter memory rows, job prompts or
delivery records again.
5. **Decide a retention/serialisation policy for `state.db`** - it currently holds every transcript
verbatim at 1.7 GB.
## Verification evidence
- Key names and lengths extracted from `~/.hermes/.env` (values never printed), then matched as exact
substrings across every TEXT/BLOB column of every table in `mnemosyne.db` including the FTS mirrors -
row counts as tabled.
- Byte-membership scans of the 10 `.db` files under `~/.hermes/config-backup/`, the shared-surface DB, the
live `cron/deliveries.db` + `cron/executions.db`, the seven uncompressed Mnemosyne rotation snapshots, and
a chunked scan of `state.db`.
- `json.dumps()` membership test of each value against every job object in `jobs.json`.
- **Not done:** no value was printed, logged or written to any wiki page; nothing was deleted, rotated or
scrubbed.
## See also
- [[infrastructure/credentials-map]] - where secrets live, by design
- [[systems/mnemosyne]] - the memory store whose tables carry them
- [[systems/scheduled-tasks]] - the Facebook Birthday jobs' schedules
- [[SCHEMA]] - the no-values rule this violates
+9 -4
View File
@@ -3,15 +3,19 @@ title: Anthony's Systems Wiki
type: collection type: collection
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-05 updated: 2026-11-06
verified_on: 2026-11-05 verified_on: 2026-11-06
confidence: high confidence: high
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence] tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident]
--- ---
# Anthony's Systems Wiki # Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure, > Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-11-05 (stocktake - **the `systems/skills-index.md` flip-flop is fixed at the root**: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> [[incidents/2026-11-05-skills-index-generator-divergence]]. Docker inventory corrected against live probes on all five hosts - the `worldmonitor-seeder-scheduled` container recorded yesterday **does not exist on any host**. Tailscale node table re-verified programmatically - 52 rows, 52 live peers, every state matching. Freshness **0 stale / 122 fresh / 3 no-date of 125**.) > model routing and automation. Last updated: 2026-11-06. **2026-11-06 (nightly audit) - the incident ledger emptied (3 -> 0), 11-05 was the first zero-`failed` day since 11-02, the 03:00 stocktake's overrun streak broke, and a new OPEN incident records live secrets stored verbatim.**
>
> **2026-11-05 (stocktake) -** **the `systems/skills-index.md` flip-flop is fixed at the root**: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> [[incidents/2026-11-05-skills-index-generator-divergence]]. Docker inventory corrected against live probes on all five hosts - the `worldmonitor-seeder-scheduled` container recorded yesterday **does not exist on any host**. Tailscale node table re-verified programmatically - 52 rows, 52 live peers, every state matching. Freshness **0 stale / 122 fresh / 3 no-date of 125**.)
>
> **2026-11-06 (nightly audit - window 2026-11-05) - a clean ledger day and an empty incident ledger:** 2026-11-05 settled **175 rows - 173 `completed` / 2 `running` / 0 `failed`** across 18 jobs, the **first zero-`failed` day since 2026-11-02**. The only two `running` rows are the **Mnemosyne Health Monitor** (12:00:45 slot, still in flight **12 h+** at audit time) and this audit itself. **Unclosed incident rows 3 -> 0** - the *Newsletter Digest* row closed at **07:07:42** and both *Job Radar - Discovery* rows at **12:36:17**, when `dca8482e4f76` finally completed after **5 h 35 m 35 s**; that closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, `delivery_failed`, streak 0); deliveries **10 rows / 10 `delivered`**; the 03:00 stocktake overran **3 h 01 m 58 s** (down from 7 h 40 m and 5 h 41 m) and **committed `2f3fc99` at 05:55**. **Fleet unchanged at 29 jobs (26 / 3)** - a fourth audit at the same count. ⚠ **New and open:** live `.env` values (`TELEGRAM_BOT_TOKEN`, `MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit verbatim in **17 rows of `mnemosyne.db`** (`memories`, `episodic_memory`, `working_memory`, `gists`, `memoria_facts.context_snippet` and the FTS mirrors) - reproduced in full by each of the **7 uncompressed rotation snapshots** - and all four values sit verbatim in the **1.7 GB session store `state.db`** too; `CAMOFOX_API_KEY` sits verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and therefore in `cron/deliveries.db` -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
> >
> **2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze:** the **11-04 fleet settled 170 rows - 167 `completed` / 2 `failed` / 1 `running`**, **non-clean `last_status` 2 -> 3**, **unclosed incident rows 1 -> 3**, **deliveries 8 rows / 8 `delivered`**. Both failures are the *same* upstream fault: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) died **36 seconds apart at 09:41-09:42**, each after **2 h 41 m**, on `RuntimeError: Upstream stream failed before completion.` - both pinned `custom:omni-101` / `smart-route`. **The day was not lost:** the 11:30 Stage 2 slot **reconstructed the missing handoff from Stage 1's own candidate files**, re-verified every candidate live, and still wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. The cover-letter route survived only on an override: `regenerate_cover_letter.py` defaults to `deepseek-paid-smart`, which threw **HTTP 402** mid-run, so the letter was pinned to `gemini/gemini-3.5-flash` via `CL_MODEL` - and that override **was never written to `.env`**, so it did not persist. And **CT299 was frozen twice, not once**: a second, **manual** `vzdump 299 ... --storage ssd0` wedged from **11-04 08:30:04** until **11-05 ~00:30** (~16 h); the live probe at 00:54:21 AWST shows `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. ⚠ **Corrected the same night:** the first draft of this entry read the fleet as **158 jobs (144 / 14)** and the > **2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze:** the **11-04 fleet settled 170 rows - 167 `completed` / 2 `failed` / 1 `running`**, **non-clean `last_status` 2 -> 3**, **unclosed incident rows 1 -> 3**, **deliveries 8 rows / 8 `delivered`**. Both failures are the *same* upstream fault: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) died **36 seconds apart at 09:41-09:42**, each after **2 h 41 m**, on `RuntimeError: Upstream stream failed before completion.` - both pinned `custom:omni-101` / `smart-route`. **The day was not lost:** the 11:30 Stage 2 slot **reconstructed the missing handoff from Stage 1's own candidate files**, re-verified every candidate live, and still wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. The cover-letter route survived only on an override: `regenerate_cover_letter.py` defaults to `deepseek-paid-smart`, which threw **HTTP 402** mid-run, so the letter was pinned to `gemini/gemini-3.5-flash` via `CL_MODEL` - and that override **was never written to `.env`**, so it did not persist. And **CT299 was frozen twice, not once**: a second, **manual** `vzdump 299 ... --storage ssd0` wedged from **11-04 08:30:04** until **11-05 ~00:30** (~16 h); the live probe at 00:54:21 AWST shows `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. ⚠ **Corrected the same night:** the first draft of this entry read the fleet as **158 jobs (144 / 14)** and the
> 11-04 ledger as closing `1 item_suppressed`; live reads give **29 jobs (26 / 3)** and a final row of **1 `running`** > 11-04 ledger as closing `1 item_suppressed`; live reads give **29 jobs (26 / 3)** and a final row of **1 `running`**
@@ -58,6 +62,7 @@ tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, gene
- [[systems/skills-index]] — Installed skills inventory (**302 skills** (**68 categories**) as of the **2026-11-05** stocktake; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (46 archived skills) — the earlier 346/347 counts included it. Freshness at the 2026-11-05 run: **0 stale / 122 fresh / 3 no-date of 125 pages** (`freshness-20261105.txt`). Note: `SKILL.md` files on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so **302 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. - [[systems/skills-index]] — Installed skills inventory (**302 skills** (**68 categories**) as of the **2026-11-05** stocktake; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (46 archived skills) — the earlier 346/347 counts included it. Freshness at the 2026-11-05 run: **0 stale / 122 fresh / 3 no-date of 125 pages** (`freshness-20261105.txt`). Note: `SKILL.md` files on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so **302 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
## Incidents ## Incidents
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **OPEN.** Live `.env` values were found stored verbatim in **17 rows of `mnemosyne.db`** (`memories` 4, `episodic_memory` 3, `working_memory` 1, `gists` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the **7 uncompressed rotation snapshots**, and all four values also sit verbatim in the **1.7 GB session store `state.db`**, which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Nothing rotated, deleted or scrubbed; no value recorded.
- [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter). - [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter).
- [[incidents/2026-11-05-skills-index-generator-divergence]] — **RESOLVED — the skills index flip-flop had two writers.** The repo `wiki-maintain.sh` (302 skills / 68 categories) and the stale `~/.hermes/scripts/wiki-maintain.sh` copy (349 / 26, because it used `Path.rglob()` and counted the parked `.archive/`) were **different scripts**, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with **no error raised by either**. The stale copy is now a **delegating wrapper**; the repo script gained `--no-build` so the wrapper honours the old interface without a second generator. - [[incidents/2026-11-05-skills-index-generator-divergence]] — **RESOLVED — the skills index flip-flop had two writers.** The repo `wiki-maintain.sh` (302 skills / 68 categories) and the stale `~/.hermes/scripts/wiki-maintain.sh` copy (349 / 26, because it used `Path.rglob()` and counted the parked `.archive/`) were **different scripts**, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with **no error raised by either**. The stale copy is now a **delegating wrapper**; the repo script gained `--no-build` so the wrapper honours the old interface without a second generator.
- [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] — **OPEN — the app's "local" voice mode never applies, and the connection test lies.** The test checks **socket reachability only**, while every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects `session.update` as an unknown event**. `voice.mode` is a **phone-side** selector — **no `mode` key exists** in `config.yaml → voice` or `conduit-settings.json` — so a "mode did not apply" cannot be diagnosed on host config. Found via session `20261031_193709_40b9c2` (19:37:09, 30 msgs / 17 tool calls); `voice.local_live.front_model` was repointed to `mistral/voxtral-small-latest` during it and **nothing was changed on the Mac**. → [[systems/conduit]] - [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] — **OPEN — the app's "local" voice mode never applies, and the connection test lies.** The test checks **socket reachability only**, while every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects `session.update` as an unknown event**. `voice.mode` is a **phone-side** selector — **no `mode` key exists** in `config.yaml → voice` or `conduit-settings.json` — so a "mode did not apply" cannot be diagnosed on host config. Found via session `20261031_193709_40b9c2` (19:37:09, 30 msgs / 17 tool calls); `voice.local_live.front_model` was repointed to `mistral/voxtral-small-latest` during it and **nothing was changed on the Mac**. → [[systems/conduit]]
+46
View File
@@ -2080,3 +2080,49 @@ embedded credential was seen during verification and is not reproduced here ([RE
`incidents/2026-08-15-skills-index-drift.md`, `wiki-maintain.sh`, `log.md`. `incidents/2026-08-15-skills-index-drift.md`, `wiki-maintain.sh`, `log.md`.
- **Not done / unchanged:** no job added, removed, disabled or re-pinned; no page removed or archived; no - **Not done / unchanged:** no job added, removed, disabled or re-pinned; no page removed or archived; no
decision reversed. **No secrets, passwords, tokens or keys written or quoted.** decision reversed. **No secrets, passwords, tokens or keys written or quoted.**
## [2026-11-06] audit - window 2026-11-05
- **Window reviewed:** 2026-11-05 00:00-24:00 AWST; live reads 2026-11-06 00:1x-01:0x AWST. Fleet unchanged at
**29 jobs (26 enabled / 3 disabled)** - a fourth consecutive audit at the same count, no job added, retired,
disabled or re-pinned.
- **New page: [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (status: OPEN).** Three live
`.env` values (`TELEGRAM_BOT_TOKEN`, `MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit in
plaintext in **17 rows of `~/.hermes/mnemosyne/data/mnemosyne.db`** - `memories.content` 4, `episodic_memory.content`
3, `working_memory.content` 1, `gists.text` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - and
`CAMOFOX_API_KEY` sits verbatim in the `prompt` of two **enabled** Facebook Birthday jobs (`0e753e9720fd`,
`fad5a86979d4`) and hence in `cron/deliveries.db`. Method: exact-substring match of each `.env` value across
**every TEXT/BLOB column of every table** in `mnemosyne.db` plus its FTS mirrors; byte-membership scans of the
10 `.db` files under `~/.hermes/config-backup/` (six are 0-byte; only `cron/deliveries.db` carries any tested
value), the shared-surface DB, the seven uncompressed rotation snapshots under `~/.hermes/mnemosyne/backups/`
(all seven reproduce the DB in full) and a chunked scan of the **1.7 GB session store `state.db`**, which
carries all four values and is the likely capture origin (capture writes verbatim session text). **Nothing was
rotated, deleted or scrubbed; no credential value was printed, logged or written to any page.**
- **Both 11-04 casualties recovered in full.** `85edf6772bca` (*Tech & AI Newsletter Digest*) completed its 07:00
run in **7 m 01 s** (07:00:42 -> 07:07:43) and `dca8482e4f76` (*Daily Marketing Job Radar - Discovery*) in
**5 h 35 m 35 s** (07:00:42 -> 12:36:18); neither re-raised `RuntimeError: Upstream stream failed before
completion.` The 11-05 ledger settled **175 rows / 18 jobs - 173 `completed` / 2 `running` / 0 `failed`**, the
**first zero-`failed` day since 2026-11-02**; the two `running` rows are `ea79a2107e36` (*Mnemosyne Health
Monitor*, 12:00:45 slot, still in flight 12 h+ with no delivery) and this audit (`cbfe434a3b5e`). Deliveries
**10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`,
*Uber Eats Monthly Spend Report*, `delivery_failed`, streak 0).
- **Incident ledger emptied.** Unclosed rows **3 -> 0**: the digest row closed **07:07:42** and both Job Radar rows
(`dca848_dfd4bea84f2a` `timeout`, `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**.
`incidents/2026-11-03-job-radar-stream-active-timeout.md` -> `status: resolved` with a Resolution section;
`incidents/2026-11-04-upstream-stream-failure-0700-cohort.md` closed out - the first empty ledger since
2026-11-02.
- **Updated [[systems/scheduled-tasks]]** - frontmatter (`updated` / `verified_on` -> 2026-11-06, `review_after`
2026-11-07, `live-sessions-db` added to `sources`), a new `## Count` bullet for the 2026-11-06 read and a new
`## Re-verified 2026-11-06` span (recoveries, zero-`failed` ledger, delivery ledger, the secrets finding).
- **Updated [[systems/mnemosyne]]** - frontmatter dates -> 2026-11-06 (`review_after` 2026-11-13), `sources` +
`live-secrets-scan, live-state-db`, and a **security entry at the head of `## Known Issues`** recording that the
store's own tables carry 17 rows of live credential material, that recall is the exposure path, and that only
rotation covers the copies already taken.
- **Updated [[current-state]], [[index]]** (`## Incidents` link for the new page, header summary rewritten to the
2026-11-06 read) and this log.
- **Freshness:** `raw/stocktake/freshness-20261106.txt` (re-run after this audit's writes) -> **0 stale (>=90d),
124 fresh, 3 no-date of 127 pages**, the new incident page included. The three no-date pages are unchanged (`SCHEMA.md`,
`experiments/active/example.md`, `log.md`).
- **Not done / unchanged:** no page removed or archived; no decision reversed; `066bb5191280` and the
`deepseek-paid-smart`-on-`omniroute` dispatch fault carried, not re-diagnosed. **No secrets, passwords, tokens
or keys written or quoted** - the incident page records where secrets live, never their values.
+140
View File
@@ -0,0 +1,140 @@
# Wiki freshness audit — 2026-11-06
Threshold: >90 days since `updated`
## STALE (0)
## FRESH (124)
- [ 17d] CONTRIBUTING.md
- [ 11d] agents/hermes-chaos.md
- [ 11d] agents/hermes-production.md
- [ 17d] agents/index.md
- [ 8d] agents/nanobot.md
- [ 17d] agents/open-claw.md
- [ 2d] agents/qwenpaw.md
- [ 17d] agents/zeroclaw.md
- [ 8d] comparisons/camoufox-vs-chrome.md
- [ 17d] comparisons/hermes-vs-nanobot.md
- [ 17d] comparisons/memory-backends.md
- [ 17d] comparisons/model-routers.md
- [ 8d] comparisons/steel-vs-camoufox.md
- [ 17d] concepts/agent-memory.md
- [ 8d] concepts/browser-automation.md
- [ 17d] concepts/model-routing.md
- [ 17d] concepts/persistent-shells.md
- [ 17d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 8d] decisions/index.md
- [ 17d] decisions/keep-openrouter-as-fallback.md
- [ 17d] decisions/persistent-shell-enabled.md
- [ 8d] decisions/rollback-steel-to-camofox.md
- [ 17d] experiments/index.md
- [ 13d] incidents/2026-07-29-mnemosyne-relay.md
- [ 17d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 13d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 15d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 1d] incidents/2026-08-15-skills-index-drift.md
- [ 2d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 12d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 17d] incidents/2026-08-29-gmail-read-state.md
- [ 16d] incidents/2026-09-02-sftpgo-outage.md
- [ 2d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 8d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 11d] incidents/2026-10-03-ct450-hermes-update.md
- [ 11d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 8d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 13d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 5d] incidents/2026-10-09-memory-recall-relevance.md
- [ 21d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 10d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 18d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 10d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 15d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 12d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 11d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 11d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 11d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 9d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 7d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 7d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 7d] incidents/2026-10-29-conduit-voice-silence.md
- [ 6d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 7d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 6d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 5d] incidents/2026-10-31-asana-access-failures.md
- [ 5d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 5d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 4d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 3d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 2d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 0d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 0d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 1d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 0d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 1d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 0d] index.md
- [ 2d] infrastructure/backups.md
- [ 17d] infrastructure/credentials-map.md
- [ 1d] infrastructure/docker-services.md
- [ 17d] infrastructure/domains-and-tunnels.md
- [ 1d] infrastructure/hosts.md
- [ 1d] infrastructure/network-map.md
- [ 5d] infrastructure/proxmox.md
- [ 1d] infrastructure/tailscale.md
- [ 2d] queries/retained-research-answers.md
- [ 12d] reports/communities-wa-social-media-audit-2026.md
- [ 5d] reports/uber-eats/2026-07.md
- [ 5d] reports/uber-eats/2026-08.md
- [ 5d] reports/uber-eats/2026-09.md
- [ 5d] reports/uber-eats/2026-10.md
- [ 5d] reports/uber-eats/index.md
- [ 17d] runbooks/backup-wiki.md
- [ 17d] runbooks/diagnose-docker.md
- [ 17d] runbooks/gateway-resource-resilience.md
- [ 1d] runbooks/job-radar.md
- [ 17d] runbooks/multiplexer-setup.md
- [ 11d] runbooks/provider-health.md
- [ 8d] runbooks/recover-docker-service.md
- [ 17d] runbooks/restart-browser.md
- [ 17d] runbooks/restart-hermes.md
- [ 17d] runbooks/restore-openrouter.md
- [ 17d] runbooks/test-openrouter.md
- [ 17d] runbooks/troubleshoot-tool-loop.md
- [ 11d] runbooks/update-hermes-safely.md
- [ 3d] systems/browser-backend.md
- [ 17d] systems/cloudflare-tunnel.md
- [ 5d] systems/conduit.md
- [ 11d] systems/current-profile.md
- [ 17d] systems/guanaco.md
- [ 17d] systems/headless-server.md
- [ 7d] systems/hermes-agent.md
- [ 2d] systems/hermes-hq-push.md
- [ 17d] systems/hermes-ui.md
- [ 2d] systems/home-assistant.md
- [ 7d] systems/homepage-dashboard.md
- [ 2d] systems/ignis.md
- [ 11d] systems/memory-backend.md
- [ 4d] systems/messaging-integrations.md
- [ 0d] systems/mnemosyne.md
- [ 11d] systems/model-providers.md
- [ 2d] systems/nanobot.md
- [ 17d] systems/nextcloud.md
- [ 17d] systems/obsidian.md
- [ 5d] systems/omniroute.md
- [ 17d] systems/opencode-go.md
- [ 17d] systems/openrouter.md
- [ 2d] systems/paseo.md
- [ 18d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 1d] systems/skills-index.md
- [ 17d] systems/terminal-backend.md
- [ 17d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 124 fresh, 3 no-date of 127 pages
+6 -4
View File
@@ -3,12 +3,12 @@ title: Mnemosyne
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-01 updated: 2026-11-06
verified_on: 2026-11-01 verified_on: 2026-11-06
review_after: 2026-11-07 review_after: 2026-11-13
confidence: high confidence: high
tags: [memory, mnemosyne, backend] tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config] sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db]
--- ---
# Mnemosyne # Mnemosyne
@@ -50,6 +50,8 @@ Primary agent memory store for Hermes.
## Known Issues ## Known Issues
**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**.
**Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below. **Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below.
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. ~~Ranking relevance is the live issue, not coverage~~ — see [[incidents/2026-10-09-memory-recall-relevance]]. **Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. ~~Ranking relevance is the live issue, not coverage~~ — see [[incidents/2026-10-09-memory-recall-relevance]].
+89 -4
View File
@@ -3,17 +3,18 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-05 updated: 2026-11-06
verified_on: 2026-11-05 verified_on: 2026-11-06
review_after: 2026-11-06 review_after: 2026-11-07
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db] sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
--- ---
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. -> [[runbooks/job-radar]] `CL_MODEL` caveat. - **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. -> [[runbooks/job-radar]] `CL_MODEL` caveat.
- **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` + - **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` / `deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` /
@@ -963,3 +964,87 @@ unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69
whether the five-compaction churn contributes; whether a stream timeout on a file-deliverable job should whether the five-compaction churn contributes; whether a stream timeout on a file-deliverable job should
**retry automatically**; `radar-ref/handoff/` retention (newest file 10-30); why Stage 2 still has no **retry automatically**; `radar-ref/handoff/` retention (newest file 10-30); why Stage 2 still has no
handoff to wait on; and why the 03:00 stocktake takes **7 h 40 m** and commits nothing. handoff to wait on; and why the 03:00 stocktake takes **7 h 40 m** and commits nothing.
## Re-verified 2026-11-06 (window 2026-11-05, AWST; live reads 2026-11-06 00:1x-00:4x)
### The day was a recovery day, not a repeat
- **Both 11-04 casualties ran clean.** `85edf6772bca` (*Tech & AI Newsletter Digest (Analytical)*, streak 1)
completed its 07:00 run in **7 m 01 s** (07:00:42 -> 07:07:43); its incident row closed at **07:07:42**.
`dca8482e4f76` (*Daily Marketing Job Radar - Discovery*, streak 2, dead twice on 11-04) also completed its
07:00 run - but took **5 h 35 m 35 s** (07:00:42 -> 12:36:18), and that duration is what held its two
incident rows open until **12:36:17**. Neither job re-raised `RuntimeError: Upstream stream failed before
completion.`
- **No `unknown`, `timeout` or `error` row exists anywhere in the 11-05 ledger** - the first such day since
11-02.
- **Deliveries: 10 rows, all `delivered`, none flagged `for_failure`** - 07:02:11, 07:07:42, 07:30:39,
08:06:00, 08:35:30, 10:41:20, 12:15:25, 18:45:49, 19:25:00, 23:27:09 AWST.
### Long runs (completed, not failures)
| Job | Slot | Duration | Against its norm |
|---|---|---|---|
| `ea79a2107e36` Mnemosyne Health Monitor | 00:00 | **7 h 01 m 28 s** (00:00:43 -> 07:02:11) | normal 17-31 m |
| `1a8e90ea2271` wiki-stocktake-maintenance | 03:00 | **3 h 01 m 58 s** (03:00:32 -> 06:02:30) | 11-02 7 h 40 m, 11-03 5 h 41 m |
| `dca8482e4f76` Job Radar - Discovery | 07:00 | **5 h 35 m 35 s** | recovered from 2 failures |
| `5fc292ff0f3b` Job Radar - Stage 2 | 14:30 | **8 h 56 m 17 s** (14:30:53 -> 23:27:10) | delivered 23:27:09 |
| `5fc292ff0f3b` Job Radar - Stage 2 | 11:30 | 10 m 33 s | normal |
| `85edf6772bca` Newsletter Digest | 07:00 | 7 m 01 s | normal |
### The 03:00 stocktake overran a third day - and committed this time
`1a8e90ea2271` ran **03:00:32 -> 06:02:30 (3 h 01 m 58 s)**, so the streak now reads
**7 h 40 m -> 5 h 41 m -> 3 h 02 m** (improving, still ~6x norm), and it **did commit** its artefacts
(`2f3fc99`, 05:55) - against the 11-04 state where it "reported incomplete and committed nothing". The vault
repo `/home/hermes/wiki` picked that up at 05:58 (`ee00b5c`). The nightly audit did **not** have to commit the
skills-index on the stocktake's behalf this pass.
### Sessions in the window
**26 sessions had activity touching 2026-11-05** - 21 `cron`, 4 `subagent`, 1 `desktop`. The only long-form
one is the morning **Mnemosyne Health Monitor** run (`local-b433c51e...`, `cron`, 17 messages, 07:02-07:50):
it requested tool approvals and received denials. No configuration change came out of it, and it is
consistent with that job's 7 h 01 m ballooning above.
### Still open (2026-11-06 - carries its predecessor forward)
Everything on the 2026-11-04 list stands unchanged and not re-probed this window (single-node no-failover
topology; the GitHub Trending Radar delivery token; Camofox containment under multi-tab load; the Conduit
health store with no retention; the unreconciled admission files; the unidentified `-15` signal source; the
30 s worker-ack window vs cold-start; whether a stream-silence timeout should retry a file-deliverable job;
`radar-ref/handoff/` retention). **New this window:**
- **Live secrets sit verbatim in Mnemosyne rows and in two cron prompts.** Three `.env` values (`TELEGRAM_BOT_TOKEN`,
`MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) appear in `memories`, `episodic_memory`,
`working_memory`, `gists` and `memoria_facts.context_snippet` plus the FTS mirrors - **17 rows**, 4 + 6 + 7 per
key - and each of the **7 uncompressed rotation snapshots** reproduces the whole DB; all four values also sit
verbatim in the **1.7 GB session store `state.db`**, the likely capture origin. The Camofox key is embedded in
the prompts of `0e753e9720fd` and `fad5a86979d4` (both **enabled** Facebook Birthday jobs) and therefore in
`cron/deliveries.db` - the only `config-backup/` DB carrying any tested value. **Found, not remediated** ->
[[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
- **`ea79a2107e36` ballooned twice out of two runs** (7 h 01 m from the 00:00 slot, then still running 12 h
after its 12:00:45 slot) - against a 17-31 m normal. Cause unexplained; the 12:00 run had delivered nothing
at audit time.
- **`066bb5191280`** (Uber Eats Monthly Spend Report) is now the fleet's **only** non-clean `last_status`
(`delivery_failed`, streak 0) - carried, not re-diagnosed this window.
- **`deepseek-paid-smart` on `omniroute` (CT600) - carried, not re-probed this window.** 11-05 read: HTTP
**503** `ALL_TARGETS_SKIPPED` on `omniroute:20128` vs HTTP **200** on `omniroute-1`
(100.78.63.20:20128 -> `deepseek/deepseek-v4-flash`). This audit remains the fleet's only
`deepseek-paid-smart` pin (provider `custom:omni-101`), which is why it still completes.
### Freshness
`raw/stocktake/freshness-20261106.txt` (re-run after this audit's writes): **0 stale (>=90d), 124 fresh, 3
no-date of 127 pages** - the new incident page included. The three no-date pages are unchanged (`SCHEMA.md`,
`experiments/active/example.md`, `log.md`).
### Files touched by this audit
- **New:** `incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md`,
`raw/stocktake/freshness-20261106.txt`.
- **Updated:** this page (+frontmatter, +Count bullet, +this section), `current-state.md`, `index.md`,
`log.md`, `incidents/2026-11-03-job-radar-stream-active-timeout.md`,
`incidents/2026-11-04-upstream-stream-failure-0700-cohort.md` (both -> `status: resolved` with closure
timestamps), and the vault mirror (`/home/hermes/wiki`, both conventions).
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
decision reversed.