audit 2026-11-06 (window 2026-11-05): incident ledger emptied 3 -> 0, first zero-failed day since 11-02, both 11-04 casualties recovered; NEW OPEN incident - live .env values verbatim in 17 mnemosyne rows, the 1.7 GB session store and two enabled Facebook Birthday prompts; freshness 0 stale / 124 fresh / 3 no-date of 127

This commit is contained in:
Tony0410
2026-11-06 00:29:52 +08:00
parent 2f3fc9952b
commit eba8062735
9 changed files with 460 additions and 22 deletions
@@ -1,10 +1,10 @@
---
title: Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
type: incident
status: open
status: resolved
created: 2026-11-04
updated: 2026-11-04
verified_on: 2026-11-04
updated: 2026-11-06
verified_on: 2026-11-06
review_after: 2026-11-11
confidence: high
tags: [cron, job-radar, stream-timeout, silent-failure, notion, context-compaction]
@@ -109,6 +109,18 @@ the pattern, not because the link is proven.
5. Should a stream timeout on a *file-deliverable* job trigger an automatic retry instead of leaving the
day's discovery to the next slot?
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
- The last open row raised here, `dca848_dfd4bea84f2a` (`timeout`, opened **2026-11-03 13:28:51**), **closed on
2026-11-05 at 12:36:17** - together with its 11-04 successor row `dca848_3b1bb1e9b003`.
- The job behind it, `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), **completed** its 2026-11-05
07:00 run in **5 h 35 m 35 s** (07:00:42 -> 12:36:18), with no repeat of
`RuntimeError: Upstream stream failed before completion.`
- **Status -> `resolved`.** The underlying questions (whether the 65 m 21 s stream silence is the watchdog
threshold or a silently dead socket; whether a stream timeout on a file-deliverable job should retry
automatically) remain open in [[systems/scheduled-tasks]] - they are no longer tracked as an incident
against this window.
## Related
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]
@@ -1,10 +1,10 @@
---
title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)"
type: incident
status: open
status: resolved
created: 2026-11-05
updated: 2026-11-05
verified_on: 2026-11-05
updated: 2026-11-06
verified_on: 2026-11-06
confidence: high
tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident]
sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json]
@@ -113,6 +113,18 @@ until re-tested** rather than as proof the route is dead. See [[runbooks/job-rad
route incident - and whether the 11-03 row should be closed.
4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]].
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
- **Both affected jobs ran clean on 2026-11-05.** `85edf6772bca` (*Tech & AI Newsletter Digest (Analytical)*)
completed its 07:00 run in **7 m 01 s** (07:00:42 -> 07:07:43), its incident row closing at **07:07:42**;
`dca8482e4f76` (*Daily Marketing Job Radar - Discovery*) completed in **5 h 35 m 35 s** and its two rows
closed at **12:36:17**. **Fleet-wide unclosed incident rows: 3 -> 0** - the ledger was empty for the first
time since 2026-11-02, and 2026-11-05 was the first zero-`failed` ledger day since then too.
- The Stage 2 salvage documented above therefore stands as the whole recovery - no follow-up failure arrived.
- The `CL_MODEL` / HTTP 402 cover-letter caveat is **unchanged and still needs a decision**: the
`gemini/gemini-3.5-flash` pin was never written to `.env`, so it did not persist.
- **Status -> `resolved`.** A recurrence should be opened as a new incident with its own window.
## Related
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
@@ -0,0 +1,113 @@
---
title: Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05)
type: incident
status: open
created: 2026-11-06
updated: 2026-11-06
verified_on: 2026-11-06
review_after: 2026-11-13
confidence: high
tags: [security, credentials, mnemosyne, memory, sessions, cron, incident]
sources: [live-env-scan, live-sqlite-mnemosyne, live-jobs-json, live-state-db, live-deliveries-db, live-config-backup]
---
# Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05)
**Status: OPEN - found by the nightly audit (2026-11-05 window), not remediated.** No key was rotated, no
memory row deleted and no cron prompt scrubbed. This page records *where* the values live and how many rows
carry them; **no credential value appears on it** ([[SCHEMA]] rule 1 - record where secrets live, never their
values).
## What was found
Three exposures, all verified by direct read on **2026-11-06 00:22-00:50 AWST**.
### 1. Live `.env` values as plaintext rows in the memory store
**17 rows** of `~/.hermes/mnemosyne/data/mnemosyne.db` carry values from `~/.hermes/.env`. Method: every
TEXT/BLOB column of every table, plus the FTS mirrors, scanned for exact substring membership.
| Key (name only) | Rows | Columns carrying it |
|---|---|---|
| `TELEGRAM_BOT_TOKEN` | 4 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1 |
| `MATTERMOST_TOKEN` | 6 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x2, `memoria_facts.context_snippet` x1, `fts_episodes.content` x1 |
| `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` | 7 | `memories.content` x2, `working_memory.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1, `fts_working.content` x1 |
- **Every hit is in `mnemosyne.db`** (27.7 MB, live). The `episodic_memory.db` and `memoria_facts.db` files
beside it are **0-byte legacy stubs** (created 2026-10-27) - the tables with those names live *inside*
`mnemosyne.db`. Nothing was found in the cross-agent shared surface
(`~/.hermes/data/shared/mnemosyne.db`).
- **The rotation snapshots carry them too.** Seven uncompressed snapshots under
`~/.hermes/mnemosyne/backups/` (`mnemosyne-pre-orphan-cleanup.db`, `-20261004_030035`, `-20261011_030046`,
`-20261018_030021`, `-20261025_030108`, `-20261101_030033`, `-20261105_140314`) contain all three values in
full; the 2026-09-27 snapshot predates them. The `.db.gz` snapshots are **compressed and were not
substring-tested** - assume the same until proven otherwise.
### 2. One live key embedded verbatim in two cron prompts
The `CAMOFOX_API_KEY` value appears in the `prompt` text of two **enabled** jobs in
`~/.hermes/cron/jobs.json`:
- `0e753e9720fd` - *Facebook Birthday Monthly Scrape* (`0 9 1 * *`)
- `fad5a86979d4` - *Facebook Birthday Weekly Registry Sync* (`0 3 * * 1`)
Consequence: the same value is inside delivery records in `~/.hermes/cron/deliveries.db`. Of the **10 `.db`
files under `~/.hermes/config-backup/`** (six of them 0-byte), `cron/deliveries.db` is the **only** one that
carries any tested value; `cron/executions.db`, `cron/notepad.db` and `data/shared/mnemosyne.db` carry none.
### 3. Session transcripts hold all four values
`~/.hermes/state.db` (1.7 GB, the session store) contains all four values. Measured as 4 MB chunks
containing the value: **`CAMOFOX_API_KEY` 178, `MATTERMOST_TOKEN` 64,
`HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` 61, `TELEGRAM_BOT_TOKEN` 22**. This is the **most likely origin of
exposure 1** - the memory-capture path writes from session text, so a session in which a key was printed
becomes a memory row - and it is itself a plaintext at-rest store that deleting memory rows does not clear.
## Why it matters
- Mnemosyne rows are **recall material**: they are pulled into a session's context by semantic or FTS match,
so a secret in a row is a secret in a prompt - echoable into any cron delivery (Telegram, Mattermost,
email), any subagent transcript, or the cross-agent shared surface.
- It turns every **backup** into a credential copy. Deleting the rows does not recall the copies;
**rotation is the only remedy that fully closes it**.
- `jobs.json` is the fleet's source of truth and is exported and examined routinely; a prompt-embedded key
is exposed to anything that reads job definitions.
- `state.db` is the fleet's largest artifact (1.7 GB) and is copied into snapshots and repair backups.
## Origin (inferred - not established)
Almost certainly the **memory-capture path storing verbatim session text** in which a key was printed by a
tool call or a `.env` read. Not verified: no capture-time or delivery-time redaction filter is known to
exist, and the affected rows carry no marker distinguishing them from ordinary content.
## Remediation options (proposed, none executed)
1. **Rotate** the three Mnemosyne-exposed values and the Camofox key - touches live services; needs a
maintenance window and explicit approval.
2. **Redact or forget the affected rows** (`mnemosyne forget <id>` per row), then **re-take the rotation
snapshots** so the superseded values leave the backup set.
3. **Move the two Facebook Birthday jobs off prompt-embedded keys** - read the key from the environment or
config at run time.
4. **Add capture-time and delivery-time redaction** so `.env` values cannot enter memory rows, job prompts or
delivery records again.
5. **Decide a retention/serialisation policy for `state.db`** - it currently holds every transcript
verbatim at 1.7 GB.
## Verification evidence
- Key names and lengths extracted from `~/.hermes/.env` (values never printed), then matched as exact
substrings across every TEXT/BLOB column of every table in `mnemosyne.db` including the FTS mirrors -
row counts as tabled.
- Byte-membership scans of the 10 `.db` files under `~/.hermes/config-backup/`, the shared-surface DB, the
live `cron/deliveries.db` + `cron/executions.db`, the seven uncompressed Mnemosyne rotation snapshots, and
a chunked scan of `state.db`.
- `json.dumps()` membership test of each value against every job object in `jobs.json`.
- **Not done:** no value was printed, logged or written to any wiki page; nothing was deleted, rotated or
scrubbed.
## See also
- [[infrastructure/credentials-map]] - where secrets live, by design
- [[systems/mnemosyne]] - the memory store whose tables carry them
- [[systems/scheduled-tasks]] - the Facebook Birthday jobs' schedules
- [[SCHEMA]] - the no-values rule this violates