Files
agent-estate-wiki/incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
T

133 lines
8.2 KiB
Markdown

---
title: Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05)
type: incident
status: open
created: 2026-11-06
updated: 2026-11-09
verified_on: 2026-11-09
review_after: 2026-11-13
confidence: high
tags: [security, credentials, mnemosyne, memory, sessions, cron, incident]
sources: [live-env-scan, live-sqlite-mnemosyne, live-jobs-json, live-state-db, live-deliveries-db, live-config-backup]
---
# Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05)
**Status: OPEN - found by the nightly audit (2026-11-05 window), not remediated.** No key was rotated, no
memory row deleted and no cron prompt scrubbed. This page records *where* the values live and how many rows
carry them; **no credential value appears on it** ([[SCHEMA]] rule 1 - record where secrets live, never their
values).
## What was found
Three exposures, all verified by direct read on **2026-11-06 00:22-00:50 AWST**.
### 1. Live `.env` values as plaintext rows in the memory store
**17 rows** of `~/.hermes/mnemosyne/data/mnemosyne.db` carry values from `~/.hermes/.env`. Method: every
TEXT/BLOB column of every table, plus the FTS mirrors, scanned for exact substring membership.
| Key (name only) | Rows | Columns carrying it |
|---|---|---|
| `TELEGRAM_BOT_TOKEN` | 4 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1 |
| `MATTERMOST_TOKEN` | 6 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x2, `memoria_facts.context_snippet` x1, `fts_episodes.content` x1 |
| `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` | 7 | `memories.content` x2, `working_memory.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1, `fts_working.content` x1 |
- **Every hit is in `mnemosyne.db`** (27.7 MB, live). The `episodic_memory.db` and `memoria_facts.db` files
beside it are **0-byte legacy stubs** (created 2026-10-27) - the tables with those names live *inside*
`mnemosyne.db`. Nothing was found in the cross-agent shared surface
(`~/.hermes/data/shared/mnemosyne.db`).
- **The rotation snapshots carry them too.** Seven uncompressed snapshots under
`~/.hermes/mnemosyne/backups/` (`mnemosyne-pre-orphan-cleanup.db`, `-20261004_030035`, `-20261011_030046`,
`-20261018_030021`, `-20261025_030108`, `-20261101_030033`, `-20261105_140314`) contain all three values in
full; the 2026-09-27 snapshot predates them. The `.db.gz` snapshots are **compressed and were not
substring-tested** - assume the same until proven otherwise.
### 2. One live key embedded verbatim in two cron prompts
The `CAMOFOX_API_KEY` value appears in the `prompt` text of two **enabled** jobs in
`~/.hermes/cron/jobs.json`:
- `0e753e9720fd` - *Facebook Birthday Monthly Scrape* (`0 9 1 * *`)
- `fad5a86979d4` - *Facebook Birthday Weekly Registry Sync* (`0 3 * * 1`)
Consequence: the same value is inside delivery records in `~/.hermes/cron/deliveries.db`. Of the **10 `.db`
files under `~/.hermes/config-backup/`** (six of them 0-byte), `cron/deliveries.db` is the **only** one that
carries any tested value; `cron/executions.db`, `cron/notepad.db` and `data/shared/mnemosyne.db` carry none.
### 3. Session transcripts hold all four values
`~/.hermes/state.db` (1.7 GB *at discovery* — **1.89 GB / 1,890,656,256 B as at 2026-11-09 00:0x AWST**, with
**2,901 sessions / 197,682 messages** and a live **40,157,672 B WAL**; the store grew while nothing was
retentioned), the session store, contains all four values. Measured as 4 MB chunks
containing the value: **`CAMOFOX_API_KEY` 178, `MATTERMOST_TOKEN` 64,
`HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` 61, `TELEGRAM_BOT_TOKEN` 22**. This is the **most likely origin of
exposure 1** - the memory-capture path writes from session text, so a session in which a key was printed
becomes a memory row - and it is itself a plaintext at-rest store that deleting memory rows does not clear.
## Why it matters
- Mnemosyne rows are **recall material**: they are pulled into a session's context by semantic or FTS match,
so a secret in a row is a secret in a prompt - echoable into any cron delivery (Telegram, Mattermost,
email), any subagent transcript, or the cross-agent shared surface.
- It turns every **backup** into a credential copy. Deleting the rows does not recall the copies;
**rotation is the only remedy that fully closes it**.
- `jobs.json` is the fleet's source of truth and is exported and examined routinely; a prompt-embedded key
is exposed to anything that reads job definitions.
- `state.db` is the fleet's largest artifact (**1.89 GB as at 2026-11-09**, up from 1.7 GB) and is copied into
snapshots and repair backups. ⚠ **It is also the load source behind the 2026-11-07/08 hook-timeout
cascade** — a 40 MB WAL checkpointing on the same spindle as a 2-core guest → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
The **2026-08-11 corruption remnant `state.db.corrupted.20260811_151052` (624.1 MB)** was deleted in the
2026-11-08 window, so one stale copy did leave the box — but **`mnemosyne/data/mnemosyne.db.corrupted.20260809_120404` (23.6 MB) is still there**, so the rotation-snapshot exposure above is untouched.
## Origin (inferred - not established)
Almost certainly the **memory-capture path storing verbatim session text** in which a key was printed by a
tool call or a `.env` read. Not verified: no capture-time or delivery-time redaction filter is known to
exist, and the affected rows carry no marker distinguishing them from ordinary content.
## Remediation options (proposed, none executed)
1. **Rotate** the three Mnemosyne-exposed values and the Camofox key - touches live services; needs a
maintenance window and explicit approval.
2. **Redact or forget the affected rows** (`mnemosyne forget <id>` per row), then **re-take the rotation
snapshots** so the superseded values leave the backup set.
3. **Move the two Facebook Birthday jobs off prompt-embedded keys** - read the key from the environment or
config at run time.
4. **Add capture-time and delivery-time redaction** so `.env` values cannot enter memory rows, job prompts or
delivery records again.
5. **Decide a retention/serialisation policy for `state.db`** - it currently holds every transcript
verbatim at 1.89 GB (2,901 sessions / 197,682 messages) and it is growing.
## Verification evidence
- Key names and lengths extracted from `~/.hermes/.env` (values never printed), then matched as exact
substrings across every TEXT/BLOB column of every table in `mnemosyne.db` including the FTS mirrors -
row counts as tabled.
- Byte-membership scans of the 10 `.db` files under `~/.hermes/config-backup/`, the shared-surface DB, the
live `cron/deliveries.db` + `cron/executions.db`, the seven uncompressed Mnemosyne rotation snapshots, and
a chunked scan of `state.db`.
- `json.dumps()` membership test of each value against every job object in `jobs.json`.
- **Not done:** no value was printed, logged or written to any wiki page; nothing was deleted, rotated or
scrubbed.
## 2026-11-08 00:5x AWST — nightly re-check (still OPEN; one narrow improvement)
- **Cron job prompts: clean.** A regex sweep of every job in `/home/hermes/.hermes/cron/jobs.json` for
secret-shaped strings (`sk-…`, `ghp_…`, and any `<key|token|password|secret> = <16+ chars>`) returned
**0 hits** on 2026-11-08 00:5x. Only the *shape* was tested — a secret that does not look like a secret
would still pass.
- **Redaction setting: not confirmable from the file.** `~/.hermes/config.yaml` still carries a
`redact_secrets:` key, but its on-disk value reads as a masked token when printed, so the effective
on/off state could not be established by reading it. The 2026-11-05 finding — *"no capture-time or
delivery-time redaction filter is known to exist"* (line 80 above) — therefore **stands unrefuted**.
- **Nothing else moved:** no key rotation, no affected row redacted or forgotten, no remediation option
executed. Both **Remediation options 2 and 4 above remain open**.
## See also
- [[infrastructure/credentials-map]] - where secrets live, by design
- [[systems/mnemosyne]] - the memory store whose tables carry them
- [[systems/scheduled-tasks]] - the Facebook Birthday jobs' schedules
- [[SCHEMA]] - the no-values rule this violates