Files
agent-estate-wiki/incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
T

8.2 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05) incident open 2026-11-06 2026-11-09 2026-11-09 2026-11-13 high
security
credentials
mnemosyne
memory
sessions
cron
incident
live-env-scan
live-sqlite-mnemosyne
live-jobs-json
live-state-db
live-deliveries-db
live-config-backup

Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05)

Status: OPEN - found by the nightly audit (2026-11-05 window), not remediated. No key was rotated, no memory row deleted and no cron prompt scrubbed. This page records where the values live and how many rows carry them; no credential value appears on it (SCHEMA rule 1 - record where secrets live, never their values).

What was found

Three exposures, all verified by direct read on 2026-11-06 00:22-00:50 AWST.

1. Live .env values as plaintext rows in the memory store

17 rows of ~/.hermes/mnemosyne/data/mnemosyne.db carry values from ~/.hermes/.env. Method: every TEXT/BLOB column of every table, plus the FTS mirrors, scanned for exact substring membership.

Key (name only) Rows Columns carrying it
TELEGRAM_BOT_TOKEN 4 memories.content x1, episodic_memory.content x1, gists.text x1, fts_episodes.content x1
MATTERMOST_TOKEN 6 memories.content x1, episodic_memory.content x1, gists.text x2, memoria_facts.context_snippet x1, fts_episodes.content x1
HERMES_CUSTOM_OMNIROUTE_20128_API_KEY 7 memories.content x2, working_memory.content x1, episodic_memory.content x1, gists.text x1, fts_episodes.content x1, fts_working.content x1
  • Every hit is in mnemosyne.db (27.7 MB, live). The episodic_memory.db and memoria_facts.db files beside it are 0-byte legacy stubs (created 2026-10-27) - the tables with those names live inside mnemosyne.db. Nothing was found in the cross-agent shared surface (~/.hermes/data/shared/mnemosyne.db).
  • The rotation snapshots carry them too. Seven uncompressed snapshots under ~/.hermes/mnemosyne/backups/ (mnemosyne-pre-orphan-cleanup.db, -20261004_030035, -20261011_030046, -20261018_030021, -20261025_030108, -20261101_030033, -20261105_140314) contain all three values in full; the 2026-09-27 snapshot predates them. The .db.gz snapshots are compressed and were not substring-tested - assume the same until proven otherwise.

2. One live key embedded verbatim in two cron prompts

The CAMOFOX_API_KEY value appears in the prompt text of two enabled jobs in ~/.hermes/cron/jobs.json:

  • 0e753e9720fd - Facebook Birthday Monthly Scrape (0 9 1 * *)
  • fad5a86979d4 - Facebook Birthday Weekly Registry Sync (0 3 * * 1)

Consequence: the same value is inside delivery records in ~/.hermes/cron/deliveries.db. Of the 10 .db files under ~/.hermes/config-backup/ (six of them 0-byte), cron/deliveries.db is the only one that carries any tested value; cron/executions.db, cron/notepad.db and data/shared/mnemosyne.db carry none.

3. Session transcripts hold all four values

~/.hermes/state.db (1.7 GB at discovery — 1.89 GB / 1,890,656,256 B as at 2026-11-09 00:0x AWST, with 2,901 sessions / 197,682 messages and a live 40,157,672 B WAL; the store grew while nothing was retentioned), the session store, contains all four values. Measured as 4 MB chunks containing the value: CAMOFOX_API_KEY 178, MATTERMOST_TOKEN 64, HERMES_CUSTOM_OMNIROUTE_20128_API_KEY 61, TELEGRAM_BOT_TOKEN 22. This is the most likely origin of exposure 1 - the memory-capture path writes from session text, so a session in which a key was printed becomes a memory row - and it is itself a plaintext at-rest store that deleting memory rows does not clear.

Why it matters

  • Mnemosyne rows are recall material: they are pulled into a session's context by semantic or FTS match, so a secret in a row is a secret in a prompt - echoable into any cron delivery (Telegram, Mattermost, email), any subagent transcript, or the cross-agent shared surface.
  • It turns every backup into a credential copy. Deleting the rows does not recall the copies; rotation is the only remedy that fully closes it.
  • jobs.json is the fleet's source of truth and is exported and examined routinely; a prompt-embedded key is exposed to anything that reads job definitions.
  • state.db is the fleet's largest artifact (1.89 GB as at 2026-11-09, up from 1.7 GB) and is copied into snapshots and repair backups. ⚠ It is also the load source behind the 2026-11-07/08 hook-timeout cascade — a 40 MB WAL checkpointing on the same spindle as a 2-core guest → incidents/2026-11-08-pre-tool-call-hook-timeout-cascade. The 2026-08-11 corruption remnant state.db.corrupted.20260811_151052 (624.1 MB) was deleted in the 2026-11-08 window, so one stale copy did leave the box — but mnemosyne/data/mnemosyne.db.corrupted.20260809_120404 (23.6 MB) is still there, so the rotation-snapshot exposure above is untouched.

Origin (inferred - not established)

Almost certainly the memory-capture path storing verbatim session text in which a key was printed by a tool call or a .env read. Not verified: no capture-time or delivery-time redaction filter is known to exist, and the affected rows carry no marker distinguishing them from ordinary content.

Remediation options (proposed, none executed)

  1. Rotate the three Mnemosyne-exposed values and the Camofox key - touches live services; needs a maintenance window and explicit approval.
  2. Redact or forget the affected rows (mnemosyne forget <id> per row), then re-take the rotation snapshots so the superseded values leave the backup set.
  3. Move the two Facebook Birthday jobs off prompt-embedded keys - read the key from the environment or config at run time.
  4. Add capture-time and delivery-time redaction so .env values cannot enter memory rows, job prompts or delivery records again.
  5. Decide a retention/serialisation policy for state.db - it currently holds every transcript verbatim at 1.89 GB (2,901 sessions / 197,682 messages) and it is growing.

Verification evidence

  • Key names and lengths extracted from ~/.hermes/.env (values never printed), then matched as exact substrings across every TEXT/BLOB column of every table in mnemosyne.db including the FTS mirrors - row counts as tabled.
  • Byte-membership scans of the 10 .db files under ~/.hermes/config-backup/, the shared-surface DB, the live cron/deliveries.db + cron/executions.db, the seven uncompressed Mnemosyne rotation snapshots, and a chunked scan of state.db.
  • json.dumps() membership test of each value against every job object in jobs.json.
  • Not done: no value was printed, logged or written to any wiki page; nothing was deleted, rotated or scrubbed.

2026-11-08 00:5x AWST — nightly re-check (still OPEN; one narrow improvement)

  • Cron job prompts: clean. A regex sweep of every job in /home/hermes/.hermes/cron/jobs.json for secret-shaped strings (sk-…, ghp_…, and any <key|token|password|secret> = <16+ chars>) returned 0 hits on 2026-11-08 00:5x. Only the shape was tested — a secret that does not look like a secret would still pass.
  • Redaction setting: not confirmable from the file. ~/.hermes/config.yaml still carries a redact_secrets: key, but its on-disk value reads as a masked token when printed, so the effective on/off state could not be established by reading it. The 2026-11-05 finding — "no capture-time or delivery-time redaction filter is known to exist" (line 80 above) — therefore stands unrefuted.
  • Nothing else moved: no key rotation, no affected row redacted or forgotten, no remediation option executed. Both Remediation options 2 and 4 above remain open.

See also