audit 2026-11-06 (window 2026-11-05): incident ledger emptied 3 -> 0, first zero-failed day since 11-02, both 11-04 casualties recovered; NEW OPEN incident - live .env values verbatim in 17 mnemosyne rows, the 1.7 GB session store and two enabled Facebook Birthday prompts; freshness 0 stale / 124 fresh / 3 no-date of 127
This commit is contained in:
@@ -1,10 +1,10 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
|
||||
type: incident
|
||||
status: open
|
||||
status: resolved
|
||||
created: 2026-11-04
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
updated: 2026-11-06
|
||||
verified_on: 2026-11-06
|
||||
review_after: 2026-11-11
|
||||
confidence: high
|
||||
tags: [cron, job-radar, stream-timeout, silent-failure, notion, context-compaction]
|
||||
@@ -109,6 +109,18 @@ the pattern, not because the link is proven.
|
||||
5. Should a stream timeout on a *file-deliverable* job trigger an automatic retry instead of leaving the
|
||||
day's discovery to the next slot?
|
||||
|
||||
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
|
||||
|
||||
- The last open row raised here, `dca848_dfd4bea84f2a` (`timeout`, opened **2026-11-03 13:28:51**), **closed on
|
||||
2026-11-05 at 12:36:17** - together with its 11-04 successor row `dca848_3b1bb1e9b003`.
|
||||
- The job behind it, `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), **completed** its 2026-11-05
|
||||
07:00 run in **5 h 35 m 35 s** (07:00:42 -> 12:36:18), with no repeat of
|
||||
`RuntimeError: Upstream stream failed before completion.`
|
||||
- **Status -> `resolved`.** The underlying questions (whether the 65 m 21 s stream silence is the watchdog
|
||||
threshold or a silently dead socket; whether a stream timeout on a file-deliverable job should retry
|
||||
automatically) remain open in [[systems/scheduled-tasks]] - they are no longer tracked as an incident
|
||||
against this window.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
---
|
||||
title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)"
|
||||
type: incident
|
||||
status: open
|
||||
status: resolved
|
||||
created: 2026-11-05
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
updated: 2026-11-06
|
||||
verified_on: 2026-11-06
|
||||
confidence: high
|
||||
tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident]
|
||||
sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json]
|
||||
@@ -113,6 +113,18 @@ until re-tested** rather than as proof the route is dead. See [[runbooks/job-rad
|
||||
route incident - and whether the 11-03 row should be closed.
|
||||
4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]].
|
||||
|
||||
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
|
||||
|
||||
- **Both affected jobs ran clean on 2026-11-05.** `85edf6772bca` (*Tech & AI Newsletter Digest (Analytical)*)
|
||||
completed its 07:00 run in **7 m 01 s** (07:00:42 -> 07:07:43), its incident row closing at **07:07:42**;
|
||||
`dca8482e4f76` (*Daily Marketing Job Radar - Discovery*) completed in **5 h 35 m 35 s** and its two rows
|
||||
closed at **12:36:17**. **Fleet-wide unclosed incident rows: 3 -> 0** - the ledger was empty for the first
|
||||
time since 2026-11-02, and 2026-11-05 was the first zero-`failed` ledger day since then too.
|
||||
- The Stage 2 salvage documented above therefore stands as the whole recovery - no follow-up failure arrived.
|
||||
- The `CL_MODEL` / HTTP 402 cover-letter caveat is **unchanged and still needs a decision**: the
|
||||
`gemini/gemini-3.5-flash` pin was never written to `.env`, so it did not persist.
|
||||
- **Status -> `resolved`.** A recurrence should be opened as a new incident with its own window.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
title: Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05)
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-11-06
|
||||
updated: 2026-11-06
|
||||
verified_on: 2026-11-06
|
||||
review_after: 2026-11-13
|
||||
confidence: high
|
||||
tags: [security, credentials, mnemosyne, memory, sessions, cron, incident]
|
||||
sources: [live-env-scan, live-sqlite-mnemosyne, live-jobs-json, live-state-db, live-deliveries-db, live-config-backup]
|
||||
---
|
||||
|
||||
# Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05)
|
||||
|
||||
**Status: OPEN - found by the nightly audit (2026-11-05 window), not remediated.** No key was rotated, no
|
||||
memory row deleted and no cron prompt scrubbed. This page records *where* the values live and how many rows
|
||||
carry them; **no credential value appears on it** ([[SCHEMA]] rule 1 - record where secrets live, never their
|
||||
values).
|
||||
|
||||
## What was found
|
||||
|
||||
Three exposures, all verified by direct read on **2026-11-06 00:22-00:50 AWST**.
|
||||
|
||||
### 1. Live `.env` values as plaintext rows in the memory store
|
||||
|
||||
**17 rows** of `~/.hermes/mnemosyne/data/mnemosyne.db` carry values from `~/.hermes/.env`. Method: every
|
||||
TEXT/BLOB column of every table, plus the FTS mirrors, scanned for exact substring membership.
|
||||
|
||||
| Key (name only) | Rows | Columns carrying it |
|
||||
|---|---|---|
|
||||
| `TELEGRAM_BOT_TOKEN` | 4 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1 |
|
||||
| `MATTERMOST_TOKEN` | 6 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x2, `memoria_facts.context_snippet` x1, `fts_episodes.content` x1 |
|
||||
| `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` | 7 | `memories.content` x2, `working_memory.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1, `fts_working.content` x1 |
|
||||
|
||||
- **Every hit is in `mnemosyne.db`** (27.7 MB, live). The `episodic_memory.db` and `memoria_facts.db` files
|
||||
beside it are **0-byte legacy stubs** (created 2026-10-27) - the tables with those names live *inside*
|
||||
`mnemosyne.db`. Nothing was found in the cross-agent shared surface
|
||||
(`~/.hermes/data/shared/mnemosyne.db`).
|
||||
- **The rotation snapshots carry them too.** Seven uncompressed snapshots under
|
||||
`~/.hermes/mnemosyne/backups/` (`mnemosyne-pre-orphan-cleanup.db`, `-20261004_030035`, `-20261011_030046`,
|
||||
`-20261018_030021`, `-20261025_030108`, `-20261101_030033`, `-20261105_140314`) contain all three values in
|
||||
full; the 2026-09-27 snapshot predates them. The `.db.gz` snapshots are **compressed and were not
|
||||
substring-tested** - assume the same until proven otherwise.
|
||||
|
||||
### 2. One live key embedded verbatim in two cron prompts
|
||||
|
||||
The `CAMOFOX_API_KEY` value appears in the `prompt` text of two **enabled** jobs in
|
||||
`~/.hermes/cron/jobs.json`:
|
||||
|
||||
- `0e753e9720fd` - *Facebook Birthday Monthly Scrape* (`0 9 1 * *`)
|
||||
- `fad5a86979d4` - *Facebook Birthday Weekly Registry Sync* (`0 3 * * 1`)
|
||||
|
||||
Consequence: the same value is inside delivery records in `~/.hermes/cron/deliveries.db`. Of the **10 `.db`
|
||||
files under `~/.hermes/config-backup/`** (six of them 0-byte), `cron/deliveries.db` is the **only** one that
|
||||
carries any tested value; `cron/executions.db`, `cron/notepad.db` and `data/shared/mnemosyne.db` carry none.
|
||||
|
||||
### 3. Session transcripts hold all four values
|
||||
|
||||
`~/.hermes/state.db` (1.7 GB, the session store) contains all four values. Measured as 4 MB chunks
|
||||
containing the value: **`CAMOFOX_API_KEY` 178, `MATTERMOST_TOKEN` 64,
|
||||
`HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` 61, `TELEGRAM_BOT_TOKEN` 22**. This is the **most likely origin of
|
||||
exposure 1** - the memory-capture path writes from session text, so a session in which a key was printed
|
||||
becomes a memory row - and it is itself a plaintext at-rest store that deleting memory rows does not clear.
|
||||
|
||||
## Why it matters
|
||||
|
||||
- Mnemosyne rows are **recall material**: they are pulled into a session's context by semantic or FTS match,
|
||||
so a secret in a row is a secret in a prompt - echoable into any cron delivery (Telegram, Mattermost,
|
||||
email), any subagent transcript, or the cross-agent shared surface.
|
||||
- It turns every **backup** into a credential copy. Deleting the rows does not recall the copies;
|
||||
**rotation is the only remedy that fully closes it**.
|
||||
- `jobs.json` is the fleet's source of truth and is exported and examined routinely; a prompt-embedded key
|
||||
is exposed to anything that reads job definitions.
|
||||
- `state.db` is the fleet's largest artifact (1.7 GB) and is copied into snapshots and repair backups.
|
||||
|
||||
## Origin (inferred - not established)
|
||||
|
||||
Almost certainly the **memory-capture path storing verbatim session text** in which a key was printed by a
|
||||
tool call or a `.env` read. Not verified: no capture-time or delivery-time redaction filter is known to
|
||||
exist, and the affected rows carry no marker distinguishing them from ordinary content.
|
||||
|
||||
## Remediation options (proposed, none executed)
|
||||
|
||||
1. **Rotate** the three Mnemosyne-exposed values and the Camofox key - touches live services; needs a
|
||||
maintenance window and explicit approval.
|
||||
2. **Redact or forget the affected rows** (`mnemosyne forget <id>` per row), then **re-take the rotation
|
||||
snapshots** so the superseded values leave the backup set.
|
||||
3. **Move the two Facebook Birthday jobs off prompt-embedded keys** - read the key from the environment or
|
||||
config at run time.
|
||||
4. **Add capture-time and delivery-time redaction** so `.env` values cannot enter memory rows, job prompts or
|
||||
delivery records again.
|
||||
5. **Decide a retention/serialisation policy for `state.db`** - it currently holds every transcript
|
||||
verbatim at 1.7 GB.
|
||||
|
||||
## Verification evidence
|
||||
|
||||
- Key names and lengths extracted from `~/.hermes/.env` (values never printed), then matched as exact
|
||||
substrings across every TEXT/BLOB column of every table in `mnemosyne.db` including the FTS mirrors -
|
||||
row counts as tabled.
|
||||
- Byte-membership scans of the 10 `.db` files under `~/.hermes/config-backup/`, the shared-surface DB, the
|
||||
live `cron/deliveries.db` + `cron/executions.db`, the seven uncompressed Mnemosyne rotation snapshots, and
|
||||
a chunked scan of `state.db`.
|
||||
- `json.dumps()` membership test of each value against every job object in `jobs.json`.
|
||||
- **Not done:** no value was printed, logged or written to any wiki page; nothing was deleted, rotated or
|
||||
scrubbed.
|
||||
|
||||
## See also
|
||||
|
||||
- [[infrastructure/credentials-map]] - where secrets live, by design
|
||||
- [[systems/mnemosyne]] - the memory store whose tables carry them
|
||||
- [[systems/scheduled-tasks]] - the Facebook Birthday jobs' schedules
|
||||
- [[SCHEMA]] - the no-values rule this violates
|
||||
Reference in New Issue
Block a user