diff --git a/current-state.md b/current-state.md index 8939c48..ba9f65a 100644 --- a/current-state.md +++ b/current-state.md @@ -3,17 +3,40 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-11-05 -verified_on: 2026-11-05 -review_after: 2026-11-07 +updated: 2026-11-06 +verified_on: 2026-11-06 +review_after: 2026-11-08 confidence: high tags: [system] -sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents] +sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db] --- # Current State Updated: 2026-11-05 +## Recent Changes (2026-11-06, nightly audit; live probes 2026-11-06 00:1x-00:4x AWST) + +- **A clean ledger day and an empty incident ledger.** 2026-11-05 settled **175 rows - 173 `completed` / 2 + `running` / 0 `failed`** across 18 jobs: the **first zero-`failed` day since 2026-11-02**. **Unclosed + incident rows 3 -> 0** - the *Tech & AI Newsletter Digest* row closed at **07:07:42** and both *Job Radar - + Discovery* rows at **12:36:17**, when that job finally completed after **5 h 35 m 35 s**. Fleet non-clean + `last_status` **3 -> 1** (only `066bb5191280`, `delivery_failed`). Deliveries **10 rows / 10 `delivered`** + -> [[systems/scheduled-tasks]]. +- **The stocktake's three-day overrun streak broke, and it committed.** `1a8e90ea2271` ran **03:00:32 -> + 06:02:30 (3 h 01 m 58 s)** - **7 h 40 m -> 5 h 41 m -> 3 h 02 m** over three days - and **committed + `2f3fc99` at 05:55**, against its 11-04 state of "reported incomplete, committed nothing". +- **Long runs, not failures:** the **Mnemosyne Health Monitor**'s 00:00 slot took **7 h 01 m 28 s** (normal + 17-31 m) and its **12:00:45 slot was still running 12 h later with no delivery**; **Stage 2**'s 14:30 slot + took **8 h 56 m 17 s** and delivered at 23:27:09. +- **New open incident - live secrets stored verbatim.** Three `.env` values (`TELEGRAM_BOT_TOKEN`, + `MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit in plaintext in Mnemosyne rows - including + the FTS mirrors) - 4 + 6 + 7 = **17 rows** - reproduced in full by each of the **7 uncompressed rotation + snapshots**; all four values also sit verbatim in the **1.7 GB session store `state.db`**, the likely capture + origin. `CAMOFOX_API_KEY` sits verbatim in the prompts of two **enabled** Facebook Birthday jobs + (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Found, not remediated; no value is recorded + anywhere -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. +- **Fleet unchanged:** **29 jobs (26 enabled / 3 disabled)**, a fourth audit at the same count; no job added, + retired, disabled or repinned. ## Recent Changes (2026-11-05, stocktake; live probes 2026-11-05 05:1x-05:3x AWST) - **The `systems/skills-index.md` flip-flop is FIXED at the root - it had two writers, not one bad writer.** diff --git a/incidents/2026-11-03-job-radar-stream-active-timeout.md b/incidents/2026-11-03-job-radar-stream-active-timeout.md index 4b708fb..9a42d9a 100644 --- a/incidents/2026-11-03-job-radar-stream-active-timeout.md +++ b/incidents/2026-11-03-job-radar-stream-active-timeout.md @@ -1,10 +1,10 @@ --- title: Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff type: incident -status: open +status: resolved created: 2026-11-04 -updated: 2026-11-04 -verified_on: 2026-11-04 +updated: 2026-11-06 +verified_on: 2026-11-06 review_after: 2026-11-11 confidence: high tags: [cron, job-radar, stream-timeout, silent-failure, notion, context-compaction] @@ -109,6 +109,18 @@ the pattern, not because the link is proven. 5. Should a stream timeout on a *file-deliverable* job trigger an automatic retry instead of leaving the day's discovery to the next slot? +## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST) + +- The last open row raised here, `dca848_dfd4bea84f2a` (`timeout`, opened **2026-11-03 13:28:51**), **closed on + 2026-11-05 at 12:36:17** - together with its 11-04 successor row `dca848_3b1bb1e9b003`. +- The job behind it, `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), **completed** its 2026-11-05 + 07:00 run in **5 h 35 m 35 s** (07:00:42 -> 12:36:18), with no repeat of + `RuntimeError: Upstream stream failed before completion.` +- **Status -> `resolved`.** The underlying questions (whether the 65 m 21 s stream silence is the watchdog + threshold or a silently dead socket; whether a stream timeout on a file-deliverable job should retry + automatically) remain open in [[systems/scheduled-tasks]] - they are no longer tracked as an incident + against this window. + ## Related - [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] diff --git a/incidents/2026-11-04-upstream-stream-failure-0700-cohort.md b/incidents/2026-11-04-upstream-stream-failure-0700-cohort.md index aac767a..82f0442 100644 --- a/incidents/2026-11-04-upstream-stream-failure-0700-cohort.md +++ b/incidents/2026-11-04-upstream-stream-failure-0700-cohort.md @@ -1,10 +1,10 @@ --- title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)" type: incident -status: open +status: resolved created: 2026-11-05 -updated: 2026-11-05 -verified_on: 2026-11-05 +updated: 2026-11-06 +verified_on: 2026-11-06 confidence: high tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident] sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json] @@ -113,6 +113,18 @@ until re-tested** rather than as proof the route is dead. See [[runbooks/job-rad route incident - and whether the 11-03 row should be closed. 4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]]. +## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST) + +- **Both affected jobs ran clean on 2026-11-05.** `85edf6772bca` (*Tech & AI Newsletter Digest (Analytical)*) + completed its 07:00 run in **7 m 01 s** (07:00:42 -> 07:07:43), its incident row closing at **07:07:42**; + `dca8482e4f76` (*Daily Marketing Job Radar - Discovery*) completed in **5 h 35 m 35 s** and its two rows + closed at **12:36:17**. **Fleet-wide unclosed incident rows: 3 -> 0** - the ledger was empty for the first + time since 2026-11-02, and 2026-11-05 was the first zero-`failed` ledger day since then too. +- The Stage 2 salvage documented above therefore stands as the whole recovery - no follow-up failure arrived. +- The `CL_MODEL` / HTTP 402 cover-letter caveat is **unchanged and still needs a decision**: the + `gemini/gemini-3.5-flash` pin was never written to `.env`, so it did not persist. +- **Status -> `resolved`.** A recurrence should be opened as a new incident with its own window. + ## Related - [[incidents/2026-11-03-job-radar-stream-active-timeout]] diff --git a/incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md b/incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md new file mode 100644 index 0000000..e7100ba --- /dev/null +++ b/incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md @@ -0,0 +1,113 @@ +--- +title: Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05) +type: incident +status: open +created: 2026-11-06 +updated: 2026-11-06 +verified_on: 2026-11-06 +review_after: 2026-11-13 +confidence: high +tags: [security, credentials, mnemosyne, memory, sessions, cron, incident] +sources: [live-env-scan, live-sqlite-mnemosyne, live-jobs-json, live-state-db, live-deliveries-db, live-config-backup] +--- + +# Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05) + +**Status: OPEN - found by the nightly audit (2026-11-05 window), not remediated.** No key was rotated, no +memory row deleted and no cron prompt scrubbed. This page records *where* the values live and how many rows +carry them; **no credential value appears on it** ([[SCHEMA]] rule 1 - record where secrets live, never their +values). + +## What was found + +Three exposures, all verified by direct read on **2026-11-06 00:22-00:50 AWST**. + +### 1. Live `.env` values as plaintext rows in the memory store + +**17 rows** of `~/.hermes/mnemosyne/data/mnemosyne.db` carry values from `~/.hermes/.env`. Method: every +TEXT/BLOB column of every table, plus the FTS mirrors, scanned for exact substring membership. + +| Key (name only) | Rows | Columns carrying it | +|---|---|---| +| `TELEGRAM_BOT_TOKEN` | 4 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1 | +| `MATTERMOST_TOKEN` | 6 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x2, `memoria_facts.context_snippet` x1, `fts_episodes.content` x1 | +| `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` | 7 | `memories.content` x2, `working_memory.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1, `fts_working.content` x1 | + +- **Every hit is in `mnemosyne.db`** (27.7 MB, live). The `episodic_memory.db` and `memoria_facts.db` files + beside it are **0-byte legacy stubs** (created 2026-10-27) - the tables with those names live *inside* + `mnemosyne.db`. Nothing was found in the cross-agent shared surface + (`~/.hermes/data/shared/mnemosyne.db`). +- **The rotation snapshots carry them too.** Seven uncompressed snapshots under + `~/.hermes/mnemosyne/backups/` (`mnemosyne-pre-orphan-cleanup.db`, `-20261004_030035`, `-20261011_030046`, + `-20261018_030021`, `-20261025_030108`, `-20261101_030033`, `-20261105_140314`) contain all three values in + full; the 2026-09-27 snapshot predates them. The `.db.gz` snapshots are **compressed and were not + substring-tested** - assume the same until proven otherwise. + +### 2. One live key embedded verbatim in two cron prompts + +The `CAMOFOX_API_KEY` value appears in the `prompt` text of two **enabled** jobs in +`~/.hermes/cron/jobs.json`: + +- `0e753e9720fd` - *Facebook Birthday Monthly Scrape* (`0 9 1 * *`) +- `fad5a86979d4` - *Facebook Birthday Weekly Registry Sync* (`0 3 * * 1`) + +Consequence: the same value is inside delivery records in `~/.hermes/cron/deliveries.db`. Of the **10 `.db` +files under `~/.hermes/config-backup/`** (six of them 0-byte), `cron/deliveries.db` is the **only** one that +carries any tested value; `cron/executions.db`, `cron/notepad.db` and `data/shared/mnemosyne.db` carry none. + +### 3. Session transcripts hold all four values + +`~/.hermes/state.db` (1.7 GB, the session store) contains all four values. Measured as 4 MB chunks +containing the value: **`CAMOFOX_API_KEY` 178, `MATTERMOST_TOKEN` 64, +`HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` 61, `TELEGRAM_BOT_TOKEN` 22**. This is the **most likely origin of +exposure 1** - the memory-capture path writes from session text, so a session in which a key was printed +becomes a memory row - and it is itself a plaintext at-rest store that deleting memory rows does not clear. + +## Why it matters + +- Mnemosyne rows are **recall material**: they are pulled into a session's context by semantic or FTS match, + so a secret in a row is a secret in a prompt - echoable into any cron delivery (Telegram, Mattermost, + email), any subagent transcript, or the cross-agent shared surface. +- It turns every **backup** into a credential copy. Deleting the rows does not recall the copies; + **rotation is the only remedy that fully closes it**. +- `jobs.json` is the fleet's source of truth and is exported and examined routinely; a prompt-embedded key + is exposed to anything that reads job definitions. +- `state.db` is the fleet's largest artifact (1.7 GB) and is copied into snapshots and repair backups. + +## Origin (inferred - not established) + +Almost certainly the **memory-capture path storing verbatim session text** in which a key was printed by a +tool call or a `.env` read. Not verified: no capture-time or delivery-time redaction filter is known to +exist, and the affected rows carry no marker distinguishing them from ordinary content. + +## Remediation options (proposed, none executed) + +1. **Rotate** the three Mnemosyne-exposed values and the Camofox key - touches live services; needs a + maintenance window and explicit approval. +2. **Redact or forget the affected rows** (`mnemosyne forget ` per row), then **re-take the rotation + snapshots** so the superseded values leave the backup set. +3. **Move the two Facebook Birthday jobs off prompt-embedded keys** - read the key from the environment or + config at run time. +4. **Add capture-time and delivery-time redaction** so `.env` values cannot enter memory rows, job prompts or + delivery records again. +5. **Decide a retention/serialisation policy for `state.db`** - it currently holds every transcript + verbatim at 1.7 GB. + +## Verification evidence + +- Key names and lengths extracted from `~/.hermes/.env` (values never printed), then matched as exact + substrings across every TEXT/BLOB column of every table in `mnemosyne.db` including the FTS mirrors - + row counts as tabled. +- Byte-membership scans of the 10 `.db` files under `~/.hermes/config-backup/`, the shared-surface DB, the + live `cron/deliveries.db` + `cron/executions.db`, the seven uncompressed Mnemosyne rotation snapshots, and + a chunked scan of `state.db`. +- `json.dumps()` membership test of each value against every job object in `jobs.json`. +- **Not done:** no value was printed, logged or written to any wiki page; nothing was deleted, rotated or + scrubbed. + +## See also + +- [[infrastructure/credentials-map]] - where secrets live, by design +- [[systems/mnemosyne]] - the memory store whose tables carry them +- [[systems/scheduled-tasks]] - the Facebook Birthday jobs' schedules +- [[SCHEMA]] - the no-values rule this violates diff --git a/index.md b/index.md index 8725279..239f6ba 100644 --- a/index.md +++ b/index.md @@ -3,15 +3,19 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-05 -verified_on: 2026-11-05 +updated: 2026-11-06 +verified_on: 2026-11-06 confidence: high -tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence] +tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident] --- # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-11-05 (stocktake - **the `systems/skills-index.md` flip-flop is fixed at the root**: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> [[incidents/2026-11-05-skills-index-generator-divergence]]. Docker inventory corrected against live probes on all five hosts - the `worldmonitor-seeder-scheduled` container recorded yesterday **does not exist on any host**. Tailscale node table re-verified programmatically - 52 rows, 52 live peers, every state matching. Freshness **0 stale / 122 fresh / 3 no-date of 125**.) +> model routing and automation. Last updated: 2026-11-06. **2026-11-06 (nightly audit) - the incident ledger emptied (3 -> 0), 11-05 was the first zero-`failed` day since 11-02, the 03:00 stocktake's overrun streak broke, and a new OPEN incident records live secrets stored verbatim.** +> +> **2026-11-05 (stocktake) -** **the `systems/skills-index.md` flip-flop is fixed at the root**: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> [[incidents/2026-11-05-skills-index-generator-divergence]]. Docker inventory corrected against live probes on all five hosts - the `worldmonitor-seeder-scheduled` container recorded yesterday **does not exist on any host**. Tailscale node table re-verified programmatically - 52 rows, 52 live peers, every state matching. Freshness **0 stale / 122 fresh / 3 no-date of 125**.) +> +> **2026-11-06 (nightly audit - window 2026-11-05) - a clean ledger day and an empty incident ledger:** 2026-11-05 settled **175 rows - 173 `completed` / 2 `running` / 0 `failed`** across 18 jobs, the **first zero-`failed` day since 2026-11-02**. The only two `running` rows are the **Mnemosyne Health Monitor** (12:00:45 slot, still in flight **12 h+** at audit time) and this audit itself. **Unclosed incident rows 3 -> 0** - the *Newsletter Digest* row closed at **07:07:42** and both *Job Radar - Discovery* rows at **12:36:17**, when `dca8482e4f76` finally completed after **5 h 35 m 35 s**; that closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, `delivery_failed`, streak 0); deliveries **10 rows / 10 `delivered`**; the 03:00 stocktake overran **3 h 01 m 58 s** (down from 7 h 40 m and 5 h 41 m) and **committed `2f3fc99` at 05:55**. **Fleet unchanged at 29 jobs (26 / 3)** - a fourth audit at the same count. ⚠ **New and open:** live `.env` values (`TELEGRAM_BOT_TOKEN`, `MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit verbatim in **17 rows of `mnemosyne.db`** (`memories`, `episodic_memory`, `working_memory`, `gists`, `memoria_facts.context_snippet` and the FTS mirrors) - reproduced in full by each of the **7 uncompressed rotation snapshots** - and all four values sit verbatim in the **1.7 GB session store `state.db`** too; `CAMOFOX_API_KEY` sits verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and therefore in `cron/deliveries.db` -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. > > **2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze:** the **11-04 fleet settled 170 rows - 167 `completed` / 2 `failed` / 1 `running`**, **non-clean `last_status` 2 -> 3**, **unclosed incident rows 1 -> 3**, **deliveries 8 rows / 8 `delivered`**. Both failures are the *same* upstream fault: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) died **36 seconds apart at 09:41-09:42**, each after **2 h 41 m**, on `RuntimeError: Upstream stream failed before completion.` - both pinned `custom:omni-101` / `smart-route`. **The day was not lost:** the 11:30 Stage 2 slot **reconstructed the missing handoff from Stage 1's own candidate files**, re-verified every candidate live, and still wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. The cover-letter route survived only on an override: `regenerate_cover_letter.py` defaults to `deepseek-paid-smart`, which threw **HTTP 402** mid-run, so the letter was pinned to `gemini/gemini-3.5-flash` via `CL_MODEL` - and that override **was never written to `.env`**, so it did not persist. And **CT299 was frozen twice, not once**: a second, **manual** `vzdump 299 ... --storage ssd0` wedged from **11-04 08:30:04** until **11-05 ~00:30** (~16 h); the live probe at 00:54:21 AWST shows `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. ⚠ **Corrected the same night:** the first draft of this entry read the fleet as **158 jobs (144 / 14)** and the > 11-04 ledger as closing `1 item_suppressed`; live reads give **29 jobs (26 / 3)** and a final row of **1 `running`** @@ -58,6 +62,7 @@ tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, gene - [[systems/skills-index]] — Installed skills inventory (**302 skills** (**68 categories**) as of the **2026-11-05** stocktake; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (46 archived skills) — the earlier 346/347 counts included it. Freshness at the 2026-11-05 run: **0 stale / 122 fresh / 3 no-date of 125 pages** (`freshness-20261105.txt`). Note: `SKILL.md` files on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so **302 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents +- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **OPEN.** Live `.env` values were found stored verbatim in **17 rows of `mnemosyne.db`** (`memories` 4, `episodic_memory` 3, `working_memory` 1, `gists` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the **7 uncompressed rotation snapshots**, and all four values also sit verbatim in the **1.7 GB session store `state.db`**, which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Nothing rotated, deleted or scrubbed; no value recorded. - [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter). - [[incidents/2026-11-05-skills-index-generator-divergence]] — **RESOLVED — the skills index flip-flop had two writers.** The repo `wiki-maintain.sh` (302 skills / 68 categories) and the stale `~/.hermes/scripts/wiki-maintain.sh` copy (349 / 26, because it used `Path.rglob()` and counted the parked `.archive/`) were **different scripts**, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with **no error raised by either**. The stale copy is now a **delegating wrapper**; the repo script gained `--no-build` so the wrapper honours the old interface without a second generator. - [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] — **OPEN — the app's "local" voice mode never applies, and the connection test lies.** The test checks **socket reachability only**, while every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects `session.update` as an unknown event**. `voice.mode` is a **phone-side** selector — **no `mode` key exists** in `config.yaml → voice` or `conduit-settings.json` — so a "mode did not apply" cannot be diagnosed on host config. Found via session `20261031_193709_40b9c2` (19:37:09, 30 msgs / 17 tool calls); `voice.local_live.front_model` was repointed to `mistral/voxtral-small-latest` during it and **nothing was changed on the Mac**. → [[systems/conduit]] diff --git a/log.md b/log.md index 5ffd268..3528b6f 100644 --- a/log.md +++ b/log.md @@ -2080,3 +2080,49 @@ embedded credential was seen during verification and is not reproduced here ([RE `incidents/2026-08-15-skills-index-drift.md`, `wiki-maintain.sh`, `log.md`. - **Not done / unchanged:** no job added, removed, disabled or re-pinned; no page removed or archived; no decision reversed. **No secrets, passwords, tokens or keys written or quoted.** + +## [2026-11-06] audit - window 2026-11-05 + +- **Window reviewed:** 2026-11-05 00:00-24:00 AWST; live reads 2026-11-06 00:1x-01:0x AWST. Fleet unchanged at + **29 jobs (26 enabled / 3 disabled)** - a fourth consecutive audit at the same count, no job added, retired, + disabled or re-pinned. +- **New page: [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (status: OPEN).** Three live + `.env` values (`TELEGRAM_BOT_TOKEN`, `MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit in + plaintext in **17 rows of `~/.hermes/mnemosyne/data/mnemosyne.db`** - `memories.content` 4, `episodic_memory.content` + 3, `working_memory.content` 1, `gists.text` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - and + `CAMOFOX_API_KEY` sits verbatim in the `prompt` of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, + `fad5a86979d4`) and hence in `cron/deliveries.db`. Method: exact-substring match of each `.env` value across + **every TEXT/BLOB column of every table** in `mnemosyne.db` plus its FTS mirrors; byte-membership scans of the + 10 `.db` files under `~/.hermes/config-backup/` (six are 0-byte; only `cron/deliveries.db` carries any tested + value), the shared-surface DB, the seven uncompressed rotation snapshots under `~/.hermes/mnemosyne/backups/` + (all seven reproduce the DB in full) and a chunked scan of the **1.7 GB session store `state.db`**, which + carries all four values and is the likely capture origin (capture writes verbatim session text). **Nothing was + rotated, deleted or scrubbed; no credential value was printed, logged or written to any page.** +- **Both 11-04 casualties recovered in full.** `85edf6772bca` (*Tech & AI Newsletter Digest*) completed its 07:00 + run in **7 m 01 s** (07:00:42 -> 07:07:43) and `dca8482e4f76` (*Daily Marketing Job Radar - Discovery*) in + **5 h 35 m 35 s** (07:00:42 -> 12:36:18); neither re-raised `RuntimeError: Upstream stream failed before + completion.` The 11-05 ledger settled **175 rows / 18 jobs - 173 `completed` / 2 `running` / 0 `failed`**, the + **first zero-`failed` day since 2026-11-02**; the two `running` rows are `ea79a2107e36` (*Mnemosyne Health + Monitor*, 12:00:45 slot, still in flight 12 h+ with no delivery) and this audit (`cbfe434a3b5e`). Deliveries + **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, + *Uber Eats Monthly Spend Report*, `delivery_failed`, streak 0). +- **Incident ledger emptied.** Unclosed rows **3 -> 0**: the digest row closed **07:07:42** and both Job Radar rows + (`dca848_dfd4bea84f2a` `timeout`, `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. + `incidents/2026-11-03-job-radar-stream-active-timeout.md` -> `status: resolved` with a Resolution section; + `incidents/2026-11-04-upstream-stream-failure-0700-cohort.md` closed out - the first empty ledger since + 2026-11-02. +- **Updated [[systems/scheduled-tasks]]** - frontmatter (`updated` / `verified_on` -> 2026-11-06, `review_after` + 2026-11-07, `live-sessions-db` added to `sources`), a new `## Count` bullet for the 2026-11-06 read and a new + `## Re-verified 2026-11-06` span (recoveries, zero-`failed` ledger, delivery ledger, the secrets finding). +- **Updated [[systems/mnemosyne]]** - frontmatter dates -> 2026-11-06 (`review_after` 2026-11-13), `sources` + + `live-secrets-scan, live-state-db`, and a **security entry at the head of `## Known Issues`** recording that the + store's own tables carry 17 rows of live credential material, that recall is the exposure path, and that only + rotation covers the copies already taken. +- **Updated [[current-state]], [[index]]** (`## Incidents` link for the new page, header summary rewritten to the + 2026-11-06 read) and this log. +- **Freshness:** `raw/stocktake/freshness-20261106.txt` (re-run after this audit's writes) -> **0 stale (>=90d), + 124 fresh, 3 no-date of 127 pages**, the new incident page included. The three no-date pages are unchanged (`SCHEMA.md`, + `experiments/active/example.md`, `log.md`). +- **Not done / unchanged:** no page removed or archived; no decision reversed; `066bb5191280` and the + `deepseek-paid-smart`-on-`omniroute` dispatch fault carried, not re-diagnosed. **No secrets, passwords, tokens + or keys written or quoted** - the incident page records where secrets live, never their values. diff --git a/raw/stocktake/freshness-20261106.txt b/raw/stocktake/freshness-20261106.txt new file mode 100644 index 0000000..71a3987 --- /dev/null +++ b/raw/stocktake/freshness-20261106.txt @@ -0,0 +1,140 @@ +# Wiki freshness audit — 2026-11-06 +Threshold: >90 days since `updated` + +## STALE (0) + + +## FRESH (124) + +- [ 17d] CONTRIBUTING.md +- [ 11d] agents/hermes-chaos.md +- [ 11d] agents/hermes-production.md +- [ 17d] agents/index.md +- [ 8d] agents/nanobot.md +- [ 17d] agents/open-claw.md +- [ 2d] agents/qwenpaw.md +- [ 17d] agents/zeroclaw.md +- [ 8d] comparisons/camoufox-vs-chrome.md +- [ 17d] comparisons/hermes-vs-nanobot.md +- [ 17d] comparisons/memory-backends.md +- [ 17d] comparisons/model-routers.md +- [ 8d] comparisons/steel-vs-camoufox.md +- [ 17d] concepts/agent-memory.md +- [ 8d] concepts/browser-automation.md +- [ 17d] concepts/model-routing.md +- [ 17d] concepts/persistent-shells.md +- [ 17d] concepts/tool-calling.md +- [ 0d] current-state.md +- [ 8d] decisions/index.md +- [ 17d] decisions/keep-openrouter-as-fallback.md +- [ 17d] decisions/persistent-shell-enabled.md +- [ 8d] decisions/rollback-steel-to-camofox.md +- [ 17d] experiments/index.md +- [ 13d] incidents/2026-07-29-mnemosyne-relay.md +- [ 17d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 13d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 15d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 1d] incidents/2026-08-15-skills-index-drift.md +- [ 2d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 12d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 17d] incidents/2026-08-29-gmail-read-state.md +- [ 16d] incidents/2026-09-02-sftpgo-outage.md +- [ 2d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 8d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 11d] incidents/2026-10-03-ct450-hermes-update.md +- [ 11d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 8d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 13d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 5d] incidents/2026-10-09-memory-recall-relevance.md +- [ 21d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 10d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 18d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 10d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 15d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 12d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 11d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 11d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 11d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 9d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 7d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 7d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md +- [ 7d] incidents/2026-10-29-conduit-voice-silence.md +- [ 6d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +- [ 7d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md +- [ 6d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 5d] incidents/2026-10-31-asana-access-failures.md +- [ 5d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md +- [ 5d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md +- [ 4d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md +- [ 3d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md +- [ 2d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md +- [ 0d] incidents/2026-11-03-job-radar-stream-active-timeout.md +- [ 0d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md +- [ 1d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md +- [ 0d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md +- [ 1d] incidents/2026-11-05-skills-index-generator-divergence.md +- [ 0d] index.md +- [ 2d] infrastructure/backups.md +- [ 17d] infrastructure/credentials-map.md +- [ 1d] infrastructure/docker-services.md +- [ 17d] infrastructure/domains-and-tunnels.md +- [ 1d] infrastructure/hosts.md +- [ 1d] infrastructure/network-map.md +- [ 5d] infrastructure/proxmox.md +- [ 1d] infrastructure/tailscale.md +- [ 2d] queries/retained-research-answers.md +- [ 12d] reports/communities-wa-social-media-audit-2026.md +- [ 5d] reports/uber-eats/2026-07.md +- [ 5d] reports/uber-eats/2026-08.md +- [ 5d] reports/uber-eats/2026-09.md +- [ 5d] reports/uber-eats/2026-10.md +- [ 5d] reports/uber-eats/index.md +- [ 17d] runbooks/backup-wiki.md +- [ 17d] runbooks/diagnose-docker.md +- [ 17d] runbooks/gateway-resource-resilience.md +- [ 1d] runbooks/job-radar.md +- [ 17d] runbooks/multiplexer-setup.md +- [ 11d] runbooks/provider-health.md +- [ 8d] runbooks/recover-docker-service.md +- [ 17d] runbooks/restart-browser.md +- [ 17d] runbooks/restart-hermes.md +- [ 17d] runbooks/restore-openrouter.md +- [ 17d] runbooks/test-openrouter.md +- [ 17d] runbooks/troubleshoot-tool-loop.md +- [ 11d] runbooks/update-hermes-safely.md +- [ 3d] systems/browser-backend.md +- [ 17d] systems/cloudflare-tunnel.md +- [ 5d] systems/conduit.md +- [ 11d] systems/current-profile.md +- [ 17d] systems/guanaco.md +- [ 17d] systems/headless-server.md +- [ 7d] systems/hermes-agent.md +- [ 2d] systems/hermes-hq-push.md +- [ 17d] systems/hermes-ui.md +- [ 2d] systems/home-assistant.md +- [ 7d] systems/homepage-dashboard.md +- [ 2d] systems/ignis.md +- [ 11d] systems/memory-backend.md +- [ 4d] systems/messaging-integrations.md +- [ 0d] systems/mnemosyne.md +- [ 11d] systems/model-providers.md +- [ 2d] systems/nanobot.md +- [ 17d] systems/nextcloud.md +- [ 17d] systems/obsidian.md +- [ 5d] systems/omniroute.md +- [ 17d] systems/opencode-go.md +- [ 17d] systems/openrouter.md +- [ 2d] systems/paseo.md +- [ 18d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 1d] systems/skills-index.md +- [ 17d] systems/terminal-backend.md +- [ 17d] systems/tool-search.md + +## NO updated DATE (3) + +- SCHEMA.md +- experiments/active/example.md +- log.md + +TOTAL: 0 stale, 124 fresh, 3 no-date of 127 pages \ No newline at end of file diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index 8cff9f9..3b1de07 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,12 +3,12 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-11-01 -verified_on: 2026-11-01 -review_after: 2026-11-07 +updated: 2026-11-06 +verified_on: 2026-11-06 +review_after: 2026-11-13 confidence: high tags: [memory, mnemosyne, backend] -sources: [live-probe, mnemosyne-db, config, ops-profile-config] +sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db] --- # Mnemosyne @@ -50,6 +50,8 @@ Primary agent memory store for Hermes. ## Known Issues +**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**. + **Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below. **Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. ~~Ranking relevance is the live issue, not coverage~~ — see [[incidents/2026-10-09-memory-recall-relevance]]. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 4685cdd..3483a5d 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,17 +3,18 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-11-05 -verified_on: 2026-11-05 -review_after: 2026-11-06 +updated: 2026-11-06 +verified_on: 2026-11-06 +review_after: 2026-11-07 confidence: high tags: [cron, scheduled, automation] -sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db] +sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db] --- # Scheduled Tasks ## Count +- **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. - **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. -> [[runbooks/job-radar]] `CL_MODEL` caveat. - **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` / @@ -963,3 +964,87 @@ unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69 whether the five-compaction churn contributes; whether a stream timeout on a file-deliverable job should **retry automatically**; `radar-ref/handoff/` retention (newest file 10-30); why Stage 2 still has no handoff to wait on; and why the 03:00 stocktake takes **7 h 40 m** and commits nothing. + +## Re-verified 2026-11-06 (window 2026-11-05, AWST; live reads 2026-11-06 00:1x-00:4x) + +### The day was a recovery day, not a repeat + +- **Both 11-04 casualties ran clean.** `85edf6772bca` (*Tech & AI Newsletter Digest (Analytical)*, streak 1) + completed its 07:00 run in **7 m 01 s** (07:00:42 -> 07:07:43); its incident row closed at **07:07:42**. + `dca8482e4f76` (*Daily Marketing Job Radar - Discovery*, streak 2, dead twice on 11-04) also completed its + 07:00 run - but took **5 h 35 m 35 s** (07:00:42 -> 12:36:18), and that duration is what held its two + incident rows open until **12:36:17**. Neither job re-raised `RuntimeError: Upstream stream failed before + completion.` +- **No `unknown`, `timeout` or `error` row exists anywhere in the 11-05 ledger** - the first such day since + 11-02. +- **Deliveries: 10 rows, all `delivered`, none flagged `for_failure`** - 07:02:11, 07:07:42, 07:30:39, + 08:06:00, 08:35:30, 10:41:20, 12:15:25, 18:45:49, 19:25:00, 23:27:09 AWST. + +### Long runs (completed, not failures) + +| Job | Slot | Duration | Against its norm | +|---|---|---|---| +| `ea79a2107e36` Mnemosyne Health Monitor | 00:00 | **7 h 01 m 28 s** (00:00:43 -> 07:02:11) | normal 17-31 m | +| `1a8e90ea2271` wiki-stocktake-maintenance | 03:00 | **3 h 01 m 58 s** (03:00:32 -> 06:02:30) | 11-02 7 h 40 m, 11-03 5 h 41 m | +| `dca8482e4f76` Job Radar - Discovery | 07:00 | **5 h 35 m 35 s** | recovered from 2 failures | +| `5fc292ff0f3b` Job Radar - Stage 2 | 14:30 | **8 h 56 m 17 s** (14:30:53 -> 23:27:10) | delivered 23:27:09 | +| `5fc292ff0f3b` Job Radar - Stage 2 | 11:30 | 10 m 33 s | normal | +| `85edf6772bca` Newsletter Digest | 07:00 | 7 m 01 s | normal | + +### The 03:00 stocktake overran a third day - and committed this time + +`1a8e90ea2271` ran **03:00:32 -> 06:02:30 (3 h 01 m 58 s)**, so the streak now reads +**7 h 40 m -> 5 h 41 m -> 3 h 02 m** (improving, still ~6x norm), and it **did commit** its artefacts +(`2f3fc99`, 05:55) - against the 11-04 state where it "reported incomplete and committed nothing". The vault +repo `/home/hermes/wiki` picked that up at 05:58 (`ee00b5c`). The nightly audit did **not** have to commit the +skills-index on the stocktake's behalf this pass. + +### Sessions in the window + +**26 sessions had activity touching 2026-11-05** - 21 `cron`, 4 `subagent`, 1 `desktop`. The only long-form +one is the morning **Mnemosyne Health Monitor** run (`local-b433c51e...`, `cron`, 17 messages, 07:02-07:50): +it requested tool approvals and received denials. No configuration change came out of it, and it is +consistent with that job's 7 h 01 m ballooning above. + +### Still open (2026-11-06 - carries its predecessor forward) + +Everything on the 2026-11-04 list stands unchanged and not re-probed this window (single-node no-failover +topology; the GitHub Trending Radar delivery token; Camofox containment under multi-tab load; the Conduit +health store with no retention; the unreconciled admission files; the unidentified `-15` signal source; the +30 s worker-ack window vs cold-start; whether a stream-silence timeout should retry a file-deliverable job; +`radar-ref/handoff/` retention). **New this window:** + +- **Live secrets sit verbatim in Mnemosyne rows and in two cron prompts.** Three `.env` values (`TELEGRAM_BOT_TOKEN`, + `MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) appear in `memories`, `episodic_memory`, + `working_memory`, `gists` and `memoria_facts.context_snippet` plus the FTS mirrors - **17 rows**, 4 + 6 + 7 per + key - and each of the **7 uncompressed rotation snapshots** reproduces the whole DB; all four values also sit + verbatim in the **1.7 GB session store `state.db`**, the likely capture origin. The Camofox key is embedded in + the prompts of `0e753e9720fd` and `fad5a86979d4` (both **enabled** Facebook Birthday jobs) and therefore in + `cron/deliveries.db` - the only `config-backup/` DB carrying any tested value. **Found, not remediated** -> + [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. +- **`ea79a2107e36` ballooned twice out of two runs** (7 h 01 m from the 00:00 slot, then still running 12 h + after its 12:00:45 slot) - against a 17-31 m normal. Cause unexplained; the 12:00 run had delivered nothing + at audit time. +- **`066bb5191280`** (Uber Eats Monthly Spend Report) is now the fleet's **only** non-clean `last_status` + (`delivery_failed`, streak 0) - carried, not re-diagnosed this window. +- **`deepseek-paid-smart` on `omniroute` (CT600) - carried, not re-probed this window.** 11-05 read: HTTP + **503** `ALL_TARGETS_SKIPPED` on `omniroute:20128` vs HTTP **200** on `omniroute-1` + (100.78.63.20:20128 -> `deepseek/deepseek-v4-flash`). This audit remains the fleet's only + `deepseek-paid-smart` pin (provider `custom:omni-101`), which is why it still completes. + +### Freshness + +`raw/stocktake/freshness-20261106.txt` (re-run after this audit's writes): **0 stale (>=90d), 124 fresh, 3 +no-date of 127 pages** - the new incident page included. The three no-date pages are unchanged (`SCHEMA.md`, +`experiments/active/example.md`, `log.md`). + +### Files touched by this audit + +- **New:** `incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md`, + `raw/stocktake/freshness-20261106.txt`. +- **Updated:** this page (+frontmatter, +Count bullet, +this section), `current-state.md`, `index.md`, + `log.md`, `incidents/2026-11-03-job-radar-stream-active-timeout.md`, + `incidents/2026-11-04-upstream-stream-failure-0700-cohort.md` (both -> `status: resolved` with closure + timestamps), and the vault mirror (`/home/hermes/wiki`, both conventions). +- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no + decision reversed.