diff --git a/current-state.md b/current-state.md index c2c7f0e..15ff4ae 100644 --- a/current-state.md +++ b/current-state.md @@ -3,16 +3,23 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-11-14 -verified_on: 2026-11-14 -review_after: 2026-11-14 +updated: 2026-11-15 +verified_on: 2026-11-15 +review_after: 2026-11-15 confidence: high tags: [system] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db] --- # Current State -Updated: 2026-11-13 +Updated: 2026-11-15 + +## Recent Changes (2026-11-14, nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) + +- **A third TTS stack landed on CT200** — new page [[systems/phonon-tts]]: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model `gradium/phonon-40m-en`), `phonon-demo` (`:8882`, WASM UI on `nginx:alpine`), `ptts-ws-server` (`:8883`, WebSocket, locally-built image). CT200 went **61 → 65 running / 68 → 72 total**; verified live from CT460 (health 200, models list, MP3 returned). ⚠ Everything after **09:03 AWST** on 11-14 is **not attributable to any retained transcript** — see the page's provenance gap before crediting it to a session. +- ⚠ **Open incident: Mnemosyne capture stalled** since **2026-11-14 09:32:59 AWST** (~15 h at this read) — new page [[incidents/2026-11-14-mnemosyne-capture-stall]]. The health monitor escalated only at **2026-11-15 00:03:35** (exit 1, `ea79a2_601037516abd`, `alerted`, unclosed), after logging the same hole as ℹ️ at 12:00 AWST. Root cause not established; the known stringified-config killer is ruled out. +- **Fleet clean apart from the monitor's own alert:** 2026-11-14 settled **167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**; `cron_incidents` **0 → 1 unclosed** (the monitor row above). Fleet **29 jobs (26 enabled / 3 disabled)**, **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, streak 1). +- [[infrastructure/docker-services]] CT200 inventory re-verified and updated — **64 non-seeder running / 72 total** (was 61 / 68); the delta is exactly the three TTS containers, no removals. ## Recent Changes (2026-11-13, nightly audit; live probes 2026-11-13 23:59 – 2026-11-14 00:2x AWST) diff --git a/incidents/2026-11-14-mnemosyne-capture-stall.md b/incidents/2026-11-14-mnemosyne-capture-stall.md new file mode 100644 index 0000000..6d35db7 --- /dev/null +++ b/incidents/2026-11-14-mnemosyne-capture-stall.md @@ -0,0 +1,108 @@ +--- +title: Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15 +type: incident +status: active +created: 2026-11-15 +updated: 2026-11-15 +verified_on: 2026-11-15 +review_after: 2026-11-17 +confidence: high +tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident] +sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db] +--- + +# Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST) + +**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**, +and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it +(exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state **`alerted`**, **unclosed**). + +Found by the **2026-11-14 nightly audit**; not caused by anything in that audit's window (it started +the previous morning and simply had not been noticed, because the monitor's own earlier reading of +the same condition was classified informational — see below). + +## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST) + +- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e. + **2026-11-14 09:32:59 AWST** (the row's own `timestamp` field reads `2026-11-14T09:32:59`). +- Newest `episodic_memory.created_at`: **`2026-11-14 01:33:34`** (09:33:34 AWST). **Both** write paths + stopped inside the same minute. +- Last 8 `working_memory` writes, all on 11-14 (UTC): `01:01:20, 01:02:39, 01:02:40, 01:03:28, + 01:04:19, 01:32:58, 01:32:59, 01:32:59` — then **nothing for ~15 hours**. +- Rows by day (UTC `created_at`): **11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15**. + So the pipeline was *low-volume but alive* before the stall, and the stall is not a normal quiet stretch. +- Totals at audit time: **working 1130 / episodic 1249**, DB **27 MB**. +- Gap: at the 00:03:35 alert the newest capture was **14 h 30 m 36 s** old; at this audit's read + (~00:40) **≈15 h 7 m**. + +## The alert (and its mislabelling) + +``` +🚨 Mnemosyne Health Monitor — exception(s): + +STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop) +has no working_memory row (last capture 2026-11-14 01:32:59) +``` +- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`, + `first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`. + Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`. +- ⚠ **The `auth` label looks wrong** — the failure is a *staleness* check, not a credential fault, and + the page records no auth error text. **Observation, not verified diagnosis.** +- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a + ~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own + log line below. + +## The earlier, quieter reading — 12:00:52 AWST 2026-11-14 + +`~/.hermes/logs/mnemosyne-health-monitor.log` shows the **12:00:52 AWST** run already looking at the +same hole and calling it benign: + +``` +ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min +``` + +That is **9 h 49.5 m** of missing capture, logged as ℹ️ informational (the delay is **negative**, i.e. +capture sits *behind* the message clock — the sign convention makes a growing gap look small). The +job exited 0 at 12:00 and 18:03; it only escalated at 00:03 the next morning. +**So the fault had ~12 hours of lead time that produced no alarm.** + +## Impact + +- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST** + (the last user message the monitor could see), all Telegram traffic, and every cron session in the + window. Recall over that period will answer from nothing. +- The **23:30 AWST sleep/consolidation cycle ran and found 0 items** to consolidate + (`Last consolidation: 2026-11-14T15:30:51 UTC (0 items)`), so consolidation is *not* compensating. +- Nothing else in the estate broke: **2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / + 1 `running`** in `executions.db`, deliveries **4 / 4 `delivered`**, and no other cron row went red. This is a + **memory-path incident only**. The one exception is the monitor's **own** job: it is now the fleet's + **single non-clean row** (`last_status: error`, `failure_streak` 1, previously `ok` / 0) and + **`cron_incidents` went 0 → 1 unclosed** (138 rows total). **2026-11-15 so far: 8 rows — 7 + `completed` / 1 `failed`** (this job), 1 delivery `delivered`. + +## Not the known config killer + +The previously recorded capture-killer on this host is a **stringified list/dict** written into +`~/.hermes/config.yaml` by `hermes config set`. Checked and **clean**: +- `config.yaml` **mtime 2026-11-13 18:03** — the file was not modified during or before the stall began. +- A full YAML walk found **no** scalar that begins with `[` or `{`; `memory.mnemosyne.*` parses as + proper lists (`ignore_patterns: []`, `skip_contexts`, `sync_roles`). + +## Open / unverified + +- **Root cause: not established.** No capture-path error lines appear in `agent.log` after 09:33 — + the writes simply stop. Candidates to test next: a stuck capture worker or watermark, the + `pulse`/heartbeat traffic that ran 12:03–19:22, session-side `memory_enabled` / + `mnemosyne.profile_isolation` interaction, or the sleep cycle's `consolidation_claimed_at` state. +- ✅ **Resolved — the 00:03 escalation DID deliver.** `deliveries.db` holds exactly one 2026-11-15 row: + **`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the + incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked; + the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]). +- Whether capture resumed after this audit's read (the monitor's next slot is the natural check). + +## Related + +- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence +- [[systems/memory-backend]] — memory backend settings +- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] — the still-open secrets-at-rest page for the same database +- [[systems/scheduled-tasks]] — the fleet's execution ledger diff --git a/index.md b/index.md index 193e774..5be0a0d 100644 --- a/index.md +++ b/index.md @@ -12,7 +12,9 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-11-14. +> model routing and automation. Last updated: 2026-11-15. +> +> **2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet.** **New page [[systems/phonon-tts]]**: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model **`gradium/phonon-40m-en`**, container created 2026-11-14 10:31:56 AWST), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`, 11:00:11), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`, 11:43:59) — verified live from CT460: `/health` 200, `/v1/models`, and a `POST /v1/audio/speech` returning a valid MP3; `/opt/ptts-voices` holds **7** voices; both ptts containers were restarted together at **17:54 AWST**. ⚠ **Provenance gap**: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are **not attributable to any session in this profile** (working hypothesis: another estate agent — unproven). **New open incident [[incidents/2026-11-14-mnemosyne-capture-stall]]**: Mnemosyne has captured **nothing since 2026-11-14 09:32:59 AWST** — `working_memory` and `episodic_memory` stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor `ea79a2107e36` **escalated only at 2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — **mislabelled**, state **`alerted`**, unclosed, alert text printing the raw **UTC** value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ **~12 h of lead time produced no alarm**: the 12:00:52 AWST run saw the same hole and logged it **ℹ️ informational** (`capture delay -589.5 min` — the negative sign hides the growth). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established**. **Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**, no other incident row moved, so `cron_incidents` went **0 → 1 unclosed** (the monitor's row, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 (26 / 3)** — a **ninth** window unchanged, but **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. [[infrastructure/docker-services]] re-verified: CT200 **64 running non-seeder / 72 total** excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is **exactly** the three TTS containers, **no removals**. The **2026-11-08 hook cascade** gained **no** new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; `errors.log` 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign `Browser Use viewing unavailable` plugin warning at ~2/hour). > **2026-11-13 (nightly audit; live probes 2026-11-13 23:59 - 2026-11-14 00:2x AWST) — the fleet went fully clean, then the hook came back.** **2026-11-13: 174 ledger rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:41) — the **third consecutive zero-`failed` day** (11-11 172/172, 11-12 176/176); deliveries **8 / 8 `delivered`**, 0 `for_failure`; `cron_incidents` **0 unclosed**. **Every job now reads `last_status: ok` with `failure_streak` 0 — the fleet has no non-clean row at all**, because **`066bb5191280` (Uber Eats Monthly Spend Report) was fixed end-to-end on 2026-11-13**, clearing a `delivery_failed` that had stood since 2026-11-01. Two stacked causes: the job's **runtime prompt had grown to 88 k chars** (it carried two heavy browser skills) so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest, and the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart (**2026-11-13 13:14**). Fix: deterministic printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, rewritten prompt with a hard output contract, skills cleared (**88 k -> 1.8 k**); verified on the 18:51 -> 18:58 slot (**154 unique / 153 completed** orders, Oct **9 orders / $301.04**), lesson saved to `cron-job-maintenance`. Fleet **29 (26 / 3)** — an **eighth** window unchanged. ⚠ **But the window's fault is the `pre_tool_call` hook, and it returned:** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40** plus **10** suppression lines through **00:06:38**, refusing this audit's own tool calls for ~1 minute (first-hand). `errors.log` by day: 11-12 **0/0**, 11-13 **0/0**, 11-14 **1/10** — the first such timeout in the retained log since the 2026-11-08 cascade (**`errors.log` now starts 2026-11-09 13:18:39**, so the 11-08 lines have rotated out and the page's 58-count stands on the 2026-11-09 read). **Load is the trigger and it has ~doubled: `load average 12.08 / 13.02 / 12.33` on 2 vCPU** (vs **5.89** at diagnosis), with **`state.db` now 2.04 GB** (**2,977 sessions / 207,577 messages**, was 1.89 GB / 2,901 / 197,682) and the root filesystem **88 % used**. The 30 -> 120 s fix is **proven insufficient**; [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] is **reopened, `resolved` -> `active`**, and `state.db` checkpoint/retention — not a bigger timeout — is now the estate's most urgent open item. Freshness (03:00 stocktake): **0 stale / 131 fresh / 3 no-date of 134**. > > **2026-11-11 → 2026-11-12 (nightly audit; live probes 2026-11-12 23:59 – 2026-11-13 00:4x AWST) — a clean day, and an upgrade that could not be installed.** The 429 row that was the estate's only open incident **closed `resolved` at 2026-11-12T00:25:36** (the next slot of the same job ran clean, carrying both windows); *cron_incidents* now reads **0 unclosed of 137**. **2026-11-12: 176 ledger rows — 175 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:54), deliveries **9 / 9 `delivered`**; **11-11: 172 / 172 `completed` / 0 `failed`**, deliveries 7 / 7. Fleet **29 (26 / 3)**, a seventh window unchanged. The window's one new open item is not a failure: the attempt to take **`hermes-hq-push`** past its installed **0.9.42** was **refused by the pre-install plugin security scan** — *"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."* The critical is the plugin's **own** `skills/dashboard-tile/SKILL.md` one-liner; **no per-verdict override exists** (`plugins.scan_on_install: false` would blind the scan for *every* future install, and `plugin_guard.py` hardcodes `trust_level="community"`). The fix is **PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file), pushed from a fork because `gh` is **read-only** upstream. Installed 0.9.42 keeps delivering. Also: the alleged *"stored hold-and-ask instruction"* for scan warnings **does not exist on disk** (184,716-file sweep, zero matches in editable text; the sentence lives only in `state.db` history) — nothing was removed or reverted, and the hunt is now the skill `stored-instruction-search-and-removal` (**skills 299 → 300**). `config.yaml` is **byte-identical to its 2026-11-12 18:14:52 backup**; the only drift since 11-09 is the **model catalogue block** (6 out, 9 in). **New page:** [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]]. @@ -57,6 +59,7 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi - [[systems/model-providers]] — LiteLLM + OmniRoute providers, routing setup - [[systems/memory-backend]] — Mnemosyne vector memory, settings - [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence +- [[systems/phonon-tts]] — **Phonon TTS stack on CT200** (`:8881` OpenAI-compatible API, `:8882` WASM demo, `:8883` WebSocket; added 2026-11-14) - [[systems/omniroute]] — OmniRoute catalog, combos, and host path - [[systems/openrouter]] — OpenRouter current state and history - [[systems/opencode-go]] — OpenCode Go subscription, model limits, pricing @@ -73,10 +76,11 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. - [[systems/hermes-hq-push]] — **Hermes HQ iPhone app plugin** (standalone, **v0.9.42**, `kind: standalone`, author *Hermes HQ*; tree mtime **2026-11-08 02:19**). `hermes plugins list` → **enabled** (state `user`); `config.yaml` → `plugins.enabled` (line **13226** as of 2026-11-13; the number drifts with each catalogue write). ⚠ **2026-11-12: the upgrade past 0.9.42 is blocked by the pre-install security scan** (dangerous verdict, one critical in the plugin's own tile skill) — see [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]]. Host side of the Hermes HQ app: Apple push (finished turns, lock-screen approvals, "waiting for a password/login/code" alerts, **live Lock Screen cards** sealed to the phone), phone **device access** (contacts, location, reminders, alarms, photos, Health, motion, music), the desktop-app **Hermes HQ theme**, the app's **Terminal page**, fast **Sessions list** on a busy host, cross-posting answers back to Telegram/Discord, a **restart guard**, and **Grok Live** voice-token minting (`XAI_API_KEY` or subscription sign-in; bearer never reaches the phone). Tools `device_contacts_search` / `device_request_result`; 7 hooks. Its manifest states it **replaces `dispatch-push`** — and the replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory remains. **Different stack** from the Cadu/Vory bridge on [[systems/push-notifications]]. ⚠ Its `pre_tool_call` guard also **blocks any tool call whose arguments contain the literal text of a launchd one-off-restart command** — describe it, never quote it. -- [[systems/scheduled-tasks]] — **As of 2026-11-12 (window 11-12): 176 rows / 175 `completed` / **0 `failed`** / 1 `running` (this audit); deliveries 9 / 9 `delivered`; unclosed incident rows **0 of 137** — the 429 row closed `resolved` at 2026-11-12T00:25:36; fleet 29 (26 / 3).** Earlier — **As of 2026-11-09 (window 11-09): 174 rows / 173 `completed` / **0 `failed`** — the first `failed`-free day since 11-05; deliveries 8/8 `delivered`; unclosed incident rows 3 → 0; the stocktake closed `completed` after 9 h 10 m with **zero edits landed**; Job Radar Stage 1 overran 9 h 09 m and landed nothing (new incident).** Earlier: **As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and `ea79a2107e36` is ENABLED again (`last_status: ok`, streak 0) after its 00:01:08 slot closed `completed` at 10:02:48; deliveries 8/8 `delivered`.** **Prior window — as of 2026-10-30 03:3x:** 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** +- [[systems/scheduled-tasks]] — **As of 2026-11-14 (window 11-14): 167 rows / 166 `completed` / 0 `failed` / 1 `running` (this audit); deliveries 4 / 4 `delivered`; `cron_incidents` 0 → 1 unclosed (138 total) — the Mnemosyne Health Monitor wrote the estate's first red row since 11-13 (`last_status: error`, streak 1); fleet 29 (26 / 3).** Earlier — **As of 2026-11-12 (window 11-12): 176 rows / 175 `completed` / **0 `failed`** / 1 `running` (this audit); deliveries 9 / 9 `delivered`; unclosed incident rows **0 of 137** — the 429 row closed `resolved` at 2026-11-12T00:25:36; fleet 29 (26 / 3).** Earlier — **As of 2026-11-09 (window 11-09): 174 rows / 173 `completed` / **0 `failed`** — the first `failed`-free day since 11-05; deliveries 8/8 `delivered`; unclosed incident rows 3 → 0; the stocktake closed `completed` after 9 h 10 m with **zero edits landed**; Job Radar Stage 1 overran 9 h 09 m and landed nothing (new incident).** Earlier: **As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and `ea79a2107e36` is ENABLED again (`last_status: ok`, streak 0) after its 00:01:08 slot closed `completed` at 10:02:48; deliveries 8/8 `delivered`.** **Prior window — as of 2026-10-30 03:3x:** 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/skills-index]] — Installed skills inventory (**300 skills** (**67 categories**) as of the **2026-11-13** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents +- [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN — Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — **mislabelled**, state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **The 12:00:52 AWST run saw the identical hole and logged it ℹ️ informational** (`capture delay -589.5 min`), i.e. **~12 h of lead time produced no alarm**. Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]]. - [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]] - **OPEN — the `hermes-hq-push` upgrade past v0.9.42 was refused by the pre-install plugin security scan.** `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` returned **"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."** The sole critical is the plugin's **own** `skills/dashboard-tile/SKILL.md:41` shell one-liner (`echo '' | python3 …` = *obfuscation*); the other 38 findings are low/medium and alone would have installed as `caution`. **`--force` lifts `caution` only**; a `dangerous` verdict is an unconditional block (`tools/plugin_guard.py:406-418`), and **no per-verdict override exists** — the only key is `plugins.scan_on_install` (default `true`, currently unset), which disables scanning for *every* future install; the `trust_level`/allowlist machinery lives in `tools/skills_guard.py` (the skills path) and `plugin_guard.py` hardcodes `trust_level="community"`. **Installed state unchanged:** v0.9.42, `enabled`, still delivering. **Fix = PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc` → `main`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file, created 2026-11-12T10:00:01Z) — pushed from a fork because this host's `gh` account has **read-only** access upstream; local clone `/home/hermes/workspace/hq-plugins-fix` (`90c9d28`). - [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion; *cron_incidents* now 0 unclosed of 137) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]). - [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - ⚠ **REOPENED 2026-11-14 — the hook was the fault, and the 30 → 120 s fix did not hold at current load.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** ⚠ **Recurred 2026-11-14 00:05:40** — **1 timeout + 10 skips**, overrunning the **120 s** budget at `load average 12.08` on 2 vCPU (**~2×** the 5.89 at diagnosis), with `state.db` grown to **2.04 GB**. Status **`resolved` → `active`**; the lever is now urgent. → [[systems/conduit]], [[systems/hermes-agent]] diff --git a/infrastructure/docker-services.md b/infrastructure/docker-services.md index 881c8dc..e028608 100644 --- a/infrastructure/docker-services.md +++ b/infrastructure/docker-services.md @@ -20,14 +20,16 @@ Docker runs on **four hosts**, and CT460's socket is not readable by the agent u | Host | Access | Containers running | |---|---|---| -| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** running / 68 total — main Docker host (**re-verified 2026-11-07**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) | +| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **65** running / 72 total — main Docker host (**re-verified 2026-11-15**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) | | CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** running / 26 total — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**re-verified 2026-11-07** — set-wise diff against live `docker ps`: no additions, no removals) | | CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total (**re-verified 2026-11-07**) — `nostalgic_keldysh`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → `flamboyant_mahavira` (2026-11-04) → **`nostalgic_keldysh`** (2026-11-06) — all `ghcr.io/github/github-mcp-server`, the live one created 2026-11-05T22:15:57Z. ⚠ **Corrected 2026-11-07:** the enumerated list still opened with `flamboyant_mahavira`, which is **not running** — live `docker ps` on CT333 returns **`nostalgic_keldysh`** (created 2026-11-05 22:15:57, `Up 29 hours`), `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1` (8 running), plus `qdrant` **exited** (9 total) | | CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** running / 9 total — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented | | CT460 `hermes` | `pct exec 460 -- docker ps` **as root** from the PVE host — socket is `root:docker srw-rw----`, the `hermes` user is not in group `docker` (its own `docker ps` → `permission denied while trying to connect to the docker API`) | `ignis` — **running** (`Up 5 days`, `ignis-subpath:0.8.9`, `0.0.0.0:8080->8080/tcp`), re-verified 2026-11-05 | -## CT200 live inventory (re-verified 2026-11-07, 61 running / 68 total, excluding 0–2 transient `worldmonitor-seeder-*` containers) -`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`. +## CT200 live inventory (re-verified 2026-11-15, 65 running / 72 total, excluding 1 transient `worldmonitor-seeder-*` container) +`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, **`phonon-demo`**, **`ptts-openai-server`**, **`ptts-ws-server`**, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`. + +**Re-verified 2026-11-15 00:4x AWST — 61 → 64 running (excluding the seeder), 68 → 72 total, and the whole delta is the TTS stack.** Live `docker ps` on CT200 returns **65 running / 72 total**, which is **64 non-seeder running + 1** transient `worldmonitor-seeder-scheduled`. A set-wise diff of the live names against this page's 2026-11-07 list found **three additions and no removals**, and the arithmetic closes exactly: **61 non-seeder → 64 non-seeder** is the **`phonon-demo`**, **`ptts-openai-server`** and **`ptts-ws-server`** trio — all created 2026-11-14, documented on [[systems/phonon-tts]]. Earlier in this pass a generic `total − prior total` subtraction suggested a fourth, unnamed addition; that was a **comparison artefact** (the 11-07 row's 61 was an explicit *non-seeder* count, today's 65 was a raw `running` count including the seeder) and is **withdrawn** — the name-level diff is exact and accounts for every added container. This pass verified **counts, the TTS trio, and the absence of removals**; it is not a full CT200 audit. **Re-verified 2026-11-06 — the 2026-11-05 "the seeder does not exist" finding was WRONG, and the cause is a sampling artefact, not a removal.** A set-wise diff against live `docker ps --format '{{.Names}}'` on CT200 diff --git a/log.md b/log.md index 393fd3e..f772c3c 100644 --- a/log.md +++ b/log.md @@ -2605,3 +2605,23 @@ hook timeout came back and reopened its incident.** 2026-11-13* section, the Uber Eats job row, frontmatter), `current-state.md` (new *Recent Changes (2026-11-13)* section), `index.md` (new blurb, the incident's RESOLVED → REOPENED label, frontmatter), `log.md` (this entry). **No page deleted, no decision reversed, no secret written.** + +## 2026-11-14 (nightly audit; window 2026-11-13 23:00 → 2026-11-14 23:59 AWST) + +**New:** [[systems/phonon-tts]] — the third TTS stack on CT200: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model `gradium/phonon-40m-en`), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`). Verified live from CT460 — `/health` 200, `/v1/models`, and a POST to `/v1/audio/speech` returning a valid MP3; `:8883` 404s on a plain GET because it is a WebSocket endpoint. `/opt/ptts-voices` holds 7 voices. Page carries the container/timeline table and flags the **provenance gap after 09:03 AWST** — no retained session in this profile covers the 10:31 / 11:00 / 11:43 / 17:54 changes. +**New:** [[incidents/2026-11-14-mnemosyne-capture-stall]] — **open incident**: Mnemosyne has captured nothing since **2026-11-14 09:32:59 AWST** (`working_memory` and `episodic_memory` stop in the same minute; last 8 working writes all 01:01–01:32 UTC). The health monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state `alerted`, unclosed) after ~12 h of lead time in which the same hole was logged **ℹ️ informational** at 12:00 AWST (`capture delay -589.5 min`). ~15 h of desktop/Telegram/cron context uncaptured; the 23:30 sleep cycle consolidated **0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime 2026-11-13 18:03; YAML walk clean). Root cause **not established**. +**Updated:** [[infrastructure/docker-services]] — CT200 re-verified 2026-11-15: **65 running / 72 total** (was 61 / 68 on 11-07), the three TTS containers added to the inventory, **no removals**. +**Updated:** [[current-state]] and [[index]] — dated change log. +**Ledger:** 2026-11-14 settled **167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit, started 23:59:37); deliveries **4 / 4 `delivered`**; **`cron_incidents` 0 → 1 unclosed** (the monitor row above, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 jobs (26 enabled / 3 disabled)** — **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. +**Not changed:** `errors.log` for 11-14 carries only the benign recurring `Browser Use viewing unavailable` plugin warning (~2/hour) plus the already-recorded `pre_tool_call timed out after 120s` at 00:05:40 — see [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] (no new occurrence in the 11-14 body). + +## 2026-11-15 00:5x — corrections pass on the 2026-11-14 entry (same audit window) + +A second read of the live DBs tightened three claims made in the entry above. All three are applied to the pages; this entry records the corrections rather than leaving two versions in the log. + +1. **WITHDRAWN — "one net-added CT200 container unidentified."** The `61 → 65` comparison mixed two different counts: the 2026-11-07 row's **61 was an explicit *non-seeder*** figure while today's **65 was a raw `running`** figure that **includes** the transient `worldmonitor-seeder-scheduled`. A set-wise name diff settles it: live = documented **+ exactly 3** (`phonon-demo`, `ptts-openai-server`, `ptts-ws-server`), **zero removals**, and **64 non-seeder running** against 61 on 11-07. **No unexplained container exists.** [[infrastructure/docker-services]] corrected. +2. **`cron_incidents` went 0 → 1 unclosed** — not "0 unclosed before the 00:03 alert". **138 rows** total; the open one is `ea79a2_601037516abd` (`state: alerted`, `alerted_at` 00:03:44.5, `failure_type: auth` — mislabelled). The monitor's **own** job is also the fleet's **single non-clean `last_status`** (`error`, `failure_streak` 1) — the **first red fleet row since 11-13**. [[systems/scheduled-tasks]], [[current-state]] and [[index]] corrected. +3. **RESOLVED (was listed as unverified) — did the 00:03 escalation actually deliver?** Yes. `deliveries.db` holds exactly one 2026-11-15 row: **`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — **3 s after** the incident row was raised, **5 s before** `alerted_at`. The **notification path worked; the detection path did not.** + +Ledger precision as read at 00:5x: **2026-11-14: 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (this audit, `cbfe434a3b5e`, 23:59:37); **2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (the monitor); deliveries 11-14 **4 / 4 `delivered`** (07:06 / 07:30 / 08:32 / 12:05). Pages also updated in this pass: [[incidents/2026-11-14-mnemosyne-capture-stall]] (delivery question closed, fleet line corrected), [[systems/mnemosyne]] (new ⚠ OPEN block at the head of *Known Issues*, frontmatter to 2026-11-15, and the naive-UTC trap note extended with the monitor's own raw-UTC alert text as a second occurrence of the same trap), [[systems/scheduled-tasks]] (new `### Open 2026-11-15` block under *Known Issues* + the monitor row in the disabled table annotated — **the threshold tests the *sign* of the capture delay, not its magnitude, which is why 12 h of a growing hole raised nothing**). + diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index a5ea796..45ffc69 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,8 +3,8 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-11-09 -verified_on: 2026-11-09 +updated: 2026-11-15 +verified_on: 2026-11-15 review_after: 2026-11-13 confidence: high tags: [memory, mnemosyne, backend] @@ -50,6 +50,8 @@ Primary agent memory store for Hermes. ## Known Issues +**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed) — but its **12:00:52** slot had already seen the identical hole and logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**; the threshold in the check script is on *sign*, not magnitude. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). → [[incidents/2026-11-14-mnemosyne-capture-stall]] + **⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**. **Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below. @@ -62,7 +64,7 @@ Primary agent memory store for Hermes. **Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check. -**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST. +**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST. **2026-11-14/15 recurrence, new flavour:** the monitor's own escalation text printed `last capture 2026-11-14 01:32:59` (raw UTC) for a write that landed at **09:32:59 AWST**, so the alert *reads* as a ~23 h hole when the true gap was **14 h 31 m** — the same trap, now inside the monitor's output rather than the audit's reading → [[incidents/2026-11-14-mnemosyne-capture-stall]]. ### Vector embedding coverage gap — RESOLVED (2026-08-02) - Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories. diff --git a/systems/phonon-tts.md b/systems/phonon-tts.md new file mode 100644 index 0000000..ae8cca1 --- /dev/null +++ b/systems/phonon-tts.md @@ -0,0 +1,120 @@ +--- +title: Phonon TTS on CT200 (ptts-openai-server, ptts-ws-server, phonon-demo) +type: system +status: active +created: 2026-11-14 +updated: 2026-11-15 +verified_on: 2026-11-15 +review_after: 2026-12-15 +confidence: high +tags: [tts, phonon, ptts, docker, ct200, voice, huggingface, websocket, openai-api, kokoro, edge-tts] +sources: [live-ct200-docker, live-ct200-docker-inspect, live-ct200-files, live-http-probe, live-state-db, live-agent-log, live-mnemosyne-db] +--- + +# Phonon TTS on CT200 + +A **third TTS stack** arrived on CT200 on **2026-11-14** — three containers that together form a +self-hosted "ptts" (Phonon TTS) service: an **OpenAI-compatible HTTP API** (`:8881`), a **static +WebAssembly demo UI** (`:8882`), and a **WebSocket TTS server** (`:8883`). It runs alongside the two +voice services that host already carried (`kokoro-tts` `:8880`, `edge-tts` `:5051`). + +Page created by the **2026-11-14 nightly audit** from live probes (see *Verification evidence*). +Everything below is a live read unless it says otherwise. + +## Containers (live `docker ps` / `docker inspect`, 2026-11-15 ~00:3x AWST) + +| Container | Image | Ports (host → container) | Created (UTC) | Last started (UTC) | Restart policy | +|---|---|---|---|---|---| +| `ptts-openai-server` | `xn-ptts-openai-server:v0.7.1` | `8881 → 10200` (**Wyoming also exposed, unmapped**) | `2026-11-14T02:31:56` = **10:31:56 AWST** | `2026-11-14T09:54:11` = **17:54:11 AWST** | `unless-stopped` | +| `phonon-demo` | `nginx:alpine` | `8882 → 80` | `2026-11-14T03:00:11` = **11:00:11 AWST** | `2026-11-14T03:00:11` | `unless-stopped` | +| `ptts-ws-server` | `ptts-ws-server:0.7.1` (**locally built**, 116 MB) | `8883 → 8080` | `2026-11-14T03:43:59` = **11:43:59 AWST** | `2026-11-14T09:54:21` = **17:54:21 AWST** | `unless-stopped` | + +Image names/tags for the first row are as reported by `docker ps`; the ws server's image is a +**local build** (`docker image ls ptts-ws-server` → `0.7.1`, no registry prefix), so it is not +re-pullable — **the build inputs are not in the wiki and were not located in this audit.** + +## Endpoints (verified live from CT460 over the tailnet, 2026-11-15) + +| URL | What answers | +|---|---| +| `http://:8881/health` | health JSON, HTTP 200 | +| `http://:8881/v1/models` | OpenAI-shaped model list → **`gradium/phonon-40m-en`** | +| `http://:8881/v1/audio/speech` | **POST works** — returned a valid MP3 (test voice `nova`) | +| `http://:8882/` | the demo UI — `nginx:alpine`, **19,212 B** body, byte-identical in size to the file below | +| `http://:8883/` | **HTTP 404 on a plain GET** — expected: it is a WebSocket endpoint (`/speech/tts`) | + +The `:8881` API is the piece that other services can consume (OpenAI-compatible `/v1/audio/speech`). + +## Files on CT200 (owner/paths verified by live `ls -la`) + +| Path | Contents | +|---|---| +| `/opt/ptts-openai-server/` | `run.sh` (861 B, `2026-11-14 09:01` local) + `.env` (**47 B, mode 600**, `2026-11-14 09:00` local — holds the HuggingFace token for the gated model; **value not recorded here**) | +| `/opt/phonon-demo/index.html` | the demo page, `2026-11-14 12:30` local; title *"Phonon TTS - WebAssembly Demo"*; talks to **`:8883`** only (no `8881` reference) | +| `/opt/phonon-demo/phonon-tts/` | the WASM/JS assets (`device.js`, `fetch.js`, `hub.js`, `gpu/`, `index.js`, `index.d.ts`, `LICENSE-APACHE`) — i.e. the upstream `phonon-demo` front-end | +| `/opt/ptts-voices/` | **7 entries** (`2026-11-14 17:54` local): six `.safetensors` voices — `anthony.safetensors`, `anthony-clone.safetensors`, `anthony-m.safetensors`, `anthony-november.safetensors`, `elizabeth.safetensors`, `grace.safetensors` — plus a `voices.jsonl` manifest | + +The ws server logged **`num_voices=6` at 2026-11-14 04:42:52 UTC (12:42:52 AWST)** and +**`num_voices=7` at 09:54:24 UTC (17:54:24 AWST)** — so the voice set was still being changed that +evening. **Provenance of the voice files is not established**: the naming suggests cloned/derived +voices, but nothing in the retained logs confirms whose they are or how they were produced. + +## Deployment timeline (2026-11-14) + +| AWST | Event | Source | +|---|---|---| +| 08:35 – 08:45 | First Phonon install session: `ptts-openai-server` brought up on `:8881`; the HuggingFace token returned **401** on the gated model and was replaced | session transcript (`state.db`) | +| 08:52 – 09:03 | Second session: model switched to `gradium/phonon-40m-en`; `/opt/ptts-openai-server/run.sh` (861 B) and its `600` `.env` written at **09:00–09:01** | session transcript + live `ls` | +| 10:31:56 | `ptts-openai-server` **re-created** (2nd recreation; the 09:01 container was not the live one) | live `docker inspect` | +| 11:00:11 | `phonon-demo` (`nginx:alpine`) created → demo UI on `:8882` | live `docker inspect` | +| 11:43:59 | `ptts-ws-server:0.7.1` created → WebSocket TTS on `:8883` | live `docker inspect` | +| 12:30 | `/opt/phonon-demo/index.html` last written | live `ls` mtime | +| 12:42:52 | ws server logs `num_voices=6` | live `docker logs` | +| 17:54:11 / 17:54:21 | **both** ptts servers restarted together; `/opt/ptts-voices` written; ws logs `num_voices=7` | live `docker inspect` + `ls` + `logs` | + +⚠ **Provenance gap.** The two retained interactive sessions in `state.db` cover only **08:35–09:03 +AWST**. Every change from **10:31 onward** — the demo UI, the ws-server build, the voice files, the +17:54 restart — is **not attributable to any retained transcript**: the only sessions active in +`state.db` between 09:03 and 23:59 on 11-14 are cron runs and a heartbeat "Pulse" session (495 +messages, none containing docker/TTS content). **Working hypothesis, not proven: the later work was +done by another estate agent or a session not retained in this profile's store.** Do not treat the +08:35/08:52 sessions as the whole of this deployment. + +## Why it is on `8881` and not `8880` + +`8880` was already taken by **`kokoro-tts`** on the same host (`ghcr.io/remsky/kokoro-fastapi-cpu:v0.2.1`, +declared in the ReadLater compose file → [[infrastructure/docker-services]]), and `edge-tts` sits on +`:5051`. Phonon is therefore the **third** voice backend on CT200 — three TTS stacks on one LXC. + +## Operations + +```bash +# reach CT200 (the PVE node is 100.96.100.82; the wiki's CT200 row carries the tailnet IP) +ssh root@100.96.100.82 'pct exec 200 -- docker ps --format "{{.Names}}\t{{.Status}}\t{{.Ports}}" | grep -Ei "ptts|phonon"' +# logs (the ws server is the talkative one) +ssh root@100.96.100.82 'pct exec 200 -- docker logs --tail 40 ptts-ws-server' +# smoke test the OpenAI-compatible path from another host on the tailnet +curl -s -X POST "http://:8881/v1/audio/speech" -H 'Content-Type: application/json' \ + -d '{"model":"gradium/phonon-40m-en","input":"test","voice":"nova"}' -o /tmp/probe.mp3 && file /tmp/probe.mp3 +``` + +## Open items / watch + +1. **The HuggingFace token is a standing single point of failure.** Both `ptts-openai-server` and + `ptts-ws-server` serve a **gated** model, and the token has already expired once (2026-11-14 + 08:35–08:45 → replaced). The token lives in `~/.hermes/.env` on CT460 and in + `/opt/ptts-openai-server/.env` (mode 600) on CT200; **whether the ws container reads the same file + was not confirmed in this audit.** A silent expiry would take the whole stack down again. +2. **`ptts-ws-server:0.7.1` is a local-only image** — no build source recorded here. +3. **Attribution gap** for everything after 09:03 AWST (see the timeline note). +4. **Nothing in the estate consumes `:8881` yet**, as far as this audit could see: it was verified by + probe, not by a caller. If it is wired into Hermes TTS later, the provider config belongs in + [[systems/hermes-agent]] / the TTS configuration, not here. +5. The upstream project's own naming is inconsistent (`ptts-*` containers, `phonon-*` model/page, a + "gradium" publisher prefix) — the same stack appears under three names. + +## Related + +- [[infrastructure/docker-services]] — CT200 inventory and container counts (65 running / 72 total as of 2026-11-15) +- [[systems/mnemosyne]] — the capture stall recorded the same night ([[incidents/2026-11-14-mnemosyne-capture-stall]]) +- [[current-state]] — dated change log diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 0b4db5f..7d946ae 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-11-14 -verified_on: 2026-11-14 -review_after: 2026-11-14 +updated: 2026-11-15 +verified_on: 2026-11-15 +review_after: 2026-11-15 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db] @@ -14,6 +14,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db # Scheduled Tasks ## Count +- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth + consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.** + Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added, + retired, disabled or repinned). Ledger — **2026-11-14: 167 rows, 166 `completed` / 0 `failed` / + 1 `running`** (this audit, `cbfe434a3b5e`, started 23:59:37). Trend: **11-12 176/176, 11-13 174/173, + 11-14 167/166**. **Deliveries 11-14: 4 rows, 4 / 4 `delivered`, 0 `for_failure`** (07:06, 07:30, 08:32, + 12:05). ⚠ **Non-clean `last_status` 0 -> 1**: **`ea79a2107e36` (Mnemosyne Health Monitor)**, `last_status: + error`, **`failure_streak` 1** — its **00:00:53 slot failed at 00:03:44 on `Script exited with code 1`** + ("STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59"), the first red row since the + Uber Eats job cleared on **2026-11-13**. **Unclosed incident rows 0 -> 1** (**138** total), the row being + `ea79a2_601037516abd` (`state: alerted`, `failure_type: auth` — **mislabelled**, `alerted_at` + 00:03:44.5) → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver** + (`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — the failure was in + *detection*, not notification: at **12:00:52** the same slot had already seen the identical hole and + logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**. + **2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (that monitor run). Next slot of the monitor: + **2026-11-15 12:00** (schedule `0 */12 * * *`) — the natural check for whether capture resumed. - **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third consecutive zero-`failed` day — and the fleet went fully clean.** Fleet **29 (26 enabled / 3 disabled)**, an **eighth consecutive window at the same count** (nothing added, retired, disabled or repinned). @@ -287,10 +304,25 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne | Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | -| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. | +| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. (⚠ **2026-11-15:** the job is enabled and now carries `last_status: error` / `failure_streak: 1` from its 00:03:44 run — [[incidents/2026-11-14-mnemosyne-capture-stall]].) The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. | ## Known Issues +### Open 2026-11-15 + +- **The Mnemosyne Health Monitor (`ea79a2107e36`) is the fleet's only non-clean row** — `last_status: error`, + `failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`, wrongly labelled + `failure_type: auth`). Its 00:00 slot exited 1 at **00:03:44** on `STALE MEMORY PIPELINE: capture missed … + last capture 2026-11-14 01:32:59` (raw **UTC** in the alert text, so it *reads* as a 23 h gap; the true + AWST gap at the alert was **14 h 31 m**). The **12:00:52** slot saw the same hole and logged it + **ℹ️ informational** — `capture delay -589.5 min`, the negative sign hiding its growth — so the estate + had **~12 h of lead time and no alarm**. The alert itself delivered (00:03:39, `for_failure: 1`). + Root cause **not established**; the known stringified-`config.yaml` killer is ruled out. Full page: + [[incidents/2026-11-14-mnemosyne-capture-stall]]. **Lessons for this fleet:** (a) a monitor that + distinguishes "capture is late" only in *sign* will never escalate on a growing hole — the threshold must + be on magnitude; (b) alert timestamps printed from raw UTC inside a +08:00 estate are a standing + misreading trap. + ### Resolved 2026-11-02 - **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.**