Compare commits
3
Commits
2d439145bf
...
402d8546e8
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
402d8546e8 | ||
|
|
646be98c31 | ||
|
|
82876abdfb |
@@ -3,9 +3,9 @@ title: Fleet-Wide Provider Timeout Storm — 2026-10-09
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-09
|
created: 2026-10-09
|
||||||
updated: 2026-10-24
|
updated: 2026-11-15
|
||||||
verified_on: 2026-10-24
|
verified_on: 2026-11-15
|
||||||
review_after: 2026-11-07
|
review_after: 2026-12-15
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, omniroute, router, timeout, incident, resolved]
|
tags: [cron, omniroute, router, timeout, incident, resolved]
|
||||||
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
|
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
|
||||||
@@ -18,6 +18,15 @@ sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-2026100
|
|||||||
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
|
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
|
||||||
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
|
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
|
||||||
The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
|
The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-15** (3 a.m. stocktake): all 12 of the storm's `cron_incidents` rows still read
|
||||||
|
> `resolved` — none reopened. `1b23c042e17d` (the weekly Friday job, last hold-out) is healthy —
|
||||||
|
> `last_status: ok`, last run 2026-11-13 08:11 AWST. One incident is currently open fleet-wide
|
||||||
|
> (`ea79a2_601037516abd`, Mnemosyne Health Monitor, first seen 2026-11-15 00:03) but it is the
|
||||||
|
> capture-stall detection firing → [[incidents/2026-11-14-mnemosyne-capture-stall]], unrelated to
|
||||||
|
> this storm. ⚠ Evidence window: `executions.db` retention now reaches back only to 2026-11-09, so
|
||||||
|
> the 2026-10-09 execution counts below are **no longer re-derivable from the DB** — they stand on
|
||||||
|
> the 2026-10-24 read (the incident rows themselves persist, which is what this check re-read).
|
||||||
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
|
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
|
||||||
|
|
||||||
## What happened
|
## What happened
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
---
|
---
|
||||||
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
|
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
|
||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: degraded
|
||||||
created: 2026-10-24
|
created: 2026-10-24
|
||||||
updated: 2026-10-25
|
updated: 2026-11-15
|
||||||
verified_on: 2026-10-25
|
verified_on: 2026-11-15
|
||||||
review_after: 2026-11-08
|
review_after: 2026-12-15
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
|
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
|
||||||
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
|
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
|
||||||
@@ -14,7 +14,8 @@ sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
|
|||||||
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
|
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
|
||||||
|
|
||||||
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
|
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
|
||||||
**Status:** Resolved 2026-10-23 18:54 (patched and verified)
|
**Status:** Resolved 2026-10-23 18:54 (patched and verified) — **but the fix was lost in the
|
||||||
|
plugin's 0.168.x restructure; `status: degraded` since 2026-11-15** (see the regression section below)
|
||||||
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
|
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
|
||||||
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
|
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
|
||||||
memory layer.
|
memory layer.
|
||||||
@@ -87,17 +88,39 @@ touching the file.
|
|||||||
not positive proof that recall was populated on those calls. The proven part is that the call
|
not positive proof that recall was populated on those calls. The proven part is that the call
|
||||||
is no longer abandoned by the memory layer.
|
is no longer abandoned by the memory layer.
|
||||||
|
|
||||||
|
## ⚠ Regression — the fix was lost in the plugin restructure (verified 2026-11-15)
|
||||||
|
|
||||||
|
The follow-up below warned that a plugin update would revert the patch. **It happened.** Live
|
||||||
|
state at the 2026-11-15 03:00 stocktake:
|
||||||
|
|
||||||
|
- The plugin tree is **clean at upstream `0.168.2`** (`296c94b`, committed 2026-11-12 US /
|
||||||
|
2026-11-13 AWST) — the patch is gone entirely, not just uncommitted.
|
||||||
|
- The memory layer **moved**: `dashboard/plugin_api.py` (now a doc index) →
|
||||||
|
`dashboard/api/memory.py`. There, `MEMORY_TIMEOUT_S = 10.0` is hard-coded again
|
||||||
|
(`memory.py:48`), with **no** `CONDUIT_MEMORY_TIMEOUT_S` env override, and the "abandoning the
|
||||||
|
call" behaviour is intact (`memory.py:256`). The restructure landed in `memory.py` on
|
||||||
|
**2026-10-29 AWST**, so the 60 s budget has been gone for ~2.5 weeks.
|
||||||
|
- The running dashboard (started 2026-11-10) loads the restructured code, so the exposure is live.
|
||||||
|
- One real call has since occurred — 2026-11-14 08:57 AWST, the first entry in
|
||||||
|
`conduit-calls.json → history` — with no abort lines in the logs afterwards. But that call met a
|
||||||
|
warm provider; it does not exercise the cold-start path. The page's original caveat stands.
|
||||||
|
- **Not re-applied by the stocktake**: porting the fix to the restructured code is a code change to
|
||||||
|
a git-managed upstream plugin that the next update will revert again — it needs Anthony's call
|
||||||
|
(re-port + commit locally, or upstream it).
|
||||||
|
|
||||||
|
The original incident window (2026-10-22 → 23) remains resolved; `status: degraded` records that
|
||||||
|
the *protection* is gone, not that the October fault returned.
|
||||||
|
|
||||||
## Follow-ups
|
## Follow-ups
|
||||||
|
|
||||||
- ⚠ **The fix is still unexercised by a call** (see the correction above): `conduit-calls.json →
|
- ~~⚠ **The fix is still unexercised by a call** …~~ **Superseded 2026-11-15**: one call has since
|
||||||
history` is empty at 2026-10-25 00:2x, and the plugin is now **five releases behind HEAD**
|
occurred (2026-11-14 08:57 AWST) — but the fix itself was lost in the restructure (section above),
|
||||||
(`0.64.0` → `0.69.0`).
|
so "unexercised" is now moot.
|
||||||
- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which
|
- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which
|
||||||
asserted the capability did not exist **without calling the tool** →
|
asserted the capability did not exist **without calling the tool** →
|
||||||
[[incidents/2026-10-24-conduit-call-request-refused]].
|
[[incidents/2026-10-24-conduit-call-request-refused]].
|
||||||
- **The patch is uncommitted.** A plugin update that resets
|
- ~~**The patch is uncommitted.** A plugin update that resets `~/.hermes/plugins/conduit_push`
|
||||||
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream
|
reverts it to the 10 s budget.~~ **Confirmed 2026-11-15** — see the regression section above.
|
||||||
(`kaishi00/hermes-conduit-notifier`).
|
|
||||||
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
|
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
|
||||||
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
|
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
|
||||||
|
|
||||||
|
|||||||
@@ -46,25 +46,60 @@ has no working_memory row (last capture 2026-11-14 01:32:59)
|
|||||||
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
|
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
|
||||||
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
|
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
|
||||||
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
|
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
|
||||||
- ⚠ **The `auth` label looks wrong** — the failure is a *staleness* check, not a credential fault, and
|
- ⚠ **The `auth` label describes the classifier, not the fault — now evidenced, not just suspected.** The
|
||||||
the page records no auth error text. **Observation, not verified diagnosis.**
|
failure is a *staleness* check and the page records no auth error text. This job's **twelve earlier
|
||||||
|
`cron_incidents` rows** read `agent` ×3, `timeout` ×1, `unknown` ×8 (`ea79a2_649d8ac46e8a`,
|
||||||
|
`_f17b45759db0`, `_573d17e10fc1`, `_6fb70b032305`, `_54d6b1f0f58a`, `_747b27556fd0`, `_c3ab2d784ba3`,
|
||||||
|
`_a4661554a3c2`, `_d93c3acdc4f2`, `_9197df32320b`, `_111e67a3d83c`, `_f317fcd843ae`), and the estate-wide
|
||||||
|
table is **53 `unknown` / 47 `agent` / 15 `timeout` / 14 `script` / 5 `auth`**. `agent` is also impossible
|
||||||
|
here: the job runs `no_agent` (plain script). Treat the field as opaque routing metadata.
|
||||||
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
|
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
|
||||||
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
|
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
|
||||||
log line below.
|
log line below.
|
||||||
|
|
||||||
## The earlier, quieter reading — 12:00:52 AWST 2026-11-14
|
## What the earlier slots actually saw — correction, re-derived 2026-11-15 01:2x AWST
|
||||||
|
|
||||||
`~/.hermes/logs/mnemosyne-health-monitor.log` shows the **12:00:52 AWST** run already looking at the
|
**An earlier revision of this page got this backwards.** It claimed the **12:00:52 AWST** run had
|
||||||
same hole and calling it benign:
|
"already seen the same hole and called it benign" (`capture delay -589.5 min`), giving the estate ~12 h
|
||||||
|
of unalarmed lead time, and blamed a "sign, not magnitude" threshold. **Both claims are withdrawn.** The
|
||||||
|
`-589.5 min` line is not the 12:00 run's — it sits inside the **00:03:35 run's own output block**.
|
||||||
|
|
||||||
```
|
Verified by line number in `~/.hermes/logs/mnemosyne-health-monitor.log` (**1103 lines**):
|
||||||
ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min
|
|
||||||
```
|
|
||||||
|
|
||||||
That is **9 h 49.5 m** of missing capture, logged as ℹ️ informational (the delay is **negative**, i.e.
|
| line | content |
|
||||||
capture sits *behind* the message clock — the sign convention makes a growing gap look small). The
|
| --- | --- |
|
||||||
job exited 0 at 12:00 and 18:03; it only escalated at 00:03 the next morning.
|
| 884 | `=== 2026-11-14 00:09:12 AWST health monitor (no_agent) ===` |
|
||||||
**So the fault had ~12 hours of lead time that produced no alarm.**
|
| 900 | `ℹ️ last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min` |
|
||||||
|
| 957 | `=== 2026-11-14 12:01:52 AWST health monitor (no_agent) ===` |
|
||||||
|
| 973 | `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay 29.4 min` |
|
||||||
|
| 1030 | `=== 2026-11-15 00:03:35 AWST health monitor (no_agent) ===` |
|
||||||
|
| 1046 | `ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min` |
|
||||||
|
| 1047 | `❌ STALE MEMORY PIPELINE: …` |
|
||||||
|
|
||||||
|
Two further corrections to the earlier revision:
|
||||||
|
|
||||||
|
- **There was no 18:03 run.** The job's schedule is `0 */12 * * *`; the only 2026-11-14 slots are
|
||||||
|
00:09:12 and 12:01:52 (both exit 0, delivery `suppressed`). The "exited 0 at 12:00 and 18:03"
|
||||||
|
sentence is withdrawn — 18:03 belongs to some other job.
|
||||||
|
- **The detection chain was not broken.** `mnemosyne_freshness_gate.py` fires on **magnitude**
|
||||||
|
(`delay_min < -CAPTURE_LAG_MIN`, `MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min) — there is no
|
||||||
|
sign-only test to blame. The 12:00:52 measurement of **+29.4 min** is the delay of the newest message
|
||||||
|
*that had been captured* (the 09:03 AWST `ios` turn, captured 09:32:59). The gate computes against the
|
||||||
|
newest **uncaptured** message and only reports a problem when one exists; with the conversation idle
|
||||||
|
from 09:03 to 19:22, every user message still had a capture row, so there was **nothing to detect**.
|
||||||
|
The first message to postdate the stall was the **19:22 AWST desktop** turn (`11:22 UTC`, the same
|
||||||
|
message the 00:03 alert names); the next scheduled slot after it was 00:03:35, which escalated
|
||||||
|
(`-589.5 min` = 09:32:59 − 19:22).
|
||||||
|
|
||||||
|
**Corrected statement of the gap:** the pipeline died at **09:32:59 AWST**; it became *detectable* at
|
||||||
|
**19:22 AWST** (first uncaptured message) and was *detected* at **00:03:35** by the next slot — a
|
||||||
|
detection latency of 4 h 41 m after detectability, driven by cadence, not by a broken threshold.
|
||||||
|
**What was genuinely unread sits in the consolidation counters:** the cycles ran every ~31 min and
|
||||||
|
returned **0 items for 29 consecutive runs** (streak opening `2026-11-14T02:04:42+00:00` = **10:04:42
|
||||||
|
AWST**; last productive cycles 4622/4621/4620 at `09:33:34` / `09:33:16` / `09:01:40`). Every run
|
||||||
|
printed `Last consolidation: … (0 items)` — including the "healthy" 12:00:52 one — but the gate tests
|
||||||
|
consolidation **age** only (`> 36 h`, default), never the item count. **An empty-but-current cycle is
|
||||||
|
the loudest signal available and the only one nothing watches.**
|
||||||
|
|
||||||
## Impact
|
## Impact
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Anthony's Systems Wiki
|
|||||||
type: collection
|
type: collection
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-14
|
updated: 2026-11-15
|
||||||
verified_on: 2026-11-14
|
verified_on: 2026-11-15
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
|
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
|
||||||
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
|
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
|
||||||
@@ -14,7 +14,7 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
|
|||||||
> Operational knowledge for Anthony's agents, infrastructure,
|
> Operational knowledge for Anthony's agents, infrastructure,
|
||||||
> model routing and automation. Last updated: 2026-11-15.
|
> model routing and automation. Last updated: 2026-11-15.
|
||||||
>
|
>
|
||||||
> **2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet.** **New page [[systems/phonon-tts]]**: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model **`gradium/phonon-40m-en`**, container created 2026-11-14 10:31:56 AWST), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`, 11:00:11), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`, 11:43:59) — verified live from CT460: `/health` 200, `/v1/models`, and a `POST /v1/audio/speech` returning a valid MP3; `/opt/ptts-voices` holds **7** voices; both ptts containers were restarted together at **17:54 AWST**. ⚠ **Provenance gap**: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are **not attributable to any session in this profile** (working hypothesis: another estate agent — unproven). **New open incident [[incidents/2026-11-14-mnemosyne-capture-stall]]**: Mnemosyne has captured **nothing since 2026-11-14 09:32:59 AWST** — `working_memory` and `episodic_memory` stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor `ea79a2107e36` **escalated only at 2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — **mislabelled**, state **`alerted`**, unclosed, alert text printing the raw **UTC** value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ **~12 h of lead time produced no alarm**: the 12:00:52 AWST run saw the same hole and logged it **ℹ️ informational** (`capture delay -589.5 min` — the negative sign hides the growth). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established**. **Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**, no other incident row moved, so `cron_incidents` went **0 → 1 unclosed** (the monitor's row, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 (26 / 3)** — a **ninth** window unchanged, but **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. [[infrastructure/docker-services]] re-verified: CT200 **64 running non-seeder / 72 total** excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is **exactly** the three TTS containers, **no removals**. The **2026-11-08 hook cascade** gained **no** new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; `errors.log` 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign `Browser Use viewing unavailable` plugin warning at ~2/hour).
|
> **2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet.** **New page [[systems/phonon-tts]]**: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model **`gradium/phonon-40m-en`**, container created 2026-11-14 10:31:56 AWST), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`, 11:00:11), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`, 11:43:59) — verified live from CT460: `/health` 200, `/v1/models`, and a `POST /v1/audio/speech` returning a valid MP3; `/opt/ptts-voices` holds **7** voices; both ptts containers were restarted together at **17:54 AWST**. ⚠ **Provenance gap**: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are **not attributable to any session in this profile** (working hypothesis: another estate agent — unproven). **New open incident [[incidents/2026-11-14-mnemosyne-capture-stall]]**: Mnemosyne has captured **nothing since 2026-11-14 09:32:59 AWST** — `working_memory` and `episodic_memory` stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor `ea79a2107e36` **escalated only at 2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault, state **`alerted`**, unclosed, alert text printing the raw **UTC** value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 run saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run's own output block** (monitor log: header 957 = 12:01:52 run, header 1030 = 00:03:35 run, `-589.5` at 1046, ❌ at 1047). The earlier "~12 h of lead time" claim is **withdrawn**, as is the "exited 0 at 12:00 and 18:03" sentence (schedule `0 */12`; there was no 18:03 slot). Detection is **activity-aware** — `mnemosyne_freshness_gate.py` fires on `delay < −120 min` (magnitude-based, no sign-only test) and can only fire once a conversational message postdates the last capture; the conversation was idle 09:03→19:22, so midnight was the first slot able to see the miss. **What actually went unread: 29 consecutive 0-item consolidation cycles** from **10:04:42 AWST**, printed as `(0 items)` every run yet tested only for age (`> 36 h`). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established**. **Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**, no other incident row moved, so `cron_incidents` went **0 → 1 unclosed** (the monitor's row, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 (26 / 3)** — a **ninth** window unchanged, but **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. [[infrastructure/docker-services]] re-verified: CT200 **64 running non-seeder / 72 total** excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is **exactly** the three TTS containers, **no removals**. The **2026-11-08 hook cascade** gained **no** new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; `errors.log` 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign `Browser Use viewing unavailable` plugin warning at ~2/hour).
|
||||||
> **2026-11-13 (nightly audit; live probes 2026-11-13 23:59 - 2026-11-14 00:2x AWST) — the fleet went fully clean, then the hook came back.** **2026-11-13: 174 ledger rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:41) — the **third consecutive zero-`failed` day** (11-11 172/172, 11-12 176/176); deliveries **8 / 8 `delivered`**, 0 `for_failure`; `cron_incidents` **0 unclosed**. **Every job now reads `last_status: ok` with `failure_streak` 0 — the fleet has no non-clean row at all**, because **`066bb5191280` (Uber Eats Monthly Spend Report) was fixed end-to-end on 2026-11-13**, clearing a `delivery_failed` that had stood since 2026-11-01. Two stacked causes: the job's **runtime prompt had grown to 88 k chars** (it carried two heavy browser skills) so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest, and the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart (**2026-11-13 13:14**). Fix: deterministic printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, rewritten prompt with a hard output contract, skills cleared (**88 k -> 1.8 k**); verified on the 18:51 -> 18:58 slot (**154 unique / 153 completed** orders, Oct **9 orders / $301.04**), lesson saved to `cron-job-maintenance`. Fleet **29 (26 / 3)** — an **eighth** window unchanged. ⚠ **But the window's fault is the `pre_tool_call` hook, and it returned:** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40** plus **10** suppression lines through **00:06:38**, refusing this audit's own tool calls for ~1 minute (first-hand). `errors.log` by day: 11-12 **0/0**, 11-13 **0/0**, 11-14 **1/10** — the first such timeout in the retained log since the 2026-11-08 cascade (**`errors.log` now starts 2026-11-09 13:18:39**, so the 11-08 lines have rotated out and the page's 58-count stands on the 2026-11-09 read). **Load is the trigger and it has ~doubled: `load average 12.08 / 13.02 / 12.33` on 2 vCPU** (vs **5.89** at diagnosis), with **`state.db` now 2.04 GB** (**2,977 sessions / 207,577 messages**, was 1.89 GB / 2,901 / 197,682) and the root filesystem **88 % used**. The 30 -> 120 s fix is **proven insufficient**; [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] is **reopened, `resolved` -> `active`**, and `state.db` checkpoint/retention — not a bigger timeout — is now the estate's most urgent open item. Freshness (03:00 stocktake): **0 stale / 131 fresh / 3 no-date of 134**.
|
> **2026-11-13 (nightly audit; live probes 2026-11-13 23:59 - 2026-11-14 00:2x AWST) — the fleet went fully clean, then the hook came back.** **2026-11-13: 174 ledger rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:41) — the **third consecutive zero-`failed` day** (11-11 172/172, 11-12 176/176); deliveries **8 / 8 `delivered`**, 0 `for_failure`; `cron_incidents` **0 unclosed**. **Every job now reads `last_status: ok` with `failure_streak` 0 — the fleet has no non-clean row at all**, because **`066bb5191280` (Uber Eats Monthly Spend Report) was fixed end-to-end on 2026-11-13**, clearing a `delivery_failed` that had stood since 2026-11-01. Two stacked causes: the job's **runtime prompt had grown to 88 k chars** (it carried two heavy browser skills) so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest, and the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart (**2026-11-13 13:14**). Fix: deterministic printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, rewritten prompt with a hard output contract, skills cleared (**88 k -> 1.8 k**); verified on the 18:51 -> 18:58 slot (**154 unique / 153 completed** orders, Oct **9 orders / $301.04**), lesson saved to `cron-job-maintenance`. Fleet **29 (26 / 3)** — an **eighth** window unchanged. ⚠ **But the window's fault is the `pre_tool_call` hook, and it returned:** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40** plus **10** suppression lines through **00:06:38**, refusing this audit's own tool calls for ~1 minute (first-hand). `errors.log` by day: 11-12 **0/0**, 11-13 **0/0**, 11-14 **1/10** — the first such timeout in the retained log since the 2026-11-08 cascade (**`errors.log` now starts 2026-11-09 13:18:39**, so the 11-08 lines have rotated out and the page's 58-count stands on the 2026-11-09 read). **Load is the trigger and it has ~doubled: `load average 12.08 / 13.02 / 12.33` on 2 vCPU** (vs **5.89** at diagnosis), with **`state.db` now 2.04 GB** (**2,977 sessions / 207,577 messages**, was 1.89 GB / 2,901 / 197,682) and the root filesystem **88 % used**. The 30 -> 120 s fix is **proven insufficient**; [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] is **reopened, `resolved` -> `active`**, and `state.db` checkpoint/retention — not a bigger timeout — is now the estate's most urgent open item. Freshness (03:00 stocktake): **0 stale / 131 fresh / 3 no-date of 134**.
|
||||||
>
|
>
|
||||||
> **2026-11-11 → 2026-11-12 (nightly audit; live probes 2026-11-12 23:59 – 2026-11-13 00:4x AWST) — a clean day, and an upgrade that could not be installed.** The 429 row that was the estate's only open incident **closed `resolved` at 2026-11-12T00:25:36** (the next slot of the same job ran clean, carrying both windows); *cron_incidents* now reads **0 unclosed of 137**. **2026-11-12: 176 ledger rows — 175 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:54), deliveries **9 / 9 `delivered`**; **11-11: 172 / 172 `completed` / 0 `failed`**, deliveries 7 / 7. Fleet **29 (26 / 3)**, a seventh window unchanged. The window's one new open item is not a failure: the attempt to take **`hermes-hq-push`** past its installed **0.9.42** was **refused by the pre-install plugin security scan** — *"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."* The critical is the plugin's **own** `skills/dashboard-tile/SKILL.md` one-liner; **no per-verdict override exists** (`plugins.scan_on_install: false` would blind the scan for *every* future install, and `plugin_guard.py` hardcodes `trust_level="community"`). The fix is **PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file), pushed from a fork because `gh` is **read-only** upstream. Installed 0.9.42 keeps delivering. Also: the alleged *"stored hold-and-ask instruction"* for scan warnings **does not exist on disk** (184,716-file sweep, zero matches in editable text; the sentence lives only in `state.db` history) — nothing was removed or reverted, and the hunt is now the skill `stored-instruction-search-and-removal` (**skills 299 → 300**). `config.yaml` is **byte-identical to its 2026-11-12 18:14:52 backup**; the only drift since 11-09 is the **model catalogue block** (6 out, 9 in). **New page:** [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]].
|
> **2026-11-11 → 2026-11-12 (nightly audit; live probes 2026-11-12 23:59 – 2026-11-13 00:4x AWST) — a clean day, and an upgrade that could not be installed.** The 429 row that was the estate's only open incident **closed `resolved` at 2026-11-12T00:25:36** (the next slot of the same job ran clean, carrying both windows); *cron_incidents* now reads **0 unclosed of 137**. **2026-11-12: 176 ledger rows — 175 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:54), deliveries **9 / 9 `delivered`**; **11-11: 172 / 172 `completed` / 0 `failed`**, deliveries 7 / 7. Fleet **29 (26 / 3)**, a seventh window unchanged. The window's one new open item is not a failure: the attempt to take **`hermes-hq-push`** past its installed **0.9.42** was **refused by the pre-install plugin security scan** — *"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."* The critical is the plugin's **own** `skills/dashboard-tile/SKILL.md` one-liner; **no per-verdict override exists** (`plugins.scan_on_install: false` would blind the scan for *every* future install, and `plugin_guard.py` hardcodes `trust_level="community"`). The fix is **PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file), pushed from a fork because `gh` is **read-only** upstream. Installed 0.9.42 keeps delivering. Also: the alleged *"stored hold-and-ask instruction"* for scan warnings **does not exist on disk** (184,716-file sweep, zero matches in editable text; the sentence lives only in `state.db` history) — nothing was removed or reverted, and the hunt is now the skill `stored-instruction-search-and-removal` (**skills 299 → 300**). `config.yaml` is **byte-identical to its 2026-11-12 18:14:52 backup**; the only drift since 11-09 is the **model catalogue block** (6 out, 9 in). **New page:** [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]].
|
||||||
@@ -80,7 +80,7 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
|
|||||||
- [[systems/skills-index]] — Installed skills inventory (**300 skills** (**67 categories**) as of the **2026-11-13** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
|
- [[systems/skills-index]] — Installed skills inventory (**300 skills** (**67 categories**) as of the **2026-11-13** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
|
||||||
|
|
||||||
## Incidents
|
## Incidents
|
||||||
- [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN — Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — **mislabelled**, state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **The 12:00:52 AWST run saw the identical hole and logged it ℹ️ informational** (`capture delay -589.5 min`), i.e. **~12 h of lead time produced no alarm**. Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]].
|
- [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN — Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault; this job's twelve earlier rows read `agent` / `timeout` / `unknown` — state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 AWST run saw nothing** (`capture delay 29.4 min`, exit 0, `suppressed`) — the `-589.5 min` ℹ️ line belongs to the **00:03:35** run itself (monitor log lines 957 / 1030 / 1046). Detection is **activity-aware** (`mnemosyne_freshness_gate.py` fires on `delay < −120 min`, i.e. on **magnitude**) and the first message to postdate the stall arrived at **19:22 AWST**, after that slot — so the ~12 h of quiet was the `0 */12` **cadence**, not a threshold bug. **What went unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, visible as `(0 items)` in every run but tested only for age (36 h). Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]].
|
||||||
- [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]] - **OPEN — the `hermes-hq-push` upgrade past v0.9.42 was refused by the pre-install plugin security scan.** `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` returned **"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."** The sole critical is the plugin's **own** `skills/dashboard-tile/SKILL.md:41` shell one-liner (`echo '<json>' | python3 …` = *obfuscation*); the other 38 findings are low/medium and alone would have installed as `caution`. **`--force` lifts `caution` only**; a `dangerous` verdict is an unconditional block (`tools/plugin_guard.py:406-418`), and **no per-verdict override exists** — the only key is `plugins.scan_on_install` (default `true`, currently unset), which disables scanning for *every* future install; the `trust_level`/allowlist machinery lives in `tools/skills_guard.py` (the skills path) and `plugin_guard.py` hardcodes `trust_level="community"`. **Installed state unchanged:** v0.9.42, `enabled`, still delivering. **Fix = PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc` → `main`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file, created 2026-11-12T10:00:01Z) — pushed from a fork because this host's `gh` account has **read-only** access upstream; local clone `/home/hermes/workspace/hq-plugins-fix` (`90c9d28`).
|
- [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]] - **OPEN — the `hermes-hq-push` upgrade past v0.9.42 was refused by the pre-install plugin security scan.** `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` returned **"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."** The sole critical is the plugin's **own** `skills/dashboard-tile/SKILL.md:41` shell one-liner (`echo '<json>' | python3 …` = *obfuscation*); the other 38 findings are low/medium and alone would have installed as `caution`. **`--force` lifts `caution` only**; a `dangerous` verdict is an unconditional block (`tools/plugin_guard.py:406-418`), and **no per-verdict override exists** — the only key is `plugins.scan_on_install` (default `true`, currently unset), which disables scanning for *every* future install; the `trust_level`/allowlist machinery lives in `tools/skills_guard.py` (the skills path) and `plugin_guard.py` hardcodes `trust_level="community"`. **Installed state unchanged:** v0.9.42, `enabled`, still delivering. **Fix = PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc` → `main`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file, created 2026-11-12T10:00:01Z) — pushed from a fork because this host's `gh` account has **read-only** access upstream; local clone `/home/hermes/workspace/hq-plugins-fix` (`90c9d28`).
|
||||||
- [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion; *cron_incidents* now 0 unclosed of 137) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]).
|
- [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion; *cron_incidents* now 0 unclosed of 137) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]).
|
||||||
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - ⚠ **REOPENED 2026-11-14 — the hook was the fault, and the 30 → 120 s fix did not hold at current load.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** ⚠ **Recurred 2026-11-14 00:05:40** — **1 timeout + 10 skips**, overrunning the **120 s** budget at `load average 12.08` on 2 vCPU (**~2×** the 5.89 at diagnosis), with `state.db` grown to **2.04 GB**. Status **`resolved` → `active`**; the lever is now urgent. → [[systems/conduit]], [[systems/hermes-agent]]
|
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - ⚠ **REOPENED 2026-11-14 — the hook was the fault, and the 30 → 120 s fix did not hold at current load.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** ⚠ **Recurred 2026-11-14 00:05:40** — **1 timeout + 10 skips**, overrunning the **120 s** budget at `load average 12.08` on 2 vCPU (**~2×** the 5.89 at diagnosis), with `state.db` grown to **2.04 GB**. Status **`resolved` → `active`**; the lever is now urgent. → [[systems/conduit]], [[systems/hermes-agent]]
|
||||||
|
|||||||
@@ -0,0 +1,149 @@
|
|||||||
|
# Wiki freshness audit — 2026-11-15
|
||||||
|
Threshold: >21 days since `updated`
|
||||||
|
|
||||||
|
## STALE (0)
|
||||||
|
|
||||||
|
|
||||||
|
## FRESH (133)
|
||||||
|
|
||||||
|
- [ 4d] CONTRIBUTING.md
|
||||||
|
- [ 7d] agents/hermes-chaos.md
|
||||||
|
- [ 7d] agents/hermes-production.md
|
||||||
|
- [ 4d] agents/index.md
|
||||||
|
- [ 17d] agents/nanobot.md
|
||||||
|
- [ 4d] agents/open-claw.md
|
||||||
|
- [ 11d] agents/qwenpaw.md
|
||||||
|
- [ 4d] agents/zeroclaw.md
|
||||||
|
- [ 17d] comparisons/camoufox-vs-chrome.md
|
||||||
|
- [ 4d] comparisons/hermes-vs-nanobot.md
|
||||||
|
- [ 4d] comparisons/memory-backends.md
|
||||||
|
- [ 4d] comparisons/model-routers.md
|
||||||
|
- [ 17d] comparisons/steel-vs-camoufox.md
|
||||||
|
- [ 4d] concepts/agent-memory.md
|
||||||
|
- [ 17d] concepts/browser-automation.md
|
||||||
|
- [ 4d] concepts/model-routing.md
|
||||||
|
- [ 4d] concepts/persistent-shells.md
|
||||||
|
- [ 4d] concepts/tool-calling.md
|
||||||
|
- [ 0d] current-state.md
|
||||||
|
- [ 17d] decisions/index.md
|
||||||
|
- [ 4d] decisions/keep-openrouter-as-fallback.md
|
||||||
|
- [ 4d] decisions/persistent-shell-enabled.md
|
||||||
|
- [ 17d] decisions/rollback-steel-to-camofox.md
|
||||||
|
- [ 4d] experiments/index.md
|
||||||
|
- [ 3d] incidents/2026-07-29-mnemosyne-relay.md
|
||||||
|
- [ 4d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||||
|
- [ 3d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||||
|
- [ 8d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||||
|
- [ 10d] incidents/2026-08-15-skills-index-drift.md
|
||||||
|
- [ 11d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||||
|
- [ 3d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||||
|
- [ 4d] incidents/2026-08-29-gmail-read-state.md
|
||||||
|
- [ 3d] incidents/2026-09-02-sftpgo-outage.md
|
||||||
|
- [ 11d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||||
|
- [ 17d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||||
|
- [ 20d] incidents/2026-10-03-ct450-hermes-update.md
|
||||||
|
- [ 20d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||||
|
- [ 17d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||||
|
- [ 0d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||||
|
- [ 14d] incidents/2026-10-09-memory-recall-relevance.md
|
||||||
|
- [ 9d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||||
|
- [ 19d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||||
|
- [ 5d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||||
|
- [ 19d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||||
|
- [ 3d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||||
|
- [ 0d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||||
|
- [ 4d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||||
|
- [ 20d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||||
|
- [ 20d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||||
|
- [ 18d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||||
|
- [ 16d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||||
|
- [ 16d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||||
|
- [ 16d] incidents/2026-10-29-conduit-voice-silence.md
|
||||||
|
- [ 15d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||||
|
- [ 16d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||||
|
- [ 15d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||||
|
- [ 14d] incidents/2026-10-31-asana-access-failures.md
|
||||||
|
- [ 14d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||||
|
- [ 14d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||||
|
- [ 13d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||||
|
- [ 12d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||||
|
- [ 11d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||||
|
- [ 9d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||||
|
- [ 9d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||||
|
- [ 10d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||||
|
- [ 6d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||||
|
- [ 10d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||||
|
- [ 8d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||||
|
- [ 6d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||||
|
- [ 1d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||||
|
- [ 4d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||||
|
- [ 4d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||||
|
- [ 3d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
|
||||||
|
- [ 2d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
|
||||||
|
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
|
||||||
|
- [ 0d] index.md
|
||||||
|
- [ 8d] infrastructure/backups.md
|
||||||
|
- [ 4d] infrastructure/credentials-map.md
|
||||||
|
- [ 8d] infrastructure/docker-services.md
|
||||||
|
- [ 4d] infrastructure/domains-and-tunnels.md
|
||||||
|
- [ 8d] infrastructure/hosts.md
|
||||||
|
- [ 8d] infrastructure/network-map.md
|
||||||
|
- [ 8d] infrastructure/proxmox.md
|
||||||
|
- [ 3d] infrastructure/tailscale.md
|
||||||
|
- [ 11d] queries/retained-research-answers.md
|
||||||
|
- [ 21d] reports/communities-wa-social-media-audit-2026.md
|
||||||
|
- [ 14d] reports/uber-eats/2026-07.md
|
||||||
|
- [ 14d] reports/uber-eats/2026-08.md
|
||||||
|
- [ 14d] reports/uber-eats/2026-09.md
|
||||||
|
- [ 2d] reports/uber-eats/2026-10.md
|
||||||
|
- [ 2d] reports/uber-eats/index.md
|
||||||
|
- [ 4d] runbooks/backup-wiki.md
|
||||||
|
- [ 4d] runbooks/diagnose-docker.md
|
||||||
|
- [ 4d] runbooks/gateway-resource-resilience.md
|
||||||
|
- [ 8d] runbooks/job-radar.md
|
||||||
|
- [ 4d] runbooks/multiplexer-setup.md
|
||||||
|
- [ 20d] runbooks/provider-health.md
|
||||||
|
- [ 17d] runbooks/recover-docker-service.md
|
||||||
|
- [ 4d] runbooks/restart-browser.md
|
||||||
|
- [ 4d] runbooks/restart-hermes.md
|
||||||
|
- [ 4d] runbooks/restore-openrouter.md
|
||||||
|
- [ 4d] runbooks/test-openrouter.md
|
||||||
|
- [ 4d] runbooks/troubleshoot-tool-loop.md
|
||||||
|
- [ 20d] runbooks/update-hermes-safely.md
|
||||||
|
- [ 7d] systems/browser-backend.md
|
||||||
|
- [ 4d] systems/cloudflare-tunnel.md
|
||||||
|
- [ 6d] systems/conduit.md
|
||||||
|
- [ 20d] systems/current-profile.md
|
||||||
|
- [ 4d] systems/guanaco.md
|
||||||
|
- [ 4d] systems/headless-server.md
|
||||||
|
- [ 6d] systems/hermes-agent.md
|
||||||
|
- [ 2d] systems/hermes-hq-push.md
|
||||||
|
- [ 4d] systems/hermes-ui.md
|
||||||
|
- [ 11d] systems/home-assistant.md
|
||||||
|
- [ 16d] systems/homepage-dashboard.md
|
||||||
|
- [ 11d] systems/ignis.md
|
||||||
|
- [ 20d] systems/memory-backend.md
|
||||||
|
- [ 13d] systems/messaging-integrations.md
|
||||||
|
- [ 0d] systems/mnemosyne.md
|
||||||
|
- [ 20d] systems/model-providers.md
|
||||||
|
- [ 11d] systems/nanobot.md
|
||||||
|
- [ 4d] systems/nextcloud.md
|
||||||
|
- [ 4d] systems/obsidian.md
|
||||||
|
- [ 5d] systems/omniroute.md
|
||||||
|
- [ 4d] systems/opencode-go.md
|
||||||
|
- [ 4d] systems/openrouter.md
|
||||||
|
- [ 11d] systems/paseo.md
|
||||||
|
- [ 0d] systems/phonon-tts.md
|
||||||
|
- [ 5d] systems/push-notifications.md
|
||||||
|
- [ 0d] systems/scheduled-tasks.md
|
||||||
|
- [ 0d] systems/skills-index.md
|
||||||
|
- [ 4d] systems/terminal-backend.md
|
||||||
|
- [ 4d] systems/tool-search.md
|
||||||
|
|
||||||
|
## NO updated DATE (3)
|
||||||
|
|
||||||
|
- SCHEMA.md
|
||||||
|
- experiments/active/example.md
|
||||||
|
- log.md
|
||||||
|
|
||||||
|
TOTAL: 0 stale, 133 fresh, 3 no-date of 136 pages
|
||||||
@@ -50,7 +50,7 @@ Primary agent memory store for Hermes.
|
|||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
|
|
||||||
**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed) — but its **12:00:52** slot had already seen the identical hole and logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**; the threshold in the check script is on *sign*, not magnitude. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). → [[incidents/2026-11-14-mnemosyne-capture-stall]]
|
**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). → [[incidents/2026-11-14-mnemosyne-capture-stall]]
|
||||||
|
|
||||||
**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**.
|
**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**.
|
||||||
|
|
||||||
|
|||||||
+44
-16
@@ -24,11 +24,15 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
|
|||||||
error`, **`failure_streak` 1** — its **00:00:53 slot failed at 00:03:44 on `Script exited with code 1`**
|
error`, **`failure_streak` 1** — its **00:00:53 slot failed at 00:03:44 on `Script exited with code 1`**
|
||||||
("STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59"), the first red row since the
|
("STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59"), the first red row since the
|
||||||
Uber Eats job cleared on **2026-11-13**. **Unclosed incident rows 0 -> 1** (**138** total), the row being
|
Uber Eats job cleared on **2026-11-13**. **Unclosed incident rows 0 -> 1** (**138** total), the row being
|
||||||
`ea79a2_601037516abd` (`state: alerted`, `failure_type: auth` — **mislabelled**, `alerted_at`
|
`ea79a2_601037516abd` (`state: alerted`, `alerted_at` 00:03:44.5; its `failure_type: auth` is **not**
|
||||||
00:03:44.5) → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver**
|
evidence of a credential fault — this job's twelve earlier rows read `agent` ×3 / `timeout` ×1 /
|
||||||
(`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — the failure was in
|
`unknown` ×8, and the estate-wide table is 53 `unknown` / 47 `agent` / 15 `timeout`) →
|
||||||
*detection*, not notification: at **12:00:52** the same slot had already seen the identical hole and
|
[[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver**
|
||||||
logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**.
|
(`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — **notification
|
||||||
|
worked, and so did detection; the earlier slots simply had nothing to see.** *(Corrected 2026-11-15
|
||||||
|
01:2x — an earlier revision of this entry claimed the **12:00:52** slot had "already seen the identical
|
||||||
|
hole" and logged it ℹ️ at `-589.5 min`, giving "~12 h of lead time"; that was wrong on both counts. See
|
||||||
|
[[#Known Issues]] for the verified mechanism.)*
|
||||||
**2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (that monitor run). Next slot of the monitor:
|
**2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (that monitor run). Next slot of the monitor:
|
||||||
**2026-11-15 12:00** (schedule `0 */12 * * *`) — the natural check for whether capture resumed.
|
**2026-11-15 12:00** (schedule `0 */12 * * *`) — the natural check for whether capture resumed.
|
||||||
- **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third
|
- **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third
|
||||||
@@ -311,17 +315,41 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
|
|||||||
### Open 2026-11-15
|
### Open 2026-11-15
|
||||||
|
|
||||||
- **The Mnemosyne Health Monitor (`ea79a2107e36`) is the fleet's only non-clean row** — `last_status: error`,
|
- **The Mnemosyne Health Monitor (`ea79a2107e36`) is the fleet's only non-clean row** — `last_status: error`,
|
||||||
`failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`, wrongly labelled
|
`failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`; the row's
|
||||||
`failure_type: auth`). Its 00:00 slot exited 1 at **00:03:44** on `STALE MEMORY PIPELINE: capture missed …
|
`failure_type: auth` is the scheduler's classifier and does **not** describe the fault — this job's
|
||||||
last capture 2026-11-14 01:32:59` (raw **UTC** in the alert text, so it *reads* as a 23 h gap; the true
|
earlier rows read `agent` / `timeout` / `unknown`). Its 00:00 slot exited 1 at **00:03:44** on
|
||||||
AWST gap at the alert was **14 h 31 m**). The **12:00:52** slot saw the same hole and logged it
|
`STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59` (raw **UTC** in the alert
|
||||||
**ℹ️ informational** — `capture delay -589.5 min`, the negative sign hiding its growth — so the estate
|
text, so it *reads* as a ~23 h gap; the true AWST gap at the alert was **14 h 31 m**). The alert itself
|
||||||
had **~12 h of lead time and no alarm**. The alert itself delivered (00:03:39, `for_failure: 1`).
|
delivered (00:03:39, `for_failure: 1`). Root cause **not established**; the known
|
||||||
Root cause **not established**; the known stringified-`config.yaml` killer is ruled out. Full page:
|
stringified-`config.yaml` killer is ruled out. Full page:
|
||||||
[[incidents/2026-11-14-mnemosyne-capture-stall]]. **Lessons for this fleet:** (a) a monitor that
|
[[incidents/2026-11-14-mnemosyne-capture-stall]].
|
||||||
distinguishes "capture is late" only in *sign* will never escalate on a growing hole — the threshold must
|
- **Verified mechanism of the escalation (re-derivation pass, 2026-11-15 01:2x AWST — corrects the first
|
||||||
be on magnitude; (b) alert timestamps printed from raw UTC inside a +08:00 estate are a standing
|
reading):** the **12:00:52** slot did **not** see the hole. `~/.hermes/logs/mnemosyne-health-monitor.log`
|
||||||
misreading trap.
|
pins it by line number — run header `2026-11-14 12:01:52 AWST` at **line 957**, next header
|
||||||
|
(`2026-11-15 00:03:35 AWST`) at **line 1030**, and the `capture delay -589.5 min` line at **line 1046**,
|
||||||
|
i.e. *inside the 00:03:35 run's own block*, immediately above the ❌ at line 1047. The 12:00:52 run's own
|
||||||
|
measurement was `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay **29.4 min**` — healthy, exit 0,
|
||||||
|
delivery `suppressed`. `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`,
|
||||||
|
`MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min; consolidation age default **36 h**), but it is
|
||||||
|
**activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last
|
||||||
|
captured message before the stall was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min); the first
|
||||||
|
message to postdate it was the **19:22 AWST desktop** turn — *after* the 12:00 slot, so midnight was the
|
||||||
|
first slot that could see the miss. **The ~12 h of quiet was the `0 */12` cadence, not a threshold bug.**
|
||||||
|
- **What genuinely went unread — consolidation item counts.** The sleep cycles kept firing every ~31 min and
|
||||||
|
returned **0 items: 29 consecutive zero-item cycles**, streak opening `2026-11-14T02:04:42+00:00`
|
||||||
|
(**10:04:42 AWST**) and running to the alert; the last productive cycles were 4622/4621/4620 at
|
||||||
|
`09:33:34` / `09:33:16` / `09:01:40` (naive **AWST** — see the format trap below). Every run printed
|
||||||
|
`Last consolidation: … (0 items)`, including the healthy 12:00 one, but the gate tests consolidation
|
||||||
|
**age** (`> 36 h`) only — never the item count. A pipeline writing nothing still reads as fresh.
|
||||||
|
**Candidate threshold: an N-consecutive-zero-item-cycle exception.**
|
||||||
|
- **Format trap in `consolidation_log.created_at` (verified this pass):** rows 4640–4651 carry a `+00:00`
|
||||||
|
offset, rows 4620–4622 are **naive and local (AWST)**. Read as one axis they misplace events by 8 h and
|
||||||
|
make the streak look as if it began after the last productive cycle. Same family as the alert text below.
|
||||||
|
- **Lessons for this fleet:** (a) alert timestamps printed from raw UTC inside a +08:00 estate are a standing
|
||||||
|
misreading trap (the alert's `01:32:59` is 09:32:59 AWST); (b) freshness gates must test a pipeline's
|
||||||
|
**output volume**, not just the age of its last run — an empty-but-current cycle is the failure signature
|
||||||
|
for a dead write path; (c) when a ledger's `failure_type` disagrees with the error text, trust the text —
|
||||||
|
the classifier is `unknown`/`agent` for most rows.
|
||||||
|
|
||||||
### Resolved 2026-11-02
|
### Resolved 2026-11-02
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-14
|
updated: 2026-11-15
|
||||||
verified_on: 2026-11-14
|
verified_on: 2026-11-15
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [skills, hermes]
|
tags: [skills, hermes]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -12,9 +12,9 @@ sources: []
|
|||||||
|
|
||||||
# Installed Skills
|
# Installed Skills
|
||||||
|
|
||||||
**Total installed: 300 skills** in `~/.hermes/skills/`.
|
**Total installed: 299 skills** in `~/.hermes/skills/`.
|
||||||
|
|
||||||
- **50 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
|
- **51 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
|
||||||
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
|
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
|
||||||
|
|
||||||
This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||||
@@ -98,7 +98,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
|||||||
|
|
||||||
- `ddia-systems` — # Designing Data-Intensive Applications Framework
|
- `ddia-systems` — # Designing Data-Intensive Applications Framework
|
||||||
|
|
||||||
## Devops (108)
|
## Devops (107)
|
||||||
|
|
||||||
- `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS)
|
- `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS)
|
||||||
- `agent-extension-audit` — # Agent Extension Audit
|
- `agent-extension-audit` — # Agent Extension Audit
|
||||||
@@ -204,7 +204,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
|||||||
- `uber-eats-monitor` — # Uber Eats Order Monitor
|
- `uber-eats-monitor` — # Uber Eats Order Monitor
|
||||||
- `vision-model-validation` — # Vision Model Validation via OmniRoute
|
- `vision-model-validation` — # Vision Model Validation via OmniRoute
|
||||||
- `watchers` — # Watchers
|
- `watchers` — # Watchers
|
||||||
- `web-content-extraction` — # Web Content Extraction — Fallback Techniques
|
|
||||||
- `weekly-comms-tracker` — # Weekly Communications Tracker
|
- `weekly-comms-tracker` — # Weekly Communications Tracker
|
||||||
- `wiki-maintenance` — # Wiki Maintenance
|
- `wiki-maintenance` — # Wiki Maintenance
|
||||||
- `zeroclaw-ops` — # ZeroClaw Operations & Configuration
|
- `zeroclaw-ops` — # ZeroClaw Operations & Configuration
|
||||||
|
|||||||
Reference in New Issue
Block a user