nightly audit 2026-11-09: pre_tool_call hook-timeout cascade (new incident page), hook budget 30->120s, state.db 1.89GB + corruption remnant, no-agent health monitor verified 73s, third silent stocktake incident, skills 299/67, freshness 0 stale/127 fresh of 130

This commit is contained in:
Tony0410
2026-11-09 00:21:01 +08:00
parent 4fd138bb18
commit e2f1ae7a0b
12 changed files with 422 additions and 29 deletions
+37 -3
View File
@@ -3,8 +3,8 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-07 updated: 2026-11-09
verified_on: 2026-11-07 verified_on: 2026-11-09
review_after: 2026-11-09 review_after: 2026-11-09
confidence: high confidence: high
tags: [system] tags: [system]
@@ -12,7 +12,41 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
--- ---
# Current State # Current State
Updated: 2026-11-08 Updated: 2026-11-09
## Recent Changes (2026-11-09, nightly audit; live probes 2026-11-08 23:5x – 2026-11-09 00:2x AWST)
- ⚠ **The `conduit_push` `pre_tool_call` hook was the day's fault, and it is fixed — `plugins.hook_callback_timeout` 30 → 120 s.**
The plugin is the **only** `pre_tool_call` registrant on the host, and its callback overran Hermes's 30 s
hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: 12
lines on 11-07, 46 on 11-08; plus **89** *"skipped after previous timeout or while still running"*
suppression lines). Every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST**;
the last timeout line is still 08:21:03, re-verified at audit time. **The plugin's caching was measured,
not assumed, and was innocent** — 3.1 s cold / 0.00 s warm (`session_titles.py:21 CACHE_SECONDS = 30.0`);
the trigger is **load** on the 2-core CT460 (load average **5.89**, `state.db` with a **40 MB WAL**).
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]], [[systems/conduit]]
- ⚠ **`state.db` is now 1.89 GB** (1,890,656,256 B; **2,901 sessions / 197,682 messages**, live
**40,157,672 B** WAL) — up from 1.7 GB, and it is the load source behind the spike above. The 2026-08-11
corruption remnant **`state.db.corrupted.20260811_151052` (624.1 MB)** was **deleted** in this window;
`state.db.recovered.recovery.json` (14,566 B) remains, and `mnemosyne.db.corrupted.20260809_120404`
(23.6 MB) was **not** cleaned. → [[systems/hermes-agent]], [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]
- ✅ **The Mnemosyne Health Monitor's no-agent conversion is verified end-to-end.** The 18:00 slot (agent
mode) ran **12:00:51 → 18:00:42 = 5 h 59 m 51 s**; the converted job's first slot ran
**2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, **silent**, writing a **159 B** output file against
**105,982 B** for the last agent-mode one. Live coverage **working 1030/1030 = 100.0 %, episodic
1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet. → [[systems/mnemosyne]], [[systems/scheduled-tasks]]
- ⚠ **The stocktake failed a third consecutive day, on a new signature.** `1a8e90ea2271` ran
**2026-11-08 03:01:11 → 04:36:08** and died on `RuntimeError: ⚠️ No reply: the turn stopped while a tool
result was still pending and the model produced no follow-up text.` — the previous two red days read
*"lost its durable fire claim ownership"*. **`failure_streak` 3.** *Working hypothesis, not proven:* the
hook above holds a tool result open in exactly the way this error describes. **Unclosed incident rows
2 → 3**, and all three are silent (`detected`, `alerted_at: null`) — but **the alert path works**: the
*Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` at **08:06:54**.
→ [[systems/scheduled-tasks]], [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- **2026-11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit). Deliveries
**8 / 8 `delivered`**. Fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count, though
`updated_at` moved to `2026-11-09T00:06:50+08:00` with no definition change. Nothing added, retired,
disabled or repinned. → [[systems/scheduled-tasks]]
## Recent Changes (2026-11-08, nightly audit; live probes 2026-11-07 23:5x – 2026-11-08 00:5x AWST) ## Recent Changes (2026-11-08, nightly audit; live probes 2026-11-07 23:5x – 2026-11-08 00:5x AWST)
@@ -3,8 +3,8 @@ title: Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and T
type: incident type: incident
status: open status: open
created: 2026-11-06 created: 2026-11-06
updated: 2026-11-06 updated: 2026-11-09
verified_on: 2026-11-06 verified_on: 2026-11-09
review_after: 2026-11-13 review_after: 2026-11-13
confidence: high confidence: high
tags: [security, credentials, mnemosyne, memory, sessions, cron, incident] tags: [security, credentials, mnemosyne, memory, sessions, cron, incident]
@@ -57,7 +57,9 @@ carries any tested value; `cron/executions.db`, `cron/notepad.db` and `data/shar
### 3. Session transcripts hold all four values ### 3. Session transcripts hold all four values
`~/.hermes/state.db` (1.7 GB, the session store) contains all four values. Measured as 4 MB chunks `~/.hermes/state.db` (1.7 GB *at discovery* — **1.89 GB / 1,890,656,256 B as at 2026-11-09 00:0x AWST**, with
**2,901 sessions / 197,682 messages** and a live **40,157,672 B WAL**; the store grew while nothing was
retentioned), the session store, contains all four values. Measured as 4 MB chunks
containing the value: **`CAMOFOX_API_KEY` 178, `MATTERMOST_TOKEN` 64, containing the value: **`CAMOFOX_API_KEY` 178, `MATTERMOST_TOKEN` 64,
`HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` 61, `TELEGRAM_BOT_TOKEN` 22**. This is the **most likely origin of `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` 61, `TELEGRAM_BOT_TOKEN` 22**. This is the **most likely origin of
exposure 1** - the memory-capture path writes from session text, so a session in which a key was printed exposure 1** - the memory-capture path writes from session text, so a session in which a key was printed
@@ -72,7 +74,11 @@ becomes a memory row - and it is itself a plaintext at-rest store that deleting
**rotation is the only remedy that fully closes it**. **rotation is the only remedy that fully closes it**.
- `jobs.json` is the fleet's source of truth and is exported and examined routinely; a prompt-embedded key - `jobs.json` is the fleet's source of truth and is exported and examined routinely; a prompt-embedded key
is exposed to anything that reads job definitions. is exposed to anything that reads job definitions.
- `state.db` is the fleet's largest artifact (1.7 GB) and is copied into snapshots and repair backups. - `state.db` is the fleet's largest artifact (**1.89 GB as at 2026-11-09**, up from 1.7 GB) and is copied into
snapshots and repair backups. ⚠ **It is also the load source behind the 2026-11-07/08 hook-timeout
cascade** — a 40 MB WAL checkpointing on the same spindle as a 2-core guest → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
The **2026-08-11 corruption remnant `state.db.corrupted.20260811_151052` (624.1 MB)** was deleted in the
2026-11-08 window, so one stale copy did leave the box — but **`mnemosyne/data/mnemosyne.db.corrupted.20260809_120404` (23.6 MB) is still there**, so the rotation-snapshot exposure above is untouched.
## Origin (inferred - not established) ## Origin (inferred - not established)
@@ -91,7 +97,7 @@ exist, and the affected rows carry no marker distinguishing them from ordinary c
4. **Add capture-time and delivery-time redaction** so `.env` values cannot enter memory rows, job prompts or 4. **Add capture-time and delivery-time redaction** so `.env` values cannot enter memory rows, job prompts or
delivery records again. delivery records again.
5. **Decide a retention/serialisation policy for `state.db`** - it currently holds every transcript 5. **Decide a retention/serialisation policy for `state.db`** - it currently holds every transcript
verbatim at 1.7 GB. verbatim at 1.89 GB (2,901 sessions / 197,682 messages) and it is growing.
## Verification evidence ## Verification evidence
@@ -3,8 +3,8 @@ title: Two cron runs lost their durable fire claims in a gateway restart drain (
type: incident type: incident
status: open status: open
created: 2026-11-08 created: 2026-11-08
updated: 2026-11-08 updated: 2026-11-09
verified_on: 2026-11-08 verified_on: 2026-11-09
review_after: 2026-11-15 review_after: 2026-11-15
confidence: high confidence: high
tags: [cron, fire-claim, scheduler, gateway-restart, drain, conduit, health-monitor, wiki-stocktake] tags: [cron, fire-claim, scheduler, gateway-restart, drain, conduit, health-monitor, wiki-stocktake]
@@ -108,6 +108,14 @@ cannot say whether their side effects ran (*"whether side effects ran is unknown
- The two open stocktake rows have **never been reconciled** (`alerted_at` null) — either resolve them or - The two open stocktake rows have **never been reconciled** (`alerted_at` null) — either resolve them or
fix the alerting path that let a `detected` row sit silent. fix the alerting path that let a `detected` row sit silent.
⚠ **2026-11-09 re-check: still open, and now three.** `1a8e90_628d914f1f97` and `1a8e90_2f370ef764f5`
(both first seen 2026-11-07) are unchanged, and a **third** arrived —
`1a8e90_a9cad99cb150`, `detected` 2026-11-08T04:35:38, `alerted_at: null`, from the 03:01:11 → 04:36:08
failure on `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending…`.
**Unclosed incident rows 2 → 3.** The alert path itself is **not** broken: the *Newsletter Digest* row
`85edf6_0877ca1e0eac` alerted at **07:23:07** and closed as `resolved` at **08:06:54** the same window.
So the gap is specific to the **stocktake's** rows.
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
- Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and - Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and
whether the 1800 s after-turn wait should refuse to drain long-running cron executions. whether the 1800 s after-turn wait should refuse to drain long-running cron executions.
@@ -0,0 +1,101 @@
---
title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
type: incident
status: resolved
created: 2026-11-09
updated: 2026-11-09
verified_on: 2026-11-09
review_after: 2026-11-16
confidence: high
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
---
# The `conduit_push` `pre_tool_call` hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
**Summary.** From **2026-11-07 14:14:03** to **2026-11-08 08:21:03** the `conduit_push` plugin's
**sole `pre_tool_call` callback** overran Hermes's **30 s hook budget** and was skipped. Each skip
suppresses the callback for a window, so the *pre-tool* side of tool dispatch degraded from *slow* to
*blocked* — this is the fault that ate the 2026-11-08 morning and stretched that day's cron runs to
**6 h** (*Mnemosyne Health Monitor* 12:00:51 → 18:00:42) and **1 h 13 m** (*FreshRSS Midday* 12:00:50 →
14:10:43). Fixed **2026-11-08 13:35 AWST** by raising `plugins.hook_callback_timeout` **30 → 120**.
## Evidence (live reads, 2026-11-08 23:5x – 2026-11-09 00:1x AWST)
grep -c "callback _pre_tool_call timed out" ~/.hermes/logs/errors.log -> 58
# 2026-11-07: 12 lines 2026-11-08: 46 lines
# first 2026-11-07 14:14:03,075 last 2026-11-08 08:21:03,863
grep -c "skipped after previous timeout" ~/.hermes/logs/*.log -> 89
The exact pair of lines (`~/.hermes/logs/errors.log`):
2026-11-08 08:19:40,942 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call skipped after previous timeout or while still running
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
**The register (correction to a plausible-but-wrong diagnosis).** `conduit_push/__init__.py:119` is the
**only** `pre_tool_call` registration on the host, and the plugin's caches are already in place —
`session_titles.py:21 CACHE_SECONDS = 30.0`. Timed in-process by the 2026-11-08 session:
**3.1 s cold, 0.00 s warm**. The *"the hook re-opens a 1.87 GB state.db per call, ~9 s each"* theory was
therefore **wrong** — warm calls were free. The timeouts track **load spikes** on the 2-core CT460
(load average **5.89** at 13:27:50; `state.db` carrying a **40 MB WAL** checkpointing on the same spindle),
not per-call database opens.
## Fix applied (2026-11-08 13:35 AWST)
~/.hermes/config.yaml
plugins:
hook_callback_timeout: 120 # was the built-in 30 s default
- **Verified live:** `plugins.hook_callback_timeout = 120`.
- **Verified effective:** no `_pre_tool_call` timeout has been logged since **08:21:03** — re-checked at
the end of the fix window and again at audit time (2026-11-09 00:0x). The **58th** line is still the last.
- **Deliberately NOT done:** rewriting the plugin's caching layer. The caches work; the budget was the
binding constraint. **Next lever is `state.db` checkpoint/retention discipline**, not more caching.
## Consequence — the same signature landed in the cron ledger
The 2026-11-08 03:00 *wiki-stocktake-maintenance* slot (`1a8e90ea2271`, 03:01:11 → 04:36:08) closed
**`failed`** on a **new terminal signature for that job**:
RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model
produced no follow-up text. Send `continue` to let it summarize.
*Working hypothesis, not proven:* a `pre_tool_call` block holding a tool result open is exactly what this
error describes, and it sits inside the same 14 h timeout window. It is **consistent**, not established —
the error string does not name the hook.
## Impact
| item | effect |
|---|---|
| *Mnemosyne Health Monitor* `ea79a2107e36` | 00:01:20 → **08:29:05** (8 h 27 m) and 12:00:51 → **18:00:42** (5 h 59 m 51 s) — its two slowest slots on record, both delivered |
| *wiki-stocktake-maintenance* `1a8e90ea2271` | 03:01:11 → 04:36:08 `failed`; **`failure_streak` 3**; a **third** unalerted `detected` incident row |
| *FreshRSS Midday Briefing* `ecc1fc13655a` | 12:00:50 → **14:10:43** (2 h 09 m 53 s) |
| *Tech & AI Newsletter Digest* `85edf6772bca` | an admission-read `database is locked` incident (`85edf6_0877ca1e0eac`) at **07:23:02**, **resolved** at 08:06:54 when the run completed |
| every interactive tool call | slow, then suppressed — the user-visible symptom that opened the investigation |
## Verification evidence
~/.hermes/logs/errors.log (58 timeout lines; last 2026-11-08 08:21:03)
~/.hermes/config.yaml (plugins.hook_callback_timeout: 120)
~/.hermes/plugins/conduit_push/__init__.py:119,257 (the only pre_tool_call registration)
~/.hermes/plugins/conduit_push/session_titles.py:21 (CACHE_SECONDS = 30.0)
~/.hermes/cron/executions.db, cron_incidents, jobs.json (run windows, streak, incident rows)
## Open items
- **`state.db` is 1.89 GB with 2,901 sessions / 197,682 messages and a live 40 MB WAL** — it is the load
source behind the spikes. Checkpoint and retention policy are still undecided
([[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
- Confirm whether the stocktake's `No reply: … tool result was still pending` failure is really the same
fault; if it recurs while the hook budget is 120 s, this hypothesis is dead.
- Decide whether a skipped `pre_tool_call` should **fail loudly** rather than silently degrade tool dispatch.
## See also
- [[systems/conduit]]
- [[systems/hermes-agent]]
- [[systems/scheduled-tasks]]
- [[systems/mnemosyne]]
- [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- [[current-state]]
+8 -5
View File
@@ -3,16 +3,18 @@ title: Anthony's Systems Wiki
type: collection type: collection
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-07 updated: 2026-11-09
verified_on: 2026-11-07 verified_on: 2026-11-09
confidence: high confidence: high
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar] tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
--- ---
# Anthony's Systems Wiki # Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure, > Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-11-08. > model routing and automation. Last updated: 2026-11-09.
>
> **2026-11-09 (nightly audit — window 2026-11-08) — the day's fault was a hook, not a schedule, and it is fixed.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`), and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** — `errors.log` holds **12** lines on 11-07 and **46** on 11-08, plus **89** *"skipped after previous timeout or while still running"* suppression lines. Every tool call degraded to slow-then-suppressed, which is what stretched the day. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 → 120** in `~/.hermes/config.yaml` — verified live, and verified *effective* (the last timeout line is still 08:21:03, re-checked at audit time). ⚠ **The obvious suspect was measured and acquitted:** the plugin's caching is fine (`session_titles.py:21 CACHE_SECONDS = 30.0`; timed in-process at **3.1 s cold, 0.00 s warm**), so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong** — the trigger is **load** on the 2-core CT460 (`load average 5.89` at 13:27:50, `state.db` checkpointing a **40 MB WAL** on the same spindle) → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. **Cost of the fault:** *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)** — its two slowest slots on record, both delivered; *FreshRSS Midday Briefing* took **2 h 09 m 53 s**; and **the stocktake failed a third consecutive day** (`1a8e90ea2271`, 03:01:11 → 04:36:08, `failure_streak: 3`) on a **new terminal signature** — `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` where the previous two red days read *"lost its durable fire claim ownership"*. *Working hypothesis, not proven:* an open tool result is exactly what a suppressed `pre_tool_call` would produce, but the string does not name the hook. **11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33); deliveries **8 / 8 `delivered`**; fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count with nothing added, retired, disabled or repinned, though `updated_at` moved to `2026-11-09T00:06:50+08:00` as a run-state stamp. **Unclosed incident rows 2 → 3**, all three the stocktake's and all silent (`detected`, `alerted_at: null`) — but **the alerting path is not the fault**: the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window. ✅ **Two verified resolutions:** the **Mnemosyne Health Monitor's no-agent conversion works** — its first script slot ran **2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, silent, writing a **159 B** output file against **105,982 B** for the last agent-mode run (coverage live at **working 1030/1030 = 100.0 %, episodic 1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet; the script uses `awk` because **`bc` is not installed on this host**); and the **1.89 GB `state.db`** was re-measured (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL) with the 2026-08-11 remnant **`state.db.corrupted.20260811_151052` (624.1 MB) deleted**. **New page:** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
> >
> **2026-11-08 (nightly audit — window 2026-11-07) — two runs orphaned mid-restart, and two incident rows that never alerted.** The gateway restart that arrived at **05:01:00** (after a requested restart sat in *after-turn* mode for the full 1800 s) drained two in-flight cron executions; both closed `failed` on `lost its durable fire claim ownership` — *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22, **second consecutive failed day**, `failure_streak: 2`). The restart was requested at ~04:29:52 — **inside the `conduit_push` self-update window** (`0.122.2 → 0.134.4`, 04:27:55→04:32:32, its own ledger recording `restarting_gateway` = *failed*) — working hypothesis, not proven; the 0.22.5 → 0.23.0 updater is excluded (its receipt starts 05:05:13). Day counts: **163 rows — 159 `completed` / 3 `failed` / 1 `running`**; the third failure is the agent-log watchdog correctly reporting the event 19 s later. ⚠ **Both stocktake incident rows are still `detected` with `alerted_at: null`** — closed 0 → 2, and neither alerted. Elsewhere in the window: **Hermes 0.22.5 → 0.23.0** on CT460 (snapshot stage **skipped again** — still no rollback point), the `camofox` browser backend arrived as an **enabled plugin (1.1.0)**, `conduit_push` self-updated to **0.134.4**, `hermes-hq-push` reinstalled **0.9.27 → 0.9.42** (routes reloaded by the 11:11 restart), and the **OpenCode Go credential is expired** (HTTP 401 `api_key_expired` — configuration alive, credential dead, nothing consuming it yet). New page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]. > **2026-11-08 (nightly audit — window 2026-11-07) — two runs orphaned mid-restart, and two incident rows that never alerted.** The gateway restart that arrived at **05:01:00** (after a requested restart sat in *after-turn* mode for the full 1800 s) drained two in-flight cron executions; both closed `failed` on `lost its durable fire claim ownership` — *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22, **second consecutive failed day**, `failure_streak: 2`). The restart was requested at ~04:29:52 — **inside the `conduit_push` self-update window** (`0.122.2 → 0.134.4`, 04:27:55→04:32:32, its own ledger recording `restarting_gateway` = *failed*) — working hypothesis, not proven; the 0.22.5 → 0.23.0 updater is excluded (its receipt starts 05:05:13). Day counts: **163 rows — 159 `completed` / 3 `failed` / 1 `running`**; the third failure is the agent-log watchdog correctly reporting the event 19 s later. ⚠ **Both stocktake incident rows are still `detected` with `alerted_at: null`** — closed 0 → 2, and neither alerted. Elsewhere in the window: **Hermes 0.22.5 → 0.23.0** on CT460 (snapshot stage **skipped again** — still no rollback point), the `camofox` browser backend arrived as an **enabled plugin (1.1.0)**, `conduit_push` self-updated to **0.134.4**, `hermes-hq-push` reinstalled **0.9.27 → 0.9.42** (routes reloaded by the 11:11 restart), and the **OpenCode Go credential is expired** (HTTP 401 `api_key_expired` — configuration alive, credential dead, nothing consuming it yet). New page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]].
> >
@@ -67,9 +69,10 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
- [[systems/skills-index]] — Installed skills inventory (**299 skills** (**67 categories**) as of the **2026-11-08** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. - [[systems/skills-index]] — Installed skills inventory (**299 skills** (**67 categories**) as of the **2026-11-08** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
## Incidents ## Incidents
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - **RESOLVED (same day) — the hook was the fault, not the schedule.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** → [[systems/conduit]], [[systems/hermes-agent]]
- [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - **OPEN.** A restart requested at **~04:29:52** sat in *after-turn* mode for the full **1800 s**, then timed out and drained at **05:00:43/47** — the same `#77184` behaviour as the 10-31 reload, but this time it killed **two** in-flight cron executions: *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42.918 -> 05:02:22.870) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31.515 -> 05:02:22.174), both closed **`failed`** on `lost its durable fire claim ownership`; systemd restarted the unit at **05:01:00** (`restart counter is at 3`). A **third** failure that night was the agent-log ERROR watchdog correctly catching the event 19 s later. The request landed **inside the `conduit_push` self-update window** (04:27:55 -> 04:32:32, its own ledger recording `restarting_gateway` = *failed*) — the **working hypothesis for the requester, not proven**; the 0.22.5 -> 0.23.0 updater is excluded (receipt starts 05:05:13). ⚠ **Closed 0 -> 2 incident rows and neither alerted**: both `1a8e90_*` rows sit `detected` with `alerted_at: null`. The stocktake slot has now failed **two days running** (`failure_streak: 2`). Job Radar ran clean. → [[systems/scheduled-tasks]], [[systems/hermes-agent]], [[systems/conduit]], [[systems/mnemosyne]] - [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - **OPEN.** A restart requested at **~04:29:52** sat in *after-turn* mode for the full **1800 s**, then timed out and drained at **05:00:43/47** — the same `#77184` behaviour as the 10-31 reload, but this time it killed **two** in-flight cron executions: *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42.918 -> 05:02:22.870) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31.515 -> 05:02:22.174), both closed **`failed`** on `lost its durable fire claim ownership`; systemd restarted the unit at **05:01:00** (`restart counter is at 3`). A **third** failure that night was the agent-log ERROR watchdog correctly catching the event 19 s later. The request landed **inside the `conduit_push` self-update window** (04:27:55 -> 04:32:32, its own ledger recording `restarting_gateway` = *failed*) — the **working hypothesis for the requester, not proven**; the 0.22.5 -> 0.23.0 updater is excluded (receipt starts 05:05:13). ⚠ **Closed 0 -> 2 incident rows and neither alerted**: both `1a8e90_*` rows sit `detected` with `alerted_at: null`. The stocktake slot has now failed **two days running** (`failure_streak: 2`). Job Radar ran clean. → [[systems/scheduled-tasks]], [[systems/hermes-agent]], [[systems/conduit]], [[systems/mnemosyne]]
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **OPEN.** A `direct`-source fire of `5fc292ff0f3b` (08:27:09 -> 13:26:22, 4 h 59 m 13 s, delivered) settled the claim while the scheduler's **own 11:30:32 slot** was still running, so that slot was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - a **new string variant** and the first claim-loss failure **with no gateway restart involved**. It raised **no `cron_incidents` row**, so a `failed` execution with no alert was the day's only red row; the discarded run left no output file. The 14:30 slot recovered the day in a normal **12 m 27 s** cycle. Candidate fixes (slot suppression when a claim is live, consuming the next slot for out-of-band fires, raising an incident row on discard) are listed on the page and **none was applied**. - [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **OPEN.** A `direct`-source fire of `5fc292ff0f3b` (08:27:09 -> 13:26:22, 4 h 59 m 13 s, delivered) settled the claim while the scheduler's **own 11:30:32 slot** was still running, so that slot was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - a **new string variant** and the first claim-loss failure **with no gateway restart involved**. It raised **no `cron_incidents` row**, so a `failed` execution with no alert was the day's only red row; the discarded run left no output file. The 14:30 slot recovered the day in a normal **12 m 27 s** cycle. Candidate fixes (slot suppression when a claim is live, consuming the next slot for out-of-band fires, raising an incident row on discard) are listed on the page and **none was applied**.
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **OPEN.** Live `.env` values were found stored verbatim in **17 rows of `mnemosyne.db`** (`memories` 4, `episodic_memory` 3, `working_memory` 1, `gists` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the **7 uncompressed rotation snapshots**, and all four values also sit verbatim in the **1.7 GB session store `state.db`**, which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Nothing rotated, deleted or scrubbed; no value recorded. - [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **OPEN.** Live `.env` values were found stored verbatim in **17 rows of `mnemosyne.db`** (`memories` 4, `episodic_memory` 3, `working_memory` 1, `gists` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the **7 uncompressed rotation snapshots**, and all four values also sit verbatim in the **1.89 GB session store `state.db`** (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL as at 2026-11-09), which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Nothing rotated, deleted or scrubbed; no value recorded.
- [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter). - [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter).
- [[incidents/2026-11-05-skills-index-generator-divergence]] — **RESOLVED — the skills index flip-flop had two writers.** The repo `wiki-maintain.sh` (302 skills / 68 categories) and the stale `~/.hermes/scripts/wiki-maintain.sh` copy (349 / 26, because it used `Path.rglob()` and counted the parked `.archive/`) were **different scripts**, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with **no error raised by either**. The stale copy is now a **delegating wrapper**; the repo script gained `--no-build` so the wrapper honours the old interface without a second generator. - [[incidents/2026-11-05-skills-index-generator-divergence]] — **RESOLVED — the skills index flip-flop had two writers.** The repo `wiki-maintain.sh` (302 skills / 68 categories) and the stale `~/.hermes/scripts/wiki-maintain.sh` copy (349 / 26, because it used `Path.rglob()` and counted the parked `.archive/`) were **different scripts**, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with **no error raised by either**. The stale copy is now a **delegating wrapper**; the repo script gained `--no-build` so the wrapper honours the old interface without a second generator.
- [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] — **OPEN — the app's "local" voice mode never applies, and the connection test lies.** The test checks **socket reachability only**, while every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects `session.update` as an unknown event**. `voice.mode` is a **phone-side** selector — **no `mode` key exists** in `config.yaml → voice` or `conduit-settings.json` — so a "mode did not apply" cannot be diagnosed on host config. Found via session `20261031_193709_40b9c2` (19:37:09, 30 msgs / 17 tool calls); `voice.local_live.front_model` was repointed to `mistral/voxtral-small-latest` during it and **nothing was changed on the Mac**. → [[systems/conduit]] - [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] — **OPEN — the app's "local" voice mode never applies, and the connection test lies.** The test checks **socket reachability only**, while every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects `session.update` as an unknown event**. `voice.mode` is a **phone-side** selector — **no `mode` key exists** in `config.yaml → voice` or `conduit-settings.json` — so a "mode did not apply" cannot be diagnosed on host config. Found via session `20261031_193709_40b9c2` (19:37:09, 30 msgs / 17 tool calls); `voice.local_live.front_model` was repointed to `mistral/voxtral-small-latest` during it and **nothing was changed on the Mac**. → [[systems/conduit]]
+58
View File
@@ -2379,8 +2379,66 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
**after** this audit's writes — skills-index regenerated (**299 / 67**), freshness report written, Quartz **after** this audit's writes — skills-index regenerated (**299 / 67**), freshness report written, Quartz
rebuilt (**129 markdown → 779 files** deployed to `public/`) and `hermes-wiki-serve.service` restarted at rebuilt (**129 markdown → 779 files** deployed to `public/`) and `hermes-wiki-serve.service` restarted at
**2026-11-08 00:38:29 AWST**. **2026-11-08 00:38:29 AWST**.
- **Wiki maintenance:** `wiki-maintain.sh` (canonical repo script) run **after** this audit's writes —
`systems/skills-index.md` regenerated (**299 skills / 67 categories**; disk shows 300 `SKILL.md`, one nested
duplicate de-duped by the generator), freshness report `raw/stocktake/freshness-20261109.txt`
(**0 stale / 127 fresh / 3 no-date of 130 pages**), Quartz rebuilt (**130 markdown → 793 files** deployed to
`public/`) and `hermes-wiki-serve.service` (a **user** unit — `systemctl --user`) restarted
**2026-11-09 00:18:30 AWST**. Published pages verified **HTTP 200** at `http://127.0.0.1:9120/<path>` **and**
on the external route `https://hermes.kangaroo-eel.ts.net/wiki/<path>`.
- **No wiki page removed or archived.** No job added, retired, disabled or repinned; no system obsoleted; no - **No wiki page removed or archived.** No job added, retired, disabled or repinned; no system obsoleted; no
decision reversed. (The four skill archives above happened on the filesystem, not in this wiki.) decision reversed. (The four skill archives above happened on the filesystem, not in this wiki.)
- **No secrets, keys, tokens or passwords written** — the opencode-go finding records the HTTP status and - **No secrets, keys, tokens or passwords written** — the opencode-go finding records the HTTP status and
the upstream error code only. the upstream error code only.
## [2026-11-09] nightly audit | Window 2026-11-08 — a 30 s hook budget, not a schedule, ate the day
- **New page:** `incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md` — **RESOLVED same day.**
- **The finding:** `conduit_push` is the **only** `pre_tool_call` registrant on the host
(`plugins/conduit_push/__init__.py:119`), and its callback overran Hermes's built-in **30 s** hook budget
**58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** — `~/.hermes/logs/errors.log`
holds **12** lines on 11-07 and **46** on 11-08, plus **89** *"skipped after previous timeout or while
still running"* suppression lines. Every tool call degraded from slow to suppressed.
- **The fix (2026-11-08 13:35 AWST):** `plugins.hook_callback_timeout` **30 → 120** in
`~/.hermes/config.yaml`. **Verified live**, and **verified effective** — the last timeout line is still
08:21:03, re-checked at audit time.
- **A plausible diagnosis measured and rejected:** the plugin's caching was *not* the fault. Timed
in-process: **3.1 s cold, 0.00 s warm** (`session_titles.py:21 CACHE_SECONDS = 30.0`). The
*"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the
2-core CT460 (`load average 5.89` at 13:27:50; `state.db` checkpointing a **40 MB WAL** on the same
spindle). **Next lever is `state.db` checkpoint/retention discipline, not more caching.**
- **Cost of the fault:** *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and
**12:00:51 → 18:00:42 (5 h 59 m 51 s)** — its two slowest slots on record, both delivered; *FreshRSS Midday
Briefing* `ecc1fc13655a` took **2 h 09 m 53 s**; and the stocktake failed a third day.
- **Cron ledger:** 11-08 settled **169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit,
23:59:33). The single failure is **`1a8e90ea2271` (wiki-stocktake-maintenance), 03:01:11 → 04:36:08**, on
a **new terminal signature**: `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still
pending and the model produced no follow-up text.` (the previous two red days read *"lost its durable fire
claim ownership"*). **`failure_streak: 3`.** *Working hypothesis, not proven:* a suppressed `pre_tool_call`
holding a tool result open is exactly what the error describes, but the string does not name the hook.
Deliveries **8 rows / 8 `delivered` / 0 `for_failure`**. Fleet **29 jobs (26 enabled / 3 disabled)** — a
**sixth** consecutive audit at the same count, nothing added, retired, disabled or repinned (fleet
`updated_at` moved to `2026-11-09T00:06:50+08:00`, read as a run-state stamp only).
- **Incidents:** unclosed rows **2 → 3** — a third silent `detected` row for the stocktake
(`1a8e90_a9cad99cb150`, `alerted_at: null`), all three still unreconciled. **The alerting path is not at
fault:** the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and closed `resolved` at
**08:06:54** (its admission-read `database is locked`, cleared when the run completed).
- ✅ **Verified resolution 1 — the Mnemosyne Health Monitor's no-agent conversion works.** The 18:00 slot
(agent mode) ran **12:00:51 → 18:00:42 = 5 h 59 m 51 s**; the converted job's first slot ran
**2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, silent, writing a **159 B** output file against
**105,982 B** for the last agent-mode run. Coverage live at **working 1030/1030 = 100.0 %**, **episodic
1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet. The script compares with `awk` because
**`bc` is not installed on this host**.
- ✅ **Verified resolution 2 — `state.db` re-measured and one corruption remnant removed.** Now
**1,890,656,256 B (1.89 GB)**, **2,901 sessions / 197,682 messages**, live **40,157,672 B** WAL.
`state.db.corrupted.20260811_151052` (**624.1 MB** + 32 KB `-shm` + 0 B `-wal`) **deleted** in-window;
`state.db.recovered.recovery.json` (14,566 B) remains and
`mnemosyne/data/mnemosyne.db.corrupted.20260809_120404` (**23.6 MB**) was **not** cleaned.
- **Pages updated:** `systems/conduit.md` (the pre-tool-call section, **total**), `systems/hermes-agent.md`
(hook budget + `state.db` size, **total**), `systems/scheduled-tasks.md` (new Count bullet, **total**),
`systems/mnemosyne.md` (health-monitor conversion verified, **total**), `current-state.md` (**total**),
`incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md` (`state.db` 1.7 → 1.89 GB in three
places, re `confident`), `incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md` (open items),
`index.md` (new entry + incident list + `state.db` figure, confidence dropped), and this log.
- **No wiki page removed or archived.** No system obsoleted; no decision reversed.
- **No secrets, keys, tokens or passwords written** — no credential value appears anywhere in this audit.
+143
View File
@@ -0,0 +1,143 @@
# Wiki freshness audit — 2026-11-09
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (127)
- [ 20d] CONTRIBUTING.md
- [ 1d] agents/hermes-chaos.md
- [ 1d] agents/hermes-production.md
- [ 20d] agents/index.md
- [ 11d] agents/nanobot.md
- [ 20d] agents/open-claw.md
- [ 5d] agents/qwenpaw.md
- [ 20d] agents/zeroclaw.md
- [ 11d] comparisons/camoufox-vs-chrome.md
- [ 20d] comparisons/hermes-vs-nanobot.md
- [ 20d] comparisons/memory-backends.md
- [ 20d] comparisons/model-routers.md
- [ 11d] comparisons/steel-vs-camoufox.md
- [ 20d] concepts/agent-memory.md
- [ 11d] concepts/browser-automation.md
- [ 20d] concepts/model-routing.md
- [ 20d] concepts/persistent-shells.md
- [ 20d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 11d] decisions/index.md
- [ 20d] decisions/keep-openrouter-as-fallback.md
- [ 20d] decisions/persistent-shell-enabled.md
- [ 11d] decisions/rollback-steel-to-camofox.md
- [ 20d] experiments/index.md
- [ 16d] incidents/2026-07-29-mnemosyne-relay.md
- [ 20d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 16d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 2d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 4d] incidents/2026-08-15-skills-index-drift.md
- [ 5d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 15d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 20d] incidents/2026-08-29-gmail-read-state.md
- [ 19d] incidents/2026-09-02-sftpgo-outage.md
- [ 5d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 11d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 14d] incidents/2026-10-03-ct450-hermes-update.md
- [ 14d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 11d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 16d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 8d] incidents/2026-10-09-memory-recall-relevance.md
- [ 3d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 13d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 21d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 13d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 18d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 15d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 14d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 14d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 14d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 12d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 10d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 10d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 10d] incidents/2026-10-29-conduit-voice-silence.md
- [ 9d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 10d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 9d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 8d] incidents/2026-10-31-asana-access-failures.md
- [ 8d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 8d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 7d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 6d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 5d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 3d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 3d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 4d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 0d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 4d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 2d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
- [ 1d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
- [ 0d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
- [ 0d] index.md
- [ 2d] infrastructure/backups.md
- [ 20d] infrastructure/credentials-map.md
- [ 2d] infrastructure/docker-services.md
- [ 20d] infrastructure/domains-and-tunnels.md
- [ 2d] infrastructure/hosts.md
- [ 2d] infrastructure/network-map.md
- [ 2d] infrastructure/proxmox.md
- [ 2d] infrastructure/tailscale.md
- [ 5d] queries/retained-research-answers.md
- [ 15d] reports/communities-wa-social-media-audit-2026.md
- [ 8d] reports/uber-eats/2026-07.md
- [ 8d] reports/uber-eats/2026-08.md
- [ 8d] reports/uber-eats/2026-09.md
- [ 8d] reports/uber-eats/2026-10.md
- [ 8d] reports/uber-eats/index.md
- [ 20d] runbooks/backup-wiki.md
- [ 20d] runbooks/diagnose-docker.md
- [ 20d] runbooks/gateway-resource-resilience.md
- [ 2d] runbooks/job-radar.md
- [ 20d] runbooks/multiplexer-setup.md
- [ 14d] runbooks/provider-health.md
- [ 11d] runbooks/recover-docker-service.md
- [ 20d] runbooks/restart-browser.md
- [ 20d] runbooks/restart-hermes.md
- [ 20d] runbooks/restore-openrouter.md
- [ 20d] runbooks/test-openrouter.md
- [ 20d] runbooks/troubleshoot-tool-loop.md
- [ 14d] runbooks/update-hermes-safely.md
- [ 1d] systems/browser-backend.md
- [ 20d] systems/cloudflare-tunnel.md
- [ 0d] systems/conduit.md
- [ 14d] systems/current-profile.md
- [ 20d] systems/guanaco.md
- [ 20d] systems/headless-server.md
- [ 0d] systems/hermes-agent.md
- [ 1d] systems/hermes-hq-push.md
- [ 20d] systems/hermes-ui.md
- [ 5d] systems/home-assistant.md
- [ 10d] systems/homepage-dashboard.md
- [ 5d] systems/ignis.md
- [ 14d] systems/memory-backend.md
- [ 7d] systems/messaging-integrations.md
- [ 0d] systems/mnemosyne.md
- [ 14d] systems/model-providers.md
- [ 5d] systems/nanobot.md
- [ 20d] systems/nextcloud.md
- [ 20d] systems/obsidian.md
- [ 8d] systems/omniroute.md
- [ 1d] systems/opencode-go.md
- [ 20d] systems/openrouter.md
- [ 5d] systems/paseo.md
- [ 21d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 20d] systems/terminal-backend.md
- [ 20d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 127 fresh, 3 no-date of 130 pages
+31 -3
View File
@@ -3,11 +3,11 @@ title: Conduit
type: system type: system
status: active status: active
created: 2026-10-24 created: 2026-10-24
updated: 2026-11-08 updated: 2026-11-09
verified_on: 2026-11-08 verified_on: 2026-11-09
review_after: 2026-12-08 review_after: 2026-12-08
confidence: high confidence: high
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl] tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl, hook, timeout, pre-tool-call]
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051] sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051]
--- ---
@@ -230,7 +230,35 @@ The plugin updated itself in-window and its own ledger records the final step as
- The plugin remained enabled and functional throughout; no user-facing conduit fault was recorded on - The plugin remained enabled and functional throughout; no user-facing conduit fault was recorded on
2026-11-07. 2026-11-07.
## 2026-11-08 13:2x–13:3x AWST — its `pre_tool_call` hook was jamming **every** tool call
The plugin is the **only** `pre_tool_call` registrant on the host (`conduit_push/__init__.py:119`), and for
**~18 hours** that callback overran Hermes's **30 s** hook budget and was skipped:
~/.hermes/logs/errors.log
2026-11-07 14:14:03,075 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
... (58 lines total)
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
- **58 timeout lines** in `errors.log` (**12** on 2026-11-07, **46** on 2026-11-08) plus **89**
*"skipped after previous timeout or while still running"* suppression lines across the logs. **The last
occurrence is 2026-11-08 08:21:03** — nothing since the fix below.
- **The plugin's caching was NOT the fault — this was measured, not assumed.** `session_titles.py:21
CACHE_SECONDS = 30.0`; timed in-process, a hook call costs **3.1 s cold and 0.00 s warm**. The
*"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**. The timeouts track **load
spikes** on the 2-core CT460 (`load average 5.89` at 13:27:50; `state.db` with a **40 MB WAL**
checkpointing on the same spindle).
- **Fix:** `plugins.hook_callback_timeout` **30 → 120** in `~/.hermes/config.yaml` (2026-11-08 13:35 AWST) —
verified live in config, and verified *effective* (no timeout logged after 08:21:03).
- **Cost of the fault:** *Mnemosyne Health Monitor* ran **8 h 27 m** and **5 h 59 m 51 s** on its two
2026-11-08 slots; *wiki-stocktake-maintenance* failed at 04:36:08 on *"No reply: the turn stopped while a
tool result was still pending"*; *FreshRSS Midday* took **2 h 09 m 53 s**.
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
## Open items ## Open items
- **Hook budget is 120 s, not 30 s — watch it.** A second overrun would eat two minutes per tool call.
The binding constraint is `state.db` (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin.
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
- ✅ **Resolved 2026-11-01** — the `plugin_api.py` timeout patch is no longer needed: the tree is clean - ✅ **Resolved 2026-11-01** — the `plugin_api.py` timeout patch is no longer needed: the tree is clean
at `0.122.2` and `MEMORY_TIMEOUT_S` ships upstream in `dashboard/api/memory.py`. at `0.122.2` and `MEMORY_TIMEOUT_S` ships upstream in `dashboard/api/memory.py`.
- **Verify a call end-to-end from the phone.** The 2026-10-29 08:49:36 call has no recorded outcome, - **Verify a call end-to-end from the phone.** The 2026-10-29 08:49:36 call has no recorded outcome,
+5 -3
View File
@@ -3,11 +3,11 @@ title: Hermes Agent
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-08 updated: 2026-11-09
verified_on: 2026-11-08 verified_on: 2026-11-09
confidence: high confidence: high
tags: [system, hermes, agent, llm] tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory] sources: [live-hermes-version, live-config, live-skill-inventory, live-errors-log, live-state-db]
--- ---
# Hermes Agent # Hermes Agent
@@ -17,6 +17,8 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Current Status ## Current Status
- **Version:** ⚠ **`0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s): `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`** (upstream commit dated 2026-11-06 03:57:38 AWST). ⚠ **The `snapshot` stage was skipped again** — `disabled by updates.pre_update_backup (mode: off)`, so there is **still no rollback point** ([[infrastructure/backups]]). Restarted: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`). The gateway was then restarted **again 11:11:44 → 11:11:55** (live process `pid 3109529`, seen logging 11:12:30 and 12:01:32). **Side effect of the same night:** two cron runs lost their fire claims in the restart drain — [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]. Previous: `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`. - **Version:** ⚠ **`0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s): `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`** (upstream commit dated 2026-11-06 03:57:38 AWST). ⚠ **The `snapshot` stage was skipped again** — `disabled by updates.pre_update_backup (mode: off)`, so there is **still no rollback point** ([[infrastructure/backups]]). Restarted: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`). The gateway was then restarted **again 11:11:44 → 11:11:55** (live process `pid 3109529`, seen logging 11:12:30 and 12:01:32). **Side effect of the same night:** two cron runs lost their fire claims in the restart drain — [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]. Previous: `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`.
- **Hook budget raised: `plugins.hook_callback_timeout: 120`** (built-in default **30 s**), set **2026-11-08 13:3x AWST** in `~/.hermes/config.yaml`. Reason: the `conduit_push` plugin's `pre_tool_call` callback — the **only** `pre_tool_call` registrant on the host — overran the 30 s budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`~/.hermes/logs/errors.log`: 12 lines on 11-07, 46 on 11-08; plus **89** *"skipped after previous timeout or while still running"* suppression lines), which degraded every tool call to slow-then-suppressed. Verified live in config, and verified effective — **no timeout line after 08:21:03**. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
- ⚠ **Session store: `~/.hermes/state.db` is 1.89 GB (1,890,656,256 B) with 2,901 sessions / 197,682 messages and a live 40,157,672 B WAL** (measured 2026-11-09 00:0x AWST). It grew **1.7 GB → 1.89 GB** since 2026-11-06; the 2-core CT460 was at load average 5.89 while it checkpointed. The `state.db.corrupted.20260811_151052` artifact (**624.1 MB**, plus a 32 KB `-shm` and a 0-byte `-wal` sibling) was **deleted** in the 2026-11-08 window; `state.db.recovered.recovery.json` (14,566 B, 2026-08-11) remains. **No retention or checkpoint policy is set** — see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
- **Install method:** git - **Install method:** git
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15) - **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
- **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on. - **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on.
+13 -4
View File
@@ -3,12 +3,12 @@ title: Mnemosyne
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-06 updated: 2026-11-09
verified_on: 2026-11-06 verified_on: 2026-11-09
review_after: 2026-11-13 review_after: 2026-11-13
confidence: high confidence: high
tags: [memory, mnemosyne, backend] tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db] sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db, live-health-monitor-log]
--- ---
# Mnemosyne # Mnemosyne
@@ -98,7 +98,16 @@ Primary agent memory store for Hermes.
## Known Management Jobs ## Known Management Jobs
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent - Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent - Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception. - Mnemosyne Health Monitor every 12h — **script**: `mnemosyne-health-monitor.sh`, mode: **no-agent** since
the 2026-11-08 conversion (job `ea79a2107e36`), reports only on exception.
⚠ **The no-agent conversion was verified end-to-end on 2026-11-08:** the 18:00 slot (agent mode) ran
**12:00:51 → 18:00:42 = 5 h 59 m 51 s**, and the converted job's first slot ran **2026-11-09 00:00:48 →
00:02:02 = 73 s**, exit 0, **silent** — `output/ea79a2107e36/2026-11-09_00-02-00.md` is 159 bytes against
105,982 B for the last agent-mode file. Live read at 2026-11-09 00:0x: **working 1030/1030 = 100.0 %,
episodic 1192/1192 = 100.0 %** (so the `< 95 %` alert is correctly quiet), 1043 embeddings generated in
24 h. The script compares with `awk`, not `bc` — **`bc` is not installed on this host**
(`mnemosyne-health-monitor.sh:47`), which is why the old monitor's arithmetic silently defaulted instead
of failing loudly.
⚠ **2026-10-30: the monitor was twice flagged and neither flag survived contact with `finished_at`.** Its ⚠ **2026-10-30: the monitor was twice flagged and neither flag survived contact with `finished_at`.** Its
00:01:08 slot was auto-paused at 01:04:22 on an *"execution unverified / gateway connection failed"* admission 00:01:08 slot was auto-paused at 01:04:22 on an *"execution unverified / gateway connection failed"* admission
([[incidents/2026-10-30-mnemosyne-health-monitor-paused]]) — but the slot was **still running** and closed ([[incidents/2026-10-30-mnemosyne-health-monitor-paused]]) — but the slot was **still running** and closed
+3 -2
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-08 updated: 2026-11-09
verified_on: 2026-11-08 verified_on: 2026-11-09
review_after: 2026-11-09 review_after: 2026-11-09
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
@@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **Current as of 2026-11-08 (window 2026-11-07; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-09 00:0x-00:1x AWST):** **29 jobs (26 enabled / 3 disabled** — `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; the count and split are unchanged across **six** consecutive audits, but fleet `updated_at` **moved** to **2026-11-09T00:06:50+08:00** — treat that as a run-state stamp, **not** a definition change, because no job was added, retired, disabled or repinned: re-verified every field on the nine jobs looked at and the enabled/disabled split is identical). **11-08 settled: 169 rows across 16 jobs — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33). The **only** failure is **`1a8e90ea2271` (wiki-stocktake-maintenance), 03:01:11 → 04:36:08 (1 h 34 m 57 s)**, and it is a **new terminal signature for that job**: `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` — the previous two red days both read *"lost its durable fire claim ownership"*. **`failure_streak` is now 3** — **three consecutive failed days**. *Working hypothesis, not proven:* the `conduit_push` `pre_tool_call` hook that was **timing out at 30 s 58 times between 2026-11-07 14:14:03 and 2026-11-08 08:21:03** holds a tool result open in exactly the way this error describes, but the string does not name the hook → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. The hole in the day was **the hook, not the schedule**: *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)**, *FreshRSS Midday Briefing* `ecc1fc13655a` **12:00:50 → 14:10:43 (2 h 09 m 53 s)**, both delivered. Deliveries **8 rows / 8 `delivered` / 0 `for_failure`** (11-07 was 7 / 7). **Unclosed incident rows 2 → 3** — a **third** silent `detected` row appeared (`1a8e90_a9cad99cb150`, `alerted_at: null`), and the two 11-07 stocktake rows are still `detected`; **the alerting path is not the fault** (the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window — its `database is locked` admission, closed when the run completed). Nothing added, retired, disabled or repinned.
- **Current as of 2026-11-07 (window 2026-11-06; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-07 00:3x-00:5x AWST):** **29 jobs (26 enabled / 3 disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; count and split unchanged across five consecutive audits, fleet `updated_at` **2026-11-07T00:39:46+08:00**). **11-06 settled: 172 rows, 16 distinct jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:31). The single `failed` row is a **new, non-restart failure signature**: the Job Radar's **11:30 Stage-2 slot** (`95347fa5…`) was **discarded as stale** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job, started 08:27:09, was still holding the claim** - that fire completed **13:26:22** and delivered, and the ledger records **no gateway restart** in the window -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; the radar still landed the day in full at 14:30 (`MISSING_COVER_LETTER_COUNT 0`, 4 same-day strong rows, 4 cover letters, 20 non-empty artifacts, *Run status: COMPLETE*). Fleet non-clean `last_status` unchanged at **1** (`066bb5191280`, `delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0** - no ledger row has `last_seen_at` on or after 2026-11-06 (newest activity remains the 2026-11-05T12:36:17 closures), so **the discarded run raised no incident row and no alert**; [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] remains the one open page. The stocktake's overrun streak **fully unwound**: `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Nothing added, retired, disabled or repinned. - **Current as of 2026-11-07 (window 2026-11-06; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-07 00:3x-00:5x AWST):** **29 jobs (26 enabled / 3 disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; count and split unchanged across five consecutive audits, fleet `updated_at` **2026-11-07T00:39:46+08:00**). **11-06 settled: 172 rows, 16 distinct jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:31). The single `failed` row is a **new, non-restart failure signature**: the Job Radar's **11:30 Stage-2 slot** (`95347fa5…`) was **discarded as stale** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job, started 08:27:09, was still holding the claim** - that fire completed **13:26:22** and delivered, and the ledger records **no gateway restart** in the window -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; the radar still landed the day in full at 14:30 (`MISSING_COVER_LETTER_COUNT 0`, 4 same-day strong rows, 4 cover letters, 20 non-empty artifacts, *Run status: COMPLETE*). Fleet non-clean `last_status` unchanged at **1** (`066bb5191280`, `delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0** - no ledger row has `last_seen_at` on or after 2026-11-06 (newest activity remains the 2026-11-05T12:36:17 closures), so **the discarded run raised no incident row and no alert**; [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] remains the one open page. The stocktake's overrun streak **fully unwound**: `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Nothing added, retired, disabled or repinned.
- **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. - **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. ⚠ **Re-verified - **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. ⚠ **Re-verified
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-08 updated: 2026-11-09
verified_on: 2026-11-08 verified_on: 2026-11-09
confidence: high confidence: high
tags: [skills, hermes] tags: [skills, hermes]
sources: [] sources: []