stocktake(2026-11-15): skills-index regenerated (299/67); storm incident re-verified (all 12 rows resolved, evidence-window note); conduit memory-timeout fix LOST in plugin 0.168.x restructure — page → degraded with regression section; 0 stale after fixes

This commit is contained in:
Tony0410
2026-11-15 03:25:18 +08:00
parent 646be98c31
commit 402d8546e8
6 changed files with 233 additions and 23 deletions
@@ -3,9 +3,9 @@ title: Fleet-Wide Provider Timeout Storm — 2026-10-09
type: incident type: incident
status: resolved status: resolved
created: 2026-10-09 created: 2026-10-09
updated: 2026-10-24 updated: 2026-11-15
verified_on: 2026-10-24 verified_on: 2026-11-15
review_after: 2026-11-07 review_after: 2026-12-15
confidence: high confidence: high
tags: [cron, omniroute, router, timeout, incident, resolved] tags: [cron, omniroute, router, timeout, incident, resolved]
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1] sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
@@ -18,6 +18,15 @@ sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-2026100
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job) (`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again. closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
The storm contributes **no** unclosed rows. Nothing here is still symptomatic. The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
> **Re-verified 2026-11-15** (3 a.m. stocktake): all 12 of the storm's `cron_incidents` rows still read
> `resolved` — none reopened. `1b23c042e17d` (the weekly Friday job, last hold-out) is healthy —
> `last_status: ok`, last run 2026-11-13 08:11 AWST. One incident is currently open fleet-wide
> (`ea79a2_601037516abd`, Mnemosyne Health Monitor, first seen 2026-11-15 00:03) but it is the
> capture-stall detection firing → [[incidents/2026-11-14-mnemosyne-capture-stall]], unrelated to
> this storm. ⚠ Evidence window: `executions.db` retention now reaches back only to 2026-11-09, so
> the 2026-10-09 execution counts below are **no longer re-derivable from the DB** — they stand on
> the 2026-10-24 read (the incident rows themselves persist, which is what this check re-read).
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08** **Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
## What happened ## What happened
@@ -1,11 +1,11 @@
--- ---
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout) title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
type: incident type: incident
status: resolved status: degraded
created: 2026-10-24 created: 2026-10-24
updated: 2026-10-25 updated: 2026-11-15
verified_on: 2026-10-25 verified_on: 2026-11-15
review_after: 2026-11-08 review_after: 2026-12-15
confidence: high confidence: high
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved] tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff] sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
@@ -14,7 +14,8 @@ sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23 # Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences) **Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
**Status:** Resolved 2026-10-23 18:54 (patched and verified) **Status:** Resolved 2026-10-23 18:54 (patched and verified) — **but the fix was lost in the
plugin's 0.168.x restructure; `status: degraded` since 2026-11-15** (see the regression section below)
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its **Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
memory layer. memory layer.
@@ -87,17 +88,39 @@ touching the file.
not positive proof that recall was populated on those calls. The proven part is that the call not positive proof that recall was populated on those calls. The proven part is that the call
is no longer abandoned by the memory layer. is no longer abandoned by the memory layer.
## ⚠ Regression — the fix was lost in the plugin restructure (verified 2026-11-15)
The follow-up below warned that a plugin update would revert the patch. **It happened.** Live
state at the 2026-11-15 03:00 stocktake:
- The plugin tree is **clean at upstream `0.168.2`** (`296c94b`, committed 2026-11-12 US /
2026-11-13 AWST) — the patch is gone entirely, not just uncommitted.
- The memory layer **moved**: `dashboard/plugin_api.py` (now a doc index) →
`dashboard/api/memory.py`. There, `MEMORY_TIMEOUT_S = 10.0` is hard-coded again
(`memory.py:48`), with **no** `CONDUIT_MEMORY_TIMEOUT_S` env override, and the "abandoning the
call" behaviour is intact (`memory.py:256`). The restructure landed in `memory.py` on
**2026-10-29 AWST**, so the 60 s budget has been gone for ~2.5 weeks.
- The running dashboard (started 2026-11-10) loads the restructured code, so the exposure is live.
- One real call has since occurred — 2026-11-14 08:57 AWST, the first entry in
`conduit-calls.json → history` — with no abort lines in the logs afterwards. But that call met a
warm provider; it does not exercise the cold-start path. The page's original caveat stands.
- **Not re-applied by the stocktake**: porting the fix to the restructured code is a code change to
a git-managed upstream plugin that the next update will revert again — it needs Anthony's call
(re-port + commit locally, or upstream it).
The original incident window (2026-10-22 → 23) remains resolved; `status: degraded` records that
the *protection* is gone, not that the October fault returned.
## Follow-ups ## Follow-ups
- ⚠ **The fix is still unexercised by a call** (see the correction above): `conduit-calls.json → - ~~⚠ **The fix is still unexercised by a call** …~~ **Superseded 2026-11-15**: one call has since
history` is empty at 2026-10-25 00:2x, and the plugin is now **five releases behind HEAD** occurred (2026-11-14 08:57 AWST) — but the fix itself was lost in the restructure (section above),
(`0.64.0` → `0.69.0`). so "unexercised" is now moot.
- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which - **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which
asserted the capability did not exist **without calling the tool** → asserted the capability did not exist **without calling the tool** →
[[incidents/2026-10-24-conduit-call-request-refused]]. [[incidents/2026-10-24-conduit-call-request-refused]].
- **The patch is uncommitted.** A plugin update that resets - ~~**The patch is uncommitted.** A plugin update that resets `~/.hermes/plugins/conduit_push`
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream reverts it to the 10 s budget.~~ **Confirmed 2026-11-15** — see the regression section above.
(`kaishi00/hermes-conduit-notifier`).
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised. - The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]] Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
+2 -2
View File
@@ -3,8 +3,8 @@ title: Anthony's Systems Wiki
type: collection type: collection
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-14 updated: 2026-11-15
verified_on: 2026-11-14 verified_on: 2026-11-15
confidence: high confidence: high
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout] tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
+31 -1
View File
@@ -1751,4 +1751,34 @@ write path, and nothing watches it.** Candidate fix: an N-consecutive-zero-item-
14 `script` / 5 `auth`. The job is `no_agent`, so `agent` is impossible — read `auth` as unexplained, not as a 14 `script` / 5 `auth`. The job is `no_agent`, so `agent` is impossible — read `auth` as unexplained, not as a
credential fault. Pages corrected: [[incidents/2026-11-14-mnemosyne-capture-stall]] (false section rewritten, `auth` bullet upgraded credential fault. Pages corrected: [[incidents/2026-11-14-mnemosyne-capture-stall]] (false section rewritten, `auth` bullet upgraded
from suspicion to evidence), [[systems/scheduled-tasks]], [[systems/mnemosyne]], [[index.md]]. The incident itself from suspicion to evidence), [[systems/scheduled-tasks]], [[systems/mnemosyne]], [[index.md]]. The incident itself
remains **OPEN / `alerted` / unclosed**; the *cause* of the stall is still **not established**. remains **OPEN / `alerted` / unclosed**; the *cause* of the stall is still **not established**.
## 2026-11-15 03:00 — stocktake (wiki-maintain.sh + drift re-verification)
Script run: skills-index regenerated (299 skills / 67 categories), freshness report
`raw/stocktake/freshness-20261115.txt` — **STALE (1)**:
`incidents/2026-10-09-cron-provider-timeout-storm.md` (22d, updated 2026-10-24). Pages ON the
threshold (21d): `incidents/2026-10-22-conduit-memory-start-timeout.md`,
`reports/communities-wa-social-media-audit-2026.md` (static deliverable, no live claims — left
unbumped). Pages at 20d (not re-probed this run, next candidates):
`runbooks/provider-health.md`, `runbooks/update-hermes-safely.md`,
`systems/current-profile.md`, `systems/memory-backend.md`, `systems/model-providers.md`,
`incidents/2026-10-03-ct450-hermes-update.md`,
`incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md`,
`incidents/2026-10-24-conduit-call-request-refused.md`,
`incidents/2026-10-25-gmail-digest-runaway-and-refire.md`.
**Re-verified live (storm page, 22d):** all 12 `cron_incidents` rows still `resolved`; weekly job
`1b23c042e17d` `last_status: ok` (2026-11-13). The one open fleet-wide incident
(`ea79a2_601037516abd`, first seen 2026-11-15 00:03) is the capture-stall monitor firing, not a
storm row. `executions.db` retention now starts 2026-11-09 — the page's 2026-10-09 execution
counts are no longer DB-re-derivable; noted on the page. Bumped with a dated verification note.
**Regression found (conduit page, 21d):** the Conduit voice memory-timeout fix was **lost in the
plugin's restructure**. `~/.hermes/plugins/conduit_push` is clean at upstream `0.168.2`; the
memory layer moved to `dashboard/api/memory.py` where `MEMORY_TIMEOUT_S = 10.0` is hard-coded
again (no `CONDUIT_MEMORY_TIMEOUT_S` env hook; restructure landed ~2026-10-29 AWST). Running
dashboard (since 2026-11-10) carries it. One real call since (2026-11-14 08:57 AWST) hit a warm
provider, so the cold-start path is untested. Page → `status: degraded` with a dated regression
section; **fix not re-applied** (needs Anthony's call: re-port + commit locally, or upstream).
Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]],
[[incidents/2026-10-22-conduit-memory-start-timeout]], [[index.md]].
+149
View File
@@ -0,0 +1,149 @@
# Wiki freshness audit — 2026-11-15
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (133)
- [ 4d] CONTRIBUTING.md
- [ 7d] agents/hermes-chaos.md
- [ 7d] agents/hermes-production.md
- [ 4d] agents/index.md
- [ 17d] agents/nanobot.md
- [ 4d] agents/open-claw.md
- [ 11d] agents/qwenpaw.md
- [ 4d] agents/zeroclaw.md
- [ 17d] comparisons/camoufox-vs-chrome.md
- [ 4d] comparisons/hermes-vs-nanobot.md
- [ 4d] comparisons/memory-backends.md
- [ 4d] comparisons/model-routers.md
- [ 17d] comparisons/steel-vs-camoufox.md
- [ 4d] concepts/agent-memory.md
- [ 17d] concepts/browser-automation.md
- [ 4d] concepts/model-routing.md
- [ 4d] concepts/persistent-shells.md
- [ 4d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 17d] decisions/index.md
- [ 4d] decisions/keep-openrouter-as-fallback.md
- [ 4d] decisions/persistent-shell-enabled.md
- [ 17d] decisions/rollback-steel-to-camofox.md
- [ 4d] experiments/index.md
- [ 3d] incidents/2026-07-29-mnemosyne-relay.md
- [ 4d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 3d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 8d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 10d] incidents/2026-08-15-skills-index-drift.md
- [ 11d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 3d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 4d] incidents/2026-08-29-gmail-read-state.md
- [ 3d] incidents/2026-09-02-sftpgo-outage.md
- [ 11d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 17d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 20d] incidents/2026-10-03-ct450-hermes-update.md
- [ 20d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 17d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 0d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 14d] incidents/2026-10-09-memory-recall-relevance.md
- [ 9d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 19d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 5d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 19d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 3d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 0d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 4d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 20d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 20d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 18d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 16d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 16d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 16d] incidents/2026-10-29-conduit-voice-silence.md
- [ 15d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 16d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 15d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 14d] incidents/2026-10-31-asana-access-failures.md
- [ 14d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 14d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 13d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 12d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 11d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 9d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 9d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 10d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 6d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 10d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 8d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
- [ 6d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
- [ 1d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
- [ 4d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
- [ 4d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
- [ 3d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
- [ 2d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
- [ 0d] index.md
- [ 8d] infrastructure/backups.md
- [ 4d] infrastructure/credentials-map.md
- [ 8d] infrastructure/docker-services.md
- [ 4d] infrastructure/domains-and-tunnels.md
- [ 8d] infrastructure/hosts.md
- [ 8d] infrastructure/network-map.md
- [ 8d] infrastructure/proxmox.md
- [ 3d] infrastructure/tailscale.md
- [ 11d] queries/retained-research-answers.md
- [ 21d] reports/communities-wa-social-media-audit-2026.md
- [ 14d] reports/uber-eats/2026-07.md
- [ 14d] reports/uber-eats/2026-08.md
- [ 14d] reports/uber-eats/2026-09.md
- [ 2d] reports/uber-eats/2026-10.md
- [ 2d] reports/uber-eats/index.md
- [ 4d] runbooks/backup-wiki.md
- [ 4d] runbooks/diagnose-docker.md
- [ 4d] runbooks/gateway-resource-resilience.md
- [ 8d] runbooks/job-radar.md
- [ 4d] runbooks/multiplexer-setup.md
- [ 20d] runbooks/provider-health.md
- [ 17d] runbooks/recover-docker-service.md
- [ 4d] runbooks/restart-browser.md
- [ 4d] runbooks/restart-hermes.md
- [ 4d] runbooks/restore-openrouter.md
- [ 4d] runbooks/test-openrouter.md
- [ 4d] runbooks/troubleshoot-tool-loop.md
- [ 20d] runbooks/update-hermes-safely.md
- [ 7d] systems/browser-backend.md
- [ 4d] systems/cloudflare-tunnel.md
- [ 6d] systems/conduit.md
- [ 20d] systems/current-profile.md
- [ 4d] systems/guanaco.md
- [ 4d] systems/headless-server.md
- [ 6d] systems/hermes-agent.md
- [ 2d] systems/hermes-hq-push.md
- [ 4d] systems/hermes-ui.md
- [ 11d] systems/home-assistant.md
- [ 16d] systems/homepage-dashboard.md
- [ 11d] systems/ignis.md
- [ 20d] systems/memory-backend.md
- [ 13d] systems/messaging-integrations.md
- [ 0d] systems/mnemosyne.md
- [ 20d] systems/model-providers.md
- [ 11d] systems/nanobot.md
- [ 4d] systems/nextcloud.md
- [ 4d] systems/obsidian.md
- [ 5d] systems/omniroute.md
- [ 4d] systems/opencode-go.md
- [ 4d] systems/openrouter.md
- [ 11d] systems/paseo.md
- [ 0d] systems/phonon-tts.md
- [ 5d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 4d] systems/terminal-backend.md
- [ 4d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 133 fresh, 3 no-date of 136 pages
+5 -6
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-14 updated: 2026-11-15
verified_on: 2026-11-14 verified_on: 2026-11-15
confidence: high confidence: high
tags: [skills, hermes] tags: [skills, hermes]
sources: [] sources: []
@@ -12,9 +12,9 @@ sources: []
# Installed Skills # Installed Skills
**Total installed: 300 skills** in `~/.hermes/skills/`. **Total installed: 299 skills** in `~/.hermes/skills/`.
- **50 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them. - **51 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory. - The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
This page is auto-regenerated from the live skill inventory. Do not hand-edit. This page is auto-regenerated from the live skill inventory. Do not hand-edit.
@@ -98,7 +98,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `ddia-systems` — # Designing Data-Intensive Applications Framework - `ddia-systems` — # Designing Data-Intensive Applications Framework
## Devops (108) ## Devops (107)
- `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS) - `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS)
- `agent-extension-audit` — # Agent Extension Audit - `agent-extension-audit` — # Agent Extension Audit
@@ -204,7 +204,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `uber-eats-monitor` — # Uber Eats Order Monitor - `uber-eats-monitor` — # Uber Eats Order Monitor
- `vision-model-validation` — # Vision Model Validation via OmniRoute - `vision-model-validation` — # Vision Model Validation via OmniRoute
- `watchers` — # Watchers - `watchers` — # Watchers
- `web-content-extraction` — # Web Content Extraction — Fallback Techniques
- `weekly-comms-tracker` — # Weekly Communications Tracker - `weekly-comms-tracker` — # Weekly Communications Tracker
- `wiki-maintenance` — # Wiki Maintenance - `wiki-maintenance` — # Wiki Maintenance
- `zeroclaw-ops` — # ZeroClaw Operations & Configuration - `zeroclaw-ops` — # ZeroClaw Operations & Configuration