stocktake(2026-11-15): skills-index regenerated (299/67); storm incident re-verified (all 12 rows resolved, evidence-window note); conduit memory-timeout fix LOST in plugin 0.168.x restructure — page → degraded with regression section; 0 stale after fixes

This commit is contained in:
Tony0410
2026-11-15 03:25:18 +08:00
parent 646be98c31
commit 402d8546e8
6 changed files with 233 additions and 23 deletions
@@ -3,9 +3,9 @@ title: Fleet-Wide Provider Timeout Storm — 2026-10-09
type: incident
status: resolved
created: 2026-10-09
updated: 2026-10-24
verified_on: 2026-10-24
review_after: 2026-11-07
updated: 2026-11-15
verified_on: 2026-11-15
review_after: 2026-12-15
confidence: high
tags: [cron, omniroute, router, timeout, incident, resolved]
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
@@ -18,6 +18,15 @@ sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-2026100
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
> **Re-verified 2026-11-15** (3 a.m. stocktake): all 12 of the storm's `cron_incidents` rows still read
> `resolved` — none reopened. `1b23c042e17d` (the weekly Friday job, last hold-out) is healthy —
> `last_status: ok`, last run 2026-11-13 08:11 AWST. One incident is currently open fleet-wide
> (`ea79a2_601037516abd`, Mnemosyne Health Monitor, first seen 2026-11-15 00:03) but it is the
> capture-stall detection firing → [[incidents/2026-11-14-mnemosyne-capture-stall]], unrelated to
> this storm. ⚠ Evidence window: `executions.db` retention now reaches back only to 2026-11-09, so
> the 2026-10-09 execution counts below are **no longer re-derivable from the DB** — they stand on
> the 2026-10-24 read (the incident rows themselves persist, which is what this check re-read).
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
## What happened
@@ -1,11 +1,11 @@
---
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
type: incident
status: resolved
status: degraded
created: 2026-10-24
updated: 2026-10-25
verified_on: 2026-10-25
review_after: 2026-11-08
updated: 2026-11-15
verified_on: 2026-11-15
review_after: 2026-12-15
confidence: high
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
@@ -14,7 +14,8 @@ sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
**Status:** Resolved 2026-10-23 18:54 (patched and verified)
**Status:** Resolved 2026-10-23 18:54 (patched and verified) — **but the fix was lost in the
plugin's 0.168.x restructure; `status: degraded` since 2026-11-15** (see the regression section below)
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
memory layer.
@@ -87,17 +88,39 @@ touching the file.
not positive proof that recall was populated on those calls. The proven part is that the call
is no longer abandoned by the memory layer.
## ⚠ Regression — the fix was lost in the plugin restructure (verified 2026-11-15)
The follow-up below warned that a plugin update would revert the patch. **It happened.** Live
state at the 2026-11-15 03:00 stocktake:
- The plugin tree is **clean at upstream `0.168.2`** (`296c94b`, committed 2026-11-12 US /
2026-11-13 AWST) — the patch is gone entirely, not just uncommitted.
- The memory layer **moved**: `dashboard/plugin_api.py` (now a doc index) →
`dashboard/api/memory.py`. There, `MEMORY_TIMEOUT_S = 10.0` is hard-coded again
(`memory.py:48`), with **no** `CONDUIT_MEMORY_TIMEOUT_S` env override, and the "abandoning the
call" behaviour is intact (`memory.py:256`). The restructure landed in `memory.py` on
**2026-10-29 AWST**, so the 60 s budget has been gone for ~2.5 weeks.
- The running dashboard (started 2026-11-10) loads the restructured code, so the exposure is live.
- One real call has since occurred — 2026-11-14 08:57 AWST, the first entry in
`conduit-calls.json → history` — with no abort lines in the logs afterwards. But that call met a
warm provider; it does not exercise the cold-start path. The page's original caveat stands.
- **Not re-applied by the stocktake**: porting the fix to the restructured code is a code change to
a git-managed upstream plugin that the next update will revert again — it needs Anthony's call
(re-port + commit locally, or upstream it).
The original incident window (2026-10-22 → 23) remains resolved; `status: degraded` records that
the *protection* is gone, not that the October fault returned.
## Follow-ups
- ⚠ **The fix is still unexercised by a call** (see the correction above): `conduit-calls.json →
history` is empty at 2026-10-25 00:2x, and the plugin is now **five releases behind HEAD**
(`0.64.0` → `0.69.0`).
- ~~⚠ **The fix is still unexercised by a call** …~~ **Superseded 2026-11-15**: one call has since
occurred (2026-11-14 08:57 AWST) — but the fix itself was lost in the restructure (section above),
so "unexercised" is now moot.
- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which
asserted the capability did not exist **without calling the tool** →
[[incidents/2026-10-24-conduit-call-request-refused]].
- **The patch is uncommitted.** A plugin update that resets
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream
(`kaishi00/hermes-conduit-notifier`).
- ~~**The patch is uncommitted.** A plugin update that resets `~/.hermes/plugins/conduit_push`
reverts it to the 10 s budget.~~ **Confirmed 2026-11-15** — see the regression section above.
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
+2 -2
View File
@@ -3,8 +3,8 @@ title: Anthony's Systems Wiki
type: collection
status: active
created: 2026-07-22
updated: 2026-11-14
verified_on: 2026-11-14
updated: 2026-11-15
verified_on: 2026-11-15
confidence: high
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
+31 -1
View File
@@ -1751,4 +1751,34 @@ write path, and nothing watches it.** Candidate fix: an N-consecutive-zero-item-
14 `script` / 5 `auth`. The job is `no_agent`, so `agent` is impossible — read `auth` as unexplained, not as a
credential fault. Pages corrected: [[incidents/2026-11-14-mnemosyne-capture-stall]] (false section rewritten, `auth` bullet upgraded
from suspicion to evidence), [[systems/scheduled-tasks]], [[systems/mnemosyne]], [[index.md]]. The incident itself
remains **OPEN / `alerted` / unclosed**; the *cause* of the stall is still **not established**.
remains **OPEN / `alerted` / unclosed**; the *cause* of the stall is still **not established**.
## 2026-11-15 03:00 — stocktake (wiki-maintain.sh + drift re-verification)
Script run: skills-index regenerated (299 skills / 67 categories), freshness report
`raw/stocktake/freshness-20261115.txt` — **STALE (1)**:
`incidents/2026-10-09-cron-provider-timeout-storm.md` (22d, updated 2026-10-24). Pages ON the
threshold (21d): `incidents/2026-10-22-conduit-memory-start-timeout.md`,
`reports/communities-wa-social-media-audit-2026.md` (static deliverable, no live claims — left
unbumped). Pages at 20d (not re-probed this run, next candidates):
`runbooks/provider-health.md`, `runbooks/update-hermes-safely.md`,
`systems/current-profile.md`, `systems/memory-backend.md`, `systems/model-providers.md`,
`incidents/2026-10-03-ct450-hermes-update.md`,
`incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md`,
`incidents/2026-10-24-conduit-call-request-refused.md`,
`incidents/2026-10-25-gmail-digest-runaway-and-refire.md`.
**Re-verified live (storm page, 22d):** all 12 `cron_incidents` rows still `resolved`; weekly job
`1b23c042e17d` `last_status: ok` (2026-11-13). The one open fleet-wide incident
(`ea79a2_601037516abd`, first seen 2026-11-15 00:03) is the capture-stall monitor firing, not a
storm row. `executions.db` retention now starts 2026-11-09 — the page's 2026-10-09 execution
counts are no longer DB-re-derivable; noted on the page. Bumped with a dated verification note.
**Regression found (conduit page, 21d):** the Conduit voice memory-timeout fix was **lost in the
plugin's restructure**. `~/.hermes/plugins/conduit_push` is clean at upstream `0.168.2`; the
memory layer moved to `dashboard/api/memory.py` where `MEMORY_TIMEOUT_S = 10.0` is hard-coded
again (no `CONDUIT_MEMORY_TIMEOUT_S` env hook; restructure landed ~2026-10-29 AWST). Running
dashboard (since 2026-11-10) carries it. One real call since (2026-11-14 08:57 AWST) hit a warm
provider, so the cold-start path is untested. Page → `status: degraded` with a dated regression
section; **fix not re-applied** (needs Anthony's call: re-port + commit locally, or upstream).
Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]],
[[incidents/2026-10-22-conduit-memory-start-timeout]], [[index.md]].
+149
View File
@@ -0,0 +1,149 @@
# Wiki freshness audit — 2026-11-15
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (133)
- [ 4d] CONTRIBUTING.md
- [ 7d] agents/hermes-chaos.md
- [ 7d] agents/hermes-production.md
- [ 4d] agents/index.md
- [ 17d] agents/nanobot.md
- [ 4d] agents/open-claw.md
- [ 11d] agents/qwenpaw.md
- [ 4d] agents/zeroclaw.md
- [ 17d] comparisons/camoufox-vs-chrome.md
- [ 4d] comparisons/hermes-vs-nanobot.md
- [ 4d] comparisons/memory-backends.md
- [ 4d] comparisons/model-routers.md
- [ 17d] comparisons/steel-vs-camoufox.md
- [ 4d] concepts/agent-memory.md
- [ 17d] concepts/browser-automation.md
- [ 4d] concepts/model-routing.md
- [ 4d] concepts/persistent-shells.md
- [ 4d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 17d] decisions/index.md
- [ 4d] decisions/keep-openrouter-as-fallback.md
- [ 4d] decisions/persistent-shell-enabled.md
- [ 17d] decisions/rollback-steel-to-camofox.md
- [ 4d] experiments/index.md
- [ 3d] incidents/2026-07-29-mnemosyne-relay.md
- [ 4d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 3d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 8d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 10d] incidents/2026-08-15-skills-index-drift.md
- [ 11d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 3d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 4d] incidents/2026-08-29-gmail-read-state.md
- [ 3d] incidents/2026-09-02-sftpgo-outage.md
- [ 11d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 17d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 20d] incidents/2026-10-03-ct450-hermes-update.md
- [ 20d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 17d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 0d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 14d] incidents/2026-10-09-memory-recall-relevance.md
- [ 9d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 19d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 5d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 19d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 3d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 0d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 4d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 20d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 20d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 18d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 16d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 16d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 16d] incidents/2026-10-29-conduit-voice-silence.md
- [ 15d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 16d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 15d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 14d] incidents/2026-10-31-asana-access-failures.md
- [ 14d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 14d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 13d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 12d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 11d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 9d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 9d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 10d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 6d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 10d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 8d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
- [ 6d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
- [ 1d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
- [ 4d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
- [ 4d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
- [ 3d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
- [ 2d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
- [ 0d] index.md
- [ 8d] infrastructure/backups.md
- [ 4d] infrastructure/credentials-map.md
- [ 8d] infrastructure/docker-services.md
- [ 4d] infrastructure/domains-and-tunnels.md
- [ 8d] infrastructure/hosts.md
- [ 8d] infrastructure/network-map.md
- [ 8d] infrastructure/proxmox.md
- [ 3d] infrastructure/tailscale.md
- [ 11d] queries/retained-research-answers.md
- [ 21d] reports/communities-wa-social-media-audit-2026.md
- [ 14d] reports/uber-eats/2026-07.md
- [ 14d] reports/uber-eats/2026-08.md
- [ 14d] reports/uber-eats/2026-09.md
- [ 2d] reports/uber-eats/2026-10.md
- [ 2d] reports/uber-eats/index.md
- [ 4d] runbooks/backup-wiki.md
- [ 4d] runbooks/diagnose-docker.md
- [ 4d] runbooks/gateway-resource-resilience.md
- [ 8d] runbooks/job-radar.md
- [ 4d] runbooks/multiplexer-setup.md
- [ 20d] runbooks/provider-health.md
- [ 17d] runbooks/recover-docker-service.md
- [ 4d] runbooks/restart-browser.md
- [ 4d] runbooks/restart-hermes.md
- [ 4d] runbooks/restore-openrouter.md
- [ 4d] runbooks/test-openrouter.md
- [ 4d] runbooks/troubleshoot-tool-loop.md
- [ 20d] runbooks/update-hermes-safely.md
- [ 7d] systems/browser-backend.md
- [ 4d] systems/cloudflare-tunnel.md
- [ 6d] systems/conduit.md
- [ 20d] systems/current-profile.md
- [ 4d] systems/guanaco.md
- [ 4d] systems/headless-server.md
- [ 6d] systems/hermes-agent.md
- [ 2d] systems/hermes-hq-push.md
- [ 4d] systems/hermes-ui.md
- [ 11d] systems/home-assistant.md
- [ 16d] systems/homepage-dashboard.md
- [ 11d] systems/ignis.md
- [ 20d] systems/memory-backend.md
- [ 13d] systems/messaging-integrations.md
- [ 0d] systems/mnemosyne.md
- [ 20d] systems/model-providers.md
- [ 11d] systems/nanobot.md
- [ 4d] systems/nextcloud.md
- [ 4d] systems/obsidian.md
- [ 5d] systems/omniroute.md
- [ 4d] systems/opencode-go.md
- [ 4d] systems/openrouter.md
- [ 11d] systems/paseo.md
- [ 0d] systems/phonon-tts.md
- [ 5d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 4d] systems/terminal-backend.md
- [ 4d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 133 fresh, 3 no-date of 136 pages
+5 -6
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system
status: active
created: 2026-07-22
updated: 2026-11-14
verified_on: 2026-11-14
updated: 2026-11-15
verified_on: 2026-11-15
confidence: high
tags: [skills, hermes]
sources: []
@@ -12,9 +12,9 @@ sources: []
# Installed Skills
**Total installed: 300 skills** in `~/.hermes/skills/`.
**Total installed: 299 skills** in `~/.hermes/skills/`.
- **50 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
- **51 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
This page is auto-regenerated from the live skill inventory. Do not hand-edit.
@@ -98,7 +98,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `ddia-systems` — # Designing Data-Intensive Applications Framework
## Devops (108)
## Devops (107)
- `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS)
- `agent-extension-audit` — # Agent Extension Audit
@@ -204,7 +204,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `uber-eats-monitor` — # Uber Eats Order Monitor
- `vision-model-validation` — # Vision Model Validation via OmniRoute
- `watchers` — # Watchers
- `web-content-extraction` — # Web Content Extraction — Fallback Techniques
- `weekly-comms-tracker` — # Weekly Communications Tracker
- `wiki-maintenance` — # Wiki Maintenance
- `zeroclaw-ops` — # ZeroClaw Operations & Configuration