From 402d8546e889357764b12a37b5f1b5b9bdf6e127 Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Sun, 15 Nov 2026 03:25:18 +0800 Subject: [PATCH] =?UTF-8?q?stocktake(2026-11-15):=20skills-index=20regener?= =?UTF-8?q?ated=20(299/67);=20storm=20incident=20re-verified=20(all=2012?= =?UTF-8?q?=20rows=20resolved,=20evidence-window=20note);=20conduit=20memo?= =?UTF-8?q?ry-timeout=20fix=20LOST=20in=20plugin=200.168.x=20restructure?= =?UTF-8?q?=20=E2=80=94=20page=20=E2=86=92=20degraded=20with=20regression?= =?UTF-8?q?=20section;=200=20stale=20after=20fixes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../2026-10-09-cron-provider-timeout-storm.md | 15 +- ...2026-10-22-conduit-memory-start-timeout.md | 45 ++++-- index.md | 4 +- log.md | 32 +++- raw/stocktake/freshness-20261115.txt | 149 ++++++++++++++++++ systems/skills-index.md | 11 +- 6 files changed, 233 insertions(+), 23 deletions(-) create mode 100644 raw/stocktake/freshness-20261115.txt diff --git a/incidents/2026-10-09-cron-provider-timeout-storm.md b/incidents/2026-10-09-cron-provider-timeout-storm.md index af484d0..f11d98d 100644 --- a/incidents/2026-10-09-cron-provider-timeout-storm.md +++ b/incidents/2026-10-09-cron-provider-timeout-storm.md @@ -3,9 +3,9 @@ title: Fleet-Wide Provider Timeout Storm — 2026-10-09 type: incident status: resolved created: 2026-10-09 -updated: 2026-10-24 -verified_on: 2026-10-24 -review_after: 2026-11-07 +updated: 2026-11-15 +verified_on: 2026-11-15 +review_after: 2026-12-15 confidence: high tags: [cron, omniroute, router, timeout, incident, resolved] sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1] @@ -18,6 +18,15 @@ sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-2026100 (`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job) closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again. The storm contributes **no** unclosed rows. Nothing here is still symptomatic. + +> **Re-verified 2026-11-15** (3 a.m. stocktake): all 12 of the storm's `cron_incidents` rows still read +> `resolved` — none reopened. `1b23c042e17d` (the weekly Friday job, last hold-out) is healthy — +> `last_status: ok`, last run 2026-11-13 08:11 AWST. One incident is currently open fleet-wide +> (`ea79a2_601037516abd`, Mnemosyne Health Monitor, first seen 2026-11-15 00:03) but it is the +> capture-stall detection firing → [[incidents/2026-11-14-mnemosyne-capture-stall]], unrelated to +> this storm. ⚠ Evidence window: `executions.db` retention now reaches back only to 2026-11-09, so +> the 2026-10-09 execution counts below are **no longer re-derivable from the DB** — they stand on +> the 2026-10-24 read (the incident rows themselves persist, which is what this check re-read). **Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08** ## What happened diff --git a/incidents/2026-10-22-conduit-memory-start-timeout.md b/incidents/2026-10-22-conduit-memory-start-timeout.md index ccecb67..f13144b 100644 --- a/incidents/2026-10-22-conduit-memory-start-timeout.md +++ b/incidents/2026-10-22-conduit-memory-start-timeout.md @@ -1,11 +1,11 @@ --- title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout) type: incident -status: resolved +status: degraded created: 2026-10-24 -updated: 2026-10-25 -verified_on: 2026-10-25 -review_after: 2026-11-08 +updated: 2026-11-15 +verified_on: 2026-11-15 +review_after: 2026-12-15 confidence: high tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved] sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff] @@ -14,7 +14,8 @@ sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff] # Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23 **Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences) -**Status:** Resolved 2026-10-23 18:54 (patched and verified) +**Status:** Resolved 2026-10-23 18:54 (patched and verified) — **but the fix was lost in the +plugin's 0.168.x restructure; `status: degraded` since 2026-11-15** (see the regression section below) **Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's memory layer. @@ -87,17 +88,39 @@ touching the file. not positive proof that recall was populated on those calls. The proven part is that the call is no longer abandoned by the memory layer. +## ⚠ Regression — the fix was lost in the plugin restructure (verified 2026-11-15) + +The follow-up below warned that a plugin update would revert the patch. **It happened.** Live +state at the 2026-11-15 03:00 stocktake: + +- The plugin tree is **clean at upstream `0.168.2`** (`296c94b`, committed 2026-11-12 US / + 2026-11-13 AWST) — the patch is gone entirely, not just uncommitted. +- The memory layer **moved**: `dashboard/plugin_api.py` (now a doc index) → + `dashboard/api/memory.py`. There, `MEMORY_TIMEOUT_S = 10.0` is hard-coded again + (`memory.py:48`), with **no** `CONDUIT_MEMORY_TIMEOUT_S` env override, and the "abandoning the + call" behaviour is intact (`memory.py:256`). The restructure landed in `memory.py` on + **2026-10-29 AWST**, so the 60 s budget has been gone for ~2.5 weeks. +- The running dashboard (started 2026-11-10) loads the restructured code, so the exposure is live. +- One real call has since occurred — 2026-11-14 08:57 AWST, the first entry in + `conduit-calls.json → history` — with no abort lines in the logs afterwards. But that call met a + warm provider; it does not exercise the cold-start path. The page's original caveat stands. +- **Not re-applied by the stocktake**: porting the fix to the restructured code is a code change to + a git-managed upstream plugin that the next update will revert again — it needs Anthony's call + (re-port + commit locally, or upstream it). + +The original incident window (2026-10-22 → 23) remains resolved; `status: degraded` records that +the *protection* is gone, not that the October fault returned. + ## Follow-ups -- ⚠ **The fix is still unexercised by a call** (see the correction above): `conduit-calls.json → - history` is empty at 2026-10-25 00:2x, and the plugin is now **five releases behind HEAD** - (`0.64.0` → `0.69.0`). +- ~~⚠ **The fix is still unexercised by a call** …~~ **Superseded 2026-11-15**: one call has since + occurred (2026-11-14 08:57 AWST) — but the fix itself was lost in the restructure (section above), + so "unexercised" is now moot. - **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which asserted the capability did not exist **without calling the tool** → [[incidents/2026-10-24-conduit-call-request-refused]]. -- **The patch is uncommitted.** A plugin update that resets - `~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream - (`kaishi00/hermes-conduit-notifier`). +- ~~**The patch is uncommitted.** A plugin update that resets `~/.hermes/plugins/conduit_push` + reverts it to the 10 s budget.~~ **Confirmed 2026-11-15** — see the regression section above. - The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised. Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]] diff --git a/index.md b/index.md index 9b1cd07..4f405ef 100644 --- a/index.md +++ b/index.md @@ -3,8 +3,8 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-14 -verified_on: 2026-11-14 +updated: 2026-11-15 +verified_on: 2026-11-15 confidence: high tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout] sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] diff --git a/log.md b/log.md index 9a24e7e..df2fb6e 100644 --- a/log.md +++ b/log.md @@ -1751,4 +1751,34 @@ write path, and nothing watches it.** Candidate fix: an N-consecutive-zero-item- 14 `script` / 5 `auth`. The job is `no_agent`, so `agent` is impossible — read `auth` as unexplained, not as a credential fault. Pages corrected: [[incidents/2026-11-14-mnemosyne-capture-stall]] (false section rewritten, `auth` bullet upgraded from suspicion to evidence), [[systems/scheduled-tasks]], [[systems/mnemosyne]], [[index.md]]. The incident itself -remains **OPEN / `alerted` / unclosed**; the *cause* of the stall is still **not established**. \ No newline at end of file +remains **OPEN / `alerted` / unclosed**; the *cause* of the stall is still **not established**. +## 2026-11-15 03:00 — stocktake (wiki-maintain.sh + drift re-verification) + +Script run: skills-index regenerated (299 skills / 67 categories), freshness report +`raw/stocktake/freshness-20261115.txt` — **STALE (1)**: +`incidents/2026-10-09-cron-provider-timeout-storm.md` (22d, updated 2026-10-24). Pages ON the +threshold (21d): `incidents/2026-10-22-conduit-memory-start-timeout.md`, +`reports/communities-wa-social-media-audit-2026.md` (static deliverable, no live claims — left +unbumped). Pages at 20d (not re-probed this run, next candidates): +`runbooks/provider-health.md`, `runbooks/update-hermes-safely.md`, +`systems/current-profile.md`, `systems/memory-backend.md`, `systems/model-providers.md`, +`incidents/2026-10-03-ct450-hermes-update.md`, +`incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md`, +`incidents/2026-10-24-conduit-call-request-refused.md`, +`incidents/2026-10-25-gmail-digest-runaway-and-refire.md`. + +**Re-verified live (storm page, 22d):** all 12 `cron_incidents` rows still `resolved`; weekly job +`1b23c042e17d` `last_status: ok` (2026-11-13). The one open fleet-wide incident +(`ea79a2_601037516abd`, first seen 2026-11-15 00:03) is the capture-stall monitor firing, not a +storm row. `executions.db` retention now starts 2026-11-09 — the page's 2026-10-09 execution +counts are no longer DB-re-derivable; noted on the page. Bumped with a dated verification note. + +**Regression found (conduit page, 21d):** the Conduit voice memory-timeout fix was **lost in the +plugin's restructure**. `~/.hermes/plugins/conduit_push` is clean at upstream `0.168.2`; the +memory layer moved to `dashboard/api/memory.py` where `MEMORY_TIMEOUT_S = 10.0` is hard-coded +again (no `CONDUIT_MEMORY_TIMEOUT_S` env hook; restructure landed ~2026-10-29 AWST). Running +dashboard (since 2026-11-10) carries it. One real call since (2026-11-14 08:57 AWST) hit a warm +provider, so the cold-start path is untested. Page → `status: degraded` with a dated regression +section; **fix not re-applied** (needs Anthony's call: re-port + commit locally, or upstream). +Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]], +[[incidents/2026-10-22-conduit-memory-start-timeout]], [[index.md]]. diff --git a/raw/stocktake/freshness-20261115.txt b/raw/stocktake/freshness-20261115.txt new file mode 100644 index 0000000..6697479 --- /dev/null +++ b/raw/stocktake/freshness-20261115.txt @@ -0,0 +1,149 @@ +# Wiki freshness audit — 2026-11-15 +Threshold: >21 days since `updated` + +## STALE (0) + + +## FRESH (133) + +- [ 4d] CONTRIBUTING.md +- [ 7d] agents/hermes-chaos.md +- [ 7d] agents/hermes-production.md +- [ 4d] agents/index.md +- [ 17d] agents/nanobot.md +- [ 4d] agents/open-claw.md +- [ 11d] agents/qwenpaw.md +- [ 4d] agents/zeroclaw.md +- [ 17d] comparisons/camoufox-vs-chrome.md +- [ 4d] comparisons/hermes-vs-nanobot.md +- [ 4d] comparisons/memory-backends.md +- [ 4d] comparisons/model-routers.md +- [ 17d] comparisons/steel-vs-camoufox.md +- [ 4d] concepts/agent-memory.md +- [ 17d] concepts/browser-automation.md +- [ 4d] concepts/model-routing.md +- [ 4d] concepts/persistent-shells.md +- [ 4d] concepts/tool-calling.md +- [ 0d] current-state.md +- [ 17d] decisions/index.md +- [ 4d] decisions/keep-openrouter-as-fallback.md +- [ 4d] decisions/persistent-shell-enabled.md +- [ 17d] decisions/rollback-steel-to-camofox.md +- [ 4d] experiments/index.md +- [ 3d] incidents/2026-07-29-mnemosyne-relay.md +- [ 4d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 3d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 8d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 10d] incidents/2026-08-15-skills-index-drift.md +- [ 11d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 3d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 4d] incidents/2026-08-29-gmail-read-state.md +- [ 3d] incidents/2026-09-02-sftpgo-outage.md +- [ 11d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 17d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 20d] incidents/2026-10-03-ct450-hermes-update.md +- [ 20d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 17d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 0d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 14d] incidents/2026-10-09-memory-recall-relevance.md +- [ 9d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 19d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 5d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 19d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 3d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 0d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 4d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 20d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 20d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 18d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 16d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 16d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md +- [ 16d] incidents/2026-10-29-conduit-voice-silence.md +- [ 15d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +- [ 16d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md +- [ 15d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 14d] incidents/2026-10-31-asana-access-failures.md +- [ 14d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md +- [ 14d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md +- [ 13d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md +- [ 12d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md +- [ 11d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md +- [ 9d] incidents/2026-11-03-job-radar-stream-active-timeout.md +- [ 9d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md +- [ 10d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md +- [ 6d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md +- [ 10d] incidents/2026-11-05-skills-index-generator-divergence.md +- [ 8d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md +- [ 6d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md +- [ 1d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md +- [ 4d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md +- [ 4d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md +- [ 3d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md +- [ 2d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md +- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md +- [ 0d] index.md +- [ 8d] infrastructure/backups.md +- [ 4d] infrastructure/credentials-map.md +- [ 8d] infrastructure/docker-services.md +- [ 4d] infrastructure/domains-and-tunnels.md +- [ 8d] infrastructure/hosts.md +- [ 8d] infrastructure/network-map.md +- [ 8d] infrastructure/proxmox.md +- [ 3d] infrastructure/tailscale.md +- [ 11d] queries/retained-research-answers.md +- [ 21d] reports/communities-wa-social-media-audit-2026.md +- [ 14d] reports/uber-eats/2026-07.md +- [ 14d] reports/uber-eats/2026-08.md +- [ 14d] reports/uber-eats/2026-09.md +- [ 2d] reports/uber-eats/2026-10.md +- [ 2d] reports/uber-eats/index.md +- [ 4d] runbooks/backup-wiki.md +- [ 4d] runbooks/diagnose-docker.md +- [ 4d] runbooks/gateway-resource-resilience.md +- [ 8d] runbooks/job-radar.md +- [ 4d] runbooks/multiplexer-setup.md +- [ 20d] runbooks/provider-health.md +- [ 17d] runbooks/recover-docker-service.md +- [ 4d] runbooks/restart-browser.md +- [ 4d] runbooks/restart-hermes.md +- [ 4d] runbooks/restore-openrouter.md +- [ 4d] runbooks/test-openrouter.md +- [ 4d] runbooks/troubleshoot-tool-loop.md +- [ 20d] runbooks/update-hermes-safely.md +- [ 7d] systems/browser-backend.md +- [ 4d] systems/cloudflare-tunnel.md +- [ 6d] systems/conduit.md +- [ 20d] systems/current-profile.md +- [ 4d] systems/guanaco.md +- [ 4d] systems/headless-server.md +- [ 6d] systems/hermes-agent.md +- [ 2d] systems/hermes-hq-push.md +- [ 4d] systems/hermes-ui.md +- [ 11d] systems/home-assistant.md +- [ 16d] systems/homepage-dashboard.md +- [ 11d] systems/ignis.md +- [ 20d] systems/memory-backend.md +- [ 13d] systems/messaging-integrations.md +- [ 0d] systems/mnemosyne.md +- [ 20d] systems/model-providers.md +- [ 11d] systems/nanobot.md +- [ 4d] systems/nextcloud.md +- [ 4d] systems/obsidian.md +- [ 5d] systems/omniroute.md +- [ 4d] systems/opencode-go.md +- [ 4d] systems/openrouter.md +- [ 11d] systems/paseo.md +- [ 0d] systems/phonon-tts.md +- [ 5d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 4d] systems/terminal-backend.md +- [ 4d] systems/tool-search.md + +## NO updated DATE (3) + +- SCHEMA.md +- experiments/active/example.md +- log.md + +TOTAL: 0 stale, 133 fresh, 3 no-date of 136 pages \ No newline at end of file diff --git a/systems/skills-index.md b/systems/skills-index.md index 2edd71d..b2ca07c 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-11-14 -verified_on: 2026-11-14 +updated: 2026-11-15 +verified_on: 2026-11-15 confidence: high tags: [skills, hermes] sources: [] @@ -12,9 +12,9 @@ sources: [] # Installed Skills -**Total installed: 300 skills** in `~/.hermes/skills/`. +**Total installed: 299 skills** in `~/.hermes/skills/`. -- **50 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them. +- **51 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them. - The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory. This page is auto-regenerated from the live skill inventory. Do not hand-edit. @@ -98,7 +98,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `ddia-systems` — # Designing Data-Intensive Applications Framework -## Devops (108) +## Devops (107) - `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS) - `agent-extension-audit` — # Agent Extension Audit @@ -204,7 +204,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `uber-eats-monitor` — # Uber Eats Order Monitor - `vision-model-validation` — # Vision Model Validation via OmniRoute - `watchers` — # Watchers -- `web-content-extraction` — # Web Content Extraction — Fallback Techniques - `weekly-comms-tracker` — # Weekly Communications Tracker - `wiki-maintenance` — # Wiki Maintenance - `zeroclaw-ops` — # ZeroClaw Operations & Configuration