2026-11-18 stocktake: re-verify 7 stale pages against live state, resolve gmail-digest refire incident

This commit is contained in:
Tony0410
2026-11-18 03:19:54 +08:00
parent 4574c4ebc3
commit 98610466cc
11 changed files with 199 additions and 20 deletions
+6 -3
View File
@@ -3,8 +3,8 @@ title: CT450 Hermes Update to v0.21.5
type: incident
status: resolved
created: 2026-10-04
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
confidence: high
tags: [hermes, ct450, chaos, update, incident]
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
@@ -49,7 +49,10 @@ Two gotchas that cost time:
- CT460 production untouched — separate config, skills, cron and memory state.
## Current state (re-verified 2026-10-26)
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. ~~The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.~~ **SUPERSEDED 2026-11-18** — CT255 is `running` and its peer online; see the 11-18 state above.
## Current state (re-verified 2026-11-18)
Both instances have moved on again and are now on **different builds**: **CT460 runs `v0.23.0` (local `2bd378ed1b8d`)**, **CT450 runs `v0.23.1` (local `3b4215785882`)** — CT450 is one patch ahead and **0 commits behind `origin/main`**. The 10-26 Mattermost condition is **superseded**: **CT255 (`mattermost`) is `running` on `pve`** and its tailnet peer is `online`, so the old "CT255 stopped / peer offline 36 d" note no longer describes reality.
## Follow-ups
- ~~CT450 reports `Update available: 3 commits behind`.~~ **SUPERSEDED 2026-10-26** — CT450 is now **0 commits behind `origin/main`** (it tracks the `NousResearch/hermes-agent` `main` branch; there is no `origin/master` on that remote). Re-run `hermes update` if the chaos box should track upstream exactly.
@@ -3,8 +3,8 @@ title: Mnemosyne Vector Recall Ran FTS-Only
type: incident
status: resolved
created: 2026-10-04
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
confidence: high
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
@@ -65,6 +65,9 @@ Both tiers are **above the 95 % working-coverage reindex threshold**, so the 6-h
monitor job `b63ed5817212` (`last_status: ok`) has nothing to fire on. The 2026-10-04
figures below are superseded.
## Current state (re-verified 2026-11-18)
The fix continues to hold. The 6-hourly coverage monitor `b63ed5817212` is `enabled`, `last_status: ok`, `failure_streak: 0`, last run 2026-11-18 00:04 — it has had nothing to fire on, so working coverage remains at/above the 95 % threshold. (The coverage script itself could not be re-run this session — the gateway's lifecycle guard blocks scripts touching the >1 MiB Mnemosyne DB — so the monitor's clean streak is the verification, per the monitor's own design.)
## Follow-ups
- ~~Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.~~ **CONFIRMED 2026-10-26** — the job is `enabled` with `last_status: ok`; working coverage is 96.8 %, above the threshold, so it correctly has nothing to fire on.
- ~~Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.~~ **DONE 2026-10-26** — `fastembed` 0.8.1 now imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`) as well as 0.8.0 in the legacy venv.
@@ -3,8 +3,8 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
type: incident
status: monitoring
created: 2026-10-18
updated: 2026-10-27
verified_on: 2026-10-27
updated: 2026-11-18
verified_on: 2026-11-18
review_after: 2026-11-03
confidence: high
tags: [cron, router, provider, infrastructure]
@@ -35,6 +35,8 @@ days** (10-16 → 10-17 and 10-19) with an identical signature on an identical n
> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are
> shut, the single-node exposure is not.
> **Re-verified 2026-11-18 — 22 days clean.** Zero execution rows carry the resource-pressure string since 2026-10-27 (verified against `executions.db`). Both nodes healthy at probe time: CT600 `/api/health` → 200 with `/models` → 200 (**6,501 models**), `omni-101` `/api/health` → 200, zero unclosed `cron_incidents` rows estate-wide. Status stays **`monitoring`**: the single-node exposure and the invisible-to-`/api/health` failure mode are unchanged.
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
@@ -3,8 +3,8 @@ title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status
type: incident
status: active
created: 2026-10-21
updated: 2026-10-27
verified_on: 2026-10-27
updated: 2026-11-18
verified_on: 2026-11-18
review_after: 2026-11-03
confidence: high
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
@@ -36,6 +36,8 @@ gateway exit since 2026-09-21.
| 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) |
| ~09:10 | the orphaned execution row closes as **`unknown`** |
> **Re-verified 2026-11-18 — no recurrence in the retained window.** Zero `status: unknown` executions exist in the retained `executions.db` (retention floor ~18 d, so the 10-20 row itself has aged out — its facts below stand on the 10-27 read). Retained `gateway.log*` files carry no UNCLEANLY line newer than **2026-08-31** (log rotation has since dropped the 10-20 entry the ledger once showed; per that rotation, `0` means *not in the retained window*, not *never happened*). Zero unclosed `cron_incidents` rows. Status unchanged — the silent-orphan failure mode still has no alerting of its own.
## Impact
- **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries
@@ -1,13 +1,13 @@
---
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
type: incident
status: open
status: resolved
created: 2026-10-26
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
review_after: 2026-11-09
confidence: medium
tags: [cron, automation, hermes, incident, unresolved]
tags: [cron, automation, hermes, incident]
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
---
@@ -55,6 +55,9 @@ scheduler admitted an unrelated session is unexplained and is the first thing to
execution row.
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
## Resolution (verified 2026-11-18)
**Resolved — no recurrence in 23 days.** The `delivery_queued` state cleared (the open question above is answered: it did **not** persist past the 2026-10-26 08:30 slot). Job `08eb86f02d37` now reads `enabled`, `last_status: ok`, `last_error: null`, `failure_streak: 0`, last run 2026-11-17 08:33. In the retained `executions.db` window (2026-11-12 → 11-17; DB retention floor ~18 d means earlier slots have aged out) it ran **exactly once per day, `completed` every time, zero failed rows** — no refires since the 10-25 loop. Root cause of the 10-25 session-adoption remains hypothesised, not confirmed; the wall-clock-budget follow-up was never implemented, so the *exposure* (a runaway can recur silently) is unchanged even though the incident itself has not recurred.
## Related
- [[systems/scheduled-tasks]]
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
+1 -1
View File
@@ -12,7 +12,7 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-11-17.
> model routing and automation. last updated: 2026-11-18
>
> **2026-11-16 (nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST) — memory dead at 62 h 54 m, and a correction to the 11-15 record.**
> The capture stall is unchanged in every respect but the counters: **1130** / **1249** rows, newest write `2026-11-14 09:32:59` AWST, **five** unclosed `alerted` incidents, monitor `failure_streak` 5, consolidator zero-item streak **120** cycles. → [[incidents/2026-11-14-mnemosyne-capture-stall]]
+12
View File
@@ -1844,3 +1844,15 @@ recorded here so the next audit does not re-litigate it.
(`status: active`, `review_after` 2026-11-18); the incident-signature fragmentation fix remains
**proposed, not applied**; the plugin-load/hook-budget health check is **proposed, not applied**.
## 2026-11-18 03:xx — 03:00 stocktake (cron)
- Ran `wiki-maintain.sh`: skills-index regenerated (299 skills / 67 categories, content unchanged vs 11-17), freshness report `raw/stocktake/freshness-20261118.txt` — 9 stale.
- Live-vs-page re-verification of stale pages; 7 re-verified with dated corrections:
- `runbooks/provider-health.md` — snippets re-executed: CT600 /models 200 (6,501), omni-101 /api/health 200, fallbacks 200.
- `runbooks/update-hermes-safely.md` — prerequisites re-verified (v0.23.0, venv + managed runtimes present).
- `incidents/2026-10-03-ct450-hermes-update.md` — CT450 now v0.23.1 (0 behind origin/main), CT460 v0.23.0; CT255 mattermost running + peer online → Mattermost-timeout claim superseded.
- `incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md` — coverage monitor ok/streak 0; fix holds (script blocked by lifecycle guard; verified via monitor).
- `incidents/2026-10-25-gmail-digest-runaway-and-refire.md` — RESOLVED: last_status ok, one clean daily run 11-12→11-17, no refires; delivery_queued cleared.
- `incidents/2026-10-16-omniroute-503-resource-pressure.md` — 22 days clean, stays monitoring.
- `incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md` — no unknown rows in retained window; 10-20 log line aged out (rotation noted on page).
- Left stale (flagged only, not verifiable this run): `reports/communities-wa-social-media-audit-2026.md` (24d), `incidents/2026-10-24-conduit-call-request-refused.md` (23d, open behavioural).
- Quartz rebuilt + deployed via second `wiki-maintain.sh` run (pattern 13).
+150
View File
@@ -0,0 +1,150 @@
# Wiki freshness audit — 2026-11-18
Threshold: >21 days since `updated`
## STALE (2)
- [ 24d] reports/communities-wa-social-media-audit-2026.md (updated 2026-10-25)
- [ 23d] incidents/2026-10-24-conduit-call-request-refused.md (updated 2026-10-26)
## FRESH (132)
- [ 7d] CONTRIBUTING.md
- [ 10d] agents/hermes-chaos.md
- [ 10d] agents/hermes-production.md
- [ 7d] agents/index.md
- [ 20d] agents/nanobot.md
- [ 7d] agents/open-claw.md
- [ 14d] agents/qwenpaw.md
- [ 7d] agents/zeroclaw.md
- [ 20d] comparisons/camoufox-vs-chrome.md
- [ 7d] comparisons/hermes-vs-nanobot.md
- [ 7d] comparisons/memory-backends.md
- [ 7d] comparisons/model-routers.md
- [ 20d] comparisons/steel-vs-camoufox.md
- [ 7d] concepts/agent-memory.md
- [ 20d] concepts/browser-automation.md
- [ 7d] concepts/model-routing.md
- [ 7d] concepts/persistent-shells.md
- [ 7d] concepts/tool-calling.md
- [ 1d] current-state.md
- [ 20d] decisions/index.md
- [ 7d] decisions/keep-openrouter-as-fallback.md
- [ 7d] decisions/persistent-shell-enabled.md
- [ 20d] decisions/rollback-steel-to-camofox.md
- [ 7d] experiments/index.md
- [ 6d] incidents/2026-07-29-mnemosyne-relay.md
- [ 7d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 6d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 11d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 13d] incidents/2026-08-15-skills-index-drift.md
- [ 14d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 6d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 7d] incidents/2026-08-29-gmail-read-state.md
- [ 6d] incidents/2026-09-02-sftpgo-outage.md
- [ 14d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 20d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 0d] incidents/2026-10-03-ct450-hermes-update.md
- [ 0d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 20d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 3d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 17d] incidents/2026-10-09-memory-recall-relevance.md
- [ 12d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 0d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 8d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 0d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 6d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 3d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 7d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 0d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 21d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 19d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 19d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 19d] incidents/2026-10-29-conduit-voice-silence.md
- [ 18d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 19d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 18d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 17d] incidents/2026-10-31-asana-access-failures.md
- [ 17d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 17d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 16d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 15d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 14d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 12d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 12d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 13d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 9d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 13d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 11d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
- [ 9d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
- [ 1d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
- [ 7d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
- [ 7d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
- [ 6d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
- [ 5d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
- [ 1d] incidents/2026-11-14-mnemosyne-capture-stall.md
- [ 1d] incidents/2026-11-15-plugin-load-timeout-storm.md
- [ 1d] index.md
- [ 11d] infrastructure/backups.md
- [ 7d] infrastructure/credentials-map.md
- [ 11d] infrastructure/docker-services.md
- [ 7d] infrastructure/domains-and-tunnels.md
- [ 11d] infrastructure/hosts.md
- [ 11d] infrastructure/network-map.md
- [ 11d] infrastructure/proxmox.md
- [ 6d] infrastructure/tailscale.md
- [ 14d] queries/retained-research-answers.md
- [ 17d] reports/uber-eats/2026-07.md
- [ 17d] reports/uber-eats/2026-08.md
- [ 17d] reports/uber-eats/2026-09.md
- [ 5d] reports/uber-eats/2026-10.md
- [ 5d] reports/uber-eats/index.md
- [ 7d] runbooks/backup-wiki.md
- [ 7d] runbooks/diagnose-docker.md
- [ 7d] runbooks/gateway-resource-resilience.md
- [ 11d] runbooks/job-radar.md
- [ 7d] runbooks/multiplexer-setup.md
- [ 0d] runbooks/provider-health.md
- [ 20d] runbooks/recover-docker-service.md
- [ 7d] runbooks/restart-browser.md
- [ 7d] runbooks/restart-hermes.md
- [ 7d] runbooks/restore-openrouter.md
- [ 7d] runbooks/test-openrouter.md
- [ 7d] runbooks/troubleshoot-tool-loop.md
- [ 0d] runbooks/update-hermes-safely.md
- [ 10d] systems/browser-backend.md
- [ 7d] systems/cloudflare-tunnel.md
- [ 1d] systems/conduit.md
- [ 1d] systems/current-profile.md
- [ 7d] systems/guanaco.md
- [ 7d] systems/headless-server.md
- [ 9d] systems/hermes-agent.md
- [ 5d] systems/hermes-hq-push.md
- [ 7d] systems/hermes-ui.md
- [ 14d] systems/home-assistant.md
- [ 19d] systems/homepage-dashboard.md
- [ 14d] systems/ignis.md
- [ 1d] systems/memory-backend.md
- [ 16d] systems/messaging-integrations.md
- [ 1d] systems/mnemosyne.md
- [ 1d] systems/model-providers.md
- [ 14d] systems/nanobot.md
- [ 7d] systems/nextcloud.md
- [ 7d] systems/obsidian.md
- [ 8d] systems/omniroute.md
- [ 7d] systems/opencode-go.md
- [ 7d] systems/openrouter.md
- [ 14d] systems/paseo.md
- [ 3d] systems/phonon-tts.md
- [ 8d] systems/push-notifications.md
- [ 1d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 7d] systems/terminal-backend.md
- [ 7d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 2 stale, 132 fresh, 3 no-date of 137 pages
+4 -2
View File
@@ -3,8 +3,8 @@ title: Provider Health Check
type: runbook
status: active
created: 2026-07-22
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
last_tested: 2026-10-26
confidence: high
tags: [runbook, providers, health, routing]
@@ -58,6 +58,8 @@ with urllib.request.urlopen(req, timeout=5) as r:
```
Current verified result (2026-10-26): reachable and returns a non-empty model catalog — **6,480 models** on `omniroute:20128` (CT600, 100.88.81.19).
> **Re-executed 2026-11-18:** step-3 snippet OK — CT600 `/models` → 200, **6,501 models** (3.6 s; one earlier attempt at the default 5 s timeout timed out, so give this probe headroom). CT600 `/api/health` → 200; `omni-101` `/api/health` → 200; fallback endpoints `opencode.ai` / `inference-api.nousresearch.com` / `ollama.com` all → 200.
### 3b. Check the default `omni-101` provider (CT101, 100.78.63.20)
The **default provider** is `custom:omni-101` → `http://100.78.63.20:20128`, keyed via
`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`. Probe it with its **own** key — the CT600 key is rejected here:
+4 -2
View File
@@ -3,8 +3,8 @@ title: Update Hermes safely
type: runbook
status: active
created: 2026-07-22
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
last_tested: 2026-10-26
confidence: high
tags: [runbook, hermes, update]
@@ -28,6 +28,8 @@ Update Hermes to a newer release without losing configuration, skills, or sessio
- Available disk space in Hermes install and backup paths
- No active Hermes chat/webui shell issues before starting
> **Re-verified 2026-11-18:** prerequisites hold on CT460 — `hermes --version` → `v0.23.0` from the git install at `/home/hermes/.hermes/hermes-agent`, legacy `venv/` present, three managed runtime venvs present under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`. (CT450 currently runs `v0.23.1` — see [[incidents/2026-10-03-ct450-hermes-update]].)
## Procedure
### 1. Create a backup first
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system
status: active
created: 2026-07-22
updated: 2026-11-17
verified_on: 2026-11-17
updated: 2026-11-18
verified_on: 2026-11-18
confidence: high
tags: [skills, hermes]
sources: []