2026-11-18 stocktake: re-verify 7 stale pages against live state, resolve gmail-digest refire incident

This commit is contained in:
Tony0410
2026-11-18 03:19:54 +08:00
parent 4574c4ebc3
commit 98610466cc
11 changed files with 199 additions and 20 deletions
+6 -3
View File
@@ -3,8 +3,8 @@ title: CT450 Hermes Update to v0.21.5
type: incident
status: resolved
created: 2026-10-04
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
confidence: high
tags: [hermes, ct450, chaos, update, incident]
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
@@ -49,7 +49,10 @@ Two gotchas that cost time:
- CT460 production untouched — separate config, skills, cron and memory state.
## Current state (re-verified 2026-10-26)
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. ~~The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.~~ **SUPERSEDED 2026-11-18** — CT255 is `running` and its peer online; see the 11-18 state above.
## Current state (re-verified 2026-11-18)
Both instances have moved on again and are now on **different builds**: **CT460 runs `v0.23.0` (local `2bd378ed1b8d`)**, **CT450 runs `v0.23.1` (local `3b4215785882`)** — CT450 is one patch ahead and **0 commits behind `origin/main`**. The 10-26 Mattermost condition is **superseded**: **CT255 (`mattermost`) is `running` on `pve`** and its tailnet peer is `online`, so the old "CT255 stopped / peer offline 36 d" note no longer describes reality.
## Follow-ups
- ~~CT450 reports `Update available: 3 commits behind`.~~ **SUPERSEDED 2026-10-26** — CT450 is now **0 commits behind `origin/main`** (it tracks the `NousResearch/hermes-agent` `main` branch; there is no `origin/master` on that remote). Re-run `hermes update` if the chaos box should track upstream exactly.
@@ -3,8 +3,8 @@ title: Mnemosyne Vector Recall Ran FTS-Only
type: incident
status: resolved
created: 2026-10-04
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
confidence: high
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
@@ -65,6 +65,9 @@ Both tiers are **above the 95 % working-coverage reindex threshold**, so the 6-h
monitor job `b63ed5817212` (`last_status: ok`) has nothing to fire on. The 2026-10-04
figures below are superseded.
## Current state (re-verified 2026-11-18)
The fix continues to hold. The 6-hourly coverage monitor `b63ed5817212` is `enabled`, `last_status: ok`, `failure_streak: 0`, last run 2026-11-18 00:04 — it has had nothing to fire on, so working coverage remains at/above the 95 % threshold. (The coverage script itself could not be re-run this session — the gateway's lifecycle guard blocks scripts touching the >1 MiB Mnemosyne DB — so the monitor's clean streak is the verification, per the monitor's own design.)
## Follow-ups
- ~~Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.~~ **CONFIRMED 2026-10-26** — the job is `enabled` with `last_status: ok`; working coverage is 96.8 %, above the threshold, so it correctly has nothing to fire on.
- ~~Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.~~ **DONE 2026-10-26** — `fastembed` 0.8.1 now imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`) as well as 0.8.0 in the legacy venv.
@@ -3,8 +3,8 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
type: incident
status: monitoring
created: 2026-10-18
updated: 2026-10-27
verified_on: 2026-10-27
updated: 2026-11-18
verified_on: 2026-11-18
review_after: 2026-11-03
confidence: high
tags: [cron, router, provider, infrastructure]
@@ -35,6 +35,8 @@ days** (10-16 → 10-17 and 10-19) with an identical signature on an identical n
> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are
> shut, the single-node exposure is not.
> **Re-verified 2026-11-18 — 22 days clean.** Zero execution rows carry the resource-pressure string since 2026-10-27 (verified against `executions.db`). Both nodes healthy at probe time: CT600 `/api/health` → 200 with `/models` → 200 (**6,501 models**), `omni-101` `/api/health` → 200, zero unclosed `cron_incidents` rows estate-wide. Status stays **`monitoring`**: the single-node exposure and the invisible-to-`/api/health` failure mode are unchanged.
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
@@ -3,8 +3,8 @@ title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status
type: incident
status: active
created: 2026-10-21
updated: 2026-10-27
verified_on: 2026-10-27
updated: 2026-11-18
verified_on: 2026-11-18
review_after: 2026-11-03
confidence: high
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
@@ -36,6 +36,8 @@ gateway exit since 2026-09-21.
| 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) |
| ~09:10 | the orphaned execution row closes as **`unknown`** |
> **Re-verified 2026-11-18 — no recurrence in the retained window.** Zero `status: unknown` executions exist in the retained `executions.db` (retention floor ~18 d, so the 10-20 row itself has aged out — its facts below stand on the 10-27 read). Retained `gateway.log*` files carry no UNCLEANLY line newer than **2026-08-31** (log rotation has since dropped the 10-20 entry the ledger once showed; per that rotation, `0` means *not in the retained window*, not *never happened*). Zero unclosed `cron_incidents` rows. Status unchanged — the silent-orphan failure mode still has no alerting of its own.
## Impact
- **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries
@@ -1,13 +1,13 @@
---
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
type: incident
status: open
status: resolved
created: 2026-10-26
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
review_after: 2026-11-09
confidence: medium
tags: [cron, automation, hermes, incident, unresolved]
tags: [cron, automation, hermes, incident]
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
---
@@ -55,6 +55,9 @@ scheduler admitted an unrelated session is unexplained and is the first thing to
execution row.
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
## Resolution (verified 2026-11-18)
**Resolved — no recurrence in 23 days.** The `delivery_queued` state cleared (the open question above is answered: it did **not** persist past the 2026-10-26 08:30 slot). Job `08eb86f02d37` now reads `enabled`, `last_status: ok`, `last_error: null`, `failure_streak: 0`, last run 2026-11-17 08:33. In the retained `executions.db` window (2026-11-12 → 11-17; DB retention floor ~18 d means earlier slots have aged out) it ran **exactly once per day, `completed` every time, zero failed rows** — no refires since the 10-25 loop. Root cause of the 10-25 session-adoption remains hypothesised, not confirmed; the wall-clock-budget follow-up was never implemented, so the *exposure* (a runaway can recur silently) is unchanged even though the incident itself has not recurred.
## Related
- [[systems/scheduled-tasks]]
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]