stocktake(2026-11-01): fix the Quartz build (YAML title), close the "unfixed memory ranking" incident, and correct live drift in tailscale/docker/proxmox

BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic.
Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an
unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a
mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)".
A scan of all 117 pages found exactly one such line; it is now properly quoted.
Quartz now builds cleanly: 690 files emitted and deployed.

RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page).
The page claimed the recall-ranking fix "was not present in live config" and that
conflict 152 was "the only open conflict". Both claims were false:
- The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5,
  fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 —
  not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the
  2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks.
- The original regression test now passes: "mnemosyne recall 'Are you up to date'"
  returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale
  Uber Eats episode no longer outranks anything.
- Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise.
All three open items closed; page moved to status: resolved.

LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01):
- infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping
  times out); m93p peer record reads false for the 4th consecutive time while the
  host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages
  refreshed; no IP changes, no missing peers.
- infrastructure/docker-services.md: CT201's Tailscale IP was wrong
  (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale
  worldmonitor-seeder-fast entry removed from the CT200 list, which is now
  regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to
  thirsty_payne.
- infrastructure/proxmox.md: storage percentages refreshed on both hosts from live
  pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an
  elevated load average recorded.
- systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md:
  re-probed and verified_on bumped.
- systems/skills-index.md: regenerated — 301 unique skills in 68 categories,
  46 archived. (301 is the unique (category, skill_name) count; find returns 302
  files due to the known nested obsidian-git-sync alias duplicate.)

Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
This commit is contained in:
Tony0410
2026-11-01 04:48:30 +08:00
parent c69681dc16
commit a0b3734c95
14 changed files with 596 additions and 111 deletions
+1 -1
View File
@@ -71,7 +71,7 @@ tags: [index, infrastructure, systems]
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved)
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST. **CLOSED — all 12 incident rows are now resolved**; the last two (`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, both on the weekly Friday job `1b23c042e17d`) closed **2026-10-23 08:04** when that slot ran clean, and the job reads `ok` again
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question. **✅ RESOLVED 2026-11-01** — the ranking weights *were* persisted, in `~/.hermes/mnemosyne/config.yaml` rather than the file the 2026-10-10 probe read; the original regression query now returns only current-state facts, and conflict `152` was auto-resolved on 2026-10-29
- [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05)
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented