Stocktake 2026-11-21: re-verify 5 stale + 2 threshold pages against live state; freshness 0 stale; skills-index 299/67

This commit is contained in:
Tony0410
2026-11-21 03:16:30 +08:00
parent 375d52fa26
commit 95e2cd8357
9 changed files with 100 additions and 27 deletions
@@ -3,9 +3,9 @@ title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion
type: incident
status: resolved
created: 2026-10-28
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-13
updated: 2026-11-21
verified_on: 2026-11-21
review_after: 2026-12-21
confidence: high
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
@@ -121,6 +121,17 @@ handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns th
The questions this page left open are answered for this run: the handoff **was** written inside budget and
consumed, and **no cover letter went missing on the first real run of the split**.
## Re-verification — 2026-11-21 (stocktake)
Live probe: the split still holds and both stages are healthy — `dca8482e4f76` (Discovery) and
`5fc292ff0f3b` (Write and Letters) are both `enabled: true`, `last_status: ok`, `failure_streak: 0`,
pins unchanged (`custom:omni-101` / `smart-route`). Handoff files are being written every weekday —
latest `~/.hermes/radar-ref/handoff/2026-11-20.json` (7.9 KB), with files present for every weekday
back to 2026-11-13. The cap/budget fault has recurred since this incident (see
[[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]],
[[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]); both ledger rows this page opened
remain closed `resolved`.
## Related
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
@@ -3,9 +3,9 @@ title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handof
type: incident
status: resolved
created: 2026-10-29
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-13
updated: 2026-11-21
verified_on: 2026-11-21
review_after: 2026-12-21
confidence: high
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
@@ -91,6 +91,14 @@ the previous two slots in the split both died on it (2026-10-27 on the tool-call
the iteration cap). Treat a `completed` run with **no handoff file** as the same fault, whatever the
execution status says → [[incidents/2026-10-27-job-radar-budget-exhausted]].
## Re-verification — 2026-11-21 (stocktake)
Live probe: `dca8482e4f76` is `enabled: true`, `last_status: ok`, `failure_streak: 0`, and handoff
files exist for every weekday back to 2026-11-13 (latest `2026-11-20.json`). The iteration-cap fault
recurred on 2026-11-02 → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] and again
on 2026-11-09 → [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] — the ⚠ "flaky, not
solved" warning above stands. This page's ledger rows remain closed `resolved`.
## Related
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
+12 -2
View File
@@ -3,8 +3,8 @@ title: Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hou
type: incident
status: resolved
created: 2026-10-30
updated: 2026-10-30
verified_on: 2026-10-30
updated: 2026-11-21
verified_on: 2026-11-21
confidence: high
tags: [conduit, voice, tts, stt, config, session-ttl, incident]
sources:
@@ -98,6 +98,16 @@ Consequence: `voice.mode` was moved **`local` → `classic`** (request queued 08
- If an app update or the settings sync ever writes the bare `/v1/` back into `voice.speech_server.url`, every call silently 403s again. Re-check it after any Conduit update.
- Watch for the Conduit settings tool re-flipping `tts.provider` should a `tts.provider` key ever appear in the app's schema.
## Re-verification — 2026-11-21 (stocktake)
All three fixes still in place live in `~/.hermes/config.yaml`: `voice.speech_server.url` =
`wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime` (the `/realtime` suffix has not been
truncated back); `dashboard.basic_auth.session_ttl_seconds` = 2592000 while
`dashboard.session_ttl_seconds` is still 43200 (exactly the two-key warning above);
`tts.provider` = `ollama_local`; `stt.provider` = `local`. Classic mode remains selected in
`~/.hermes/conduit-settings.json`. A follow-on Conduit voice fault is recorded separately →
[[incidents/2026-10-31-conduit-voice-local-mode-not-applying]].
## Related
- [Conduit](../systems/conduit.md) — service page (voice pipeline, call path, pairing)
@@ -3,8 +3,8 @@ title: CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026
type: incident
status: resolved
created: 2026-10-31
updated: 2026-10-31
verified_on: 2026-10-31
updated: 2026-11-21
verified_on: 2026-11-21
confidence: high
tags: [cron, gateway, host, outage, pve, ct460, incident]
sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-agent-log, live-deliveries-db, ssh-pve-uptime]
@@ -16,6 +16,14 @@ sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-a
closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not
established.** No monitor alerted; the gap was found by this audit.
## Re-verification — 2026-11-21 (stocktake)
CT460's host has rebooted once more since this incident: current `uptime -s` reads
**2026-11-10 22:03:51**, matching the 2026-11-10 unclean gateway exit in `gateway.log`
(previous life `pid=3109529`, started 2026-11-07, SIGKILL/OOM-class exit, `suspected_oom=false`).
This page's 2026-10-30 window stands as recorded; later scheduler gaps were recorded as separate
incidents.
## Summary
The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until
@@ -3,9 +3,9 @@ title: m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start
type: concept
status: resolved
created: 2026-10-30
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-30
updated: 2026-11-21
verified_on: 2026-11-21
review_after: 2026-12-21
confidence: high
tags: [proxmox, m93p, nfs, lxc, outage, freshrss]
sources: [live-probe, ssh-m93p, systemctl-status, pct-list, freshrss-script-output]
@@ -75,6 +75,14 @@ If a service on m93p is unreachable and `pct list` shows it `stopped`:
3. `pct list` to confirm `running`; then check the service from the Hermes host
4. FreshRSS specifically: `curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/`
## Re-verification — 2026-11-21 (stocktake)
`ssh root@100.104.51.98` probe: all **three** NFS mounts present (`ssd0-39`, `proxmox-31`,
`proxmox-54`), `m93p-nfs-recover.service` **enabled and active**. The guest fleet has grown since
this incident: **23 CTs, 21 running** — `nightbridge` (998) and `openmausbot` (1264) are stopped
(stop cause for 1264 not re-probed this run; it has a history of tailscale-binary absence).
No repeat of the boot-time mount loss has been recorded since the fix was installed.
## Lessons
- A Google Reader API `Auth failed ... timed out` error is an availability symptom first, an auth
@@ -3,8 +3,8 @@ title: Mnemosyne Health Monitor auto-paused after gateway-connection failure
type: incident
status: resolved
created: 2026-10-30
updated: 2026-10-31
verified_on: 2026-10-31
updated: 2026-11-21
verified_on: 2026-11-21
confidence: high
tags: [cron, mnemosyne, gateway, incident, delivery]
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
@@ -24,6 +24,15 @@ host outage** and fired late in the 16:37:21 catch-up burst, still `running` at
**Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
## Re-verification — 2026-11-21 (stocktake)
`ea79a2107e36` is still `enabled: true` and firing every 12 h, but `last_status: error`,
`failure_streak: 4` — the monitor is (correctly) reporting the **Mnemosyne capture stall**:
its latest run found a user message with no `working_memory` row and `last capture
2026-11-18 11:25`. That is a live memory-pipeline fault the monitor is doing its job about, not a
delivery or gateway fault → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The 2026-10-30
resolution above (auto-close, deliveries resumed) stands.
## Summary
The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its