From 6fc109e546e5df47ff383de0ab8cf8d015ce50e3 Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Mon, 12 Oct 2026 00:14:11 +0800 Subject: [PATCH] audit(2026-10-12): cron fleet clean 2nd day (10-11 171/0, 10-10 finalised 175/0); UTC-vs-AWST false 'capture stalled' alarm corrected; proxmox storage table + host mounts; queries/retained-research-answers stub filled (VNC); capture-pipeline incident -> superseded; index stale labels fixed --- current-state.md | 15 +++- .../2026-08-18-mnemosyne-capture-pipeline.md | 19 ++++- index.md | 13 ++-- infrastructure/proxmox.md | 19 ++++- log.md | 33 ++++++++ queries/retained-research-answers.md | 75 ++++++++++++++++--- systems/mnemosyne.md | 8 +- systems/scheduled-tasks.md | 52 ++++++++++++- 8 files changed, 205 insertions(+), 29 deletions(-) diff --git a/current-state.md b/current-state.md index d1483ed..81441c4 100644 --- a/current-state.md +++ b/current-state.md @@ -3,8 +3,8 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-11 -verified_on: 2026-10-11 +updated: 2026-10-12 +verified_on: 2026-10-12 review_after: 2026-10-25 confidence: high tags: [system] @@ -12,7 +12,16 @@ sources: [live-config, live-cron-list, live-hermes-version] --- # Current State -Updated: 2026-10-11 +Updated: 2026-10-12 + +## Recent Changes (2026-10-11 → 2026-10-12, nightly audit; live probes 2026-10-12 00:0x-00:2x AWST) + +- **The cron fleet is clean for a second consecutive day: 2026-10-11 ran `171 completed / 0 failed`** (+1 `running` = this audit), after 10-07 `1` / 10-08 `2` / 10-09 `15` failed. 10-10 finalises at **`175 / 0`**. Fleet **29 jobs — 26 enabled / 3 disabled**; non-clean `last_status` steady at **5**; **0 incidents opened, 0 closed — 5 unresolved** (3 `alerted`, 2 `detected`). The two streak-1 jobs (`dca8482e4f76`, `1b23c042e17d`) carry Friday 2026-10-09 timeouts and have not had a scheduled slot since. → [[systems/scheduled-tasks]] +- **The 2026-10-11 00:08 "memory capture stalled" EXCEPTION was a timezone mislabel, not a capture fault.** `working_memory.created_at` is naive UTC; the monitor compared it against AWST `now` and reported a 2.7 h gap as 10.7 h. Capture verified healthy: 786 rows, +2 in the window, coverage 98.3 % working / 99.6 % episodic. → [[systems/mnemosyne]] +- **Proxmox storage documented for the first time:** `/mnt/ssd0` is the pressure point — 808 GiB used, **~63 GiB free (93 % by `df`, 88.1 % by `pvesm`)**; `SSD1` 86.1 %, `fred-qnap` 76.8 %, `iMacHDD` inactive. → [[infrastructure/proxmox]] +- **First retained research answer filed:** fastest VNC clients (asked 2026-10-11) — the server-side encoder is the real bottleneck, TigerVNC native viewer for standard RFB, `-CompressLevel` is a *client* parameter, RealVNC's "4.7×" is self-referential. → [[queries/retained-research-answers]] +- **Only one interactive session in the window** (`talaria-ios`, 19:13 → 19:51, 41 messages); the other 17 were `cron`. The 03:00 stocktake `1a8e90ea2271` completed on schedule (03:00:40 → 03:11:29) and committed `d981e30`. +- **Stale labels corrected:** [[incidents/2026-08-18-mnemosyne-capture-pipeline]] moved from `active` to `superseded` (evidence above); the index's count of open 2026-10-09 storm incidents trimmed to the 2 that are genuinely open. ## Recent Changes (2026-10-11, 03:00 stocktake; live probes 03:00-03:30 AWST) diff --git a/incidents/2026-08-18-mnemosyne-capture-pipeline.md b/incidents/2026-08-18-mnemosyne-capture-pipeline.md index 7782600..036b613 100644 --- a/incidents/2026-08-18-mnemosyne-capture-pipeline.md +++ b/incidents/2026-08-18-mnemosyne-capture-pipeline.md @@ -1,10 +1,10 @@ --- title: "Mnemosyne Capture Pipeline Broken (Aug 18)" type: incident -status: active +status: superseded created: 2026-08-18 -updated: 2026-08-18 -verified_on: 2026-08-18 +updated: 2026-10-12 +verified_on: 2026-10-12 confidence: high tags: [memory, mnemosyne, hermes, incident, capture] sources: [mnemosyne-db, health-monitor, run_agent.py] @@ -59,6 +59,19 @@ print('Last WM capture:', cur.fetchone()[0]) # Should be recent " ``` +## Current Status (verified 2026-10-12) + +**Superseded — capture has been healthy since, and this page is kept for the diagnostic trail.** The 2026-10-12 +nightly audit found the full path working: `working_memory` at **786 rows**, +2 on 2026-10-11 (11:19:37 and +11:51:39 UTC = 19:19 / 19:51 AWST, both from that day's interactive session), `episodic_memory` 1,056, +embedding coverage **98.3 % working / 99.6 % episodic**, FTS aligned (`fts_working` 786, `fts_facts` 876). +The three guard conditions diagnosed below were never individually confirmed as the fix — the path resumed +capturing and has captured through every window since. + +**Caution for the next alarm:** the health monitor raised a false EXCEPTION on 2026-10-11 00:08 by comparing +naive-UTC `created_at` against AWST `now` ("stalled 10.7 h" — real gap 2.7 h). Do not triage that as a repeat +of this incident; see [[systems/mnemosyne]]. + ## Prevention - Never use `hermes config set memory.mnemosyne.` for list/dict/null values - After any config change, verify YAML types with `yaml.safe_load` + `type()` on each field diff --git a/index.md b/index.md index f5a4951..42bc405 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-11 +> model routing and automation. Last updated: 2026-10-12 ## Start here - [[current-state]] — Current systems status @@ -29,11 +29,11 @@ ### Integrations - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. -- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **5 carry a non-clean `last_status`** as of 2026-10-11 00:14 (down from 10). The 2026-10-10 fleet ran **clean — 174 `completed`, 0 `failed`** — ending the 2026-10-09 provider timeout storm (15 failed executions / 10 jobs; 7 of its incidents auto-closed during 2026-10-10, 2 remain open). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. See [[incidents/2026-10-09-cron-provider-timeout-storm]] -- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **no regeneration committed since 2026-10-07** — the 10-08 run's output was left uncommitted and the 10-09 run failed, see [[systems/scheduled-tasks]]) +- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **5 carry a non-clean `last_status`** as of 2026-10-12 00:10. **Two consecutive clean days: 2026-10-11 ran `171 completed` / 0 `failed`** (2026-10-10 finalised `175 / 0`), ending the 2026-10-09 provider timeout storm (15 failed executions / 10 jobs; 7 of its incidents auto-closed on 2026-10-10, 2 of its jobs remain `alerted`). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. See [[incidents/2026-10-09-cron-provider-timeout-storm]] +- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **regenerated daily — the 10-10 and 10-11 stocktake runs both committed (`d981e30`)**) ## Incidents -- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; 8 incidents still open (unresolved) +- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; 7 of its incidents auto-closed on 2026-10-10, **2 remain `alerted`** (`dca8482e4f76`, `1b23c042e17d` — both weekly jobs still carrying the Friday `RuntimeError: Request timed out.`) - [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved) - [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05) - [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic @@ -41,7 +41,7 @@ - [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); **closed 2026-10-05** when all three were repinned off the pin - [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26) - [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. **Recurred 2026-10-10 inside the *gateway* runtime env** — `mnemosyne` and `elevenlabs` missing from the committed uv env the gateway loads; both repaired and verified the same evening -- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured +- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (Aug 2026, **superseded**): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded - [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration ### Monitoring & Dashboards @@ -52,6 +52,7 @@ - [[reports/uber-eats/index]] — Uber Eats monthly spend reports - [[reports/communities-wa-social-media-audit-2026]] — Department of Communities WA social media audit and first-day brief - [[systems/obsidian]] — Obsidian vault structure and conventions +- [[queries/retained-research-answers]] — Retained research answers (fastest VNC clients, 2026-10-11) ## Infrastructure ### Services @@ -63,7 +64,7 @@ - [[infrastructure/backups]] — hermes backup CLI, retention, and gaps ### Networking -- [[infrastructure/proxmox]] — PVE host, VMs, LXC containers +- [[infrastructure/proxmox]] — PVE host, VMs, LXC containers, storage datastores (`/mnt/ssd0` 93 % / ~63 GiB free, 2026-10-12) - [[infrastructure/tailscale]] — Tailscale serve routes and VPN - [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly diff --git a/infrastructure/proxmox.md b/infrastructure/proxmox.md index e56d357..3ca2059 100644 --- a/infrastructure/proxmox.md +++ b/infrastructure/proxmox.md @@ -3,8 +3,8 @@ title: Proxmox VE type: infrastructure status: active created: 2026-07-26 -updated: 2026-10-11 -verified_on: 2026-10-11 +updated: 2026-10-12 +verified_on: 2026-10-12 confidence: high tags: [proxmox, virtualization, infrastructure, homelab] --- @@ -61,6 +61,21 @@ Hypervisor hosting all VMs and LXC containers for the homelab. **Not in either list:** VM 403 (`open-claw`) and VM 401 (`hermes-old`) are referenced by the wiki but no longer appear in `qm list`. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry. +## Storage (live, 2026-10-12 00:1x AWST — `df -h /mnt/ssd0` + `pvesm status`) + +| Datastore | Type | Status | Total | Used | Used % | +|---|---|---|---|---|---| +| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 808 GiB | **93 %** (`df`) / 88.1 % (`pvesm`) | +| `SSD1` | lvmthin | active | 931 GiB | 802 GiB | 86.1 % | +| `fred-qnap` | cifs | active | 13.4 TiB | 10.3 TiB | 76.8 % | +| `local` | dir | active | 67.7 GiB | 55.5 GiB | 82.0 % | +| `local-lvm` | lvmthin | active | 141 GiB | 19.2 GiB | 13.6 % | +| `iMacHDD` | cifs | **inactive** | — | — | — | + +**`/mnt/ssd0` is the capacity pressure point: 93 % with only ~63 GiB free.** Both figures describe the same ~808 GiB used and differ only in denominator — `df` excludes reserved blocks (808/871 GiB), `pvesm` divides by the full 916 GiB (88.14 %). Treat the 93 % as the number that matters before adding a disk-heavy guest (VM 110 `omarchy` is ~106 GB on its own). `iMacHDD` is inactive (`storage 'iMacHDD' is not online`) — check the CIFS mount before using it as a target. + +**Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`). + ## API Tokens - `homepage@pve!homepage` — PVEAuditor role, privsep=0 (used by Homepage dashboard) diff --git a/log.md b/log.md index 0124c5d..45f3224 100644 --- a/log.md +++ b/log.md @@ -472,3 +472,36 @@ No secrets written; job/incident IDs, paths and tailnet addresses only. - **Flagged for follow-up, root cause not investigated:** `mycompose-db-1` on CT200 is in `Restarting (1)` — crash-looping. - Mattermost CT255 confirmed stopped, matching its Tailscale peer offline 21d. - Left as stale-without-fix: 62 pages still older than 21 days (concepts, decisions, comparisons, most runbooks) — these were not probed against a live source this pass, so they were reported, not rewritten. + +## [2026-10-12] wiki audit | window 2026-10-11 -> 2026-10-12 (ran 2026-10-11 23:59 -> 2026-10-12 00:2x AWST) + +**Cron.** 29 jobs, 26 enabled / 3 disabled (`jobs.json` 2026-10-12T00:10). Day tally: 10-07 `177/1`, +10-08 `178/2`, 10-09 `163/15`, 10-10 **`175 completed / 0 failed`** (finalised), 10-11 **`171 completed / +0 failed`** + 1 `running` — **two consecutive clean days**. Non-clean `last_status` steady at 5; +**0 incidents opened, 0 closed — 5 unresolved** (3 `alerted`, 2 `detected`). + +**Corrected a false alarm instead of logging a fault.** The 2026-10-11 00:08 Mnemosyne Health Monitor +EXCEPTION ("capture stalled 10.7 h") was a **naive-UTC vs AWST comparison error** — real gap 2.7 h, the row it +cited landed inside the session that wrote it. Recorded in [[systems/mnemosyne]] and +[[systems/scheduled-tasks]] so the next run of that monitor is not triaged as a capture outage. Live counts +refreshed: 786 WM / 1,056 EP, coverage 98.3 % / 99.6 %. + +**New content.** [[queries/retained-research-answers]] was a 2026-07-22 stub and is now a real page — the +2026-10-11 VNC-client research (server-side encoder is the bottleneck; TigerVNC native viewer for standard RFB; +`-CompressLevel` 2 is client-side; RealVNC's "4.7×" is self-referential) with its sources. Proxmox storage +was undocumented; the live datastore table plus host mount points are now in [[infrastructure/proxmox]] +(`/mnt/ssd0` 93 % / ~63 GiB free is the pressure point). + +**Stale labels fixed.** [[incidents/2026-08-18-mnemosyne-capture-pipeline]] carried `status: active` from +August despite capture being verifiably healthy — now `superseded`, with the live evidence and a caution about +the UTC comparison. Index's "8 incidents still open" from the 2026-10-09 storm trimmed to the 2 genuinely open, +and the skills-index blurb no longer claims no regeneration since 2026-10-07. + +**Verified, not changed.** The `blocked-page-recovery` lesson that session reported patching +(`recovered: false` = archive routes failed, not task failed; API-first rung) is present in the live skill. + +**Pages touched:** `systems/scheduled-tasks.md`, `systems/mnemosyne.md`, `infrastructure/proxmox.md`, +`queries/retained-research-answers.md`, `incidents/2026-08-18-mnemosyne-capture-pipeline.md`, +`current-state.md`, `index.md`, `log.md`. + +No secrets written; job/incident IDs, paths and tailnet addresses only. diff --git a/queries/retained-research-answers.md b/queries/retained-research-answers.md index c1f1217..82c6e4f 100644 --- a/queries/retained-research-answers.md +++ b/queries/retained-research-answers.md @@ -1,16 +1,73 @@ --- title: Retained Research Answers type: query -status: unresolved +status: active created: 2026-07-22 -updated: 2026-07-22 -review_after: 2026-08-22 -confidence: low -tags: [query, research] -sources: [] +updated: 2026-10-12 +verified_on: 2026-10-12 +review_after: 2027-01-12 +confidence: medium +tags: [query, research, vnc, remote-desktop] +sources: [web-research, turboVNC-benchmark-pdf, superuser-15337, tigervnc-manpage, session-20261011] --- -> **Stub — structure only.** Headings exist but content has not been verified/written yet. Flagged during 2026-07-22 wiki audit; fill in with verified facts or mark deprecated if no longer relevant. - # Retained Research Answers -Filed research answers worth reusing. + +Research answers worth keeping: what was asked, the answer, the evidence it rests on, and what the answer does +**not** establish. Entries are point-in-time; re-verify anything load-bearing before acting on it. + +> The 2026-07-22 stub was filled on 2026-10-12 with the first retained answer (asked in the 2026-10-11 +> interactive session `talaria-ios`, 41 messages). + +## Fastest VNC client (asked 2026-10-11) + +**Answer for the CT460/CT200 TigerVNC + Xtigervnc servers:** use the **TigerVNC native viewer** with +AutoSelect and the defaults — free, and the fastest standard-RFB client. Lowest bandwidth is **KasmVNC** +(WebP, ~30 % less) but that is a *server* swap, not a client. **RealVNC Connect** is not a faster decoder, it is +a forked protocol (RFB 6, audio, no port-forwarding) and needs a RealVNC server. **TurboVNC** remains the +reference for lossless 3D/CAD. Genuinely low latency means leaving RFB entirely: **Parsec** or +**Sunshine + Moonlight**. + +**The load-bearing finding: the server-side encoder is the bottleneck, not the client.** A viewer can only +decode what it is sent. Two independent sources say the same thing — the TurboVNC benchmark report (encoder +throughput measurements) and the accepted Super User answer (RFB ships pixel rectangles, so cost scales with +resolution and bit depth). + +**Practical levers, server side (in order of payoff):** +1. **Compression level 1–2.** The single biggest lever; the 2012 report found compression got *worse* above + level 6 while costing 4.4× the CPU. +2. **16-bit colour** where banding is acceptable — halves the pixel payload. +3. **Local cursor rendering** (`-UseLocalCursor`) — large perceived win on slow links. +4. **Let AutoSelect work**; forcing `Tight` on a fast link or `Raw` on a slow link just wastes CPU/bandwidth. +5. **Tunnel over SSH** — latency and loss usually dominate codec choice. +6. `-FrameRate` (~5) is worth setting over the internet; if the requested rate exceeds what the link delivers + you get visible catch-up lag. + +**Client side:** `-CompressLevel` defaults to **2** and is a *client* parameter (0 = low, 9 = high) — do not +confuse it with the server-side encoder setting from the 2012 report. TigerVNC's viewer has an in-tree +**H.264 decoder** only when built with `ENABLE_H264=ON` (FFmpeg or WMF backend); server-side H.264 support is +still the blocker. Decode speed depends heavily on libjpeg-turbo SIMD — client CPU matters as much as client +software (a thin client with no SIMD headroom decodes slowly regardless of viewer). +**noVNC is convenience, not speed** — the JS/Canvas path is pure overhead versus the native client. + +**Evidence / sources:** +- TurboVNC "TigerVNC vs TurboVNC" benchmark, primary source (2012 data on 2012 CPUs): + https://turbovnc.org/pmwiki/uploads/About/turbototiger.pdf — after the TigerVNC 1.2 encoder rework the two + matched within ~2 % at compression level 1, and TigerVNC led by up to 16 % on indexed-colour workloads. +- Super User question 15337 (recovered via the **Stack Exchange API**, no browser needed): RFB transmits pixel + rectangles while RDP/X send drawing instructions — which is why VNC only wins at very low bitrates while + remaining the most portable option. +- TigerVNC man page, pulled from GitHub master because **`tigervnc.com` is dead** (domain parked at Gandi): + `-CompressLevel` default 2, client-side. +- RealVNC performance / viewer-comparison pages (recovered via Camofox): the "4.7× faster video / 4.5× faster + CAD" claim is **RealVNC against the previous RealVNC** at 37 Mbps / 33 ms / 0.3 % loss — not a comparison + against TigerVNC or TurboVNC. Their differentiator is proprietary RFB 6. +- TigerVNC issue #1187 — H.264 server support status. + +**Does NOT establish:** exact throughput multipliers on current hardware (the benchmark is 2012); no +end-to-end benchmark was run against Anthony's own hosts. If speed becomes a real complaint, measure the +existing server before changing clients. + +## Related +- [[systems/hermes-agent]] — the estate's own VNC/Xtigervnc setup lives under [[systems/headless-server]] +- [[log]] — audit entries diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index ba36e3d..809315a 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,8 +3,8 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-10-11 -verified_on: 2026-10-11 +updated: 2026-10-12 +verified_on: 2026-10-12 confidence: high tags: [memory, mnemosyne, backend] sources: [live-probe, mnemosyne-db, config, ops-profile-config] @@ -51,6 +51,10 @@ Primary agent memory store for Hermes. **Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]]. +**Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check. + +**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST. + ### Vector embedding coverage gap — RESOLVED (2026-08-02) - Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories. - **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 65ef39f..acc47bc 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,8 +3,8 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-11 -verified_on: 2026-10-11 +updated: 2026-10-12 +verified_on: 2026-10-12 review_after: 2026-10-25 confidence: high tags: [cron, scheduled, automation] @@ -14,7 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-10-06) +- **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, 2026-10-12; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10) - **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. See [[incidents/2026-10-05-dead-model-pin-repin]]. - Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04). - **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window. @@ -25,7 +25,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 `completed`, 0 `failed`, 1 `running` (the audit itself); 14 sessions, all `cron` — **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`). - **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice. - **Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00):** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — every `failure_streak` reads 0. Window tally (`executions.db`): **176 `completed`, 1 `failed`, 1 `running`** (the audit itself). The single failure is `62619aa6bcb3` (DHL Tracking, 08:00) — a Camofox HTTP read timeout (`ensure_browser()` → `TimeoutError: timed out`), notified 08:00:58 and **auto-resolved 11:01:13** when the next 3-hourly run succeeded (incident `62619a_f485ccfd4a0d`). 13 sessions started in the window, all `source = cron` — **no interactive sessions**; 12 deliveries, **0 delivery failures**. The same three stale pre-repin `last_status: error` jobs remain (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`); the stocktake job `1a8e90ea2271` ran on schedule at 03:00 for the second consecutive day. -- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08) +- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08) → **29, 0 failing** (2026-10-10) → **29, 0 failing** (2026-10-11) ## Job Table @@ -214,3 +214,47 @@ the absence of job output rather than a positive lookup. Removals must be verifi off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the 15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]). + +--- + +## Re-verified 2026-10-12 (audit window 2026-10-11 00:00 → 2026-10-12 00:2x AWST; live probes 00:0x-00:2x) + +**Fleet:** **29 jobs — 26 enabled / 3 disabled** (`jobs.json` `updated_at` 2026-10-12T00:10:42+08:00). Disabled +set unchanged: `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher), +`0a7f933d06e2` (AgentMail Rhino Inbox Poll — legacy). + +**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` — 10-08 `178 / 2` — +10-09 `163 / 15` — 10-10 **`175 completed / 0 failed`** (finalised; the 10-11 audit saw 174 + 1 still +`running`) — 10-11 **`171 completed / 0 failed`** + 1 `running` (this audit). **Two consecutive +zero-failure days** after the 2026-10-09 provider-timeout storm. Heaviest jobs on 10-11: runtime-deps guard +`96eb95ce12f2` 96 runs (15-min cadence), `0c771463c3cd` 46, `2fd50c87b0d0` 12, coverage monitor +`b63ed5817212` 3, health monitor `ea79a2107e36` 2, stocktake `1a8e90ea2271` 1. + +**Non-clean `last_status`: 5 jobs — unchanged from 2026-10-10.** `0e753e9720fd` (`error`, streak 0), +`fad5a86979d4` (`error`, streak 0), `dca8482e4f76` (`error`, streak 1), `1b23c042e17d` (`error`, streak 1), +`066bb5191280` (`delivery_failed`, streak 0). The two streak-1 jobs are the Mon-Fri and Friday-only schedulers: +their `last_error` is `RuntimeError: Request timed out.` from **Friday 2026-10-09** (the storm) and neither has +had a scheduled slot since — `dca8482e4f76` next runs 07:00 Mon, `1b23c042e17d` Fri 08:00, so both should +clear themselves. Everything else reads `last_status: ok` (this build reports `ok`, not `success`). + +**Incidents: 0 opened / 0 closed in the window — 5 unresolved, unchanged.** `alerted` 3: +`fad5a8_c80b59a907f0` (weekly registry sync, scope `agent`, first seen 2026-09-30, `deepseek-v4-flash` HTTP 400), +`dca848_23fcd2b18aaa` + `1b23c0_c35703f637b8` (`timeout`, both first seen 2026-10-09 — the last two left from +the storm). `detected` 2: `fad5a8_a13bbc837e1c` (2026-09-28, every provider in the fallback chain exhausted), +`0e753e_e7f8194a7d06` (2026-10-01, same `deepseek-v4-flash` 400). Table totals: alerted 3 / detected 2 / +closed 8 / resolved 63. Last closures remain the seven on 2026-10-10. + +**Sessions in the window:** 18 — 17 `cron` + 1 interactive (`talaria-ios`, 2026-10-11 19:13 → 19:51, +41 messages, "best VNC clients" research). The 03:00 stocktake `1a8e90ea2271` ran clean on schedule +(`completed` 03:00:40 → 03:11:29) and committed `d981e30`. + +**False alarm — the 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION was a timezone mislabel, not a +capture stall.** The run compared a naive-UTC `working_memory.created_at` (`2026-10-10 13:24:53`) against AWST +`now` and reported "10.7 h of no captures"; the true gap was 2.7 h (the row landed 21:24:53 AWST, inside the +session that produced it). No pipeline fault. The same job's 12:05 and 2026-10-12 00:02 runs returned +`[SILENT]` (green). → [[systems/mnemosyne]] + +**Still open (unchanged):** Quartz rebuild step inert (`wiki-maintain.sh` step 3 — no `quartz.config.ts`); +skills-index off-by-one (347 `SKILL.md` on disk vs 346 reported); the 5 non-clean `last_status` rows; +Mattermost platform unreachable ([[systems/messaging-integrations]]); gateway runtime env outside the +15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).