audit(2026-10-12): cron fleet clean 2nd day (10-11 171/0, 10-10 finalised 175/0); UTC-vs-AWST false 'capture stalled' alarm corrected; proxmox storage table + host mounts; queries/retained-research-answers stub filled (VNC); capture-pipeline incident -> superseded; index stale labels fixed
This commit is contained in:
+12
-3
@@ -3,8 +3,8 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-10-12
|
||||
verified_on: 2026-10-12
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [system]
|
||||
@@ -12,7 +12,16 @@ sources: [live-config, live-cron-list, live-hermes-version]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-11
|
||||
Updated: 2026-10-12
|
||||
|
||||
## Recent Changes (2026-10-11 → 2026-10-12, nightly audit; live probes 2026-10-12 00:0x-00:2x AWST)
|
||||
|
||||
- **The cron fleet is clean for a second consecutive day: 2026-10-11 ran `171 completed / 0 failed`** (+1 `running` = this audit), after 10-07 `1` / 10-08 `2` / 10-09 `15` failed. 10-10 finalises at **`175 / 0`**. Fleet **29 jobs — 26 enabled / 3 disabled**; non-clean `last_status` steady at **5**; **0 incidents opened, 0 closed — 5 unresolved** (3 `alerted`, 2 `detected`). The two streak-1 jobs (`dca8482e4f76`, `1b23c042e17d`) carry Friday 2026-10-09 timeouts and have not had a scheduled slot since. → [[systems/scheduled-tasks]]
|
||||
- **The 2026-10-11 00:08 "memory capture stalled" EXCEPTION was a timezone mislabel, not a capture fault.** `working_memory.created_at` is naive UTC; the monitor compared it against AWST `now` and reported a 2.7 h gap as 10.7 h. Capture verified healthy: 786 rows, +2 in the window, coverage 98.3 % working / 99.6 % episodic. → [[systems/mnemosyne]]
|
||||
- **Proxmox storage documented for the first time:** `/mnt/ssd0` is the pressure point — 808 GiB used, **~63 GiB free (93 % by `df`, 88.1 % by `pvesm`)**; `SSD1` 86.1 %, `fred-qnap` 76.8 %, `iMacHDD` inactive. → [[infrastructure/proxmox]]
|
||||
- **First retained research answer filed:** fastest VNC clients (asked 2026-10-11) — the server-side encoder is the real bottleneck, TigerVNC native viewer for standard RFB, `-CompressLevel` is a *client* parameter, RealVNC's "4.7×" is self-referential. → [[queries/retained-research-answers]]
|
||||
- **Only one interactive session in the window** (`talaria-ios`, 19:13 → 19:51, 41 messages); the other 17 were `cron`. The 03:00 stocktake `1a8e90ea2271` completed on schedule (03:00:40 → 03:11:29) and committed `d981e30`.
|
||||
- **Stale labels corrected:** [[incidents/2026-08-18-mnemosyne-capture-pipeline]] moved from `active` to `superseded` (evidence above); the index's count of open 2026-10-09 storm incidents trimmed to the 2 that are genuinely open.
|
||||
|
||||
## Recent Changes (2026-10-11, 03:00 stocktake; live probes 03:00-03:30 AWST)
|
||||
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
---
|
||||
title: "Mnemosyne Capture Pipeline Broken (Aug 18)"
|
||||
type: incident
|
||||
status: active
|
||||
status: superseded
|
||||
created: 2026-08-18
|
||||
updated: 2026-08-18
|
||||
verified_on: 2026-08-18
|
||||
updated: 2026-10-12
|
||||
verified_on: 2026-10-12
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, hermes, incident, capture]
|
||||
sources: [mnemosyne-db, health-monitor, run_agent.py]
|
||||
@@ -59,6 +59,19 @@ print('Last WM capture:', cur.fetchone()[0]) # Should be recent
|
||||
"
|
||||
```
|
||||
|
||||
## Current Status (verified 2026-10-12)
|
||||
|
||||
**Superseded — capture has been healthy since, and this page is kept for the diagnostic trail.** The 2026-10-12
|
||||
nightly audit found the full path working: `working_memory` at **786 rows**, +2 on 2026-10-11 (11:19:37 and
|
||||
11:51:39 UTC = 19:19 / 19:51 AWST, both from that day's interactive session), `episodic_memory` 1,056,
|
||||
embedding coverage **98.3 % working / 99.6 % episodic**, FTS aligned (`fts_working` 786, `fts_facts` 876).
|
||||
The three guard conditions diagnosed below were never individually confirmed as the fix — the path resumed
|
||||
capturing and has captured through every window since.
|
||||
|
||||
**Caution for the next alarm:** the health monitor raised a false EXCEPTION on 2026-10-11 00:08 by comparing
|
||||
naive-UTC `created_at` against AWST `now` ("stalled 10.7 h" — real gap 2.7 h). Do not triage that as a repeat
|
||||
of this incident; see [[systems/mnemosyne]].
|
||||
|
||||
## Prevention
|
||||
- Never use `hermes config set memory.mnemosyne.<field>` for list/dict/null values
|
||||
- After any config change, verify YAML types with `yaml.safe_load` + `type()` on each field
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. Last updated: 2026-10-11
|
||||
> model routing and automation. Last updated: 2026-10-12
|
||||
|
||||
## Start here
|
||||
- [[current-state]] — Current systems status
|
||||
@@ -29,11 +29,11 @@
|
||||
|
||||
### Integrations
|
||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
||||
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **5 carry a non-clean `last_status`** as of 2026-10-11 00:14 (down from 10). The 2026-10-10 fleet ran **clean — 174 `completed`, 0 `failed`** — ending the 2026-10-09 provider timeout storm (15 failed executions / 10 jobs; 7 of its incidents auto-closed during 2026-10-10, 2 remain open). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
|
||||
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **no regeneration committed since 2026-10-07** — the 10-08 run's output was left uncommitted and the 10-09 run failed, see [[systems/scheduled-tasks]])
|
||||
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **5 carry a non-clean `last_status`** as of 2026-10-12 00:10. **Two consecutive clean days: 2026-10-11 ran `171 completed` / 0 `failed`** (2026-10-10 finalised `175 / 0`), ending the 2026-10-09 provider timeout storm (15 failed executions / 10 jobs; 7 of its incidents auto-closed on 2026-10-10, 2 of its jobs remain `alerted`). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
|
||||
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **regenerated daily — the 10-10 and 10-11 stocktake runs both committed (`d981e30`)**)
|
||||
|
||||
## Incidents
|
||||
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; 8 incidents still open (unresolved)
|
||||
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; 7 of its incidents auto-closed on 2026-10-10, **2 remain `alerted`** (`dca8482e4f76`, `1b23c042e17d` — both weekly jobs still carrying the Friday `RuntimeError: Request timed out.`)
|
||||
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)
|
||||
- [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05)
|
||||
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
|
||||
@@ -41,7 +41,7 @@
|
||||
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); **closed 2026-10-05** when all three were repinned off the pin
|
||||
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
|
||||
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. **Recurred 2026-10-10 inside the *gateway* runtime env** — `mnemosyne` and `elevenlabs` missing from the committed uv env the gateway loads; both repaired and verified the same evening
|
||||
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured
|
||||
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (Aug 2026, **superseded**): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded
|
||||
- [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
|
||||
|
||||
### Monitoring & Dashboards
|
||||
@@ -52,6 +52,7 @@
|
||||
- [[reports/uber-eats/index]] — Uber Eats monthly spend reports
|
||||
- [[reports/communities-wa-social-media-audit-2026]] — Department of Communities WA social media audit and first-day brief
|
||||
- [[systems/obsidian]] — Obsidian vault structure and conventions
|
||||
- [[queries/retained-research-answers]] — Retained research answers (fastest VNC clients, 2026-10-11)
|
||||
|
||||
## Infrastructure
|
||||
### Services
|
||||
@@ -63,7 +64,7 @@
|
||||
- [[infrastructure/backups]] — hermes backup CLI, retention, and gaps
|
||||
|
||||
### Networking
|
||||
- [[infrastructure/proxmox]] — PVE host, VMs, LXC containers
|
||||
- [[infrastructure/proxmox]] — PVE host, VMs, LXC containers, storage datastores (`/mnt/ssd0` 93 % / ~63 GiB free, 2026-10-12)
|
||||
- [[infrastructure/tailscale]] — Tailscale serve routes and VPN
|
||||
- [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly
|
||||
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Proxmox VE
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-26
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-10-12
|
||||
verified_on: 2026-10-12
|
||||
confidence: high
|
||||
tags: [proxmox, virtualization, infrastructure, homelab]
|
||||
---
|
||||
@@ -61,6 +61,21 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
|
||||
|
||||
**Not in either list:** VM 403 (`open-claw`) and VM 401 (`hermes-old`) are referenced by the wiki but no longer appear in `qm list`. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry.
|
||||
|
||||
## Storage (live, 2026-10-12 00:1x AWST — `df -h /mnt/ssd0` + `pvesm status`)
|
||||
|
||||
| Datastore | Type | Status | Total | Used | Used % |
|
||||
|---|---|---|---|---|---|
|
||||
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 808 GiB | **93 %** (`df`) / 88.1 % (`pvesm`) |
|
||||
| `SSD1` | lvmthin | active | 931 GiB | 802 GiB | 86.1 % |
|
||||
| `fred-qnap` | cifs | active | 13.4 TiB | 10.3 TiB | 76.8 % |
|
||||
| `local` | dir | active | 67.7 GiB | 55.5 GiB | 82.0 % |
|
||||
| `local-lvm` | lvmthin | active | 141 GiB | 19.2 GiB | 13.6 % |
|
||||
| `iMacHDD` | cifs | **inactive** | — | — | — |
|
||||
|
||||
**`/mnt/ssd0` is the capacity pressure point: 93 % with only ~63 GiB free.** Both figures describe the same ~808 GiB used and differ only in denominator — `df` excludes reserved blocks (808/871 GiB), `pvesm` divides by the full 916 GiB (88.14 %). Treat the 93 % as the number that matters before adding a disk-heavy guest (VM 110 `omarchy` is ~106 GB on its own). `iMacHDD` is inactive (`storage 'iMacHDD' is not online`) — check the CIFS mount before using it as a target.
|
||||
|
||||
**Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`).
|
||||
|
||||
## API Tokens
|
||||
- `homepage@pve!homepage` — PVEAuditor role, privsep=0 (used by Homepage dashboard)
|
||||
|
||||
|
||||
@@ -472,3 +472,36 @@ No secrets written; job/incident IDs, paths and tailnet addresses only.
|
||||
- **Flagged for follow-up, root cause not investigated:** `mycompose-db-1` on CT200 is in `Restarting (1)` — crash-looping.
|
||||
- Mattermost CT255 confirmed stopped, matching its Tailscale peer offline 21d.
|
||||
- Left as stale-without-fix: 62 pages still older than 21 days (concepts, decisions, comparisons, most runbooks) — these were not probed against a live source this pass, so they were reported, not rewritten.
|
||||
|
||||
## [2026-10-12] wiki audit | window 2026-10-11 -> 2026-10-12 (ran 2026-10-11 23:59 -> 2026-10-12 00:2x AWST)
|
||||
|
||||
**Cron.** 29 jobs, 26 enabled / 3 disabled (`jobs.json` 2026-10-12T00:10). Day tally: 10-07 `177/1`,
|
||||
10-08 `178/2`, 10-09 `163/15`, 10-10 **`175 completed / 0 failed`** (finalised), 10-11 **`171 completed /
|
||||
0 failed`** + 1 `running` — **two consecutive clean days**. Non-clean `last_status` steady at 5;
|
||||
**0 incidents opened, 0 closed — 5 unresolved** (3 `alerted`, 2 `detected`).
|
||||
|
||||
**Corrected a false alarm instead of logging a fault.** The 2026-10-11 00:08 Mnemosyne Health Monitor
|
||||
EXCEPTION ("capture stalled 10.7 h") was a **naive-UTC vs AWST comparison error** — real gap 2.7 h, the row it
|
||||
cited landed inside the session that wrote it. Recorded in [[systems/mnemosyne]] and
|
||||
[[systems/scheduled-tasks]] so the next run of that monitor is not triaged as a capture outage. Live counts
|
||||
refreshed: 786 WM / 1,056 EP, coverage 98.3 % / 99.6 %.
|
||||
|
||||
**New content.** [[queries/retained-research-answers]] was a 2026-07-22 stub and is now a real page — the
|
||||
2026-10-11 VNC-client research (server-side encoder is the bottleneck; TigerVNC native viewer for standard RFB;
|
||||
`-CompressLevel` 2 is client-side; RealVNC's "4.7×" is self-referential) with its sources. Proxmox storage
|
||||
was undocumented; the live datastore table plus host mount points are now in [[infrastructure/proxmox]]
|
||||
(`/mnt/ssd0` 93 % / ~63 GiB free is the pressure point).
|
||||
|
||||
**Stale labels fixed.** [[incidents/2026-08-18-mnemosyne-capture-pipeline]] carried `status: active` from
|
||||
August despite capture being verifiably healthy — now `superseded`, with the live evidence and a caution about
|
||||
the UTC comparison. Index's "8 incidents still open" from the 2026-10-09 storm trimmed to the 2 genuinely open,
|
||||
and the skills-index blurb no longer claims no regeneration since 2026-10-07.
|
||||
|
||||
**Verified, not changed.** The `blocked-page-recovery` lesson that session reported patching
|
||||
(`recovered: false` = archive routes failed, not task failed; API-first rung) is present in the live skill.
|
||||
|
||||
**Pages touched:** `systems/scheduled-tasks.md`, `systems/mnemosyne.md`, `infrastructure/proxmox.md`,
|
||||
`queries/retained-research-answers.md`, `incidents/2026-08-18-mnemosyne-capture-pipeline.md`,
|
||||
`current-state.md`, `index.md`, `log.md`.
|
||||
|
||||
No secrets written; job/incident IDs, paths and tailnet addresses only.
|
||||
|
||||
@@ -1,16 +1,73 @@
|
||||
---
|
||||
title: Retained Research Answers
|
||||
type: query
|
||||
status: unresolved
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-07-22
|
||||
review_after: 2026-08-22
|
||||
confidence: low
|
||||
tags: [query, research]
|
||||
sources: []
|
||||
updated: 2026-10-12
|
||||
verified_on: 2026-10-12
|
||||
review_after: 2027-01-12
|
||||
confidence: medium
|
||||
tags: [query, research, vnc, remote-desktop]
|
||||
sources: [web-research, turboVNC-benchmark-pdf, superuser-15337, tigervnc-manpage, session-20261011]
|
||||
---
|
||||
|
||||
> **Stub — structure only.** Headings exist but content has not been verified/written yet. Flagged during 2026-07-22 wiki audit; fill in with verified facts or mark deprecated if no longer relevant.
|
||||
|
||||
# Retained Research Answers
|
||||
Filed research answers worth reusing.
|
||||
|
||||
Research answers worth keeping: what was asked, the answer, the evidence it rests on, and what the answer does
|
||||
**not** establish. Entries are point-in-time; re-verify anything load-bearing before acting on it.
|
||||
|
||||
> The 2026-07-22 stub was filled on 2026-10-12 with the first retained answer (asked in the 2026-10-11
|
||||
> interactive session `talaria-ios`, 41 messages).
|
||||
|
||||
## Fastest VNC client (asked 2026-10-11)
|
||||
|
||||
**Answer for the CT460/CT200 TigerVNC + Xtigervnc servers:** use the **TigerVNC native viewer** with
|
||||
AutoSelect and the defaults — free, and the fastest standard-RFB client. Lowest bandwidth is **KasmVNC**
|
||||
(WebP, ~30 % less) but that is a *server* swap, not a client. **RealVNC Connect** is not a faster decoder, it is
|
||||
a forked protocol (RFB 6, audio, no port-forwarding) and needs a RealVNC server. **TurboVNC** remains the
|
||||
reference for lossless 3D/CAD. Genuinely low latency means leaving RFB entirely: **Parsec** or
|
||||
**Sunshine + Moonlight**.
|
||||
|
||||
**The load-bearing finding: the server-side encoder is the bottleneck, not the client.** A viewer can only
|
||||
decode what it is sent. Two independent sources say the same thing — the TurboVNC benchmark report (encoder
|
||||
throughput measurements) and the accepted Super User answer (RFB ships pixel rectangles, so cost scales with
|
||||
resolution and bit depth).
|
||||
|
||||
**Practical levers, server side (in order of payoff):**
|
||||
1. **Compression level 1–2.** The single biggest lever; the 2012 report found compression got *worse* above
|
||||
level 6 while costing 4.4× the CPU.
|
||||
2. **16-bit colour** where banding is acceptable — halves the pixel payload.
|
||||
3. **Local cursor rendering** (`-UseLocalCursor`) — large perceived win on slow links.
|
||||
4. **Let AutoSelect work**; forcing `Tight` on a fast link or `Raw` on a slow link just wastes CPU/bandwidth.
|
||||
5. **Tunnel over SSH** — latency and loss usually dominate codec choice.
|
||||
6. `-FrameRate` (~5) is worth setting over the internet; if the requested rate exceeds what the link delivers
|
||||
you get visible catch-up lag.
|
||||
|
||||
**Client side:** `-CompressLevel` defaults to **2** and is a *client* parameter (0 = low, 9 = high) — do not
|
||||
confuse it with the server-side encoder setting from the 2012 report. TigerVNC's viewer has an in-tree
|
||||
**H.264 decoder** only when built with `ENABLE_H264=ON` (FFmpeg or WMF backend); server-side H.264 support is
|
||||
still the blocker. Decode speed depends heavily on libjpeg-turbo SIMD — client CPU matters as much as client
|
||||
software (a thin client with no SIMD headroom decodes slowly regardless of viewer).
|
||||
**noVNC is convenience, not speed** — the JS/Canvas path is pure overhead versus the native client.
|
||||
|
||||
**Evidence / sources:**
|
||||
- TurboVNC "TigerVNC vs TurboVNC" benchmark, primary source (2012 data on 2012 CPUs):
|
||||
https://turbovnc.org/pmwiki/uploads/About/turbototiger.pdf — after the TigerVNC 1.2 encoder rework the two
|
||||
matched within ~2 % at compression level 1, and TigerVNC led by up to 16 % on indexed-colour workloads.
|
||||
- Super User question 15337 (recovered via the **Stack Exchange API**, no browser needed): RFB transmits pixel
|
||||
rectangles while RDP/X send drawing instructions — which is why VNC only wins at very low bitrates while
|
||||
remaining the most portable option.
|
||||
- TigerVNC man page, pulled from GitHub master because **`tigervnc.com` is dead** (domain parked at Gandi):
|
||||
`-CompressLevel` default 2, client-side.
|
||||
- RealVNC performance / viewer-comparison pages (recovered via Camofox): the "4.7× faster video / 4.5× faster
|
||||
CAD" claim is **RealVNC against the previous RealVNC** at 37 Mbps / 33 ms / 0.3 % loss — not a comparison
|
||||
against TigerVNC or TurboVNC. Their differentiator is proprietary RFB 6.
|
||||
- TigerVNC issue #1187 — H.264 server support status.
|
||||
|
||||
**Does NOT establish:** exact throughput multipliers on current hardware (the benchmark is 2012); no
|
||||
end-to-end benchmark was run against Anthony's own hosts. If speed becomes a real complaint, measure the
|
||||
existing server before changing clients.
|
||||
|
||||
## Related
|
||||
- [[systems/hermes-agent]] — the estate's own VNC/Xtigervnc setup lives under [[systems/headless-server]]
|
||||
- [[log]] — audit entries
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Mnemosyne
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-10-12
|
||||
verified_on: 2026-10-12
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, backend]
|
||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||||
@@ -51,6 +51,10 @@ Primary agent memory store for Hermes.
|
||||
|
||||
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]].
|
||||
|
||||
**Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check.
|
||||
|
||||
**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST.
|
||||
|
||||
### Vector embedding coverage gap — RESOLVED (2026-08-02)
|
||||
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
|
||||
- **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-10-12
|
||||
verified_on: 2026-10-12
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
@@ -14,7 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-10-06)
|
||||
- **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, 2026-10-12; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10)
|
||||
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. See [[incidents/2026-10-05-dead-model-pin-repin]].
|
||||
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
|
||||
- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window.
|
||||
@@ -25,7 +25,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
- **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 `completed`, 0 `failed`, 1 `running` (the audit itself); 14 sessions, all `cron` — **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`).
|
||||
- **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice.
|
||||
- **Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00):** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — every `failure_streak` reads 0. Window tally (`executions.db`): **176 `completed`, 1 `failed`, 1 `running`** (the audit itself). The single failure is `62619aa6bcb3` (DHL Tracking, 08:00) — a Camofox HTTP read timeout (`ensure_browser()` → `TimeoutError: timed out`), notified 08:00:58 and **auto-resolved 11:01:13** when the next 3-hourly run succeeded (incident `62619a_f485ccfd4a0d`). 13 sessions started in the window, all `source = cron` — **no interactive sessions**; 12 deliveries, **0 delivery failures**. The same three stale pre-repin `last_status: error` jobs remain (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`); the stocktake job `1a8e90ea2271` ran on schedule at 03:00 for the second consecutive day.
|
||||
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08)
|
||||
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08) → **29, 0 failing** (2026-10-10) → **29, 0 failing** (2026-10-11)
|
||||
|
||||
## Job Table
|
||||
|
||||
@@ -214,3 +214,47 @@ the absence of job output rather than a positive lookup. Removals must be verifi
|
||||
off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost
|
||||
platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the
|
||||
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).
|
||||
|
||||
---
|
||||
|
||||
## Re-verified 2026-10-12 (audit window 2026-10-11 00:00 → 2026-10-12 00:2x AWST; live probes 00:0x-00:2x)
|
||||
|
||||
**Fleet:** **29 jobs — 26 enabled / 3 disabled** (`jobs.json` `updated_at` 2026-10-12T00:10:42+08:00). Disabled
|
||||
set unchanged: `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher),
|
||||
`0a7f933d06e2` (AgentMail Rhino Inbox Poll — legacy).
|
||||
|
||||
**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` — 10-08 `178 / 2` —
|
||||
10-09 `163 / 15` — 10-10 **`175 completed / 0 failed`** (finalised; the 10-11 audit saw 174 + 1 still
|
||||
`running`) — 10-11 **`171 completed / 0 failed`** + 1 `running` (this audit). **Two consecutive
|
||||
zero-failure days** after the 2026-10-09 provider-timeout storm. Heaviest jobs on 10-11: runtime-deps guard
|
||||
`96eb95ce12f2` 96 runs (15-min cadence), `0c771463c3cd` 46, `2fd50c87b0d0` 12, coverage monitor
|
||||
`b63ed5817212` 3, health monitor `ea79a2107e36` 2, stocktake `1a8e90ea2271` 1.
|
||||
|
||||
**Non-clean `last_status`: 5 jobs — unchanged from 2026-10-10.** `0e753e9720fd` (`error`, streak 0),
|
||||
`fad5a86979d4` (`error`, streak 0), `dca8482e4f76` (`error`, streak 1), `1b23c042e17d` (`error`, streak 1),
|
||||
`066bb5191280` (`delivery_failed`, streak 0). The two streak-1 jobs are the Mon-Fri and Friday-only schedulers:
|
||||
their `last_error` is `RuntimeError: Request timed out.` from **Friday 2026-10-09** (the storm) and neither has
|
||||
had a scheduled slot since — `dca8482e4f76` next runs 07:00 Mon, `1b23c042e17d` Fri 08:00, so both should
|
||||
clear themselves. Everything else reads `last_status: ok` (this build reports `ok`, not `success`).
|
||||
|
||||
**Incidents: 0 opened / 0 closed in the window — 5 unresolved, unchanged.** `alerted` 3:
|
||||
`fad5a8_c80b59a907f0` (weekly registry sync, scope `agent`, first seen 2026-09-30, `deepseek-v4-flash` HTTP 400),
|
||||
`dca848_23fcd2b18aaa` + `1b23c0_c35703f637b8` (`timeout`, both first seen 2026-10-09 — the last two left from
|
||||
the storm). `detected` 2: `fad5a8_a13bbc837e1c` (2026-09-28, every provider in the fallback chain exhausted),
|
||||
`0e753e_e7f8194a7d06` (2026-10-01, same `deepseek-v4-flash` 400). Table totals: alerted 3 / detected 2 /
|
||||
closed 8 / resolved 63. Last closures remain the seven on 2026-10-10.
|
||||
|
||||
**Sessions in the window:** 18 — 17 `cron` + 1 interactive (`talaria-ios`, 2026-10-11 19:13 → 19:51,
|
||||
41 messages, "best VNC clients" research). The 03:00 stocktake `1a8e90ea2271` ran clean on schedule
|
||||
(`completed` 03:00:40 → 03:11:29) and committed `d981e30`.
|
||||
|
||||
**False alarm — the 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION was a timezone mislabel, not a
|
||||
capture stall.** The run compared a naive-UTC `working_memory.created_at` (`2026-10-10 13:24:53`) against AWST
|
||||
`now` and reported "10.7 h of no captures"; the true gap was 2.7 h (the row landed 21:24:53 AWST, inside the
|
||||
session that produced it). No pipeline fault. The same job's 12:05 and 2026-10-12 00:02 runs returned
|
||||
`[SILENT]` (green). → [[systems/mnemosyne]]
|
||||
|
||||
**Still open (unchanged):** Quartz rebuild step inert (`wiki-maintain.sh` step 3 — no `quartz.config.ts`);
|
||||
skills-index off-by-one (347 `SKILL.md` on disk vs 346 reported); the 5 non-clean `last_status` rows;
|
||||
Mattermost platform unreachable ([[systems/messaging-integrations]]); gateway runtime env outside the
|
||||
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).
|
||||
|
||||
Reference in New Issue
Block a user