audit(2026-10-12): cron fleet clean 2nd day (10-11 171/0, 10-10 finalised 175/0); UTC-vs-AWST false 'capture stalled' alarm corrected; proxmox storage table + host mounts; queries/retained-research-answers stub filled (VNC); capture-pipeline incident -> superseded; index stale labels fixed

This commit is contained in:
Tony0410
2026-10-12 00:14:11 +08:00
parent d981e30128
commit 6fc109e546
8 changed files with 205 additions and 29 deletions
+7 -6
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-11
> model routing and automation. Last updated: 2026-10-12
## Start here
- [[current-state]] — Current systems status
@@ -29,11 +29,11 @@
### Integrations
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **5 carry a non-clean `last_status`** as of 2026-10-11 00:14 (down from 10). The 2026-10-10 fleet ran **clean — 174 `completed`, 0 `failed`** — ending the 2026-10-09 provider timeout storm (15 failed executions / 10 jobs; 7 of its incidents auto-closed during 2026-10-10, 2 remain open). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **no regeneration committed since 2026-10-07** — the 10-08 run's output was left uncommitted and the 10-09 run failed, see [[systems/scheduled-tasks]])
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **5 carry a non-clean `last_status`** as of 2026-10-12 00:10. **Two consecutive clean days: 2026-10-11 ran `171 completed` / 0 `failed`** (2026-10-10 finalised `175 / 0`), ending the 2026-10-09 provider timeout storm (15 failed executions / 10 jobs; 7 of its incidents auto-closed on 2026-10-10, 2 of its jobs remain `alerted`). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **regenerated daily — the 10-10 and 10-11 stocktake runs both committed (`d981e30`)**)
## Incidents
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; 8 incidents still open (unresolved)
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; 7 of its incidents auto-closed on 2026-10-10, **2 remain `alerted`** (`dca8482e4f76`, `1b23c042e17d` — both weekly jobs still carrying the Friday `RuntimeError: Request timed out.`)
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)
- [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05)
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
@@ -41,7 +41,7 @@
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); **closed 2026-10-05** when all three were repinned off the pin
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. **Recurred 2026-10-10 inside the *gateway* runtime env** — `mnemosyne` and `elevenlabs` missing from the committed uv env the gateway loads; both repaired and verified the same evening
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (Aug 2026, **superseded**): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded
- [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
### Monitoring & Dashboards
@@ -52,6 +52,7 @@
- [[reports/uber-eats/index]] — Uber Eats monthly spend reports
- [[reports/communities-wa-social-media-audit-2026]] — Department of Communities WA social media audit and first-day brief
- [[systems/obsidian]] — Obsidian vault structure and conventions
- [[queries/retained-research-answers]] — Retained research answers (fastest VNC clients, 2026-10-11)
## Infrastructure
### Services
@@ -63,7 +64,7 @@
- [[infrastructure/backups]] — hermes backup CLI, retention, and gaps
### Networking
- [[infrastructure/proxmox]] — PVE host, VMs, LXC containers
- [[infrastructure/proxmox]] — PVE host, VMs, LXC containers, storage datastores (`/mnt/ssd0` 93 % / ~63 GiB free, 2026-10-12)
- [[infrastructure/tailscale]] — Tailscale serve routes and VPN
- [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly