audit(2026-10-19): degenerate-stream incident page + push-notifications page; refresh scheduled-tasks, current-state, index, log, messaging, mnemosyne, tailscale, proxmox

This commit is contained in:
Tony0410
2026-10-19 00:21:26 +08:00
parent dd53e5fbe1
commit 0ed522bfb8
10 changed files with 312 additions and 24 deletions
+7 -5
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-18
> model routing and automation. Last updated: 2026-10-19
>
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
@@ -32,11 +32,13 @@
- [[systems/tool-search]] — Always-on lazy discovery for external tools
### Integrations
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
- [[systems/scheduled-tasks]] — **28 cron jobs (25 enabled, 3 disabled)**; **8 carry a non-clean `last_status`** as of 2026-10-18 00:3x — down from 12. The **`omniroute-1` 503 resource-pressure window (2026-10-16 08:00 → 10-17 03:00)** cost **9 model-call failures across 7 jobs** and was followed by a clean 10-17 06:00 → 10-18 window; the Mosquito Tracker was removed (29 → 28 jobs). See [[incidents/2026-10-16-omniroute-503-resource-pressure]]. Prior incident: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs** and ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. ⚠ **2026-10-17** stocktake slot failed (`HTTP 503`), so `freshness-20261017.txt` is missing and the index is stamped **10-16**; the **10-18 03:00** slot closes it. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP).
- [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip.
- [[systems/scheduled-tasks]] — **28 cron jobs (25 enabled, 3 disabled)**; **7 carry a non-clean `last_status`** as of 2026-10-19 00:2x — down from 12. The **2026-10-18 window was clean except one run**: `ecc1fc13655a` (FreshRSS Midday) delivered a canned non-answer to Telegram and then died on a model repetition loop → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. The **`omniroute-1` 503 resource-pressure window (2026-10-16 08:00 → 10-17 03:00)** cost **9 model-call failures across 7 jobs** and was followed by a clean 10-17 06:00 → 10-18 window; the Mosquito Tracker was removed (29 → 28 jobs). See [[incidents/2026-10-16-omniroute-503-resource-pressure]]. Prior incident: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs** and ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. The **2026-10-17** slot failed (`HTTP 503`), but the **10-18 03:00** slot closed the hole: `freshness-20261018.txt` written, index regenerated and committed (`dd53e5f`) — freshness **47 stale / 41 fresh / 7 no-date**. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
## Incidents
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route?
- [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **OmniRoute's second failure mode, 2026-10-16 → 10-17:** `omniroute-1` load-shed model calls with **`HTTP 503 … resource pressure`** — **9 model calls across 7 jobs** over ~19 h, while `/api/health` stayed `200 ok` and `/v1/models` stayed `401`. **Self-cleared** (last 503 10-17 03:00:28; 10-17 closed 162 / 5, 10-18 so far 7 / 0). 9 incidents opened, 6 closed; **8 open** overall, and the ledger tags **7 of the 9 `failure_type: unknown`**. **Single-node topology still open.**
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved)
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)
@@ -71,7 +73,7 @@
### Networking
- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 8.x + `m93p` 9.2.0, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 88.17 % / ~63 GiB free, 2026-10-14)
- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **45-node** tailnet inventory (2026-10-14)
- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **48-node** tailnet inventory (re-verified 2026-10-19: `guest-vps`, `openmausbot`, `openmausbot-vm` added; `mattermost` offline 29d)
- [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly
## Concepts