audit(2026-10-19): degenerate-stream incident page + push-notifications page; refresh scheduled-tasks, current-state, index, log, messaging, mnemosyne, tailscale, proxmox
This commit is contained in:
+37
-3
@@ -3,8 +3,8 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-18
|
||||
verified_on: 2026-10-18
|
||||
updated: 2026-10-19
|
||||
verified_on: 2026-10-19
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [system]
|
||||
@@ -12,7 +12,41 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-18
|
||||
Updated: 2026-10-19
|
||||
|
||||
## Recent Changes (2026-10-19, nightly audit; live probes 2026-10-19 00:04-00:2x AWST)
|
||||
|
||||
- **A model stream started repeating itself — and Hermes delivered the placeholder to Telegram before it
|
||||
aborted.** The FreshRSS Midday slot (`ecc1fc13655a`, 12:00) ended `failed` at 12:08:25 with
|
||||
`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the
|
||||
degenerate response.` The run had already written and **delivered** a canned line — *"Your request was
|
||||
not processed. Send it again if you still want me to carry it out."* — at 12:08:11
|
||||
(`delivery_outcome: delivered`), so the digest topic received a non-answer that does not read as an
|
||||
error. **A delivered message is not evidence of delivered content.** Open question: should the guard
|
||||
suppress delivery or re-queue, instead of substituting text on the job's normal route?
|
||||
→ [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
||||
- **Window tally: `10-18 = 168 completed / 1 failed`** — that row is the only failure of the day;
|
||||
`10-17 = 163 / 5`, and `10-19` so far `5 / 0` plus this audit. **Non-clean `last_status` 8 → 7**;
|
||||
**open incidents 8 → 7** (6 `alerted` / 1 `detected`), with **1 opened and 2 resolved** in-window
|
||||
(`cbfe434a3b5e` 00:42:54, `1a8e90ea2271` 03:01:22 — both auto-closed by their own next clean slot).
|
||||
**11 sessions in the window, all `cron`; no interactive sessions.** → [[systems/scheduled-tasks]]
|
||||
- **The 10-18 03:00 stocktake landed and closed the two-day gap** the last audit flagged: `1a8e90ea2271`
|
||||
exited 0, `raw/stocktake/freshness-20261018.txt` exists, [[systems/skills-index]] was regenerated
|
||||
(346 skills / 70 categories; freshness **47 stale / 41 fresh / 7 no-date**) and committed (`dd53e5f`).
|
||||
- **Mattermost's cause is finally known: CT255 is stopped.** The adapter's 5-minute reconnect loop is
|
||||
unchanged (`attempt 471` at 00:14:07; `mattermost connect timed out after 30s`), the peer is offline
|
||||
(last seen **29d**), and the host answers neither TCP `:8065` nor ICMP from CT460. The 10-11 entry's
|
||||
"cause not determined" is now superseded. → [[systems/messaging-integrations]]
|
||||
- **New page: [[systems/push-notifications]] — and the push bridge is broken, and has been for a long
|
||||
time.** `status.json` says `connected: false` with `connection refused (nothing listens there on this
|
||||
machine)`: the bridge dials `HERMES_PUSH_GATEWAY_URL=https://hermes.kangaroo-eel.ts.net`, which resolves
|
||||
on this host to **`127.0.1.1`** via `/etc/hosts` — where nothing serves 443 — while the gateway itself is
|
||||
healthy (`0.0.0.0:9119`, `/` → 302) and the tailnet Serve route is intact. The error string appears
|
||||
**23,469** times in the retained logs. Fix not applied (change the dial target, or the name resolution).
|
||||
→ [[systems/push-notifications]]
|
||||
- **One more standing alert, unchanged: the GitHub Trending Radar delivery token is still dead**
|
||||
(`getMe` → 401, re-verified 10-18), and its next exercise is **Mon 2026-10-19 09:00**. Not repaired — a
|
||||
replacement bot token is Anthony's call. → [[systems/scheduled-tasks]]
|
||||
|
||||
## Recent Changes (2026-10-18, nightly audit; live probes 2026-10-18 00:31-00:36 AWST)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user