audit(2026-10-19): degenerate-stream incident page + push-notifications page; refresh scheduled-tasks, current-state, index, log, messaging, mnemosyne, tailscale, proxmox

This commit is contained in:
Tony0410
2026-10-19 00:21:26 +08:00
parent dd53e5fbe1
commit 0ed522bfb8
10 changed files with 312 additions and 24 deletions
+22 -2
View File
@@ -3,8 +3,8 @@ title: Messaging Integrations
type: system
status: active
created: 2026-07-22
updated: 2026-10-11
verified_on: 2026-10-11
updated: 2026-10-19
verified_on: 2026-10-19
confidence: high
tags: [messaging, telegram, discord, email, integrations]
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
@@ -93,3 +93,23 @@ not merely an application-level refusal.
**Open item:** re-probe the host; if the host is back but the adapter stays down, restart the
gateway's Mattermost platform.
## 2026-10-19 00:15 AWST — Mattermost adapter: cause identified (CT255 stopped), still down
This pass replaced the 10-11 observation's "cause not determined" with a definite cause. Live probes,
2026-10-19 00:0x-00:15 AWST:
- `tailscale status`: peer `mattermost` (`100.98.116.14`) — **offline, last seen 29d ago**, `tx 7176 rx 0`.
- `gateway.log`: the reconnect loop is unchanged — `Reconnecting mattermost (attempt 470)` at 00:08:36
and `(attempt 471)` at 00:14:07, each followed by `Mattermost: disconnected` and
`Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s`
(5-minute cadence; 3,992 such lines in the retained `gateway.log`).
- Probed from CT460 at the configured endpoint (`MATTERMOST_URL=http://100.98.116.14:8065/`):
**TCP 8065 refused/closed and 100 % ICMP loss** — the host is unreachable, not the application.
- **Cause: CT255 (`mattermost`) is `stopped` on the PVE host** (see [[infrastructure/proxmox]]).
The attempt counter restarted at the gateway restart of 2026-10-17 05:10 and now reads 471.
- Impact unchanged: messaging path only. No cron job delivers via Mattermost, and the cron fleet
performed normally throughout.
**Open item:** start CT255, or retire the platform block. The 5-minute reconnect loop continues until
one of those happens — restarting the adapter while the host is stopped achieves nothing.
+4 -2
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system
status: active
created: 2026-07-22
updated: 2026-10-18
verified_on: 2026-10-18
updated: 2026-10-19
verified_on: 2026-10-19
confidence: high
tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
@@ -49,6 +49,8 @@ Primary agent memory store for Hermes.
## Known Issues
**Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below.
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]].
**Tier counts (live, 2026-10-12 12:02 — health job `ea79a2107e36`, green):** working **760/786 embedded (96.7%)**, episodic **1050/1056 (99.4%)** — both above the 95% floor, no `reindex` needed (`b63ed5817212` continues to backfill). Orphan embeddings **0**. FTS5 exactly aligned: `fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876. **0 unconsolidated** working rows; **1 unresolved conflict** (threshold 10); last consolidation 0.3 h earlier; 2,252 memories total (786 working / 1,056 episodic / 1 triple); disk 15 GB free (81% used). Interpretation: coverage is *not* the live issue — ranking relevance is (see [[incidents/2026-10-09-memory-recall-relevance]]).
+96
View File
@@ -0,0 +1,96 @@
---
title: Push Notifications (Cadu / Vory bridge)
type: system
status: degraded
created: 2026-10-19
updated: 2026-10-19
verified_on: 2026-10-19
review_after: 2026-10-26
confidence: medium
tags: [push, notifications, plugins, gateway, integrations]
sources: [live-probe, live-config, live-push-dir, live-logs]
---
# Push Notifications (Cadu / Vory bridge)
## Purpose
Delivers Hermes activity to Anthony's phone (APNs push) and carries remote
answer/approval traffic back to the gateway. Two plugin layers plus a standalone bridge script.
## Components
| Piece | Where | Notes |
|---|---|---|
| `hermes-push` | plugin (enabled) | Cadu push layer — device delivery |
| `cadu-device`, `cadu-mail`, `cadu-rich-cards`, `cadu-secrets-vault` | plugins (enabled) | Cadu device/mail/card/vault surfaces |
| `vory-push` | plugin (enabled) | runs the push bridge alongside the gateway |
| `hermes_push.py` | `~/.hermes/push/hermes_push.py` (91 KB, **v1.0.37**, sha256 `59cabfcc…`) | the bridge itself; connects to the gateway's `/api/ws` as an ordinary client |
| `install.sh` | `~/.hermes/push/install.sh` | installer (Vory) |
| `hermes-push.conf` | `~/.hermes/push/hermes-push.conf` | written by Vory, read by the bridge via `HERMES_PUSH_CONFIG` |
`hermes-push.conf` holds `HERMES_PUSH_GATEWAY_URL`, `HERMES_PUSH_PUBLIC_URL`,
`HERMES_PUSH_GATEWAY_BEARER`, `HERMES_PUSH_GATEWAY_REFRESH_TOKEN` and `HERMES_PUSH_DEVICES_DIR`.
**The two tokens are JWT-shaped secrets and are deliberately not recorded here.** State dirs:
`devices/`, `goals/`, `origins/`, plus `status.json` and `relay.lock`.
Config (live `config.yaml`): `plugins.enabled` lists `cadu-device`, `cadu-mail`, `cadu-rich-cards`, `cadu-secrets-vault`, `hermes-push`, `vory-push` (alongside `platforms/mattermost`, `homeassistant`, `hermes-browser-stream`, `a2a-platform`); under `platforms:`, `push.gateway_restart_notification: false` and `relay.enabled: true`.
## Status file — how to read it
`~/.hermes/push/status.json` is rewritten every retry cycle. Fields seen live:
`version`, `script_sha256`, `pid`, `updated_at`, `profile`, `connected`, `gateway`, `transport`, `error`.
## Current state (live, 2026-10-19 00:18 AWST) — BROKEN, and it has been for a while
```json
{"version": "1.0.37", "pid": 2496376, "profile": "default", "connected": false,
"gateway": "https://hermes.kangaroo-eel.ts.net", "transport": "",
"error": "https://hermes.kangaroo-eel.ts.net: connection refused (nothing listens there on this machine)"}
```
- **Cause (verified):** `hermes.kangaroo-eel.ts.net` resolves **on this host to `127.0.1.1`**
(`getent hosts` → `127.0.1.1 hermes.kangaroo-eel.ts.net hermes`, from the host's `/etc/hosts`
convention), so the bridge dials loopback port 443 — where nothing listens. `/etc/hosts` wins over
the tailnet DNS name.
- **The gateway and the Serve route are both healthy** — this is a resolution/dial-target problem,
not a gateway outage: the gateway listens on `0.0.0.0:9119` (`curl http://127.0.0.1:9119/` → **302**),
`tailscale serve status` still shows `https://hermes.kangaroo-eel.ts.net (tailnet only)` with
`/ → http://127.0.0.1:9119`, and the public name is served on Funnel `:8443`.
- **Chronic, not a blip:** the `connection refused (nothing listens there on this machine)` string
appears **23,469** times across the retained log set (`agent.log`, `agent.log.1-3`,
`errors.log`, `errors.log.1-2`). `relay.lock` was last touched 2026-10-19 00:03.
- **Impact:** phone push delivery and any remote answer/approve path through the bridge are down.
Nothing else observably depends on it — no cron job's delivery uses it.
### Fix options (diagnosed, NOT applied)
1. Point `HERMES_PUSH_GATEWAY_URL` at an address that actually answers from this host —
`http://127.0.0.1:9119` (the gateway's own listener) or the tailnet IP
(`100.118.5.51:443`, where `tailscaled` serves the name) — instead of the FQDN that resolves to
`127.0.1.1`.
2. Or give the FQDN a resolvable answer on this host: remove/qualify the `/etc/hosts` line that maps
`hermes`/`hermes.kangaroo-eel.ts.net` to `127.0.1.1`, letting tailnet DNS resolve it.
3. Either way, confirm WSS/WS scheme matches what the bridge expects before changing the value.
**Not yet traced in this pass:** which process launches the bridge and under which unit/service, and
whether `HERMES_PUSH_PUBLIC_URL` also embeds the broken FQDN.
## Relay contention (benign, recurring)
`relay.lock` serialises bridge instances. The log line
```
vory-push relay running in another Hermes process; waiting for it to stop
```
appears when a second process contends for the relay (last recorded 2026-10-15; `relay.lock`
touched 2026-10-19 00:03). It resolves without intervention.
## Related
- [[systems/messaging-integrations]]
- [[infrastructure/tailscale]]
- [[systems/hermes-agent]]
- [[current-state]]
No secrets recorded; key names, hashes, ports, status codes and paths only.
+7 -6
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-18
verified_on: 2026-10-18
updated: 2026-10-19
verified_on: 2026-10-19
review_after: 2026-10-25
confidence: high
tags: [cron, scheduled, automation]
@@ -14,10 +14,11 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **✅ Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **✅ Current as of 2026-10-19 00:20 AWST (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
- **Total jobs:** **28 — 25 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-18 00:31 AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs.
- **Non-clean `last_status`:** **8 as of 2026-10-18 00:31** — `1b23c042e17d` (`error`, streak 2), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`, streak 0). History: **3** at the 2026-10-12 audit (when `fad5a86979d4` and `dca8482e4f76` cleared on schedule) → **12** at the 2026-10-16 storm peak → 8 now. The six jobs that fixed themselves did so on their next scheduled slot; the remaining weekly ones (`1b23c042e17d` Fri, `0e519874d280` Mon, `06a0a1c8704f` + `c8d52904e1d6` Thu) cannot clear before their next run.
- **Total jobs:** **28 — 25 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-19 00:20 AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs.
- **Non-clean `last_status`:** **7 as of 2026-10-19 00:20** — `1b23c042e17d` (`error`, streak 2), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `ecc1fc13655a` (`error`, streak 1 — degenerate stream, 10-18 12:08), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`, streak 0). `cbfe434a3b5e` and `1a8e90ea2271` cleared on schedule (10-18 00:42:54 and 03:01:22). History: **3** at the 2026-10-12 audit (when `fad5a86979d4` and `dca8482e4f76` cleared on schedule) → **12** at the 2026-10-16 storm peak → 8 at the 2026-10-18 audit → **7 now**. The six jobs that fixed themselves did so on their next scheduled slot; the remaining weekly ones (`1b23c042e17d` Fri, `0e519874d280` Mon, `06a0a1c8704f` + `c8d52904e1d6` Thu) cannot clear before their next run.
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]].
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window.
@@ -45,7 +46,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic |
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). Live `last_status: ok`, streak 0. ⚠ Bare `telegram` target should move to a group topic |
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed the incident. Live `last_status: ok` |
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed that incident. ⚠ **Then 2026-10-18 12:00 failed differently** — a model repetition loop aborted the run at 12:08:25 **after** it had delivered a canned "Your request was not processed…" line to the topic (12:08:11) → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Live `last_status: error`, `failure_streak: 1`, next slot **2026-10-19 12:00** |
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. `last_status: ok` because the 401 is inside the script, not the cron run. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** |
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); cleared by the 10-17 08:31 run, which auto-closed the incident. Live `last_status: ok` |
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |