audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd

Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown.

NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active)
NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved)
UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section)
UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h)
UPDATED index.md, log.md
This commit is contained in:
Tony0410
2026-11-02 00:27:09 +08:00
parent 8bb3d196b1
commit fda56e53fe
6 changed files with 383 additions and 9 deletions
+33 -2
View File
@@ -3,8 +3,8 @@ title: Messaging Integrations
type: system
status: active
created: 2026-07-22
updated: 2026-11-01
verified_on: 2026-11-01
updated: 2026-11-02
verified_on: 2026-11-02
confidence: high
tags: [messaging, telegram, discord, email, integrations]
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
@@ -152,3 +152,34 @@ the adapter stayed connected since.
[[infrastructure/tailscale]]
- **Open item now closed:** the platform is no longer an outage. The home channel startup notification
was delivered successfully on reconnect (`Sent home-channel startup notification to mattermost:…`).
## 2026-11-02 00:4x AWST — Telegram flapped all day; the relay adapter has now been absent ~29 h
Live probes on `gateway.log` for the 11-01 window plus `deliveries.db`:
- **Telegram transport degraded repeatedly, and it never fully failed.** The recurring shape is a
**dual-stack `api.telegram.org` `ConnectTimeout`** on the first hop, followed by a **fallback to an
IPv4 literal** that succeeds (`Telegram] Disconnected from Telegram` at **20:51:53**, with polling
restarts at **22:33:30** and **23:12:23**). The host's resolver currently returns **only an AAAA
record** for `api.telegram.org` (`getent hosts` → `2001:67c:4e8:f004::9`), so the dual-stack path the
client tries first is the one DNS actually hands it. **Consistent with, but not proof of, a
systematic timeout on that path** — recorded as an observation, not a diagnosed cause.
- **`066bb5191280` is the second variant of the thread-mismatch bug, confirmed with hard evidence.**
Its 11-01 delivery row is `failed` with
`configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id`.
The target is the **home DM** (`1793951355`) carrying a *thread* id, so the sanctioned thread-less
fallback fires, the content still reaches the DM, and the **run is recorded `failed` regardless** —
i.e. a delivery that landed is reported as a failure. Same 2026-10-01 semantics documented above,
now the second confirmed instance.
- **The `relay` platform has no adapter in the running gateway and has not had one for ~29 h:**
`Reconnecting relay` … `no adapter yet` repeats every ~300 s, **attempt 147 at 10-31 18:39:24 →
attempt 336 at 11-02 00:10:47**, counting without a reset. **The counter is monotonic across both the
18:00 handoff failure and the three `-15` kills, which is the proof that the gateway process never
restarted in that window** — see [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]].
The feed of retries is cosmetic for Telegram/email delivery (both work), but it makes
`Reconnecting <platform>` log lines useless as a restart detector once an adapter is absent.
- **Net delivery health for 11-01: 8 rows, 7 `delivered`, 1 `failed`** — and **nothing after 11:01:12**
(the missing Weekly Review). The Telegram adapter itself carried every Telegram-bound delivery
successfully, so the 11-01 degradation is a **latency/retry** problem, not a delivery outage.
→ [[systems/scheduled-tasks]], [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]