audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd
Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown. NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active) NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved) UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section) UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h) UPDATED index.md, log.md
This commit is contained in:
@@ -3,8 +3,8 @@ title: Messaging Integrations
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
updated: 2026-11-02
|
||||
verified_on: 2026-11-02
|
||||
confidence: high
|
||||
tags: [messaging, telegram, discord, email, integrations]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
|
||||
@@ -152,3 +152,34 @@ the adapter stayed connected since.
|
||||
[[infrastructure/tailscale]]
|
||||
- **Open item now closed:** the platform is no longer an outage. The home channel startup notification
|
||||
was delivered successfully on reconnect (`Sent home-channel startup notification to mattermost:…`).
|
||||
|
||||
## 2026-11-02 00:4x AWST — Telegram flapped all day; the relay adapter has now been absent ~29 h
|
||||
|
||||
Live probes on `gateway.log` for the 11-01 window plus `deliveries.db`:
|
||||
|
||||
- **Telegram transport degraded repeatedly, and it never fully failed.** The recurring shape is a
|
||||
**dual-stack `api.telegram.org` `ConnectTimeout`** on the first hop, followed by a **fallback to an
|
||||
IPv4 literal** that succeeds (`Telegram] Disconnected from Telegram` at **20:51:53**, with polling
|
||||
restarts at **22:33:30** and **23:12:23**). The host's resolver currently returns **only an AAAA
|
||||
record** for `api.telegram.org` (`getent hosts` → `2001:67c:4e8:f004::9`), so the dual-stack path the
|
||||
client tries first is the one DNS actually hands it. **Consistent with, but not proof of, a
|
||||
systematic timeout on that path** — recorded as an observation, not a diagnosed cause.
|
||||
- **`066bb5191280` is the second variant of the thread-mismatch bug, confirmed with hard evidence.**
|
||||
Its 11-01 delivery row is `failed` with
|
||||
`configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id`.
|
||||
The target is the **home DM** (`1793951355`) carrying a *thread* id, so the sanctioned thread-less
|
||||
fallback fires, the content still reaches the DM, and the **run is recorded `failed` regardless** —
|
||||
i.e. a delivery that landed is reported as a failure. Same 2026-10-01 semantics documented above,
|
||||
now the second confirmed instance.
|
||||
- **The `relay` platform has no adapter in the running gateway and has not had one for ~29 h:**
|
||||
`Reconnecting relay` … `no adapter yet` repeats every ~300 s, **attempt 147 at 10-31 18:39:24 →
|
||||
attempt 336 at 11-02 00:10:47**, counting without a reset. **The counter is monotonic across both the
|
||||
18:00 handoff failure and the three `-15` kills, which is the proof that the gateway process never
|
||||
restarted in that window** — see [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]].
|
||||
The feed of retries is cosmetic for Telegram/email delivery (both work), but it makes
|
||||
`Reconnecting <platform>` log lines useless as a restart detector once an adapter is absent.
|
||||
- **Net delivery health for 11-01: 8 rows, 7 `delivered`, 1 `failed`** — and **nothing after 11:01:12**
|
||||
(the missing Weekly Review). The Telegram adapter itself carried every Telegram-bound delivery
|
||||
successfully, so the 11-01 degradation is a **latency/retry** problem, not a delivery outage.
|
||||
|
||||
→ [[systems/scheduled-tasks]], [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]
|
||||
|
||||
Reference in New Issue
Block a user