Files
agent-estate-wiki/systems/messaging-integrations.md
T
Tony0410 fda56e53fe audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd
Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown.

NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active)
NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved)
UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section)
UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h)
UPDATED index.md, log.md
2026-11-02 00:27:09 +08:00

186 lines
14 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Messaging Integrations
type: system
status: active
created: 2026-07-22
updated: 2026-11-02
verified_on: 2026-11-02
confidence: high
tags: [messaging, telegram, discord, email, integrations]
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
---
# Messaging Integrations
## Telegram
- **Status:** enabled, active
- **allowed_chats:** `-1004321904721` (primary), `1793951355` (DM), plus the remaining chat IDs required by both multiplexed profiles on the default adapter — verified 2026-09-30
- **Rich messages:** true
- **Platform toolsets:** cli, telegram (full suite)
- **Delivery targets seen in cron:** telegram, `telegram:1793951355`, `telegram:-1004321904721`, `telegram:-1004321904721:notifications`, `telegram:-1004321904721:102`, `telegram:-1004321904721:273`, `telegram:-1004321904721:346`, `telegram:-1004321904721:1030`
- **Notes:** Primary delivery target for scheduled jobs
### Delivery target semantics (verified 2026-10-01)
- A **bare `deliver: "telegram"` resolves to the home DM (`1793951355`)** — *not* to the primary group. It is not a "send to the main chat" shorthand.
- To reach a forum topic you must use the full form **`telegram:<chat_id>:<thread_id>`**, e.g. `telegram:-1004321904721:102`.
- Consequence: a job that names a group topic ID but is actually pointed at the DM fails with `configured thread_id <n> … was not found` / `Message thread not found`. **Two variants are observed, and they behave differently** (verified 2026-10-02 against `~/.hermes/cron/executions.db`):
- `Message thread not found` — hard failure, **the message is lost** (the 2026-09-30 incident variant).
- `configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id` — a **thread-less fallback send** does occur, yet the run is still recorded as `last_status: delivery_failed` / `delivery_outcome: failed`. Content reaches the DM; the delivery is marked failed anyway, so the job record reads like a lost message.
- Do not assume "delivery failed" means the message never arrived — check the error variant before reporting a report as missing.
## Discord
- **Status:** enabled, streaming true
- **Platform:** `hermes-discord`
- **Platform toolsets:** cli, discord
## Email
- **Libraries:** himalaya CLI available
- **Platform toolsets:** cli
- **Status: ⚠ partially broken — the personal account works, the `work` account does not.**
- **Personal (Gmail):** connected and usable. Google still permits app-password auth.
- **`work` (Outlook / Microsoft 365):** **unreachable.** `himalaya envelope list -a work` fails with `Error: cannot build IMAP client / Caused by: login not supported`. The account is configured for password auth (`backend.auth.type = "password"` + `~/.config/himalaya/get_work_outlook_password.sh`) while the tenant advertises **only `XOAuth2`** — basic auth for IMAP/SMTP is disabled tenant-wide. The TCP/TLS connection to `outlook.office365.com:993` succeeds; only the login is refused. This is a policy wall, **not a wrong password** — re-running the credential helper cannot fix it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
- Fix path: OAuth2 (`backend.auth.type = "oauth2"` + token store) after an app registration with IMAP/SMTP scopes; Anthony's normal Office sign-in is number-matching approval in Microsoft Authenticator, which is the device-code flow this needs. Tenant admin consent may be required.
- Consequence for tooling: `email/mail-lookup` resolves the account before searching, so a request for work mail will reach this wall and must report the account as unreachable — not as "no results found".
## Slack
- **Status:** available (toolsets: cli, slack-platform)
- **Config noted:** slack entries in config but Slack enabled status unclear from config grep
## Other Platforms Available
- **Mattermost:** toolsets present (`hermes-mattermost`)
- **Google Chat:** toolsets present (`hermes-google_chat`)
- **QQ:** toolsets present (`hermes-qqbot`)
- **Signal:** toolsets present (`hermes-signal`)
- **WhatsApp:** toolsets present (`hermes-whatsapp`)
- **ntfy:** noted in memory config outputs but runtime status unverified
- **Yuanbao:** toolsets present (`hermes-yuanbao`)
- **Home Assistant:** homeassistant-platform enabled
## Known Issues
- **Telegram thread-delivery failures — resolved 2026-09-30.** Jobs `fad5a86979d4` and `1b23c042e17d` reported `configured thread_id 34795 … was not found`, and `c8d52904e1d6` reported `Message thread not found` (`last_status: delivery_failed`). Root cause was the *target*, not the topic: those jobs used a bare `deliver: "telegram"`, which resolves to the home DM where group topic IDs do not exist. `1b23c042e17d` and `c8d52904e1d6` were retargeted to `telegram:-1004321904721:102` and verified end-to-end. See [[systems/scheduled-tasks]].
- **Three jobs still deliver to the home DM via a bare `telegram` target** (`0e753e9720fd`, `fad5a86979d4`, `066bb5191280`). **No longer theoretical — two of them failed delivery on 2026-10-01:** `066bb5191280` (02:11) and `0e753e9720fd` (09:01) both recorded `delivery_outcome: failed` with the thread-less-fallback variant above, in the same failure class as the 2026-09-30 incident. Retarget them to `telegram:-1004321904721:<topic>`. See [[systems/scheduled-tasks]].
## Multi-bot group chat (open question, raised 2026-09-30)
Anthony asked whether his other Hermes instances (the estate VMs/CTs) could share a group chat.
- **Preferred shape:** a Telegram supergroup with **one forum topic per bot**, so each agent posts into its own thread and no two bots share a message stream. Mattermost was floated and rejected.
- **Estate reachability:** CT460's Hermes serves the WebUI/API on `:9119` (locally fronted by nginx on `:19119`). Whether the other estate hosts expose `:9119` still needs checking before any fan-out design.
- **Status:** design question only — nothing has been deployed or changed.
## Related
- [[systems/hermes-agent]]
- [[systems/scheduled-tasks]]
- [[infrastructure/tailscale]]
---
## 2026-10-11 00:20 AWST - Mattermost platform unreachable (observed, not fixed)
The gateway's Mattermost adapter has been down since at least 2026-10-10. `gateway.log` shows a
5-minute retry cadence with `Mattermost: disconnected` followed by
`Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s`
(attempt 34 at 00:15, attempt 35 at 00:20).
Probe of the configured endpoint (`MATTERMOST_URL=http://100.98.116.14:8065/`) from CT460:
**100 % ICMP loss and TCP 8065 refuses** - the Mattermost host itself is unreachable from CT460,
not merely an application-level refusal.
- Cause **not determined**; not investigated further during this audit (no host/VM/proxy probe).
- Impact is limited to the messaging path. No cron job depends on Mattermost, and the 2026-10-10
cron fleet ran clean.
- Config: plugin block `plugins...mattermost` (`enabled: true`, home channel
`5ipo6wfe3j88xxox1gkta3d5mw`, name `Rhino`); the bot token is env-only and is **not** recorded here.
**Open item:** re-probe the host; if the host is back but the adapter stays down, restart the
gateway's Mattermost platform.
## 2026-10-19 00:15 AWST — Mattermost adapter: cause identified (CT255 stopped), still down
This pass replaced the 10-11 observation's "cause not determined" with a definite cause. Live probes,
2026-10-19 00:0x-00:15 AWST:
- `tailscale status`: peer `mattermost` (`100.98.116.14`) — **offline, last seen 29d ago**, `tx 7176 rx 0`.
- `gateway.log`: the reconnect loop is unchanged — `Reconnecting mattermost (attempt 470)` at 00:08:36
and `(attempt 471)` at 00:14:07, each followed by `Mattermost: disconnected` and
`Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s`
(5-minute cadence; 3,992 such lines in the retained `gateway.log`).
- Probed from CT460 at the configured endpoint (`MATTERMOST_URL=http://100.98.116.14:8065/`):
**TCP 8065 refused/closed and 100 % ICMP loss** — the host is unreachable, not the application.
- **Cause: CT255 (`mattermost`) is `stopped` on the PVE host** (see [[infrastructure/proxmox]]).
The attempt counter restarted at the gateway restart of 2026-10-17 05:10 and now reads 471.
- Impact unchanged: messaging path only. No cron job delivers via Mattermost, and the cron fleet
performed normally throughout.
**Open item:** start CT255, or retire the platform block. The 5-minute reconnect loop continues until
one of those happens — restarting the adapter while the host is stopped achieves nothing.
## 2026-10-26 00:1x AWST — Mattermost still down; attempt counter reset by the 10-25 restart
- **Cause unchanged: CT255 (`mattermost`) is `stopped`.** `tailscale status` shows the peer **offline, last seen
36 d ago**, `tx 312 / rx 0` — `rx` has read 0 in every audit since 2026-10-11, i.e. no packet has come back.
- **Reconnect loop still running:** **268** `mattermost connect timed out` lines on 2026-10-25 vs **264** on
2026-10-24 (≈1 per 5 min, as designed); **2** so far on 2026-10-26.
- **The attempt counter reset to 54** with the 19:27:13 gateway restart (`Reconnecting mattermost (attempt 54)`
at 2026-10-26 00:07:06). The previous reset was the 2026-10-17 05:10 restart. ⚠ The counter measures **this
process's** attempts, not the outage's length — do not quote it as downtime.
- **Impact unchanged:** messaging path only; no cron job delivers via Mattermost.
- **Not attempted:** `pct start 255` — starting a service on the PVE host is Anthony's call.
## 2026-10-31 06:2x AWST — ✅ Mattermost RECOVERED (outage over)
**The ~40-day Mattermost outage is over.** All three live checks pass, and the gateway log confirms
the adapter stayed connected since.
| Check | Live result (2026-10-31) |
|---|---|
| `pct list` on `pve` | **CT255 (`mattermost`) → `running`** (was `stopped`) |
| `pct exec 255 -- systemctl is-active mattermost` | **`active`** |
| `pct exec 255 -- ss -ltn` | `LISTEN *:8065` |
| `tailscale ping 100.98.116.14` | **`pong from mattermost ... in 0s`** |
| `curl http://100.98.116.14:8065/api/v4/system/ping` | **HTTP 200** |
| Tailscale peer record | **`Online: true`** — first time since the outage began |
- **Gateway reconnection (CT460's own `gateway.log`):** `Mattermost: authenticated as @rhino-bot` at
**2026-10-30 07:19:37**, with `✓ mattermost reconnected successfully` and a successful WebSocket
handshake at the same second. A second connect cycle at **2026-10-30 16:36:07** produced
`✓ mattermost connected` at **16:36:17**.
- **No errors since.** Grepping every `mattermost` line in `gateway.log` after `2026-10-30 16:36:17`
returns **zero** `error` / `timeout` / `disconnect` / `fail` lines — the 5-minute reconnect loop that
produced 264–268 timeouts a day is **no longer running**.
- **The prior "cause: CT255 is stopped" finding is superseded.** CT255 is running; the gateway can
reach `http://100.98.116.14:8065/` and authenticate. → [[infrastructure/proxmox]],
[[infrastructure/tailscale]]
- **Open item now closed:** the platform is no longer an outage. The home channel startup notification
was delivered successfully on reconnect (`Sent home-channel startup notification to mattermost:…`).
## 2026-11-02 00:4x AWST — Telegram flapped all day; the relay adapter has now been absent ~29 h
Live probes on `gateway.log` for the 11-01 window plus `deliveries.db`:
- **Telegram transport degraded repeatedly, and it never fully failed.** The recurring shape is a
**dual-stack `api.telegram.org` `ConnectTimeout`** on the first hop, followed by a **fallback to an
IPv4 literal** that succeeds (`Telegram] Disconnected from Telegram` at **20:51:53**, with polling
restarts at **22:33:30** and **23:12:23**). The host's resolver currently returns **only an AAAA
record** for `api.telegram.org` (`getent hosts` → `2001:67c:4e8:f004::9`), so the dual-stack path the
client tries first is the one DNS actually hands it. **Consistent with, but not proof of, a
systematic timeout on that path** — recorded as an observation, not a diagnosed cause.
- **`066bb5191280` is the second variant of the thread-mismatch bug, confirmed with hard evidence.**
Its 11-01 delivery row is `failed` with
`configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id`.
The target is the **home DM** (`1793951355`) carrying a *thread* id, so the sanctioned thread-less
fallback fires, the content still reaches the DM, and the **run is recorded `failed` regardless** —
i.e. a delivery that landed is reported as a failure. Same 2026-10-01 semantics documented above,
now the second confirmed instance.
- **The `relay` platform has no adapter in the running gateway and has not had one for ~29 h:**
`Reconnecting relay` … `no adapter yet` repeats every ~300 s, **attempt 147 at 10-31 18:39:24 →
attempt 336 at 11-02 00:10:47**, counting without a reset. **The counter is monotonic across both the
18:00 handoff failure and the three `-15` kills, which is the proof that the gateway process never
restarted in that window** — see [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]].
The feed of retries is cosmetic for Telegram/email delivery (both work), but it makes
`Reconnecting <platform>` log lines useless as a restart detector once an adapter is absent.
- **Net delivery health for 11-01: 8 rows, 7 `delivered`, 1 `failed`** — and **nothing after 11:01:12**
(the missing Weekly Review). The Telegram adapter itself carried every Telegram-bound delivery
successfully, so the 11-01 degradation is a **latency/retry** problem, not a delivery outage.
→ [[systems/scheduled-tasks]], [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]