Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown. NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active) NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved) UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section) UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h) UPDATED index.md, log.md
186 lines
14 KiB
Markdown
186 lines
14 KiB
Markdown
---
|
||
title: Messaging Integrations
|
||
type: system
|
||
status: active
|
||
created: 2026-07-22
|
||
updated: 2026-11-02
|
||
verified_on: 2026-11-02
|
||
confidence: high
|
||
tags: [messaging, telegram, discord, email, integrations]
|
||
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
|
||
---
|
||
|
||
# Messaging Integrations
|
||
|
||
## Telegram
|
||
- **Status:** enabled, active
|
||
- **allowed_chats:** `-1004321904721` (primary), `1793951355` (DM), plus the remaining chat IDs required by both multiplexed profiles on the default adapter — verified 2026-09-30
|
||
- **Rich messages:** true
|
||
- **Platform toolsets:** cli, telegram (full suite)
|
||
- **Delivery targets seen in cron:** telegram, `telegram:1793951355`, `telegram:-1004321904721`, `telegram:-1004321904721:notifications`, `telegram:-1004321904721:102`, `telegram:-1004321904721:273`, `telegram:-1004321904721:346`, `telegram:-1004321904721:1030`
|
||
- **Notes:** Primary delivery target for scheduled jobs
|
||
|
||
### Delivery target semantics (verified 2026-10-01)
|
||
- A **bare `deliver: "telegram"` resolves to the home DM (`1793951355`)** — *not* to the primary group. It is not a "send to the main chat" shorthand.
|
||
- To reach a forum topic you must use the full form **`telegram:<chat_id>:<thread_id>`**, e.g. `telegram:-1004321904721:102`.
|
||
- Consequence: a job that names a group topic ID but is actually pointed at the DM fails with `configured thread_id <n> … was not found` / `Message thread not found`. **Two variants are observed, and they behave differently** (verified 2026-10-02 against `~/.hermes/cron/executions.db`):
|
||
- `Message thread not found` — hard failure, **the message is lost** (the 2026-09-30 incident variant).
|
||
- `configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id` — a **thread-less fallback send** does occur, yet the run is still recorded as `last_status: delivery_failed` / `delivery_outcome: failed`. Content reaches the DM; the delivery is marked failed anyway, so the job record reads like a lost message.
|
||
- Do not assume "delivery failed" means the message never arrived — check the error variant before reporting a report as missing.
|
||
|
||
## Discord
|
||
- **Status:** enabled, streaming true
|
||
- **Platform:** `hermes-discord`
|
||
- **Platform toolsets:** cli, discord
|
||
|
||
## Email
|
||
- **Libraries:** himalaya CLI available
|
||
- **Platform toolsets:** cli
|
||
- **Status: ⚠ partially broken — the personal account works, the `work` account does not.**
|
||
- **Personal (Gmail):** connected and usable. Google still permits app-password auth.
|
||
- **`work` (Outlook / Microsoft 365):** **unreachable.** `himalaya envelope list -a work` fails with `Error: cannot build IMAP client / Caused by: login not supported`. The account is configured for password auth (`backend.auth.type = "password"` + `~/.config/himalaya/get_work_outlook_password.sh`) while the tenant advertises **only `XOAuth2`** — basic auth for IMAP/SMTP is disabled tenant-wide. The TCP/TLS connection to `outlook.office365.com:993` succeeds; only the login is refused. This is a policy wall, **not a wrong password** — re-running the credential helper cannot fix it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
|
||
- Fix path: OAuth2 (`backend.auth.type = "oauth2"` + token store) after an app registration with IMAP/SMTP scopes; Anthony's normal Office sign-in is number-matching approval in Microsoft Authenticator, which is the device-code flow this needs. Tenant admin consent may be required.
|
||
- Consequence for tooling: `email/mail-lookup` resolves the account before searching, so a request for work mail will reach this wall and must report the account as unreachable — not as "no results found".
|
||
|
||
## Slack
|
||
- **Status:** available (toolsets: cli, slack-platform)
|
||
- **Config noted:** slack entries in config but Slack enabled status unclear from config grep
|
||
|
||
## Other Platforms Available
|
||
- **Mattermost:** toolsets present (`hermes-mattermost`)
|
||
- **Google Chat:** toolsets present (`hermes-google_chat`)
|
||
- **QQ:** toolsets present (`hermes-qqbot`)
|
||
- **Signal:** toolsets present (`hermes-signal`)
|
||
- **WhatsApp:** toolsets present (`hermes-whatsapp`)
|
||
- **ntfy:** noted in memory config outputs but runtime status unverified
|
||
- **Yuanbao:** toolsets present (`hermes-yuanbao`)
|
||
- **Home Assistant:** homeassistant-platform enabled
|
||
|
||
## Known Issues
|
||
- **Telegram thread-delivery failures — resolved 2026-09-30.** Jobs `fad5a86979d4` and `1b23c042e17d` reported `configured thread_id 34795 … was not found`, and `c8d52904e1d6` reported `Message thread not found` (`last_status: delivery_failed`). Root cause was the *target*, not the topic: those jobs used a bare `deliver: "telegram"`, which resolves to the home DM where group topic IDs do not exist. `1b23c042e17d` and `c8d52904e1d6` were retargeted to `telegram:-1004321904721:102` and verified end-to-end. See [[systems/scheduled-tasks]].
|
||
- **Three jobs still deliver to the home DM via a bare `telegram` target** (`0e753e9720fd`, `fad5a86979d4`, `066bb5191280`). **No longer theoretical — two of them failed delivery on 2026-10-01:** `066bb5191280` (02:11) and `0e753e9720fd` (09:01) both recorded `delivery_outcome: failed` with the thread-less-fallback variant above, in the same failure class as the 2026-09-30 incident. Retarget them to `telegram:-1004321904721:<topic>`. See [[systems/scheduled-tasks]].
|
||
|
||
## Multi-bot group chat (open question, raised 2026-09-30)
|
||
Anthony asked whether his other Hermes instances (the estate VMs/CTs) could share a group chat.
|
||
- **Preferred shape:** a Telegram supergroup with **one forum topic per bot**, so each agent posts into its own thread and no two bots share a message stream. Mattermost was floated and rejected.
|
||
- **Estate reachability:** CT460's Hermes serves the WebUI/API on `:9119` (locally fronted by nginx on `:19119`). Whether the other estate hosts expose `:9119` still needs checking before any fan-out design.
|
||
- **Status:** design question only — nothing has been deployed or changed.
|
||
|
||
## Related
|
||
- [[systems/hermes-agent]]
|
||
- [[systems/scheduled-tasks]]
|
||
- [[infrastructure/tailscale]]
|
||
|
||
|
||
---
|
||
|
||
## 2026-10-11 00:20 AWST - Mattermost platform unreachable (observed, not fixed)
|
||
|
||
The gateway's Mattermost adapter has been down since at least 2026-10-10. `gateway.log` shows a
|
||
5-minute retry cadence with `Mattermost: disconnected` followed by
|
||
`Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s`
|
||
(attempt 34 at 00:15, attempt 35 at 00:20).
|
||
|
||
Probe of the configured endpoint (`MATTERMOST_URL=http://100.98.116.14:8065/`) from CT460:
|
||
**100 % ICMP loss and TCP 8065 refuses** - the Mattermost host itself is unreachable from CT460,
|
||
not merely an application-level refusal.
|
||
|
||
- Cause **not determined**; not investigated further during this audit (no host/VM/proxy probe).
|
||
- Impact is limited to the messaging path. No cron job depends on Mattermost, and the 2026-10-10
|
||
cron fleet ran clean.
|
||
- Config: plugin block `plugins...mattermost` (`enabled: true`, home channel
|
||
`5ipo6wfe3j88xxox1gkta3d5mw`, name `Rhino`); the bot token is env-only and is **not** recorded here.
|
||
|
||
**Open item:** re-probe the host; if the host is back but the adapter stays down, restart the
|
||
gateway's Mattermost platform.
|
||
|
||
## 2026-10-19 00:15 AWST — Mattermost adapter: cause identified (CT255 stopped), still down
|
||
|
||
This pass replaced the 10-11 observation's "cause not determined" with a definite cause. Live probes,
|
||
2026-10-19 00:0x-00:15 AWST:
|
||
|
||
- `tailscale status`: peer `mattermost` (`100.98.116.14`) — **offline, last seen 29d ago**, `tx 7176 rx 0`.
|
||
- `gateway.log`: the reconnect loop is unchanged — `Reconnecting mattermost (attempt 470)` at 00:08:36
|
||
and `(attempt 471)` at 00:14:07, each followed by `Mattermost: disconnected` and
|
||
`Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s`
|
||
(5-minute cadence; 3,992 such lines in the retained `gateway.log`).
|
||
- Probed from CT460 at the configured endpoint (`MATTERMOST_URL=http://100.98.116.14:8065/`):
|
||
**TCP 8065 refused/closed and 100 % ICMP loss** — the host is unreachable, not the application.
|
||
- **Cause: CT255 (`mattermost`) is `stopped` on the PVE host** (see [[infrastructure/proxmox]]).
|
||
The attempt counter restarted at the gateway restart of 2026-10-17 05:10 and now reads 471.
|
||
- Impact unchanged: messaging path only. No cron job delivers via Mattermost, and the cron fleet
|
||
performed normally throughout.
|
||
|
||
**Open item:** start CT255, or retire the platform block. The 5-minute reconnect loop continues until
|
||
one of those happens — restarting the adapter while the host is stopped achieves nothing.
|
||
|
||
## 2026-10-26 00:1x AWST — Mattermost still down; attempt counter reset by the 10-25 restart
|
||
|
||
- **Cause unchanged: CT255 (`mattermost`) is `stopped`.** `tailscale status` shows the peer **offline, last seen
|
||
36 d ago**, `tx 312 / rx 0` — `rx` has read 0 in every audit since 2026-10-11, i.e. no packet has come back.
|
||
- **Reconnect loop still running:** **268** `mattermost connect timed out` lines on 2026-10-25 vs **264** on
|
||
2026-10-24 (≈1 per 5 min, as designed); **2** so far on 2026-10-26.
|
||
- **The attempt counter reset to 54** with the 19:27:13 gateway restart (`Reconnecting mattermost (attempt 54)`
|
||
at 2026-10-26 00:07:06). The previous reset was the 2026-10-17 05:10 restart. ⚠ The counter measures **this
|
||
process's** attempts, not the outage's length — do not quote it as downtime.
|
||
- **Impact unchanged:** messaging path only; no cron job delivers via Mattermost.
|
||
- **Not attempted:** `pct start 255` — starting a service on the PVE host is Anthony's call.
|
||
|
||
## 2026-10-31 06:2x AWST — ✅ Mattermost RECOVERED (outage over)
|
||
|
||
**The ~40-day Mattermost outage is over.** All three live checks pass, and the gateway log confirms
|
||
the adapter stayed connected since.
|
||
|
||
| Check | Live result (2026-10-31) |
|
||
|---|---|
|
||
| `pct list` on `pve` | **CT255 (`mattermost`) → `running`** (was `stopped`) |
|
||
| `pct exec 255 -- systemctl is-active mattermost` | **`active`** |
|
||
| `pct exec 255 -- ss -ltn` | `LISTEN *:8065` |
|
||
| `tailscale ping 100.98.116.14` | **`pong from mattermost ... in 0s`** |
|
||
| `curl http://100.98.116.14:8065/api/v4/system/ping` | **HTTP 200** |
|
||
| Tailscale peer record | **`Online: true`** — first time since the outage began |
|
||
|
||
- **Gateway reconnection (CT460's own `gateway.log`):** `Mattermost: authenticated as @rhino-bot` at
|
||
**2026-10-30 07:19:37**, with `✓ mattermost reconnected successfully` and a successful WebSocket
|
||
handshake at the same second. A second connect cycle at **2026-10-30 16:36:07** produced
|
||
`✓ mattermost connected` at **16:36:17**.
|
||
- **No errors since.** Grepping every `mattermost` line in `gateway.log` after `2026-10-30 16:36:17`
|
||
returns **zero** `error` / `timeout` / `disconnect` / `fail` lines — the 5-minute reconnect loop that
|
||
produced 264–268 timeouts a day is **no longer running**.
|
||
- **The prior "cause: CT255 is stopped" finding is superseded.** CT255 is running; the gateway can
|
||
reach `http://100.98.116.14:8065/` and authenticate. → [[infrastructure/proxmox]],
|
||
[[infrastructure/tailscale]]
|
||
- **Open item now closed:** the platform is no longer an outage. The home channel startup notification
|
||
was delivered successfully on reconnect (`Sent home-channel startup notification to mattermost:…`).
|
||
|
||
## 2026-11-02 00:4x AWST — Telegram flapped all day; the relay adapter has now been absent ~29 h
|
||
|
||
Live probes on `gateway.log` for the 11-01 window plus `deliveries.db`:
|
||
|
||
- **Telegram transport degraded repeatedly, and it never fully failed.** The recurring shape is a
|
||
**dual-stack `api.telegram.org` `ConnectTimeout`** on the first hop, followed by a **fallback to an
|
||
IPv4 literal** that succeeds (`Telegram] Disconnected from Telegram` at **20:51:53**, with polling
|
||
restarts at **22:33:30** and **23:12:23**). The host's resolver currently returns **only an AAAA
|
||
record** for `api.telegram.org` (`getent hosts` → `2001:67c:4e8:f004::9`), so the dual-stack path the
|
||
client tries first is the one DNS actually hands it. **Consistent with, but not proof of, a
|
||
systematic timeout on that path** — recorded as an observation, not a diagnosed cause.
|
||
- **`066bb5191280` is the second variant of the thread-mismatch bug, confirmed with hard evidence.**
|
||
Its 11-01 delivery row is `failed` with
|
||
`configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id`.
|
||
The target is the **home DM** (`1793951355`) carrying a *thread* id, so the sanctioned thread-less
|
||
fallback fires, the content still reaches the DM, and the **run is recorded `failed` regardless** —
|
||
i.e. a delivery that landed is reported as a failure. Same 2026-10-01 semantics documented above,
|
||
now the second confirmed instance.
|
||
- **The `relay` platform has no adapter in the running gateway and has not had one for ~29 h:**
|
||
`Reconnecting relay` … `no adapter yet` repeats every ~300 s, **attempt 147 at 10-31 18:39:24 →
|
||
attempt 336 at 11-02 00:10:47**, counting without a reset. **The counter is monotonic across both the
|
||
18:00 handoff failure and the three `-15` kills, which is the proof that the gateway process never
|
||
restarted in that window** — see [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]].
|
||
The feed of retries is cosmetic for Telegram/email delivery (both work), but it makes
|
||
`Reconnecting <platform>` log lines useless as a restart detector once an adapter is absent.
|
||
- **Net delivery health for 11-01: 8 rows, 7 `delivered`, 1 `failed`** — and **nothing after 11:01:12**
|
||
(the missing Weekly Review). The Telegram adapter itself carried every Telegram-bound delivery
|
||
successfully, so the 11-01 degradation is a **latency/retry** problem, not a delivery outage.
|
||
|
||
→ [[systems/scheduled-tasks]], [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]
|