Files
agent-estate-wiki/systems/messaging-integrations.md
T
Tony0410 fda56e53fe audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd
Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown.

NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active)
NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved)
UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section)
UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h)
UPDATED index.md, log.md
2026-11-02 00:27:09 +08:00

14 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Messaging Integrations system active 2026-07-22 2026-11-02 2026-11-02 high
messaging
telegram
discord
email
integrations
raw/configs/hermes-config-sanitized.txt
live-jobs-json
live-himalaya-debug

Messaging Integrations

Telegram

  • Status: enabled, active
  • allowed_chats: -1004321904721 (primary), 1793951355 (DM), plus the remaining chat IDs required by both multiplexed profiles on the default adapter — verified 2026-09-30
  • Rich messages: true
  • Platform toolsets: cli, telegram (full suite)
  • Delivery targets seen in cron: telegram, telegram:1793951355, telegram:-1004321904721, telegram:-1004321904721:notifications, telegram:-1004321904721:102, telegram:-1004321904721:273, telegram:-1004321904721:346, telegram:-1004321904721:1030
  • Notes: Primary delivery target for scheduled jobs

Delivery target semantics (verified 2026-10-01)

  • A bare deliver: "telegram" resolves to the home DM (1793951355) — not to the primary group. It is not a "send to the main chat" shorthand.
  • To reach a forum topic you must use the full form telegram:<chat_id>:<thread_id>, e.g. telegram:-1004321904721:102.
  • Consequence: a job that names a group topic ID but is actually pointed at the DM fails with configured thread_id <n> … was not found / Message thread not found. Two variants are observed, and they behave differently (verified 2026-10-02 against ~/.hermes/cron/executions.db):
    • Message thread not found — hard failure, the message is lost (the 2026-09-30 incident variant).
    • configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id — a thread-less fallback send does occur, yet the run is still recorded as last_status: delivery_failed / delivery_outcome: failed. Content reaches the DM; the delivery is marked failed anyway, so the job record reads like a lost message.
    • Do not assume "delivery failed" means the message never arrived — check the error variant before reporting a report as missing.

Discord

  • Status: enabled, streaming true
  • Platform: hermes-discord
  • Platform toolsets: cli, discord

Email

  • Libraries: himalaya CLI available
  • Platform toolsets: cli
  • Status: ⚠ partially broken — the personal account works, the work account does not.
    • Personal (Gmail): connected and usable. Google still permits app-password auth.
    • work (Outlook / Microsoft 365): unreachable. himalaya envelope list -a work fails with Error: cannot build IMAP client / Caused by: login not supported. The account is configured for password auth (backend.auth.type = "password" + ~/.config/himalaya/get_work_outlook_password.sh) while the tenant advertises only XOAuth2 — basic auth for IMAP/SMTP is disabled tenant-wide. The TCP/TLS connection to outlook.office365.com:993 succeeds; only the login is refused. This is a policy wall, not a wrong password — re-running the credential helper cannot fix it. See incidents/2026-08-26-work-outlook-basic-auth-disabled.
    • Fix path: OAuth2 (backend.auth.type = "oauth2" + token store) after an app registration with IMAP/SMTP scopes; Anthony's normal Office sign-in is number-matching approval in Microsoft Authenticator, which is the device-code flow this needs. Tenant admin consent may be required.
    • Consequence for tooling: email/mail-lookup resolves the account before searching, so a request for work mail will reach this wall and must report the account as unreachable — not as "no results found".

Slack

  • Status: available (toolsets: cli, slack-platform)
  • Config noted: slack entries in config but Slack enabled status unclear from config grep

Other Platforms Available

  • Mattermost: toolsets present (hermes-mattermost)
  • Google Chat: toolsets present (hermes-google_chat)
  • QQ: toolsets present (hermes-qqbot)
  • Signal: toolsets present (hermes-signal)
  • WhatsApp: toolsets present (hermes-whatsapp)
  • ntfy: noted in memory config outputs but runtime status unverified
  • Yuanbao: toolsets present (hermes-yuanbao)
  • Home Assistant: homeassistant-platform enabled

Known Issues

  • Telegram thread-delivery failures — resolved 2026-09-30. Jobs fad5a86979d4 and 1b23c042e17d reported configured thread_id 34795 … was not found, and c8d52904e1d6 reported Message thread not found (last_status: delivery_failed). Root cause was the target, not the topic: those jobs used a bare deliver: "telegram", which resolves to the home DM where group topic IDs do not exist. 1b23c042e17d and c8d52904e1d6 were retargeted to telegram:-1004321904721:102 and verified end-to-end. See systems/scheduled-tasks.
  • Three jobs still deliver to the home DM via a bare telegram target (0e753e9720fd, fad5a86979d4, 066bb5191280). No longer theoretical — two of them failed delivery on 2026-10-01: 066bb5191280 (02:11) and 0e753e9720fd (09:01) both recorded delivery_outcome: failed with the thread-less-fallback variant above, in the same failure class as the 2026-09-30 incident. Retarget them to telegram:-1004321904721:<topic>. See systems/scheduled-tasks.

Multi-bot group chat (open question, raised 2026-09-30)

Anthony asked whether his other Hermes instances (the estate VMs/CTs) could share a group chat.

  • Preferred shape: a Telegram supergroup with one forum topic per bot, so each agent posts into its own thread and no two bots share a message stream. Mattermost was floated and rejected.
  • Estate reachability: CT460's Hermes serves the WebUI/API on :9119 (locally fronted by nginx on :19119). Whether the other estate hosts expose :9119 still needs checking before any fan-out design.
  • Status: design question only — nothing has been deployed or changed.

2026-10-11 00:20 AWST - Mattermost platform unreachable (observed, not fixed)

The gateway's Mattermost adapter has been down since at least 2026-10-10. gateway.log shows a 5-minute retry cadence with Mattermost: disconnected followed by Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s (attempt 34 at 00:15, attempt 35 at 00:20).

Probe of the configured endpoint (MATTERMOST_URL=http://100.98.116.14:8065/) from CT460: 100 % ICMP loss and TCP 8065 refuses - the Mattermost host itself is unreachable from CT460, not merely an application-level refusal.

  • Cause not determined; not investigated further during this audit (no host/VM/proxy probe).
  • Impact is limited to the messaging path. No cron job depends on Mattermost, and the 2026-10-10 cron fleet ran clean.
  • Config: plugin block plugins...mattermost (enabled: true, home channel 5ipo6wfe3j88xxox1gkta3d5mw, name Rhino); the bot token is env-only and is not recorded here.

Open item: re-probe the host; if the host is back but the adapter stays down, restart the gateway's Mattermost platform.

2026-10-19 00:15 AWST — Mattermost adapter: cause identified (CT255 stopped), still down

This pass replaced the 10-11 observation's "cause not determined" with a definite cause. Live probes, 2026-10-19 00:0x-00:15 AWST:

  • tailscale status: peer mattermost (100.98.116.14) — offline, last seen 29d ago, tx 7176 rx 0.
  • gateway.log: the reconnect loop is unchanged — Reconnecting mattermost (attempt 470) at 00:08:36 and (attempt 471) at 00:14:07, each followed by Mattermost: disconnected and Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s (5-minute cadence; 3,992 such lines in the retained gateway.log).
  • Probed from CT460 at the configured endpoint (MATTERMOST_URL=http://100.98.116.14:8065/): TCP 8065 refused/closed and 100 % ICMP loss — the host is unreachable, not the application.
  • Cause: CT255 (mattermost) is stopped on the PVE host (see infrastructure/proxmox). The attempt counter restarted at the gateway restart of 2026-10-17 05:10 and now reads 471.
  • Impact unchanged: messaging path only. No cron job delivers via Mattermost, and the cron fleet performed normally throughout.

Open item: start CT255, or retire the platform block. The 5-minute reconnect loop continues until one of those happens — restarting the adapter while the host is stopped achieves nothing.

2026-10-26 00:1x AWST — Mattermost still down; attempt counter reset by the 10-25 restart

  • Cause unchanged: CT255 (mattermost) is stopped. tailscale status shows the peer offline, last seen 36 d ago, tx 312 / rx 0 — rx has read 0 in every audit since 2026-10-11, i.e. no packet has come back.
  • Reconnect loop still running: 268 mattermost connect timed out lines on 2026-10-25 vs 264 on 2026-10-24 (≈1 per 5 min, as designed); 2 so far on 2026-10-26.
  • The attempt counter reset to 54 with the 19:27:13 gateway restart (Reconnecting mattermost (attempt 54) at 2026-10-26 00:07:06). The previous reset was the 2026-10-17 05:10 restart. ⚠ The counter measures this process's attempts, not the outage's length — do not quote it as downtime.
  • Impact unchanged: messaging path only; no cron job delivers via Mattermost.
  • Not attempted: pct start 255 — starting a service on the PVE host is Anthony's call.

2026-10-31 06:2x AWST — ✅ Mattermost RECOVERED (outage over)

The ~40-day Mattermost outage is over. All three live checks pass, and the gateway log confirms the adapter stayed connected since.

Check Live result (2026-10-31)
pct list on pve CT255 (mattermost) → running (was stopped)
pct exec 255 -- systemctl is-active mattermost active
pct exec 255 -- ss -ltn LISTEN *:8065
tailscale ping 100.98.116.14 pong from mattermost ... in 0s
curl http://100.98.116.14:8065/api/v4/system/ping HTTP 200
Tailscale peer record Online: true — first time since the outage began
  • Gateway reconnection (CT460's own gateway.log): Mattermost: authenticated as @rhino-bot at 2026-10-30 07:19:37, with ✓ mattermost reconnected successfully and a successful WebSocket handshake at the same second. A second connect cycle at 2026-10-30 16:36:07 produced ✓ mattermost connected at 16:36:17.
  • No errors since. Grepping every mattermost line in gateway.log after 2026-10-30 16:36:17 returns zero error / timeout / disconnect / fail lines — the 5-minute reconnect loop that produced 264–268 timeouts a day is no longer running.
  • The prior "cause: CT255 is stopped" finding is superseded. CT255 is running; the gateway can reach http://100.98.116.14:8065/ and authenticate. → infrastructure/proxmox, infrastructure/tailscale
  • Open item now closed: the platform is no longer an outage. The home channel startup notification was delivered successfully on reconnect (Sent home-channel startup notification to mattermost:…).

2026-11-02 00:4x AWST — Telegram flapped all day; the relay adapter has now been absent ~29 h

Live probes on gateway.log for the 11-01 window plus deliveries.db:

  • Telegram transport degraded repeatedly, and it never fully failed. The recurring shape is a dual-stack api.telegram.org ConnectTimeout on the first hop, followed by a fallback to an IPv4 literal that succeeds (Telegram] Disconnected from Telegram at 20:51:53, with polling restarts at 22:33:30 and 23:12:23). The host's resolver currently returns only an AAAA record for api.telegram.org (getent hosts → 2001:67c:4e8:f004::9), so the dual-stack path the client tries first is the one DNS actually hands it. Consistent with, but not proof of, a systematic timeout on that path — recorded as an observation, not a diagnosed cause.
  • 066bb5191280 is the second variant of the thread-mismatch bug, confirmed with hard evidence. Its 11-01 delivery row is failed with configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id. The target is the home DM (1793951355) carrying a thread id, so the sanctioned thread-less fallback fires, the content still reaches the DM, and the run is recorded failed regardless — i.e. a delivery that landed is reported as a failure. Same 2026-10-01 semantics documented above, now the second confirmed instance.
  • The relay platform has no adapter in the running gateway and has not had one for ~29 h: Reconnecting relay … no adapter yet repeats every ~300 s, attempt 147 at 10-31 18:39:24 → attempt 336 at 11-02 00:10:47, counting without a reset. The counter is monotonic across both the 18:00 handoff failure and the three -15 kills, which is the proof that the gateway process never restarted in that window — see incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure. The feed of retries is cosmetic for Telegram/email delivery (both work), but it makes Reconnecting <platform> log lines useless as a restart detector once an adapter is absent.
  • Net delivery health for 11-01: 8 rows, 7 delivered, 1 failed — and nothing after 11:01:12 (the missing Weekly Review). The Telegram adapter itself carried every Telegram-bound delivery successfully, so the 11-01 degradation is a latency/retry problem, not a delivery outage.

→ systems/scheduled-tasks, incidents/2026-11-01-cron-worker-sigterm-orphaned-claims