BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic. Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)". A scan of all 117 pages found exactly one such line; it is now properly quoted. Quartz now builds cleanly: 690 files emitted and deployed. RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page). The page claimed the recall-ranking fix "was not present in live config" and that conflict 152 was "the only open conflict". Both claims were false: - The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5, fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 — not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the 2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks. - The original regression test now passes: "mnemosyne recall 'Are you up to date'" returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale Uber Eats episode no longer outranks anything. - Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise. All three open items closed; page moved to status: resolved. LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01): - infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping times out); m93p peer record reads false for the 4th consecutive time while the host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages refreshed; no IP changes, no missing peers. - infrastructure/docker-services.md: CT201's Tailscale IP was wrong (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale worldmonitor-seeder-fast entry removed from the CT200 list, which is now regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to thirsty_payne. - infrastructure/proxmox.md: storage percentages refreshed on both hosts from live pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an elevated load average recorded. - systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md: re-probed and verified_on bumped. - systems/skills-index.md: regenerated — 301 unique skills in 68 categories, 46 archived. (301 is the unique (category, skill_name) count; find returns 302 files due to the known nested obsidian-git-sync alias duplicate.) Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
155 lines
11 KiB
Markdown
155 lines
11 KiB
Markdown
---
|
||
title: Messaging Integrations
|
||
type: system
|
||
status: active
|
||
created: 2026-07-22
|
||
updated: 2026-11-01
|
||
verified_on: 2026-11-01
|
||
confidence: high
|
||
tags: [messaging, telegram, discord, email, integrations]
|
||
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
|
||
---
|
||
|
||
# Messaging Integrations
|
||
|
||
## Telegram
|
||
- **Status:** enabled, active
|
||
- **allowed_chats:** `-1004321904721` (primary), `1793951355` (DM), plus the remaining chat IDs required by both multiplexed profiles on the default adapter — verified 2026-09-30
|
||
- **Rich messages:** true
|
||
- **Platform toolsets:** cli, telegram (full suite)
|
||
- **Delivery targets seen in cron:** telegram, `telegram:1793951355`, `telegram:-1004321904721`, `telegram:-1004321904721:notifications`, `telegram:-1004321904721:102`, `telegram:-1004321904721:273`, `telegram:-1004321904721:346`, `telegram:-1004321904721:1030`
|
||
- **Notes:** Primary delivery target for scheduled jobs
|
||
|
||
### Delivery target semantics (verified 2026-10-01)
|
||
- A **bare `deliver: "telegram"` resolves to the home DM (`1793951355`)** — *not* to the primary group. It is not a "send to the main chat" shorthand.
|
||
- To reach a forum topic you must use the full form **`telegram:<chat_id>:<thread_id>`**, e.g. `telegram:-1004321904721:102`.
|
||
- Consequence: a job that names a group topic ID but is actually pointed at the DM fails with `configured thread_id <n> … was not found` / `Message thread not found`. **Two variants are observed, and they behave differently** (verified 2026-10-02 against `~/.hermes/cron/executions.db`):
|
||
- `Message thread not found` — hard failure, **the message is lost** (the 2026-09-30 incident variant).
|
||
- `configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id` — a **thread-less fallback send** does occur, yet the run is still recorded as `last_status: delivery_failed` / `delivery_outcome: failed`. Content reaches the DM; the delivery is marked failed anyway, so the job record reads like a lost message.
|
||
- Do not assume "delivery failed" means the message never arrived — check the error variant before reporting a report as missing.
|
||
|
||
## Discord
|
||
- **Status:** enabled, streaming true
|
||
- **Platform:** `hermes-discord`
|
||
- **Platform toolsets:** cli, discord
|
||
|
||
## Email
|
||
- **Libraries:** himalaya CLI available
|
||
- **Platform toolsets:** cli
|
||
- **Status: ⚠ partially broken — the personal account works, the `work` account does not.**
|
||
- **Personal (Gmail):** connected and usable. Google still permits app-password auth.
|
||
- **`work` (Outlook / Microsoft 365):** **unreachable.** `himalaya envelope list -a work` fails with `Error: cannot build IMAP client / Caused by: login not supported`. The account is configured for password auth (`backend.auth.type = "password"` + `~/.config/himalaya/get_work_outlook_password.sh`) while the tenant advertises **only `XOAuth2`** — basic auth for IMAP/SMTP is disabled tenant-wide. The TCP/TLS connection to `outlook.office365.com:993` succeeds; only the login is refused. This is a policy wall, **not a wrong password** — re-running the credential helper cannot fix it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
|
||
- Fix path: OAuth2 (`backend.auth.type = "oauth2"` + token store) after an app registration with IMAP/SMTP scopes; Anthony's normal Office sign-in is number-matching approval in Microsoft Authenticator, which is the device-code flow this needs. Tenant admin consent may be required.
|
||
- Consequence for tooling: `email/mail-lookup` resolves the account before searching, so a request for work mail will reach this wall and must report the account as unreachable — not as "no results found".
|
||
|
||
## Slack
|
||
- **Status:** available (toolsets: cli, slack-platform)
|
||
- **Config noted:** slack entries in config but Slack enabled status unclear from config grep
|
||
|
||
## Other Platforms Available
|
||
- **Mattermost:** toolsets present (`hermes-mattermost`)
|
||
- **Google Chat:** toolsets present (`hermes-google_chat`)
|
||
- **QQ:** toolsets present (`hermes-qqbot`)
|
||
- **Signal:** toolsets present (`hermes-signal`)
|
||
- **WhatsApp:** toolsets present (`hermes-whatsapp`)
|
||
- **ntfy:** noted in memory config outputs but runtime status unverified
|
||
- **Yuanbao:** toolsets present (`hermes-yuanbao`)
|
||
- **Home Assistant:** homeassistant-platform enabled
|
||
|
||
## Known Issues
|
||
- **Telegram thread-delivery failures — resolved 2026-09-30.** Jobs `fad5a86979d4` and `1b23c042e17d` reported `configured thread_id 34795 … was not found`, and `c8d52904e1d6` reported `Message thread not found` (`last_status: delivery_failed`). Root cause was the *target*, not the topic: those jobs used a bare `deliver: "telegram"`, which resolves to the home DM where group topic IDs do not exist. `1b23c042e17d` and `c8d52904e1d6` were retargeted to `telegram:-1004321904721:102` and verified end-to-end. See [[systems/scheduled-tasks]].
|
||
- **Three jobs still deliver to the home DM via a bare `telegram` target** (`0e753e9720fd`, `fad5a86979d4`, `066bb5191280`). **No longer theoretical — two of them failed delivery on 2026-10-01:** `066bb5191280` (02:11) and `0e753e9720fd` (09:01) both recorded `delivery_outcome: failed` with the thread-less-fallback variant above, in the same failure class as the 2026-09-30 incident. Retarget them to `telegram:-1004321904721:<topic>`. See [[systems/scheduled-tasks]].
|
||
|
||
## Multi-bot group chat (open question, raised 2026-09-30)
|
||
Anthony asked whether his other Hermes instances (the estate VMs/CTs) could share a group chat.
|
||
- **Preferred shape:** a Telegram supergroup with **one forum topic per bot**, so each agent posts into its own thread and no two bots share a message stream. Mattermost was floated and rejected.
|
||
- **Estate reachability:** CT460's Hermes serves the WebUI/API on `:9119` (locally fronted by nginx on `:19119`). Whether the other estate hosts expose `:9119` still needs checking before any fan-out design.
|
||
- **Status:** design question only — nothing has been deployed or changed.
|
||
|
||
## Related
|
||
- [[systems/hermes-agent]]
|
||
- [[systems/scheduled-tasks]]
|
||
- [[infrastructure/tailscale]]
|
||
|
||
|
||
---
|
||
|
||
## 2026-10-11 00:20 AWST - Mattermost platform unreachable (observed, not fixed)
|
||
|
||
The gateway's Mattermost adapter has been down since at least 2026-10-10. `gateway.log` shows a
|
||
5-minute retry cadence with `Mattermost: disconnected` followed by
|
||
`Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s`
|
||
(attempt 34 at 00:15, attempt 35 at 00:20).
|
||
|
||
Probe of the configured endpoint (`MATTERMOST_URL=http://100.98.116.14:8065/`) from CT460:
|
||
**100 % ICMP loss and TCP 8065 refuses** - the Mattermost host itself is unreachable from CT460,
|
||
not merely an application-level refusal.
|
||
|
||
- Cause **not determined**; not investigated further during this audit (no host/VM/proxy probe).
|
||
- Impact is limited to the messaging path. No cron job depends on Mattermost, and the 2026-10-10
|
||
cron fleet ran clean.
|
||
- Config: plugin block `plugins...mattermost` (`enabled: true`, home channel
|
||
`5ipo6wfe3j88xxox1gkta3d5mw`, name `Rhino`); the bot token is env-only and is **not** recorded here.
|
||
|
||
**Open item:** re-probe the host; if the host is back but the adapter stays down, restart the
|
||
gateway's Mattermost platform.
|
||
|
||
## 2026-10-19 00:15 AWST — Mattermost adapter: cause identified (CT255 stopped), still down
|
||
|
||
This pass replaced the 10-11 observation's "cause not determined" with a definite cause. Live probes,
|
||
2026-10-19 00:0x-00:15 AWST:
|
||
|
||
- `tailscale status`: peer `mattermost` (`100.98.116.14`) — **offline, last seen 29d ago**, `tx 7176 rx 0`.
|
||
- `gateway.log`: the reconnect loop is unchanged — `Reconnecting mattermost (attempt 470)` at 00:08:36
|
||
and `(attempt 471)` at 00:14:07, each followed by `Mattermost: disconnected` and
|
||
`Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s`
|
||
(5-minute cadence; 3,992 such lines in the retained `gateway.log`).
|
||
- Probed from CT460 at the configured endpoint (`MATTERMOST_URL=http://100.98.116.14:8065/`):
|
||
**TCP 8065 refused/closed and 100 % ICMP loss** — the host is unreachable, not the application.
|
||
- **Cause: CT255 (`mattermost`) is `stopped` on the PVE host** (see [[infrastructure/proxmox]]).
|
||
The attempt counter restarted at the gateway restart of 2026-10-17 05:10 and now reads 471.
|
||
- Impact unchanged: messaging path only. No cron job delivers via Mattermost, and the cron fleet
|
||
performed normally throughout.
|
||
|
||
**Open item:** start CT255, or retire the platform block. The 5-minute reconnect loop continues until
|
||
one of those happens — restarting the adapter while the host is stopped achieves nothing.
|
||
|
||
## 2026-10-26 00:1x AWST — Mattermost still down; attempt counter reset by the 10-25 restart
|
||
|
||
- **Cause unchanged: CT255 (`mattermost`) is `stopped`.** `tailscale status` shows the peer **offline, last seen
|
||
36 d ago**, `tx 312 / rx 0` — `rx` has read 0 in every audit since 2026-10-11, i.e. no packet has come back.
|
||
- **Reconnect loop still running:** **268** `mattermost connect timed out` lines on 2026-10-25 vs **264** on
|
||
2026-10-24 (≈1 per 5 min, as designed); **2** so far on 2026-10-26.
|
||
- **The attempt counter reset to 54** with the 19:27:13 gateway restart (`Reconnecting mattermost (attempt 54)`
|
||
at 2026-10-26 00:07:06). The previous reset was the 2026-10-17 05:10 restart. ⚠ The counter measures **this
|
||
process's** attempts, not the outage's length — do not quote it as downtime.
|
||
- **Impact unchanged:** messaging path only; no cron job delivers via Mattermost.
|
||
- **Not attempted:** `pct start 255` — starting a service on the PVE host is Anthony's call.
|
||
|
||
## 2026-10-31 06:2x AWST — ✅ Mattermost RECOVERED (outage over)
|
||
|
||
**The ~40-day Mattermost outage is over.** All three live checks pass, and the gateway log confirms
|
||
the adapter stayed connected since.
|
||
|
||
| Check | Live result (2026-10-31) |
|
||
|---|---|
|
||
| `pct list` on `pve` | **CT255 (`mattermost`) → `running`** (was `stopped`) |
|
||
| `pct exec 255 -- systemctl is-active mattermost` | **`active`** |
|
||
| `pct exec 255 -- ss -ltn` | `LISTEN *:8065` |
|
||
| `tailscale ping 100.98.116.14` | **`pong from mattermost ... in 0s`** |
|
||
| `curl http://100.98.116.14:8065/api/v4/system/ping` | **HTTP 200** |
|
||
| Tailscale peer record | **`Online: true`** — first time since the outage began |
|
||
|
||
- **Gateway reconnection (CT460's own `gateway.log`):** `Mattermost: authenticated as @rhino-bot` at
|
||
**2026-10-30 07:19:37**, with `✓ mattermost reconnected successfully` and a successful WebSocket
|
||
handshake at the same second. A second connect cycle at **2026-10-30 16:36:07** produced
|
||
`✓ mattermost connected` at **16:36:17**.
|
||
- **No errors since.** Grepping every `mattermost` line in `gateway.log` after `2026-10-30 16:36:17`
|
||
returns **zero** `error` / `timeout` / `disconnect` / `fail` lines — the 5-minute reconnect loop that
|
||
produced 264–268 timeouts a day is **no longer running**.
|
||
- **The prior "cause: CT255 is stopped" finding is superseded.** CT255 is running; the gateway can
|
||
reach `http://100.98.116.14:8065/` and authenticate. → [[infrastructure/proxmox]],
|
||
[[infrastructure/tailscale]]
|
||
- **Open item now closed:** the platform is no longer an outage. The home channel startup notification
|
||
was delivered successfully on reconnect (`Sent home-channel startup notification to mattermost:…`).
|