Files
agent-estate-wiki/systems/messaging-integrations.md
T
Tony0410 a0b3734c95 stocktake(2026-11-01): fix the Quartz build (YAML title), close the "unfixed memory ranking" incident, and correct live drift in tailscale/docker/proxmox
BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic.
Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an
unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a
mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)".
A scan of all 117 pages found exactly one such line; it is now properly quoted.
Quartz now builds cleanly: 690 files emitted and deployed.

RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page).
The page claimed the recall-ranking fix "was not present in live config" and that
conflict 152 was "the only open conflict". Both claims were false:
- The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5,
  fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 —
  not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the
  2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks.
- The original regression test now passes: "mnemosyne recall 'Are you up to date'"
  returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale
  Uber Eats episode no longer outranks anything.
- Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise.
All three open items closed; page moved to status: resolved.

LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01):
- infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping
  times out); m93p peer record reads false for the 4th consecutive time while the
  host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages
  refreshed; no IP changes, no missing peers.
- infrastructure/docker-services.md: CT201's Tailscale IP was wrong
  (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale
  worldmonitor-seeder-fast entry removed from the CT200 list, which is now
  regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to
  thirsty_payne.
- infrastructure/proxmox.md: storage percentages refreshed on both hosts from live
  pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an
  elevated load average recorded.
- systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md:
  re-probed and verified_on bumped.
- systems/skills-index.md: regenerated — 301 unique skills in 68 categories,
  46 archived. (301 is the unique (category, skill_name) count; find returns 302
  files due to the known nested obsidian-git-sync alias duplicate.)

Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
2026-11-01 04:48:30 +08:00

11 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Messaging Integrations system active 2026-07-22 2026-11-01 2026-11-01 high
messaging
telegram
discord
email
integrations
raw/configs/hermes-config-sanitized.txt
live-jobs-json
live-himalaya-debug

Messaging Integrations

Telegram

  • Status: enabled, active
  • allowed_chats: -1004321904721 (primary), 1793951355 (DM), plus the remaining chat IDs required by both multiplexed profiles on the default adapter — verified 2026-09-30
  • Rich messages: true
  • Platform toolsets: cli, telegram (full suite)
  • Delivery targets seen in cron: telegram, telegram:1793951355, telegram:-1004321904721, telegram:-1004321904721:notifications, telegram:-1004321904721:102, telegram:-1004321904721:273, telegram:-1004321904721:346, telegram:-1004321904721:1030
  • Notes: Primary delivery target for scheduled jobs

Delivery target semantics (verified 2026-10-01)

  • A bare deliver: "telegram" resolves to the home DM (1793951355) — not to the primary group. It is not a "send to the main chat" shorthand.
  • To reach a forum topic you must use the full form telegram:<chat_id>:<thread_id>, e.g. telegram:-1004321904721:102.
  • Consequence: a job that names a group topic ID but is actually pointed at the DM fails with configured thread_id <n> … was not found / Message thread not found. Two variants are observed, and they behave differently (verified 2026-10-02 against ~/.hermes/cron/executions.db):
    • Message thread not found — hard failure, the message is lost (the 2026-09-30 incident variant).
    • configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id — a thread-less fallback send does occur, yet the run is still recorded as last_status: delivery_failed / delivery_outcome: failed. Content reaches the DM; the delivery is marked failed anyway, so the job record reads like a lost message.
    • Do not assume "delivery failed" means the message never arrived — check the error variant before reporting a report as missing.

Discord

  • Status: enabled, streaming true
  • Platform: hermes-discord
  • Platform toolsets: cli, discord

Email

  • Libraries: himalaya CLI available
  • Platform toolsets: cli
  • Status: ⚠ partially broken — the personal account works, the work account does not.
    • Personal (Gmail): connected and usable. Google still permits app-password auth.
    • work (Outlook / Microsoft 365): unreachable. himalaya envelope list -a work fails with Error: cannot build IMAP client / Caused by: login not supported. The account is configured for password auth (backend.auth.type = "password" + ~/.config/himalaya/get_work_outlook_password.sh) while the tenant advertises only XOAuth2 — basic auth for IMAP/SMTP is disabled tenant-wide. The TCP/TLS connection to outlook.office365.com:993 succeeds; only the login is refused. This is a policy wall, not a wrong password — re-running the credential helper cannot fix it. See incidents/2026-08-26-work-outlook-basic-auth-disabled.
    • Fix path: OAuth2 (backend.auth.type = "oauth2" + token store) after an app registration with IMAP/SMTP scopes; Anthony's normal Office sign-in is number-matching approval in Microsoft Authenticator, which is the device-code flow this needs. Tenant admin consent may be required.
    • Consequence for tooling: email/mail-lookup resolves the account before searching, so a request for work mail will reach this wall and must report the account as unreachable — not as "no results found".

Slack

  • Status: available (toolsets: cli, slack-platform)
  • Config noted: slack entries in config but Slack enabled status unclear from config grep

Other Platforms Available

  • Mattermost: toolsets present (hermes-mattermost)
  • Google Chat: toolsets present (hermes-google_chat)
  • QQ: toolsets present (hermes-qqbot)
  • Signal: toolsets present (hermes-signal)
  • WhatsApp: toolsets present (hermes-whatsapp)
  • ntfy: noted in memory config outputs but runtime status unverified
  • Yuanbao: toolsets present (hermes-yuanbao)
  • Home Assistant: homeassistant-platform enabled

Known Issues

  • Telegram thread-delivery failures — resolved 2026-09-30. Jobs fad5a86979d4 and 1b23c042e17d reported configured thread_id 34795 … was not found, and c8d52904e1d6 reported Message thread not found (last_status: delivery_failed). Root cause was the target, not the topic: those jobs used a bare deliver: "telegram", which resolves to the home DM where group topic IDs do not exist. 1b23c042e17d and c8d52904e1d6 were retargeted to telegram:-1004321904721:102 and verified end-to-end. See systems/scheduled-tasks.
  • Three jobs still deliver to the home DM via a bare telegram target (0e753e9720fd, fad5a86979d4, 066bb5191280). No longer theoretical — two of them failed delivery on 2026-10-01: 066bb5191280 (02:11) and 0e753e9720fd (09:01) both recorded delivery_outcome: failed with the thread-less-fallback variant above, in the same failure class as the 2026-09-30 incident. Retarget them to telegram:-1004321904721:<topic>. See systems/scheduled-tasks.

Multi-bot group chat (open question, raised 2026-09-30)

Anthony asked whether his other Hermes instances (the estate VMs/CTs) could share a group chat.

  • Preferred shape: a Telegram supergroup with one forum topic per bot, so each agent posts into its own thread and no two bots share a message stream. Mattermost was floated and rejected.
  • Estate reachability: CT460's Hermes serves the WebUI/API on :9119 (locally fronted by nginx on :19119). Whether the other estate hosts expose :9119 still needs checking before any fan-out design.
  • Status: design question only — nothing has been deployed or changed.

2026-10-11 00:20 AWST - Mattermost platform unreachable (observed, not fixed)

The gateway's Mattermost adapter has been down since at least 2026-10-10. gateway.log shows a 5-minute retry cadence with Mattermost: disconnected followed by Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s (attempt 34 at 00:15, attempt 35 at 00:20).

Probe of the configured endpoint (MATTERMOST_URL=http://100.98.116.14:8065/) from CT460: 100 % ICMP loss and TCP 8065 refuses - the Mattermost host itself is unreachable from CT460, not merely an application-level refusal.

  • Cause not determined; not investigated further during this audit (no host/VM/proxy probe).
  • Impact is limited to the messaging path. No cron job depends on Mattermost, and the 2026-10-10 cron fleet ran clean.
  • Config: plugin block plugins...mattermost (enabled: true, home channel 5ipo6wfe3j88xxox1gkta3d5mw, name Rhino); the bot token is env-only and is not recorded here.

Open item: re-probe the host; if the host is back but the adapter stays down, restart the gateway's Mattermost platform.

2026-10-19 00:15 AWST — Mattermost adapter: cause identified (CT255 stopped), still down

This pass replaced the 10-11 observation's "cause not determined" with a definite cause. Live probes, 2026-10-19 00:0x-00:15 AWST:

  • tailscale status: peer mattermost (100.98.116.14) — offline, last seen 29d ago, tx 7176 rx 0.
  • gateway.log: the reconnect loop is unchanged — Reconnecting mattermost (attempt 470) at 00:08:36 and (attempt 471) at 00:14:07, each followed by Mattermost: disconnected and Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s (5-minute cadence; 3,992 such lines in the retained gateway.log).
  • Probed from CT460 at the configured endpoint (MATTERMOST_URL=http://100.98.116.14:8065/): TCP 8065 refused/closed and 100 % ICMP loss — the host is unreachable, not the application.
  • Cause: CT255 (mattermost) is stopped on the PVE host (see infrastructure/proxmox). The attempt counter restarted at the gateway restart of 2026-10-17 05:10 and now reads 471.
  • Impact unchanged: messaging path only. No cron job delivers via Mattermost, and the cron fleet performed normally throughout.

Open item: start CT255, or retire the platform block. The 5-minute reconnect loop continues until one of those happens — restarting the adapter while the host is stopped achieves nothing.

2026-10-26 00:1x AWST — Mattermost still down; attempt counter reset by the 10-25 restart

  • Cause unchanged: CT255 (mattermost) is stopped. tailscale status shows the peer offline, last seen 36 d ago, tx 312 / rx 0 — rx has read 0 in every audit since 2026-10-11, i.e. no packet has come back.
  • Reconnect loop still running: 268 mattermost connect timed out lines on 2026-10-25 vs 264 on 2026-10-24 (≈1 per 5 min, as designed); 2 so far on 2026-10-26.
  • The attempt counter reset to 54 with the 19:27:13 gateway restart (Reconnecting mattermost (attempt 54) at 2026-10-26 00:07:06). The previous reset was the 2026-10-17 05:10 restart. ⚠ The counter measures this process's attempts, not the outage's length — do not quote it as downtime.
  • Impact unchanged: messaging path only; no cron job delivers via Mattermost.
  • Not attempted: pct start 255 — starting a service on the PVE host is Anthony's call.

2026-10-31 06:2x AWST — ✅ Mattermost RECOVERED (outage over)

The ~40-day Mattermost outage is over. All three live checks pass, and the gateway log confirms the adapter stayed connected since.

Check Live result (2026-10-31)
pct list on pve CT255 (mattermost) → running (was stopped)
pct exec 255 -- systemctl is-active mattermost active
pct exec 255 -- ss -ltn LISTEN *:8065
tailscale ping 100.98.116.14 pong from mattermost ... in 0s
curl http://100.98.116.14:8065/api/v4/system/ping HTTP 200
Tailscale peer record Online: true — first time since the outage began
  • Gateway reconnection (CT460's own gateway.log): Mattermost: authenticated as @rhino-bot at 2026-10-30 07:19:37, with ✓ mattermost reconnected successfully and a successful WebSocket handshake at the same second. A second connect cycle at 2026-10-30 16:36:07 produced ✓ mattermost connected at 16:36:17.
  • No errors since. Grepping every mattermost line in gateway.log after 2026-10-30 16:36:17 returns zero error / timeout / disconnect / fail lines — the 5-minute reconnect loop that produced 264–268 timeouts a day is no longer running.
  • The prior "cause: CT255 is stopped" finding is superseded. CT255 is running; the gateway can reach http://100.98.116.14:8065/ and authenticate. → infrastructure/proxmox, infrastructure/tailscale
  • Open item now closed: the platform is no longer an outage. The home channel startup notification was delivered successfully on reconnect (Sent home-channel startup notification to mattermost:…).