audit(2026-10-10): DHL monitor retired (29 jobs / 26 enabled, verified by ID lookup); 10-10 fleet clean 174/0, 7 incidents closed - 5 open; gateway runtime env recurrence (mnemosyne + elevenlabs) verified fixed; new systems/paseo.md; mattermost unreachable; current-state + index + log updated

This commit is contained in:
hermes
2026-10-11 00:22:20 +08:00
parent c11f11e652
commit 7bffeab157
9 changed files with 295 additions and 18 deletions
+23 -2
View File
@@ -3,8 +3,8 @@ title: Hermes Agent
type: system
status: active
created: 2026-07-22
updated: 2026-10-04
verified_on: 2026-10-04
updated: 2026-10-11
verified_on: 2026-10-11
confidence: high
tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory]
@@ -148,3 +148,24 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
- [[messaging-integrations]]
- [[scheduled-tasks]]
- [[current-state]]
---
## 2026-10-10 - gateway runtime env, TTS, Paseo
- **Gateway process shape (verified 2026-10-11 00:20):** PID 230494, started 2026-10-10 21:24:44,
launched as `/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '<bootstrap>' gateway run`
(RSS ~371 MB). `activate_dependencies()` replaces `sys.path` with a single committed uv
environment (`~/.hermes/installs/<id>/environments/795020dbf24343cd9c52df24723dca83/venv`), and the
user site directory is invisible to it - packages the gateway needs must be installed **into that
env**. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Mnemosyne provider healthy again:** `Memory provider 'mnemosyne' registered (40 tools)` +
`activated` from 2026-10-10 22:01:19 onward (repeated 23:59 / 00:00 / 00:02); `hermes memory
status` reports available.
- **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env
and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS
output appears only when explicitly requested.
- **Mattermost platform unreachable** since 2026-10-10; the gateway keeps retrying on a 5-minute
cadence - [[systems/messaging-integrations]].
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
+25 -2
View File
@@ -3,8 +3,8 @@ title: Messaging Integrations
type: system
status: active
created: 2026-07-22
updated: 2026-10-03
verified_on: 2026-10-03
updated: 2026-10-11
verified_on: 2026-10-11
confidence: high
tags: [messaging, telegram, discord, email, integrations]
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
@@ -70,3 +70,26 @@ Anthony asked whether his other Hermes instances (the estate VMs/CTs) could shar
- [[systems/hermes-agent]]
- [[systems/scheduled-tasks]]
- [[infrastructure/tailscale]]
---
## 2026-10-11 00:20 AWST - Mattermost platform unreachable (observed, not fixed)
The gateway's Mattermost adapter has been down since at least 2026-10-10. `gateway.log` shows a
5-minute retry cadence with `Mattermost: disconnected` followed by
`Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s`
(attempt 34 at 00:15, attempt 35 at 00:20).
Probe of the configured endpoint (`MATTERMOST_URL=http://100.98.116.14:8065/`) from CT460:
**100 % ICMP loss and TCP 8065 refuses** - the Mattermost host itself is unreachable from CT460,
not merely an application-level refusal.
- Cause **not determined**; not investigated further during this audit (no host/VM/proxy probe).
- Impact is limited to the messaging path. No cron job depends on Mattermost, and the 2026-10-10
cron fleet ran clean.
- Config: plugin block `plugins...mattermost` (`enabled: true`, home channel
`5ipo6wfe3j88xxox1gkta3d5mw`, name `Rhino`); the bot token is env-only and is **not** recorded here.
**Open item:** re-probe the host; if the host is back but the adapter stays down, restart the
gateway's Mattermost platform.
+22 -2
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system
status: active
created: 2026-07-22
updated: 2026-10-04
verified_on: 2026-10-04
updated: 2026-10-11
verified_on: 2026-10-11
confidence: high
tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
@@ -105,3 +105,23 @@ Primary agent memory store for Hermes.
- [[incidents/2026-08-01-mnemosyne-capture-failure]]
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]]
- [[incidents/2026-07-29-mnemosyne-relay]]
---
## Re-verified 2026-10-10 (audit window; live probes 2026-10-11 00:14-00:20 AWST)
- **Gateway load path repaired.** The gateway's `activate_dependencies()` selects one committed uv
env (`795020dbf24343cd9c52df24723dca83`); it was missing `mnemosyne` / `mnemosyne_hermes` and the
native dependencies. Repaired 2026-10-10 21:26-21:34 and confirmed live from 22:01:19
(`registered (40 tools)` -> `activated`). Detail:
[[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **"Capture silent since 10-09 15:20" was a false alarm.** It resolved to *no interactive sessions
in the window* - the window was cron-only traffic. What the surrounding hours did capture was
dominated by gateway background-process notifications stored as
`[USER] [Anthony Martin] [IMPORTANT: Background process ...]` rows: recall-visible, low signal.
A capture filter would help; **no fix applied** (carried open item).
- **Coverage script executable again:** `~/.hermes/scripts/check_embedding_coverage.py` now carries
mode `0755`, so the monitor's non-executable finding is resolved.
- **Open, unverified:** 13 orphan rows in `memory_embeddings` (oldest 2026-08-22) reported by a
2026-10-10 monitor run - neither confirmed nor cleaned by this audit.
+92
View File
@@ -0,0 +1,92 @@
---
title: Paseo Daemon
type: system
status: active
created: 2026-10-11
updated: 2026-10-11
verified_on: 2026-10-11
review_after: 2026-11-11
confidence: high
tags: [system, paseo, coding-agent, ct460, systemd]
sources: [live-systemctl-user, live-ps, live-paseo-config, live-session-20261010]
---
# Paseo Daemon (CT460)
Paseo is a **coding-agent orchestrator with a browser terminal workspace**. It runs on
CT460 (`hermes`), hosts terminal sessions against repositories under `/home/hermes`,
and brokers turns to external coding agents — Claude Code, Codex, Cline, OpenCode, and
Hermes itself via ACP.
- Local: `http://127.0.0.1:6767`
- Tailnet: `https://hermes.kangaroo-eel.ts.net:6767`
The daemon spawns provider processes on demand; a live example is
`opencode serve --port 42021` running as a child of the unit.
## State (verified 2026-10-11 00:16 AWST)
| Item | Value |
|---|---|
| Unit | `paseo-daemon.service` — **user** unit, `enabled` |
| Unit file | `/home/hermes/.config/systemd/user/paseo-daemon.service` |
| State | `active (running)` since **2026-10-10 14:47:28 AWST** |
| Daemon version | **0.11.2** (upgraded from 0.7.0 on 2026-10-10) |
| Main PID | `163127` (`node --disable-warning=DEP0040 /home/hermes/.npm-global/bin/paseo daemon run`) |
| Inner processes | supervisor `163168`, `Paseo Daemon` `163198`, terminal worker, `opencode serve` |
| Listen | `0.0.0.0:6767` |
| Binary used by the unit | `/home/hermes/.npm-global/bin/paseo` |
| Config | `/home/hermes/.paseo/config.json` (mode `0600`) |
| Logs | `/home/hermes/.paseo/<YYYYMMDD-HHMM>-NN-daemon.log` (10 MB rotation) and `journalctl --user -u paseo-daemon` |
## Maintenance
`sudo systemctl` will **not** see this unit — it belongs to the user manager:
systemctl --user status paseo-daemon
systemctl --user restart paseo-daemon
journalctl --user -u paseo-daemon -f
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:6767/
Upgrade: `npm install -g @getpaseo/cli@latest`, then restart the unit. **The daemon keeps
running the old version until the unit restarts.**
## Why `ExecStart` uses `daemon run`, not `daemon start`
`paseo daemon start` daemonises and returns immediately, so the foreground process exits
`0`. Under `Type=simple` + `Restart=always` systemd reads that as a clean exit and
restart-loops the service (the flapping observed during the 2026-10-10 repair).
`paseo daemon run` stays in the foreground and is the correct supervised command.
The unit's `PATH` must include both `/home/hermes/.npm-global/bin` and `~/.local/bin`:
agent CLIs the daemon shells out to are resolved through the *unit's* environment, not an
interactive shell's.
## Known pitfalls
- **The `paseo` on `PATH` is not the daemon's version.** `/usr/bin/paseo` reports
**0.7.2** while the unit runs the npm-global **0.11.2**. `paseo --version` therefore
tells you nothing about what the daemon is running — read the unit's `ExecStart`, or
the `daemonVersion` field in the daemon log.
- **A truncated agent binary looks like a provider fault.** A partly-written `cline`
binary (107 KB ELF instead of a full ~3 MB) made that one provider fail while every
other provider worked. Check binary size before chasing provider config; reinstalling
to 3.0.70 fixed it.
- **Hermes as a Paseo provider needs the ACP extra** — `hermes pm install --extra acp`
before the Hermes provider will load (its check reports `Hermes ACP check OK` once
installed).
- **Config edits need a restart.** `baseUrl`/CORS changes in `~/.paseo/config.json` only
take effect after `systemctl --user restart paseo-daemon`. Backups are kept beside the
config (`config.json.bak-20261010-corsfix`, `config.json.bak-20260809-codex-fix`).
- **Cosmetic CLI error while the daemon is healthy (unresolved 2026-10-10).** When a
`daemon.get_status.request` takes ~2.2 s the CLI prints
`daemon detail request to 0.0.0.0:6767 failed` even though `127.0.0.1:6767` returns
HTTP 200. Verify with the health checks above before treating a single such line as an
outage.
- **Pairing payloads are credentials.** The `#offer=…` URL and `~/.paseo/daemon-keypair.json`
carry the daemon key — never paste them into the wiki or a shared channel.
## Related
- [[systems/hermes-agent]]
- [[systems/terminal-backend]]
- [[infrastructure/network-map]]
+46 -3
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-09
verified_on: 2026-10-09
review_after: 2026-10-23
updated: 2026-10-11
verified_on: 2026-10-11
review_after: 2026-10-25
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -171,3 +171,46 @@ See [[incidents/2026-10-09-memory-recall-relevance]].
5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open.
6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.
---
## Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST)
**Fleet: 29 jobs - 26 enabled - 3 disabled** (`jobs.json.updated_at 2026-10-11T00:19:44+08:00`).
The count fell from 30 to 29 because **`62619aa6bcb3` (DHL Tracking - Parcel `8052838382`) is gone**:
open item #1 of the 2026-10-09 audit is **CLOSED** (below). Disabled: `d40085da631c` (Important-mail
monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher), `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` - 10-08 `178 / 2` -
10-09 `163 / 15` - **10-10 `174 completed / 0 failed`** (+1 left `running`). The 2026-10-10 fleet was
wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm.
**Non-clean `last_status`: 5 jobs, down from 10.**
| Job | ID | `last_status` | streak |
|---|---|---|---|
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `error` | 0 |
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `error` | 0 |
| Daily Marketing Job Radar | `dca8482e4f76` | `error` | 1 |
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `error` | 1 |
| Uber Eats Monthly Spend Report | `066bb5191280` | `delivery_failed` | 0 |
The three `streak 0` rows are stale `last_status` fields, not live failure streaks. `1a8e90ea2271`
(stocktake) is back to `last_status: ok`.
**Incidents: 7 closed during 2026-10-10, 5 open.** The 10-09 storm unwound as each job next ran clean
(`ea79a2` 00:03, `cbfe43` 00:27, `1a8e90ea2271` x2 03:06, `85edf6` 07:04, `08eb86` 08:32, `ecc1fc`
12:05). Still open: `1b23c042e17d` + `dca8482e4f76` (`timeout`, `alerted`, both first seen
2026-10-09) and `0e753e9720fd` + `fad5a86979d4` (`agent`; open since 2026-10-01 / 2026-09-30).
**Open item #1 CLOSED - the DHL monitor is retired.** Anthony instructed removal again on the evening
of 2026-10-10 and the job is **absent from live `jobs.json`**, verified 2026-10-11 00:19 by direct ID
lookup (no job id contains `62619`; no job name matches dhl/parcel). This is the third "deleted"
claim for this job - the 2026-10-07 and 2026-10-09 reports were both **false** because the check was
the absence of job output rather than a positive lookup. Removals must be verified by ID lookup.
**Still open:** the Quartz rebuild step in `wiki-maintain.sh` (no `quartz.config.ts`) - skills-index
off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost
platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).