audit(2026-10-10): DHL monitor retired (29 jobs / 26 enabled, verified by ID lookup); 10-10 fleet clean 174/0, 7 incidents closed - 5 open; gateway runtime env recurrence (mnemosyne + elevenlabs) verified fixed; new systems/paseo.md; mattermost unreachable; current-state + index + log updated
This commit is contained in:
+23
-2
@@ -3,8 +3,8 @@ title: Hermes Agent
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
confidence: high
|
||||
tags: [system, hermes, agent, llm]
|
||||
sources: [live-hermes-version, live-config, live-skill-inventory]
|
||||
@@ -148,3 +148,24 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
||||
- [[messaging-integrations]]
|
||||
- [[scheduled-tasks]]
|
||||
- [[current-state]]
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-10 - gateway runtime env, TTS, Paseo
|
||||
|
||||
- **Gateway process shape (verified 2026-10-11 00:20):** PID 230494, started 2026-10-10 21:24:44,
|
||||
launched as `/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '<bootstrap>' gateway run`
|
||||
(RSS ~371 MB). `activate_dependencies()` replaces `sys.path` with a single committed uv
|
||||
environment (`~/.hermes/installs/<id>/environments/795020dbf24343cd9c52df24723dca83/venv`), and the
|
||||
user site directory is invisible to it - packages the gateway needs must be installed **into that
|
||||
env**. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||
- **Mnemosyne provider healthy again:** `Memory provider 'mnemosyne' registered (40 tools)` +
|
||||
`activated` from 2026-10-10 22:01:19 onward (repeated 23:59 / 00:00 / 00:02); `hermes memory
|
||||
status` reports available.
|
||||
- **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env
|
||||
and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS
|
||||
output appears only when explicitly requested.
|
||||
- **Mattermost platform unreachable** since 2026-10-10; the gateway keeps retrying on a 5-minute
|
||||
cadence - [[systems/messaging-integrations]].
|
||||
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Messaging Integrations
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-03
|
||||
verified_on: 2026-10-03
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
confidence: high
|
||||
tags: [messaging, telegram, discord, email, integrations]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
|
||||
@@ -70,3 +70,26 @@ Anthony asked whether his other Hermes instances (the estate VMs/CTs) could shar
|
||||
- [[systems/hermes-agent]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[infrastructure/tailscale]]
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-11 00:20 AWST - Mattermost platform unreachable (observed, not fixed)
|
||||
|
||||
The gateway's Mattermost adapter has been down since at least 2026-10-10. `gateway.log` shows a
|
||||
5-minute retry cadence with `Mattermost: disconnected` followed by
|
||||
`Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s`
|
||||
(attempt 34 at 00:15, attempt 35 at 00:20).
|
||||
|
||||
Probe of the configured endpoint (`MATTERMOST_URL=http://100.98.116.14:8065/`) from CT460:
|
||||
**100 % ICMP loss and TCP 8065 refuses** - the Mattermost host itself is unreachable from CT460,
|
||||
not merely an application-level refusal.
|
||||
|
||||
- Cause **not determined**; not investigated further during this audit (no host/VM/proxy probe).
|
||||
- Impact is limited to the messaging path. No cron job depends on Mattermost, and the 2026-10-10
|
||||
cron fleet ran clean.
|
||||
- Config: plugin block `plugins...mattermost` (`enabled: true`, home channel
|
||||
`5ipo6wfe3j88xxox1gkta3d5mw`, name `Rhino`); the bot token is env-only and is **not** recorded here.
|
||||
|
||||
**Open item:** re-probe the host; if the host is back but the adapter stays down, restart the
|
||||
gateway's Mattermost platform.
|
||||
|
||||
+22
-2
@@ -3,8 +3,8 @@ title: Mnemosyne
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, backend]
|
||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||||
@@ -105,3 +105,23 @@ Primary agent memory store for Hermes.
|
||||
- [[incidents/2026-08-01-mnemosyne-capture-failure]]
|
||||
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]]
|
||||
- [[incidents/2026-07-29-mnemosyne-relay]]
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Re-verified 2026-10-10 (audit window; live probes 2026-10-11 00:14-00:20 AWST)
|
||||
|
||||
- **Gateway load path repaired.** The gateway's `activate_dependencies()` selects one committed uv
|
||||
env (`795020dbf24343cd9c52df24723dca83`); it was missing `mnemosyne` / `mnemosyne_hermes` and the
|
||||
native dependencies. Repaired 2026-10-10 21:26-21:34 and confirmed live from 22:01:19
|
||||
(`registered (40 tools)` -> `activated`). Detail:
|
||||
[[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||
- **"Capture silent since 10-09 15:20" was a false alarm.** It resolved to *no interactive sessions
|
||||
in the window* - the window was cron-only traffic. What the surrounding hours did capture was
|
||||
dominated by gateway background-process notifications stored as
|
||||
`[USER] [Anthony Martin] [IMPORTANT: Background process ...]` rows: recall-visible, low signal.
|
||||
A capture filter would help; **no fix applied** (carried open item).
|
||||
- **Coverage script executable again:** `~/.hermes/scripts/check_embedding_coverage.py` now carries
|
||||
mode `0755`, so the monitor's non-executable finding is resolved.
|
||||
- **Open, unverified:** 13 orphan rows in `memory_embeddings` (oldest 2026-08-22) reported by a
|
||||
2026-10-10 monitor run - neither confirmed nor cleaned by this audit.
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
title: Paseo Daemon
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-10-11
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
review_after: 2026-11-11
|
||||
confidence: high
|
||||
tags: [system, paseo, coding-agent, ct460, systemd]
|
||||
sources: [live-systemctl-user, live-ps, live-paseo-config, live-session-20261010]
|
||||
---
|
||||
|
||||
# Paseo Daemon (CT460)
|
||||
|
||||
Paseo is a **coding-agent orchestrator with a browser terminal workspace**. It runs on
|
||||
CT460 (`hermes`), hosts terminal sessions against repositories under `/home/hermes`,
|
||||
and brokers turns to external coding agents — Claude Code, Codex, Cline, OpenCode, and
|
||||
Hermes itself via ACP.
|
||||
|
||||
- Local: `http://127.0.0.1:6767`
|
||||
- Tailnet: `https://hermes.kangaroo-eel.ts.net:6767`
|
||||
|
||||
The daemon spawns provider processes on demand; a live example is
|
||||
`opencode serve --port 42021` running as a child of the unit.
|
||||
|
||||
## State (verified 2026-10-11 00:16 AWST)
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Unit | `paseo-daemon.service` — **user** unit, `enabled` |
|
||||
| Unit file | `/home/hermes/.config/systemd/user/paseo-daemon.service` |
|
||||
| State | `active (running)` since **2026-10-10 14:47:28 AWST** |
|
||||
| Daemon version | **0.11.2** (upgraded from 0.7.0 on 2026-10-10) |
|
||||
| Main PID | `163127` (`node --disable-warning=DEP0040 /home/hermes/.npm-global/bin/paseo daemon run`) |
|
||||
| Inner processes | supervisor `163168`, `Paseo Daemon` `163198`, terminal worker, `opencode serve` |
|
||||
| Listen | `0.0.0.0:6767` |
|
||||
| Binary used by the unit | `/home/hermes/.npm-global/bin/paseo` |
|
||||
| Config | `/home/hermes/.paseo/config.json` (mode `0600`) |
|
||||
| Logs | `/home/hermes/.paseo/<YYYYMMDD-HHMM>-NN-daemon.log` (10 MB rotation) and `journalctl --user -u paseo-daemon` |
|
||||
|
||||
## Maintenance
|
||||
|
||||
`sudo systemctl` will **not** see this unit — it belongs to the user manager:
|
||||
|
||||
systemctl --user status paseo-daemon
|
||||
systemctl --user restart paseo-daemon
|
||||
journalctl --user -u paseo-daemon -f
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:6767/
|
||||
|
||||
Upgrade: `npm install -g @getpaseo/cli@latest`, then restart the unit. **The daemon keeps
|
||||
running the old version until the unit restarts.**
|
||||
|
||||
## Why `ExecStart` uses `daemon run`, not `daemon start`
|
||||
|
||||
`paseo daemon start` daemonises and returns immediately, so the foreground process exits
|
||||
`0`. Under `Type=simple` + `Restart=always` systemd reads that as a clean exit and
|
||||
restart-loops the service (the flapping observed during the 2026-10-10 repair).
|
||||
`paseo daemon run` stays in the foreground and is the correct supervised command.
|
||||
|
||||
The unit's `PATH` must include both `/home/hermes/.npm-global/bin` and `~/.local/bin`:
|
||||
agent CLIs the daemon shells out to are resolved through the *unit's* environment, not an
|
||||
interactive shell's.
|
||||
|
||||
## Known pitfalls
|
||||
|
||||
- **The `paseo` on `PATH` is not the daemon's version.** `/usr/bin/paseo` reports
|
||||
**0.7.2** while the unit runs the npm-global **0.11.2**. `paseo --version` therefore
|
||||
tells you nothing about what the daemon is running — read the unit's `ExecStart`, or
|
||||
the `daemonVersion` field in the daemon log.
|
||||
- **A truncated agent binary looks like a provider fault.** A partly-written `cline`
|
||||
binary (107 KB ELF instead of a full ~3 MB) made that one provider fail while every
|
||||
other provider worked. Check binary size before chasing provider config; reinstalling
|
||||
to 3.0.70 fixed it.
|
||||
- **Hermes as a Paseo provider needs the ACP extra** — `hermes pm install --extra acp`
|
||||
before the Hermes provider will load (its check reports `Hermes ACP check OK` once
|
||||
installed).
|
||||
- **Config edits need a restart.** `baseUrl`/CORS changes in `~/.paseo/config.json` only
|
||||
take effect after `systemctl --user restart paseo-daemon`. Backups are kept beside the
|
||||
config (`config.json.bak-20261010-corsfix`, `config.json.bak-20260809-codex-fix`).
|
||||
- **Cosmetic CLI error while the daemon is healthy (unresolved 2026-10-10).** When a
|
||||
`daemon.get_status.request` takes ~2.2 s the CLI prints
|
||||
`daemon detail request to 0.0.0.0:6767 failed` even though `127.0.0.1:6767` returns
|
||||
HTTP 200. Verify with the health checks above before treating a single such line as an
|
||||
outage.
|
||||
- **Pairing payloads are credentials.** The `#offer=…` URL and `~/.paseo/daemon-keypair.json`
|
||||
carry the daemon key — never paste them into the wiki or a shared channel.
|
||||
|
||||
## Related
|
||||
- [[systems/hermes-agent]]
|
||||
- [[systems/terminal-backend]]
|
||||
- [[infrastructure/network-map]]
|
||||
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-09
|
||||
verified_on: 2026-10-09
|
||||
review_after: 2026-10-23
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
@@ -171,3 +171,46 @@ See [[incidents/2026-10-09-memory-recall-relevance]].
|
||||
5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open.
|
||||
6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST)
|
||||
|
||||
**Fleet: 29 jobs - 26 enabled - 3 disabled** (`jobs.json.updated_at 2026-10-11T00:19:44+08:00`).
|
||||
The count fell from 30 to 29 because **`62619aa6bcb3` (DHL Tracking - Parcel `8052838382`) is gone**:
|
||||
open item #1 of the 2026-10-09 audit is **CLOSED** (below). Disabled: `d40085da631c` (Important-mail
|
||||
monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher), `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
|
||||
|
||||
**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` - 10-08 `178 / 2` -
|
||||
10-09 `163 / 15` - **10-10 `174 completed / 0 failed`** (+1 left `running`). The 2026-10-10 fleet was
|
||||
wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm.
|
||||
|
||||
**Non-clean `last_status`: 5 jobs, down from 10.**
|
||||
|
||||
| Job | ID | `last_status` | streak |
|
||||
|---|---|---|---|
|
||||
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `error` | 0 |
|
||||
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `error` | 0 |
|
||||
| Daily Marketing Job Radar | `dca8482e4f76` | `error` | 1 |
|
||||
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `error` | 1 |
|
||||
| Uber Eats Monthly Spend Report | `066bb5191280` | `delivery_failed` | 0 |
|
||||
|
||||
The three `streak 0` rows are stale `last_status` fields, not live failure streaks. `1a8e90ea2271`
|
||||
(stocktake) is back to `last_status: ok`.
|
||||
|
||||
**Incidents: 7 closed during 2026-10-10, 5 open.** The 10-09 storm unwound as each job next ran clean
|
||||
(`ea79a2` 00:03, `cbfe43` 00:27, `1a8e90ea2271` x2 03:06, `85edf6` 07:04, `08eb86` 08:32, `ecc1fc`
|
||||
12:05). Still open: `1b23c042e17d` + `dca8482e4f76` (`timeout`, `alerted`, both first seen
|
||||
2026-10-09) and `0e753e9720fd` + `fad5a86979d4` (`agent`; open since 2026-10-01 / 2026-09-30).
|
||||
|
||||
**Open item #1 CLOSED - the DHL monitor is retired.** Anthony instructed removal again on the evening
|
||||
of 2026-10-10 and the job is **absent from live `jobs.json`**, verified 2026-10-11 00:19 by direct ID
|
||||
lookup (no job id contains `62619`; no job name matches dhl/parcel). This is the third "deleted"
|
||||
claim for this job - the 2026-10-07 and 2026-10-09 reports were both **false** because the check was
|
||||
the absence of job output rather than a positive lookup. Removals must be verified by ID lookup.
|
||||
|
||||
**Still open:** the Quartz rebuild step in `wiki-maintain.sh` (no `quartz.config.ts`) - skills-index
|
||||
off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost
|
||||
platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the
|
||||
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).
|
||||
|
||||
Reference in New Issue
Block a user