audit(2026-10-10): DHL monitor retired (29 jobs / 26 enabled, verified by ID lookup); 10-10 fleet clean 174/0, 7 incidents closed - 5 open; gateway runtime env recurrence (mnemosyne + elevenlabs) verified fixed; new systems/paseo.md; mattermost unreachable; current-state + index + log updated
This commit is contained in:
+15
-4
@@ -3,16 +3,27 @@ title: Current State
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-10
|
updated: 2026-10-11
|
||||||
verified_on: 2026-10-10
|
verified_on: 2026-10-11
|
||||||
review_after: 2026-10-31
|
review_after: 2026-10-25
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system]
|
tags: [system]
|
||||||
sources: [live-config, live-cron-list, live-hermes-version]
|
sources: [live-config, live-cron-list, live-hermes-version]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Current State
|
# Current State
|
||||||
Updated: 2026-10-10
|
Updated: 2026-10-11
|
||||||
|
|
||||||
|
## Recent Changes (2026-10-10 -> 10-11, nightly audit of the 2026-10-09 -> 2026-10-10 window; probes 2026-10-11 00:14-00:20 AWST)
|
||||||
|
|
||||||
|
_The manual 2026-10-10 stocktake entry follows beneath this one._
|
||||||
|
|
||||||
|
- **The cron fleet is clean again: 174 `completed`, 0 `failed` on 2026-10-10** (AWST day, `executions.db`) - after 10-07 `1` / 10-08 `2` / 10-09 `15` failures. 7 of the 10 storm incidents auto-closed during the day; **5 remain open** over 3 jobs (`1b23c042e17d` + `dca8482e4f76` `timeout`/`alerted` since 2026-10-09; `0e753e9720fd` + `fad5a86979d4` `agent`). Non-clean `last_status` down from 10 to **5**. Job count **30 -> 29 (26 enabled / 3 disabled)**. -> [[systems/scheduled-tasks]]
|
||||||
|
- **The DHL monitor `62619aa6bcb3` is retired - open item #1 closed.** Anthony instructed removal again on the evening of 2026-10-10; the job is **absent from live `jobs.json`** (verified 2026-10-11 00:19 by direct ID lookup). The 2026-10-07 and 2026-10-09 "deleted" reports were false - they inferred removal from missing output instead of looking up the ID.
|
||||||
|
- **The 2026-09-28 runtime-dependency class recurred - in the *gateway* env this time.** The gateway's `activate_dependencies()` loads one committed uv env (`795020dbf24343cd9c52df24723dca83`); it was missing `mnemosyne` / `mnemosyne_hermes` plus native deps, and `elevenlabs` was missing from the live managed env. Repaired 21:26-21:34, verified live from **22:01:19** (`Memory provider 'mnemosyne' registered (40 tools)` -> `activated`; repeated 23:59 / 00:00 / 00:02). The 15-minute guard does **not** cover this env - open. -> [[incidents/2026-09-28-hermes-managed-runtime-deps]]
|
||||||
|
- **New page: [[systems/paseo]]** - Paseo coding-agent daemon, user unit `paseo-daemon.service`, v0.11.2, `0.0.0.0:6767`, running since 2026-10-10 14:47:28. Pitfalls recorded: `daemon run` vs `daemon start`, `/usr/bin/paseo` is 0.7.2 while the daemon is 0.11.2, and a truncated `cline` binary presenting as a provider fault.
|
||||||
|
- **Mattermost unreachable from CT460** - configured endpoint `100.98.116.14:8065` gives 100 % ICMP loss and TCP refuse; the gateway retries every 5 minutes. Cause not determined. -> [[systems/messaging-integrations]]
|
||||||
|
- **Health-monitor follow-ups (2026-10-10):** "capture silent" was a false alarm (cron-only window); `check_embedding_coverage.py` now carries mode `0755`; 13 orphan `memory_embeddings` rows (oldest 2026-08-22) remain unverified. -> [[systems/mnemosyne]]
|
||||||
|
|
||||||
## Recent Changes (2026-10-10)
|
## Recent Changes (2026-10-10)
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Hermes Managed-Runtime Dependency Failure
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-09-30
|
created: 2026-09-30
|
||||||
updated: 2026-09-30
|
updated: 2026-10-11
|
||||||
verified_on: 2026-09-30
|
verified_on: 2026-10-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, cron, runtime, python, worker, incident]
|
tags: [hermes, cron, runtime, python, worker, incident]
|
||||||
sources: [live-jobs-json, live-guard-script, live-cron-list]
|
sources: [live-jobs-json, live-guard-script, live-cron-list]
|
||||||
@@ -92,3 +92,42 @@ Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the
|
|||||||
- [[systems/hermes-agent]]
|
- [[systems/hermes-agent]]
|
||||||
- [[systems/mnemosyne]]
|
- [[systems/mnemosyne]]
|
||||||
- [[runbooks/restart-hermes]]
|
- [[runbooks/restart-hermes]]
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Recurrence - 2026-10-10 (this time the *gateway* runtime env, plus a new failure mode)
|
||||||
|
|
||||||
|
**The class recurred twice in one day, on packages the 15-minute guard does not cover.**
|
||||||
|
|
||||||
|
**1. `mnemosyne` missing from the committed gateway env (repaired 21:26-21:34, verified 22:01:19).**
|
||||||
|
The gateway does not run from a normal venv: it launches as
|
||||||
|
`/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '<bootstrap>' gateway run`
|
||||||
|
and `activate_dependencies()` **replaces `sys.path`** with one committed uv environment -
|
||||||
|
`~/.hermes/installs/<id>/environments/795020dbf24343cd9c52df24723dca83/venv`. Of the uv
|
||||||
|
environments on CT460, that one alone was missing `mnemosyne` / `mnemosyne_hermes`, so sessions
|
||||||
|
logged `mnemosyne_batch helpers unavailable (No module named 'mnemosyne')` and
|
||||||
|
`Memory provider 'mnemosyne' is selected but reports unavailable`. Fix: install
|
||||||
|
`hermes_memory_provider` + `mnemosyne_hermes` + `mnemosyne` **together with the native dependencies**
|
||||||
|
(`fastembed`, `onnxruntime`, `numpy` + `numpy.libs`, `tokenizers`, `huggingface_hub`, `sqlite-vec`,
|
||||||
|
`loguru`, `mmh3`) into that committed venv. Verified live: `Memory provider 'mnemosyne' registered
|
||||||
|
(40 tools)` -> `activated` at 2026-10-10 22:01:19, again at 23:59 / 00:00 / 00:02; and
|
||||||
|
`hermes memory status` -> `Plugin: installed / Status: available`.
|
||||||
|
|
||||||
|
**2. `elevenlabs` missing from the live managed env (21:13 -> 21:24).** TTS was configured but the
|
||||||
|
package was absent from the managed runtime in use; reinstalled and verified by generating a real
|
||||||
|
clip.
|
||||||
|
|
||||||
|
**Trap: `~/.local/lib/python3.14/site-packages` is invisible to the gateway.** The first repair
|
||||||
|
attempt installed into the user site directory; with `ENABLE_USER_SITE=False` **and** a replaced
|
||||||
|
`sys.path`, the gateway never sees it. Install into the committed uv env that
|
||||||
|
`activate_dependencies()` selects - not the user site dir, not a sibling venv.
|
||||||
|
|
||||||
|
**Trap: `hermes memory status` can be green while the gateway is still broken.** It is a separate
|
||||||
|
process with its own runtime resolution. The gateway's own log lines
|
||||||
|
(`registered (40 tools)` / `activated` versus `reports unavailable`) are the authoritative check.
|
||||||
|
|
||||||
|
**Trap: this recurrence produced no cron failures at all** (2026-10-10: 174 `completed`, 0 `failed`).
|
||||||
|
It degraded the gateway's memory and TTS paths only. A green cron day does **not** mean the runtime
|
||||||
|
is healthy - and the 15-minute guard still does not check the gateway's committed env, so **guard
|
||||||
|
coverage remains open.**
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Anthony's Systems Wiki
|
# Anthony's Systems Wiki
|
||||||
> Operational knowledge for Anthony's agents, infrastructure,
|
> Operational knowledge for Anthony's agents, infrastructure,
|
||||||
> model routing and automation. Last updated: 2026-10-10
|
> model routing and automation. Last updated: 2026-10-11
|
||||||
|
|
||||||
## Start here
|
## Start here
|
||||||
- [[current-state]] — Current systems status
|
- [[current-state]] — Current systems status
|
||||||
@@ -22,13 +22,14 @@
|
|||||||
- [[systems/nanobot]] — Nanobot current state and history
|
- [[systems/nanobot]] — Nanobot current state and history
|
||||||
- [[systems/browser-backend]] — Camoufox (primary) + Steel (disabled), CDP config
|
- [[systems/browser-backend]] — Camoufox (primary) + Steel (disabled), CDP config
|
||||||
- [[systems/hermes-ui]] — Browser Hermes Desktop (przbadu/hermes-ui)
|
- [[systems/hermes-ui]] — Browser Hermes Desktop (przbadu/hermes-ui)
|
||||||
|
- [[systems/paseo]] — Paseo coding-agent daemon + web terminal workspace (CT460, :6767)
|
||||||
- [[systems/ignis]] — Ignis browser-native Obsidian (wiki interface)
|
- [[systems/ignis]] — Ignis browser-native Obsidian (wiki interface)
|
||||||
- [[systems/terminal-backend]] — Shell persistence, working directory
|
- [[systems/terminal-backend]] — Shell persistence, working directory
|
||||||
- [[systems/tool-search]] — Always-on lazy discovery for external tools
|
- [[systems/tool-search]] — Always-on lazy discovery for external tools
|
||||||
|
|
||||||
### Integrations
|
### Integrations
|
||||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
||||||
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; **10 carry a non-clean `last_status` as of 2026-10-10 00:22** — 9 from the 2026-10-09 fleet-wide provider timeout storm (15 failed executions, 10 jobs) plus 1 `delivery_failed`; the 7 dead-model-pin jobs repinned 2026-10-05 stay healthy. See [[incidents/2026-10-09-cron-provider-timeout-storm]])
|
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **5 carry a non-clean `last_status`** as of 2026-10-11 00:14 (down from 10). The 2026-10-10 fleet ran **clean — 174 `completed`, 0 `failed`** — ending the 2026-10-09 provider timeout storm (15 failed executions / 10 jobs; 7 of its incidents auto-closed during 2026-10-10, 2 remain open). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
|
||||||
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **no regeneration committed since 2026-10-07** — the 10-08 run's output was left uncommitted and the 10-09 run failed, see [[systems/scheduled-tasks]])
|
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **no regeneration committed since 2026-10-07** — the 10-08 run's output was left uncommitted and the 10-09 run failed, see [[systems/scheduled-tasks]])
|
||||||
|
|
||||||
## Incidents
|
## Incidents
|
||||||
@@ -39,7 +40,7 @@
|
|||||||
- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
|
- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
|
||||||
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); **closed 2026-10-05** when all three were repinned off the pin
|
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); **closed 2026-10-05** when all three were repinned off the pin
|
||||||
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
|
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
|
||||||
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min
|
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. **Recurred 2026-10-10 inside the *gateway* runtime env** — `mnemosyne` and `elevenlabs` missing from the committed uv env the gateway loads; both repaired and verified the same evening
|
||||||
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured
|
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured
|
||||||
- [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
|
- [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
|
||||||
|
|
||||||
|
|||||||
@@ -432,3 +432,30 @@ No secrets written; credential-shaped values referenced by job ID and connection
|
|||||||
- **Sessions in the 2026-10-09 window (`state.db`, epoch REAL):** 19 total — 6 `cron`, 11 `ios`, 1 `talaria-ios`, 1 `telegram`. Delivered 16 / suppressed 160 / suppressed-acked 2 (0 delivery failures — the `delivered` count includes failure notices).
|
- **Sessions in the 2026-10-09 window (`state.db`, epoch REAL):** 19 total — 6 `cron`, 11 `ios`, 1 `talaria-ios`, 1 `telegram`. Delivered 16 / suppressed 160 / suppressed-acked 2 (0 delivery failures — the `delivered` count includes failure notices).
|
||||||
|
|
||||||
No secrets written; job/incident IDs only.
|
No secrets written; job/incident IDs only.
|
||||||
|
|
||||||
|
## [2026-10-10] wiki audit | window 2026-10-09 -> 2026-10-10 (ran 2026-10-10 23:59 -> 2026-10-11 00:2x AWST)
|
||||||
|
|
||||||
|
**Cron.** 29 jobs, 26 enabled / 3 disabled (`jobs.json` 2026-10-11T00:19). Day tally: 10-07 `177/1`,
|
||||||
|
10-08 `178/2`, 10-09 `163/15`, **10-10 `174 completed / 0 failed`**. Non-clean `last_status` 10 -> 5.
|
||||||
|
Incidents: **7 closed during 10-10, 5 open** (3 jobs). **Open item #1 closed: the DHL monitor
|
||||||
|
`62619aa6bcb3` is retired and verified absent from `jobs.json` by ID lookup** (the 10-07 and 10-09
|
||||||
|
"deleted" claims were false).
|
||||||
|
|
||||||
|
**Runtime/memory.** The 2026-09-28 managed-runtime-dependency class **recurred in the gateway env**:
|
||||||
|
`mnemosyne` (+native deps) and `elevenlabs` were missing from the committed uv env
|
||||||
|
`795020dbf24343cd9c52df24723dca83` that `activate_dependencies()` loads. Repaired 21:26-21:34 and
|
||||||
|
verified live from 22:01:19 (`Memory provider 'mnemosyne' registered (40 tools)` -> `activated`;
|
||||||
|
repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - left open.
|
||||||
|
|
||||||
|
**New page.** [[systems/paseo]] - Paseo daemon v0.11.2 on `0.0.0.0:6767` (user unit
|
||||||
|
`paseo-daemon.service`); recorded the `daemon run` vs `daemon start` supervisor trap, the
|
||||||
|
`/usr/bin/paseo` 0.7.2 vs daemon 0.11.2 version split, and the truncated-`cline` false provider fault.
|
||||||
|
|
||||||
|
**Platform.** Mattermost unreachable from CT460 - configured endpoint `100.98.116.14:8065` answers
|
||||||
|
neither ICMP nor TCP; gateway retrying every 5 minutes. Cause not determined.
|
||||||
|
|
||||||
|
**Pages touched:** `systems/paseo.md` (new), `systems/scheduled-tasks.md`, `systems/hermes-agent.md`,
|
||||||
|
`systems/mnemosyne.md`, `systems/messaging-integrations.md`,
|
||||||
|
`incidents/2026-09-28-hermes-managed-runtime-deps.md`, `current-state.md`, `index.md`, `log.md`.
|
||||||
|
|
||||||
|
No secrets written; job/incident IDs, paths and tailnet addresses only.
|
||||||
|
|||||||
+23
-2
@@ -3,8 +3,8 @@ title: Hermes Agent
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-04
|
updated: 2026-10-11
|
||||||
verified_on: 2026-10-04
|
verified_on: 2026-10-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system, hermes, agent, llm]
|
tags: [system, hermes, agent, llm]
|
||||||
sources: [live-hermes-version, live-config, live-skill-inventory]
|
sources: [live-hermes-version, live-config, live-skill-inventory]
|
||||||
@@ -148,3 +148,24 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
|||||||
- [[messaging-integrations]]
|
- [[messaging-integrations]]
|
||||||
- [[scheduled-tasks]]
|
- [[scheduled-tasks]]
|
||||||
- [[current-state]]
|
- [[current-state]]
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2026-10-10 - gateway runtime env, TTS, Paseo
|
||||||
|
|
||||||
|
- **Gateway process shape (verified 2026-10-11 00:20):** PID 230494, started 2026-10-10 21:24:44,
|
||||||
|
launched as `/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '<bootstrap>' gateway run`
|
||||||
|
(RSS ~371 MB). `activate_dependencies()` replaces `sys.path` with a single committed uv
|
||||||
|
environment (`~/.hermes/installs/<id>/environments/795020dbf24343cd9c52df24723dca83/venv`), and the
|
||||||
|
user site directory is invisible to it - packages the gateway needs must be installed **into that
|
||||||
|
env**. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||||
|
- **Mnemosyne provider healthy again:** `Memory provider 'mnemosyne' registered (40 tools)` +
|
||||||
|
`activated` from 2026-10-10 22:01:19 onward (repeated 23:59 / 00:00 / 00:02); `hermes memory
|
||||||
|
status` reports available.
|
||||||
|
- **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env
|
||||||
|
and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS
|
||||||
|
output appears only when explicitly requested.
|
||||||
|
- **Mattermost platform unreachable** since 2026-10-10; the gateway keeps retrying on a 5-minute
|
||||||
|
cadence - [[systems/messaging-integrations]].
|
||||||
|
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Messaging Integrations
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-03
|
updated: 2026-10-11
|
||||||
verified_on: 2026-10-03
|
verified_on: 2026-10-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [messaging, telegram, discord, email, integrations]
|
tags: [messaging, telegram, discord, email, integrations]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
|
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
|
||||||
@@ -70,3 +70,26 @@ Anthony asked whether his other Hermes instances (the estate VMs/CTs) could shar
|
|||||||
- [[systems/hermes-agent]]
|
- [[systems/hermes-agent]]
|
||||||
- [[systems/scheduled-tasks]]
|
- [[systems/scheduled-tasks]]
|
||||||
- [[infrastructure/tailscale]]
|
- [[infrastructure/tailscale]]
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2026-10-11 00:20 AWST - Mattermost platform unreachable (observed, not fixed)
|
||||||
|
|
||||||
|
The gateway's Mattermost adapter has been down since at least 2026-10-10. `gateway.log` shows a
|
||||||
|
5-minute retry cadence with `Mattermost: disconnected` followed by
|
||||||
|
`Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s`
|
||||||
|
(attempt 34 at 00:15, attempt 35 at 00:20).
|
||||||
|
|
||||||
|
Probe of the configured endpoint (`MATTERMOST_URL=http://100.98.116.14:8065/`) from CT460:
|
||||||
|
**100 % ICMP loss and TCP 8065 refuses** - the Mattermost host itself is unreachable from CT460,
|
||||||
|
not merely an application-level refusal.
|
||||||
|
|
||||||
|
- Cause **not determined**; not investigated further during this audit (no host/VM/proxy probe).
|
||||||
|
- Impact is limited to the messaging path. No cron job depends on Mattermost, and the 2026-10-10
|
||||||
|
cron fleet ran clean.
|
||||||
|
- Config: plugin block `plugins...mattermost` (`enabled: true`, home channel
|
||||||
|
`5ipo6wfe3j88xxox1gkta3d5mw`, name `Rhino`); the bot token is env-only and is **not** recorded here.
|
||||||
|
|
||||||
|
**Open item:** re-probe the host; if the host is back but the adapter stays down, restart the
|
||||||
|
gateway's Mattermost platform.
|
||||||
|
|||||||
+22
-2
@@ -3,8 +3,8 @@ title: Mnemosyne
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-04
|
updated: 2026-10-11
|
||||||
verified_on: 2026-10-04
|
verified_on: 2026-10-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [memory, mnemosyne, backend]
|
tags: [memory, mnemosyne, backend]
|
||||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||||||
@@ -105,3 +105,23 @@ Primary agent memory store for Hermes.
|
|||||||
- [[incidents/2026-08-01-mnemosyne-capture-failure]]
|
- [[incidents/2026-08-01-mnemosyne-capture-failure]]
|
||||||
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]]
|
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]]
|
||||||
- [[incidents/2026-07-29-mnemosyne-relay]]
|
- [[incidents/2026-07-29-mnemosyne-relay]]
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Re-verified 2026-10-10 (audit window; live probes 2026-10-11 00:14-00:20 AWST)
|
||||||
|
|
||||||
|
- **Gateway load path repaired.** The gateway's `activate_dependencies()` selects one committed uv
|
||||||
|
env (`795020dbf24343cd9c52df24723dca83`); it was missing `mnemosyne` / `mnemosyne_hermes` and the
|
||||||
|
native dependencies. Repaired 2026-10-10 21:26-21:34 and confirmed live from 22:01:19
|
||||||
|
(`registered (40 tools)` -> `activated`). Detail:
|
||||||
|
[[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||||
|
- **"Capture silent since 10-09 15:20" was a false alarm.** It resolved to *no interactive sessions
|
||||||
|
in the window* - the window was cron-only traffic. What the surrounding hours did capture was
|
||||||
|
dominated by gateway background-process notifications stored as
|
||||||
|
`[USER] [Anthony Martin] [IMPORTANT: Background process ...]` rows: recall-visible, low signal.
|
||||||
|
A capture filter would help; **no fix applied** (carried open item).
|
||||||
|
- **Coverage script executable again:** `~/.hermes/scripts/check_embedding_coverage.py` now carries
|
||||||
|
mode `0755`, so the monitor's non-executable finding is resolved.
|
||||||
|
- **Open, unverified:** 13 orphan rows in `memory_embeddings` (oldest 2026-08-22) reported by a
|
||||||
|
2026-10-10 monitor run - neither confirmed nor cleaned by this audit.
|
||||||
|
|||||||
@@ -0,0 +1,92 @@
|
|||||||
|
---
|
||||||
|
title: Paseo Daemon
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-10-11
|
||||||
|
updated: 2026-10-11
|
||||||
|
verified_on: 2026-10-11
|
||||||
|
review_after: 2026-11-11
|
||||||
|
confidence: high
|
||||||
|
tags: [system, paseo, coding-agent, ct460, systemd]
|
||||||
|
sources: [live-systemctl-user, live-ps, live-paseo-config, live-session-20261010]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Paseo Daemon (CT460)
|
||||||
|
|
||||||
|
Paseo is a **coding-agent orchestrator with a browser terminal workspace**. It runs on
|
||||||
|
CT460 (`hermes`), hosts terminal sessions against repositories under `/home/hermes`,
|
||||||
|
and brokers turns to external coding agents — Claude Code, Codex, Cline, OpenCode, and
|
||||||
|
Hermes itself via ACP.
|
||||||
|
|
||||||
|
- Local: `http://127.0.0.1:6767`
|
||||||
|
- Tailnet: `https://hermes.kangaroo-eel.ts.net:6767`
|
||||||
|
|
||||||
|
The daemon spawns provider processes on demand; a live example is
|
||||||
|
`opencode serve --port 42021` running as a child of the unit.
|
||||||
|
|
||||||
|
## State (verified 2026-10-11 00:16 AWST)
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|---|---|
|
||||||
|
| Unit | `paseo-daemon.service` — **user** unit, `enabled` |
|
||||||
|
| Unit file | `/home/hermes/.config/systemd/user/paseo-daemon.service` |
|
||||||
|
| State | `active (running)` since **2026-10-10 14:47:28 AWST** |
|
||||||
|
| Daemon version | **0.11.2** (upgraded from 0.7.0 on 2026-10-10) |
|
||||||
|
| Main PID | `163127` (`node --disable-warning=DEP0040 /home/hermes/.npm-global/bin/paseo daemon run`) |
|
||||||
|
| Inner processes | supervisor `163168`, `Paseo Daemon` `163198`, terminal worker, `opencode serve` |
|
||||||
|
| Listen | `0.0.0.0:6767` |
|
||||||
|
| Binary used by the unit | `/home/hermes/.npm-global/bin/paseo` |
|
||||||
|
| Config | `/home/hermes/.paseo/config.json` (mode `0600`) |
|
||||||
|
| Logs | `/home/hermes/.paseo/<YYYYMMDD-HHMM>-NN-daemon.log` (10 MB rotation) and `journalctl --user -u paseo-daemon` |
|
||||||
|
|
||||||
|
## Maintenance
|
||||||
|
|
||||||
|
`sudo systemctl` will **not** see this unit — it belongs to the user manager:
|
||||||
|
|
||||||
|
systemctl --user status paseo-daemon
|
||||||
|
systemctl --user restart paseo-daemon
|
||||||
|
journalctl --user -u paseo-daemon -f
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:6767/
|
||||||
|
|
||||||
|
Upgrade: `npm install -g @getpaseo/cli@latest`, then restart the unit. **The daemon keeps
|
||||||
|
running the old version until the unit restarts.**
|
||||||
|
|
||||||
|
## Why `ExecStart` uses `daemon run`, not `daemon start`
|
||||||
|
|
||||||
|
`paseo daemon start` daemonises and returns immediately, so the foreground process exits
|
||||||
|
`0`. Under `Type=simple` + `Restart=always` systemd reads that as a clean exit and
|
||||||
|
restart-loops the service (the flapping observed during the 2026-10-10 repair).
|
||||||
|
`paseo daemon run` stays in the foreground and is the correct supervised command.
|
||||||
|
|
||||||
|
The unit's `PATH` must include both `/home/hermes/.npm-global/bin` and `~/.local/bin`:
|
||||||
|
agent CLIs the daemon shells out to are resolved through the *unit's* environment, not an
|
||||||
|
interactive shell's.
|
||||||
|
|
||||||
|
## Known pitfalls
|
||||||
|
|
||||||
|
- **The `paseo` on `PATH` is not the daemon's version.** `/usr/bin/paseo` reports
|
||||||
|
**0.7.2** while the unit runs the npm-global **0.11.2**. `paseo --version` therefore
|
||||||
|
tells you nothing about what the daemon is running — read the unit's `ExecStart`, or
|
||||||
|
the `daemonVersion` field in the daemon log.
|
||||||
|
- **A truncated agent binary looks like a provider fault.** A partly-written `cline`
|
||||||
|
binary (107 KB ELF instead of a full ~3 MB) made that one provider fail while every
|
||||||
|
other provider worked. Check binary size before chasing provider config; reinstalling
|
||||||
|
to 3.0.70 fixed it.
|
||||||
|
- **Hermes as a Paseo provider needs the ACP extra** — `hermes pm install --extra acp`
|
||||||
|
before the Hermes provider will load (its check reports `Hermes ACP check OK` once
|
||||||
|
installed).
|
||||||
|
- **Config edits need a restart.** `baseUrl`/CORS changes in `~/.paseo/config.json` only
|
||||||
|
take effect after `systemctl --user restart paseo-daemon`. Backups are kept beside the
|
||||||
|
config (`config.json.bak-20261010-corsfix`, `config.json.bak-20260809-codex-fix`).
|
||||||
|
- **Cosmetic CLI error while the daemon is healthy (unresolved 2026-10-10).** When a
|
||||||
|
`daemon.get_status.request` takes ~2.2 s the CLI prints
|
||||||
|
`daemon detail request to 0.0.0.0:6767 failed` even though `127.0.0.1:6767` returns
|
||||||
|
HTTP 200. Verify with the health checks above before treating a single such line as an
|
||||||
|
outage.
|
||||||
|
- **Pairing payloads are credentials.** The `#offer=…` URL and `~/.paseo/daemon-keypair.json`
|
||||||
|
carry the daemon key — never paste them into the wiki or a shared channel.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
|
- [[systems/terminal-backend]]
|
||||||
|
- [[infrastructure/network-map]]
|
||||||
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-09
|
updated: 2026-10-11
|
||||||
verified_on: 2026-10-09
|
verified_on: 2026-10-11
|
||||||
review_after: 2026-10-23
|
review_after: 2026-10-25
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, scheduled, automation]
|
tags: [cron, scheduled, automation]
|
||||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||||
@@ -171,3 +171,46 @@ See [[incidents/2026-10-09-memory-recall-relevance]].
|
|||||||
5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open.
|
5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open.
|
||||||
6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.
|
6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST)
|
||||||
|
|
||||||
|
**Fleet: 29 jobs - 26 enabled - 3 disabled** (`jobs.json.updated_at 2026-10-11T00:19:44+08:00`).
|
||||||
|
The count fell from 30 to 29 because **`62619aa6bcb3` (DHL Tracking - Parcel `8052838382`) is gone**:
|
||||||
|
open item #1 of the 2026-10-09 audit is **CLOSED** (below). Disabled: `d40085da631c` (Important-mail
|
||||||
|
monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher), `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
|
||||||
|
|
||||||
|
**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` - 10-08 `178 / 2` -
|
||||||
|
10-09 `163 / 15` - **10-10 `174 completed / 0 failed`** (+1 left `running`). The 2026-10-10 fleet was
|
||||||
|
wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm.
|
||||||
|
|
||||||
|
**Non-clean `last_status`: 5 jobs, down from 10.**
|
||||||
|
|
||||||
|
| Job | ID | `last_status` | streak |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `error` | 0 |
|
||||||
|
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `error` | 0 |
|
||||||
|
| Daily Marketing Job Radar | `dca8482e4f76` | `error` | 1 |
|
||||||
|
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `error` | 1 |
|
||||||
|
| Uber Eats Monthly Spend Report | `066bb5191280` | `delivery_failed` | 0 |
|
||||||
|
|
||||||
|
The three `streak 0` rows are stale `last_status` fields, not live failure streaks. `1a8e90ea2271`
|
||||||
|
(stocktake) is back to `last_status: ok`.
|
||||||
|
|
||||||
|
**Incidents: 7 closed during 2026-10-10, 5 open.** The 10-09 storm unwound as each job next ran clean
|
||||||
|
(`ea79a2` 00:03, `cbfe43` 00:27, `1a8e90ea2271` x2 03:06, `85edf6` 07:04, `08eb86` 08:32, `ecc1fc`
|
||||||
|
12:05). Still open: `1b23c042e17d` + `dca8482e4f76` (`timeout`, `alerted`, both first seen
|
||||||
|
2026-10-09) and `0e753e9720fd` + `fad5a86979d4` (`agent`; open since 2026-10-01 / 2026-09-30).
|
||||||
|
|
||||||
|
**Open item #1 CLOSED - the DHL monitor is retired.** Anthony instructed removal again on the evening
|
||||||
|
of 2026-10-10 and the job is **absent from live `jobs.json`**, verified 2026-10-11 00:19 by direct ID
|
||||||
|
lookup (no job id contains `62619`; no job name matches dhl/parcel). This is the third "deleted"
|
||||||
|
claim for this job - the 2026-10-07 and 2026-10-09 reports were both **false** because the check was
|
||||||
|
the absence of job output rather than a positive lookup. Removals must be verified by ID lookup.
|
||||||
|
|
||||||
|
**Still open:** the Quartz rebuild step in `wiki-maintain.sh` (no `quartz.config.ts`) - skills-index
|
||||||
|
off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost
|
||||||
|
platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the
|
||||||
|
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).
|
||||||
|
|||||||
Reference in New Issue
Block a user