audit(2026-10-10): DHL monitor retired (29 jobs / 26 enabled, verified by ID lookup); 10-10 fleet clean 174/0, 7 incidents closed - 5 open; gateway runtime env recurrence (mnemosyne + elevenlabs) verified fixed; new systems/paseo.md; mattermost unreachable; current-state + index + log updated

This commit is contained in:
hermes
2026-10-11 00:22:20 +08:00
parent c11f11e652
commit 7bffeab157
9 changed files with 295 additions and 18 deletions
+15 -4
View File
@@ -3,16 +3,27 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-10-10
verified_on: 2026-10-10
review_after: 2026-10-31
updated: 2026-10-11
verified_on: 2026-10-11
review_after: 2026-10-25
confidence: high
tags: [system]
sources: [live-config, live-cron-list, live-hermes-version]
---
# Current State
Updated: 2026-10-10
Updated: 2026-10-11
## Recent Changes (2026-10-10 -> 10-11, nightly audit of the 2026-10-09 -> 2026-10-10 window; probes 2026-10-11 00:14-00:20 AWST)
_The manual 2026-10-10 stocktake entry follows beneath this one._
- **The cron fleet is clean again: 174 `completed`, 0 `failed` on 2026-10-10** (AWST day, `executions.db`) - after 10-07 `1` / 10-08 `2` / 10-09 `15` failures. 7 of the 10 storm incidents auto-closed during the day; **5 remain open** over 3 jobs (`1b23c042e17d` + `dca8482e4f76` `timeout`/`alerted` since 2026-10-09; `0e753e9720fd` + `fad5a86979d4` `agent`). Non-clean `last_status` down from 10 to **5**. Job count **30 -> 29 (26 enabled / 3 disabled)**. -> [[systems/scheduled-tasks]]
- **The DHL monitor `62619aa6bcb3` is retired - open item #1 closed.** Anthony instructed removal again on the evening of 2026-10-10; the job is **absent from live `jobs.json`** (verified 2026-10-11 00:19 by direct ID lookup). The 2026-10-07 and 2026-10-09 "deleted" reports were false - they inferred removal from missing output instead of looking up the ID.
- **The 2026-09-28 runtime-dependency class recurred - in the *gateway* env this time.** The gateway's `activate_dependencies()` loads one committed uv env (`795020dbf24343cd9c52df24723dca83`); it was missing `mnemosyne` / `mnemosyne_hermes` plus native deps, and `elevenlabs` was missing from the live managed env. Repaired 21:26-21:34, verified live from **22:01:19** (`Memory provider 'mnemosyne' registered (40 tools)` -> `activated`; repeated 23:59 / 00:00 / 00:02). The 15-minute guard does **not** cover this env - open. -> [[incidents/2026-09-28-hermes-managed-runtime-deps]]
- **New page: [[systems/paseo]]** - Paseo coding-agent daemon, user unit `paseo-daemon.service`, v0.11.2, `0.0.0.0:6767`, running since 2026-10-10 14:47:28. Pitfalls recorded: `daemon run` vs `daemon start`, `/usr/bin/paseo` is 0.7.2 while the daemon is 0.11.2, and a truncated `cline` binary presenting as a provider fault.
- **Mattermost unreachable from CT460** - configured endpoint `100.98.116.14:8065` gives 100 % ICMP loss and TCP refuse; the gateway retries every 5 minutes. Cause not determined. -> [[systems/messaging-integrations]]
- **Health-monitor follow-ups (2026-10-10):** "capture silent" was a false alarm (cron-only window); `check_embedding_coverage.py` now carries mode `0755`; 13 orphan `memory_embeddings` rows (oldest 2026-08-22) remain unverified. -> [[systems/mnemosyne]]
## Recent Changes (2026-10-10)
@@ -3,8 +3,8 @@ title: Hermes Managed-Runtime Dependency Failure
type: incident
status: resolved
created: 2026-09-30
updated: 2026-09-30
verified_on: 2026-09-30
updated: 2026-10-11
verified_on: 2026-10-11
confidence: high
tags: [hermes, cron, runtime, python, worker, incident]
sources: [live-jobs-json, live-guard-script, live-cron-list]
@@ -92,3 +92,42 @@ Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the
- [[systems/hermes-agent]]
- [[systems/mnemosyne]]
- [[runbooks/restart-hermes]]
---
## Recurrence - 2026-10-10 (this time the *gateway* runtime env, plus a new failure mode)
**The class recurred twice in one day, on packages the 15-minute guard does not cover.**
**1. `mnemosyne` missing from the committed gateway env (repaired 21:26-21:34, verified 22:01:19).**
The gateway does not run from a normal venv: it launches as
`/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '<bootstrap>' gateway run`
and `activate_dependencies()` **replaces `sys.path`** with one committed uv environment -
`~/.hermes/installs/<id>/environments/795020dbf24343cd9c52df24723dca83/venv`. Of the uv
environments on CT460, that one alone was missing `mnemosyne` / `mnemosyne_hermes`, so sessions
logged `mnemosyne_batch helpers unavailable (No module named 'mnemosyne')` and
`Memory provider 'mnemosyne' is selected but reports unavailable`. Fix: install
`hermes_memory_provider` + `mnemosyne_hermes` + `mnemosyne` **together with the native dependencies**
(`fastembed`, `onnxruntime`, `numpy` + `numpy.libs`, `tokenizers`, `huggingface_hub`, `sqlite-vec`,
`loguru`, `mmh3`) into that committed venv. Verified live: `Memory provider 'mnemosyne' registered
(40 tools)` -> `activated` at 2026-10-10 22:01:19, again at 23:59 / 00:00 / 00:02; and
`hermes memory status` -> `Plugin: installed / Status: available`.
**2. `elevenlabs` missing from the live managed env (21:13 -> 21:24).** TTS was configured but the
package was absent from the managed runtime in use; reinstalled and verified by generating a real
clip.
**Trap: `~/.local/lib/python3.14/site-packages` is invisible to the gateway.** The first repair
attempt installed into the user site directory; with `ENABLE_USER_SITE=False` **and** a replaced
`sys.path`, the gateway never sees it. Install into the committed uv env that
`activate_dependencies()` selects - not the user site dir, not a sibling venv.
**Trap: `hermes memory status` can be green while the gateway is still broken.** It is a separate
process with its own runtime resolution. The gateway's own log lines
(`registered (40 tools)` / `activated` versus `reports unavailable`) are the authoritative check.
**Trap: this recurrence produced no cron failures at all** (2026-10-10: 174 `completed`, 0 `failed`).
It degraded the gateway's memory and TTS paths only. A green cron day does **not** mean the runtime
is healthy - and the 15-minute guard still does not check the gateway's committed env, so **guard
coverage remains open.**
+4 -3
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-10
> model routing and automation. Last updated: 2026-10-11
## Start here
- [[current-state]] — Current systems status
@@ -22,13 +22,14 @@
- [[systems/nanobot]] — Nanobot current state and history
- [[systems/browser-backend]] — Camoufox (primary) + Steel (disabled), CDP config
- [[systems/hermes-ui]] — Browser Hermes Desktop (przbadu/hermes-ui)
- [[systems/paseo]] — Paseo coding-agent daemon + web terminal workspace (CT460, :6767)
- [[systems/ignis]] — Ignis browser-native Obsidian (wiki interface)
- [[systems/terminal-backend]] — Shell persistence, working directory
- [[systems/tool-search]] — Always-on lazy discovery for external tools
### Integrations
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; **10 carry a non-clean `last_status` as of 2026-10-10 00:22** — 9 from the 2026-10-09 fleet-wide provider timeout storm (15 failed executions, 10 jobs) plus 1 `delivery_failed`; the 7 dead-model-pin jobs repinned 2026-10-05 stay healthy. See [[incidents/2026-10-09-cron-provider-timeout-storm]])
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **5 carry a non-clean `last_status`** as of 2026-10-11 00:14 (down from 10). The 2026-10-10 fleet ran **clean — 174 `completed`, 0 `failed`** — ending the 2026-10-09 provider timeout storm (15 failed executions / 10 jobs; 7 of its incidents auto-closed during 2026-10-10, 2 remain open). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **no regeneration committed since 2026-10-07** — the 10-08 run's output was left uncommitted and the 10-09 run failed, see [[systems/scheduled-tasks]])
## Incidents
@@ -39,7 +40,7 @@
- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); **closed 2026-10-05** when all three were repinned off the pin
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. **Recurred 2026-10-10 inside the *gateway* runtime env** — `mnemosyne` and `elevenlabs` missing from the committed uv env the gateway loads; both repaired and verified the same evening
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured
- [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
+27
View File
@@ -432,3 +432,30 @@ No secrets written; credential-shaped values referenced by job ID and connection
- **Sessions in the 2026-10-09 window (`state.db`, epoch REAL):** 19 total — 6 `cron`, 11 `ios`, 1 `talaria-ios`, 1 `telegram`. Delivered 16 / suppressed 160 / suppressed-acked 2 (0 delivery failures — the `delivered` count includes failure notices).
No secrets written; job/incident IDs only.
## [2026-10-10] wiki audit | window 2026-10-09 -> 2026-10-10 (ran 2026-10-10 23:59 -> 2026-10-11 00:2x AWST)
**Cron.** 29 jobs, 26 enabled / 3 disabled (`jobs.json` 2026-10-11T00:19). Day tally: 10-07 `177/1`,
10-08 `178/2`, 10-09 `163/15`, **10-10 `174 completed / 0 failed`**. Non-clean `last_status` 10 -> 5.
Incidents: **7 closed during 10-10, 5 open** (3 jobs). **Open item #1 closed: the DHL monitor
`62619aa6bcb3` is retired and verified absent from `jobs.json` by ID lookup** (the 10-07 and 10-09
"deleted" claims were false).
**Runtime/memory.** The 2026-09-28 managed-runtime-dependency class **recurred in the gateway env**:
`mnemosyne` (+native deps) and `elevenlabs` were missing from the committed uv env
`795020dbf24343cd9c52df24723dca83` that `activate_dependencies()` loads. Repaired 21:26-21:34 and
verified live from 22:01:19 (`Memory provider 'mnemosyne' registered (40 tools)` -> `activated`;
repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - left open.
**New page.** [[systems/paseo]] - Paseo daemon v0.11.2 on `0.0.0.0:6767` (user unit
`paseo-daemon.service`); recorded the `daemon run` vs `daemon start` supervisor trap, the
`/usr/bin/paseo` 0.7.2 vs daemon 0.11.2 version split, and the truncated-`cline` false provider fault.
**Platform.** Mattermost unreachable from CT460 - configured endpoint `100.98.116.14:8065` answers
neither ICMP nor TCP; gateway retrying every 5 minutes. Cause not determined.
**Pages touched:** `systems/paseo.md` (new), `systems/scheduled-tasks.md`, `systems/hermes-agent.md`,
`systems/mnemosyne.md`, `systems/messaging-integrations.md`,
`incidents/2026-09-28-hermes-managed-runtime-deps.md`, `current-state.md`, `index.md`, `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.
+23 -2
View File
@@ -3,8 +3,8 @@ title: Hermes Agent
type: system
status: active
created: 2026-07-22
updated: 2026-10-04
verified_on: 2026-10-04
updated: 2026-10-11
verified_on: 2026-10-11
confidence: high
tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory]
@@ -148,3 +148,24 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
- [[messaging-integrations]]
- [[scheduled-tasks]]
- [[current-state]]
---
## 2026-10-10 - gateway runtime env, TTS, Paseo
- **Gateway process shape (verified 2026-10-11 00:20):** PID 230494, started 2026-10-10 21:24:44,
launched as `/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '<bootstrap>' gateway run`
(RSS ~371 MB). `activate_dependencies()` replaces `sys.path` with a single committed uv
environment (`~/.hermes/installs/<id>/environments/795020dbf24343cd9c52df24723dca83/venv`), and the
user site directory is invisible to it - packages the gateway needs must be installed **into that
env**. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Mnemosyne provider healthy again:** `Memory provider 'mnemosyne' registered (40 tools)` +
`activated` from 2026-10-10 22:01:19 onward (repeated 23:59 / 00:00 / 00:02); `hermes memory
status` reports available.
- **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env
and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS
output appears only when explicitly requested.
- **Mattermost platform unreachable** since 2026-10-10; the gateway keeps retrying on a 5-minute
cadence - [[systems/messaging-integrations]].
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
+25 -2
View File
@@ -3,8 +3,8 @@ title: Messaging Integrations
type: system
status: active
created: 2026-07-22
updated: 2026-10-03
verified_on: 2026-10-03
updated: 2026-10-11
verified_on: 2026-10-11
confidence: high
tags: [messaging, telegram, discord, email, integrations]
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
@@ -70,3 +70,26 @@ Anthony asked whether his other Hermes instances (the estate VMs/CTs) could shar
- [[systems/hermes-agent]]
- [[systems/scheduled-tasks]]
- [[infrastructure/tailscale]]
---
## 2026-10-11 00:20 AWST - Mattermost platform unreachable (observed, not fixed)
The gateway's Mattermost adapter has been down since at least 2026-10-10. `gateway.log` shows a
5-minute retry cadence with `Mattermost: disconnected` followed by
`Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s`
(attempt 34 at 00:15, attempt 35 at 00:20).
Probe of the configured endpoint (`MATTERMOST_URL=http://100.98.116.14:8065/`) from CT460:
**100 % ICMP loss and TCP 8065 refuses** - the Mattermost host itself is unreachable from CT460,
not merely an application-level refusal.
- Cause **not determined**; not investigated further during this audit (no host/VM/proxy probe).
- Impact is limited to the messaging path. No cron job depends on Mattermost, and the 2026-10-10
cron fleet ran clean.
- Config: plugin block `plugins...mattermost` (`enabled: true`, home channel
`5ipo6wfe3j88xxox1gkta3d5mw`, name `Rhino`); the bot token is env-only and is **not** recorded here.
**Open item:** re-probe the host; if the host is back but the adapter stays down, restart the
gateway's Mattermost platform.
+22 -2
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system
status: active
created: 2026-07-22
updated: 2026-10-04
verified_on: 2026-10-04
updated: 2026-10-11
verified_on: 2026-10-11
confidence: high
tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
@@ -105,3 +105,23 @@ Primary agent memory store for Hermes.
- [[incidents/2026-08-01-mnemosyne-capture-failure]]
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]]
- [[incidents/2026-07-29-mnemosyne-relay]]
---
## Re-verified 2026-10-10 (audit window; live probes 2026-10-11 00:14-00:20 AWST)
- **Gateway load path repaired.** The gateway's `activate_dependencies()` selects one committed uv
env (`795020dbf24343cd9c52df24723dca83`); it was missing `mnemosyne` / `mnemosyne_hermes` and the
native dependencies. Repaired 2026-10-10 21:26-21:34 and confirmed live from 22:01:19
(`registered (40 tools)` -> `activated`). Detail:
[[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **"Capture silent since 10-09 15:20" was a false alarm.** It resolved to *no interactive sessions
in the window* - the window was cron-only traffic. What the surrounding hours did capture was
dominated by gateway background-process notifications stored as
`[USER] [Anthony Martin] [IMPORTANT: Background process ...]` rows: recall-visible, low signal.
A capture filter would help; **no fix applied** (carried open item).
- **Coverage script executable again:** `~/.hermes/scripts/check_embedding_coverage.py` now carries
mode `0755`, so the monitor's non-executable finding is resolved.
- **Open, unverified:** 13 orphan rows in `memory_embeddings` (oldest 2026-08-22) reported by a
2026-10-10 monitor run - neither confirmed nor cleaned by this audit.
+92
View File
@@ -0,0 +1,92 @@
---
title: Paseo Daemon
type: system
status: active
created: 2026-10-11
updated: 2026-10-11
verified_on: 2026-10-11
review_after: 2026-11-11
confidence: high
tags: [system, paseo, coding-agent, ct460, systemd]
sources: [live-systemctl-user, live-ps, live-paseo-config, live-session-20261010]
---
# Paseo Daemon (CT460)
Paseo is a **coding-agent orchestrator with a browser terminal workspace**. It runs on
CT460 (`hermes`), hosts terminal sessions against repositories under `/home/hermes`,
and brokers turns to external coding agents — Claude Code, Codex, Cline, OpenCode, and
Hermes itself via ACP.
- Local: `http://127.0.0.1:6767`
- Tailnet: `https://hermes.kangaroo-eel.ts.net:6767`
The daemon spawns provider processes on demand; a live example is
`opencode serve --port 42021` running as a child of the unit.
## State (verified 2026-10-11 00:16 AWST)
| Item | Value |
|---|---|
| Unit | `paseo-daemon.service` — **user** unit, `enabled` |
| Unit file | `/home/hermes/.config/systemd/user/paseo-daemon.service` |
| State | `active (running)` since **2026-10-10 14:47:28 AWST** |
| Daemon version | **0.11.2** (upgraded from 0.7.0 on 2026-10-10) |
| Main PID | `163127` (`node --disable-warning=DEP0040 /home/hermes/.npm-global/bin/paseo daemon run`) |
| Inner processes | supervisor `163168`, `Paseo Daemon` `163198`, terminal worker, `opencode serve` |
| Listen | `0.0.0.0:6767` |
| Binary used by the unit | `/home/hermes/.npm-global/bin/paseo` |
| Config | `/home/hermes/.paseo/config.json` (mode `0600`) |
| Logs | `/home/hermes/.paseo/<YYYYMMDD-HHMM>-NN-daemon.log` (10 MB rotation) and `journalctl --user -u paseo-daemon` |
## Maintenance
`sudo systemctl` will **not** see this unit — it belongs to the user manager:
systemctl --user status paseo-daemon
systemctl --user restart paseo-daemon
journalctl --user -u paseo-daemon -f
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:6767/
Upgrade: `npm install -g @getpaseo/cli@latest`, then restart the unit. **The daemon keeps
running the old version until the unit restarts.**
## Why `ExecStart` uses `daemon run`, not `daemon start`
`paseo daemon start` daemonises and returns immediately, so the foreground process exits
`0`. Under `Type=simple` + `Restart=always` systemd reads that as a clean exit and
restart-loops the service (the flapping observed during the 2026-10-10 repair).
`paseo daemon run` stays in the foreground and is the correct supervised command.
The unit's `PATH` must include both `/home/hermes/.npm-global/bin` and `~/.local/bin`:
agent CLIs the daemon shells out to are resolved through the *unit's* environment, not an
interactive shell's.
## Known pitfalls
- **The `paseo` on `PATH` is not the daemon's version.** `/usr/bin/paseo` reports
**0.7.2** while the unit runs the npm-global **0.11.2**. `paseo --version` therefore
tells you nothing about what the daemon is running — read the unit's `ExecStart`, or
the `daemonVersion` field in the daemon log.
- **A truncated agent binary looks like a provider fault.** A partly-written `cline`
binary (107 KB ELF instead of a full ~3 MB) made that one provider fail while every
other provider worked. Check binary size before chasing provider config; reinstalling
to 3.0.70 fixed it.
- **Hermes as a Paseo provider needs the ACP extra** — `hermes pm install --extra acp`
before the Hermes provider will load (its check reports `Hermes ACP check OK` once
installed).
- **Config edits need a restart.** `baseUrl`/CORS changes in `~/.paseo/config.json` only
take effect after `systemctl --user restart paseo-daemon`. Backups are kept beside the
config (`config.json.bak-20261010-corsfix`, `config.json.bak-20260809-codex-fix`).
- **Cosmetic CLI error while the daemon is healthy (unresolved 2026-10-10).** When a
`daemon.get_status.request` takes ~2.2 s the CLI prints
`daemon detail request to 0.0.0.0:6767 failed` even though `127.0.0.1:6767` returns
HTTP 200. Verify with the health checks above before treating a single such line as an
outage.
- **Pairing payloads are credentials.** The `#offer=…` URL and `~/.paseo/daemon-keypair.json`
carry the daemon key — never paste them into the wiki or a shared channel.
## Related
- [[systems/hermes-agent]]
- [[systems/terminal-backend]]
- [[infrastructure/network-map]]
+46 -3
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-09
verified_on: 2026-10-09
review_after: 2026-10-23
updated: 2026-10-11
verified_on: 2026-10-11
review_after: 2026-10-25
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -171,3 +171,46 @@ See [[incidents/2026-10-09-memory-recall-relevance]].
5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open.
6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.
---
## Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST)
**Fleet: 29 jobs - 26 enabled - 3 disabled** (`jobs.json.updated_at 2026-10-11T00:19:44+08:00`).
The count fell from 30 to 29 because **`62619aa6bcb3` (DHL Tracking - Parcel `8052838382`) is gone**:
open item #1 of the 2026-10-09 audit is **CLOSED** (below). Disabled: `d40085da631c` (Important-mail
monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher), `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` - 10-08 `178 / 2` -
10-09 `163 / 15` - **10-10 `174 completed / 0 failed`** (+1 left `running`). The 2026-10-10 fleet was
wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm.
**Non-clean `last_status`: 5 jobs, down from 10.**
| Job | ID | `last_status` | streak |
|---|---|---|---|
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `error` | 0 |
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `error` | 0 |
| Daily Marketing Job Radar | `dca8482e4f76` | `error` | 1 |
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `error` | 1 |
| Uber Eats Monthly Spend Report | `066bb5191280` | `delivery_failed` | 0 |
The three `streak 0` rows are stale `last_status` fields, not live failure streaks. `1a8e90ea2271`
(stocktake) is back to `last_status: ok`.
**Incidents: 7 closed during 2026-10-10, 5 open.** The 10-09 storm unwound as each job next ran clean
(`ea79a2` 00:03, `cbfe43` 00:27, `1a8e90ea2271` x2 03:06, `85edf6` 07:04, `08eb86` 08:32, `ecc1fc`
12:05). Still open: `1b23c042e17d` + `dca8482e4f76` (`timeout`, `alerted`, both first seen
2026-10-09) and `0e753e9720fd` + `fad5a86979d4` (`agent`; open since 2026-10-01 / 2026-09-30).
**Open item #1 CLOSED - the DHL monitor is retired.** Anthony instructed removal again on the evening
of 2026-10-10 and the job is **absent from live `jobs.json`**, verified 2026-10-11 00:19 by direct ID
lookup (no job id contains `62619`; no job name matches dhl/parcel). This is the third "deleted"
claim for this job - the 2026-10-07 and 2026-10-09 reports were both **false** because the check was
the absence of job output rather than a positive lookup. Removals must be verified by ID lookup.
**Still open:** the Quartz rebuild step in `wiki-maintain.sh` (no `quartz.config.ts`) - skills-index
off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost
platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).