diff --git a/current-state.md b/current-state.md index 5991dec..c3d200a 100644 --- a/current-state.md +++ b/current-state.md @@ -3,16 +3,27 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-10 -verified_on: 2026-10-10 -review_after: 2026-10-31 +updated: 2026-10-11 +verified_on: 2026-10-11 +review_after: 2026-10-25 confidence: high tags: [system] sources: [live-config, live-cron-list, live-hermes-version] --- # Current State -Updated: 2026-10-10 +Updated: 2026-10-11 + +## Recent Changes (2026-10-10 -> 10-11, nightly audit of the 2026-10-09 -> 2026-10-10 window; probes 2026-10-11 00:14-00:20 AWST) + +_The manual 2026-10-10 stocktake entry follows beneath this one._ + +- **The cron fleet is clean again: 174 `completed`, 0 `failed` on 2026-10-10** (AWST day, `executions.db`) - after 10-07 `1` / 10-08 `2` / 10-09 `15` failures. 7 of the 10 storm incidents auto-closed during the day; **5 remain open** over 3 jobs (`1b23c042e17d` + `dca8482e4f76` `timeout`/`alerted` since 2026-10-09; `0e753e9720fd` + `fad5a86979d4` `agent`). Non-clean `last_status` down from 10 to **5**. Job count **30 -> 29 (26 enabled / 3 disabled)**. -> [[systems/scheduled-tasks]] +- **The DHL monitor `62619aa6bcb3` is retired - open item #1 closed.** Anthony instructed removal again on the evening of 2026-10-10; the job is **absent from live `jobs.json`** (verified 2026-10-11 00:19 by direct ID lookup). The 2026-10-07 and 2026-10-09 "deleted" reports were false - they inferred removal from missing output instead of looking up the ID. +- **The 2026-09-28 runtime-dependency class recurred - in the *gateway* env this time.** The gateway's `activate_dependencies()` loads one committed uv env (`795020dbf24343cd9c52df24723dca83`); it was missing `mnemosyne` / `mnemosyne_hermes` plus native deps, and `elevenlabs` was missing from the live managed env. Repaired 21:26-21:34, verified live from **22:01:19** (`Memory provider 'mnemosyne' registered (40 tools)` -> `activated`; repeated 23:59 / 00:00 / 00:02). The 15-minute guard does **not** cover this env - open. -> [[incidents/2026-09-28-hermes-managed-runtime-deps]] +- **New page: [[systems/paseo]]** - Paseo coding-agent daemon, user unit `paseo-daemon.service`, v0.11.2, `0.0.0.0:6767`, running since 2026-10-10 14:47:28. Pitfalls recorded: `daemon run` vs `daemon start`, `/usr/bin/paseo` is 0.7.2 while the daemon is 0.11.2, and a truncated `cline` binary presenting as a provider fault. +- **Mattermost unreachable from CT460** - configured endpoint `100.98.116.14:8065` gives 100 % ICMP loss and TCP refuse; the gateway retries every 5 minutes. Cause not determined. -> [[systems/messaging-integrations]] +- **Health-monitor follow-ups (2026-10-10):** "capture silent" was a false alarm (cron-only window); `check_embedding_coverage.py` now carries mode `0755`; 13 orphan `memory_embeddings` rows (oldest 2026-08-22) remain unverified. -> [[systems/mnemosyne]] ## Recent Changes (2026-10-10) diff --git a/incidents/2026-09-28-hermes-managed-runtime-deps.md b/incidents/2026-09-28-hermes-managed-runtime-deps.md index 1704e78..bf6c399 100644 --- a/incidents/2026-09-28-hermes-managed-runtime-deps.md +++ b/incidents/2026-09-28-hermes-managed-runtime-deps.md @@ -3,8 +3,8 @@ title: Hermes Managed-Runtime Dependency Failure type: incident status: resolved created: 2026-09-30 -updated: 2026-09-30 -verified_on: 2026-09-30 +updated: 2026-10-11 +verified_on: 2026-10-11 confidence: high tags: [hermes, cron, runtime, python, worker, incident] sources: [live-jobs-json, live-guard-script, live-cron-list] @@ -92,3 +92,42 @@ Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the - [[systems/hermes-agent]] - [[systems/mnemosyne]] - [[runbooks/restart-hermes]] + + +--- + +## Recurrence - 2026-10-10 (this time the *gateway* runtime env, plus a new failure mode) + +**The class recurred twice in one day, on packages the 15-minute guard does not cover.** + +**1. `mnemosyne` missing from the committed gateway env (repaired 21:26-21:34, verified 22:01:19).** +The gateway does not run from a normal venv: it launches as +`/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '' gateway run` +and `activate_dependencies()` **replaces `sys.path`** with one committed uv environment - +`~/.hermes/installs//environments/795020dbf24343cd9c52df24723dca83/venv`. Of the uv +environments on CT460, that one alone was missing `mnemosyne` / `mnemosyne_hermes`, so sessions +logged `mnemosyne_batch helpers unavailable (No module named 'mnemosyne')` and +`Memory provider 'mnemosyne' is selected but reports unavailable`. Fix: install +`hermes_memory_provider` + `mnemosyne_hermes` + `mnemosyne` **together with the native dependencies** +(`fastembed`, `onnxruntime`, `numpy` + `numpy.libs`, `tokenizers`, `huggingface_hub`, `sqlite-vec`, +`loguru`, `mmh3`) into that committed venv. Verified live: `Memory provider 'mnemosyne' registered +(40 tools)` -> `activated` at 2026-10-10 22:01:19, again at 23:59 / 00:00 / 00:02; and +`hermes memory status` -> `Plugin: installed / Status: available`. + +**2. `elevenlabs` missing from the live managed env (21:13 -> 21:24).** TTS was configured but the +package was absent from the managed runtime in use; reinstalled and verified by generating a real +clip. + +**Trap: `~/.local/lib/python3.14/site-packages` is invisible to the gateway.** The first repair +attempt installed into the user site directory; with `ENABLE_USER_SITE=False` **and** a replaced +`sys.path`, the gateway never sees it. Install into the committed uv env that +`activate_dependencies()` selects - not the user site dir, not a sibling venv. + +**Trap: `hermes memory status` can be green while the gateway is still broken.** It is a separate +process with its own runtime resolution. The gateway's own log lines +(`registered (40 tools)` / `activated` versus `reports unavailable`) are the authoritative check. + +**Trap: this recurrence produced no cron failures at all** (2026-10-10: 174 `completed`, 0 `failed`). +It degraded the gateway's memory and TTS paths only. A green cron day does **not** mean the runtime +is healthy - and the 15-minute guard still does not check the gateway's committed env, so **guard +coverage remains open.** diff --git a/index.md b/index.md index 0d6bf53..f5a4951 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-10 +> model routing and automation. Last updated: 2026-10-11 ## Start here - [[current-state]] — Current systems status @@ -22,13 +22,14 @@ - [[systems/nanobot]] — Nanobot current state and history - [[systems/browser-backend]] — Camoufox (primary) + Steel (disabled), CDP config - [[systems/hermes-ui]] — Browser Hermes Desktop (przbadu/hermes-ui) +- [[systems/paseo]] — Paseo coding-agent daemon + web terminal workspace (CT460, :6767) - [[systems/ignis]] — Ignis browser-native Obsidian (wiki interface) - [[systems/terminal-backend]] — Shell persistence, working directory - [[systems/tool-search]] — Always-on lazy discovery for external tools ### Integrations - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. -- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; **10 carry a non-clean `last_status` as of 2026-10-10 00:22** — 9 from the 2026-10-09 fleet-wide provider timeout storm (15 failed executions, 10 jobs) plus 1 `delivery_failed`; the 7 dead-model-pin jobs repinned 2026-10-05 stay healthy. See [[incidents/2026-10-09-cron-provider-timeout-storm]]) +- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **5 carry a non-clean `last_status`** as of 2026-10-11 00:14 (down from 10). The 2026-10-10 fleet ran **clean — 174 `completed`, 0 `failed`** — ending the 2026-10-09 provider timeout storm (15 failed executions / 10 jobs; 7 of its incidents auto-closed during 2026-10-10, 2 remain open). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. See [[incidents/2026-10-09-cron-provider-timeout-storm]] - [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **no regeneration committed since 2026-10-07** — the 10-08 run's output was left uncommitted and the 10-09 run failed, see [[systems/scheduled-tasks]]) ## Incidents @@ -39,7 +40,7 @@ - [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented - [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); **closed 2026-10-05** when all three were repinned off the pin - [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26) -- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min +- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. **Recurred 2026-10-10 inside the *gateway* runtime env** — `mnemosyne` and `elevenlabs` missing from the committed uv env the gateway loads; both repaired and verified the same evening - [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured - [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration diff --git a/log.md b/log.md index 5473d8d..0c71b67 100644 --- a/log.md +++ b/log.md @@ -432,3 +432,30 @@ No secrets written; credential-shaped values referenced by job ID and connection - **Sessions in the 2026-10-09 window (`state.db`, epoch REAL):** 19 total — 6 `cron`, 11 `ios`, 1 `talaria-ios`, 1 `telegram`. Delivered 16 / suppressed 160 / suppressed-acked 2 (0 delivery failures — the `delivered` count includes failure notices). No secrets written; job/incident IDs only. + +## [2026-10-10] wiki audit | window 2026-10-09 -> 2026-10-10 (ran 2026-10-10 23:59 -> 2026-10-11 00:2x AWST) + +**Cron.** 29 jobs, 26 enabled / 3 disabled (`jobs.json` 2026-10-11T00:19). Day tally: 10-07 `177/1`, +10-08 `178/2`, 10-09 `163/15`, **10-10 `174 completed / 0 failed`**. Non-clean `last_status` 10 -> 5. +Incidents: **7 closed during 10-10, 5 open** (3 jobs). **Open item #1 closed: the DHL monitor +`62619aa6bcb3` is retired and verified absent from `jobs.json` by ID lookup** (the 10-07 and 10-09 +"deleted" claims were false). + +**Runtime/memory.** The 2026-09-28 managed-runtime-dependency class **recurred in the gateway env**: +`mnemosyne` (+native deps) and `elevenlabs` were missing from the committed uv env +`795020dbf24343cd9c52df24723dca83` that `activate_dependencies()` loads. Repaired 21:26-21:34 and +verified live from 22:01:19 (`Memory provider 'mnemosyne' registered (40 tools)` -> `activated`; +repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - left open. + +**New page.** [[systems/paseo]] - Paseo daemon v0.11.2 on `0.0.0.0:6767` (user unit +`paseo-daemon.service`); recorded the `daemon run` vs `daemon start` supervisor trap, the +`/usr/bin/paseo` 0.7.2 vs daemon 0.11.2 version split, and the truncated-`cline` false provider fault. + +**Platform.** Mattermost unreachable from CT460 - configured endpoint `100.98.116.14:8065` answers +neither ICMP nor TCP; gateway retrying every 5 minutes. Cause not determined. + +**Pages touched:** `systems/paseo.md` (new), `systems/scheduled-tasks.md`, `systems/hermes-agent.md`, +`systems/mnemosyne.md`, `systems/messaging-integrations.md`, +`incidents/2026-09-28-hermes-managed-runtime-deps.md`, `current-state.md`, `index.md`, `log.md`. + +No secrets written; job/incident IDs, paths and tailnet addresses only. diff --git a/systems/hermes-agent.md b/systems/hermes-agent.md index 969c369..3b85bd8 100644 --- a/systems/hermes-agent.md +++ b/systems/hermes-agent.md @@ -3,8 +3,8 @@ title: Hermes Agent type: system status: active created: 2026-07-22 -updated: 2026-10-04 -verified_on: 2026-10-04 +updated: 2026-10-11 +verified_on: 2026-10-11 confidence: high tags: [system, hermes, agent, llm] sources: [live-hermes-version, live-config, live-skill-inventory] @@ -148,3 +148,24 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi - [[messaging-integrations]] - [[scheduled-tasks]] - [[current-state]] + + +--- + +## 2026-10-10 - gateway runtime env, TTS, Paseo + +- **Gateway process shape (verified 2026-10-11 00:20):** PID 230494, started 2026-10-10 21:24:44, + launched as `/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '' gateway run` + (RSS ~371 MB). `activate_dependencies()` replaces `sys.path` with a single committed uv + environment (`~/.hermes/installs//environments/795020dbf24343cd9c52df24723dca83/venv`), and the + user site directory is invisible to it - packages the gateway needs must be installed **into that + env**. See [[incidents/2026-09-28-hermes-managed-runtime-deps]]. +- **Mnemosyne provider healthy again:** `Memory provider 'mnemosyne' registered (40 tools)` + + `activated` from 2026-10-10 22:01:19 onward (repeated 23:59 / 00:00 / 00:02); `hermes memory + status` reports available. +- **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env + and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS + output appears only when explicitly requested. +- **Mattermost platform unreachable** since 2026-10-10; the gateway keeps retrying on a 5-minute + cadence - [[systems/messaging-integrations]]. +- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]]. diff --git a/systems/messaging-integrations.md b/systems/messaging-integrations.md index ad27f8b..5584385 100644 --- a/systems/messaging-integrations.md +++ b/systems/messaging-integrations.md @@ -3,8 +3,8 @@ title: Messaging Integrations type: system status: active created: 2026-07-22 -updated: 2026-10-03 -verified_on: 2026-10-03 +updated: 2026-10-11 +verified_on: 2026-10-11 confidence: high tags: [messaging, telegram, discord, email, integrations] sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug] @@ -70,3 +70,26 @@ Anthony asked whether his other Hermes instances (the estate VMs/CTs) could shar - [[systems/hermes-agent]] - [[systems/scheduled-tasks]] - [[infrastructure/tailscale]] + + +--- + +## 2026-10-11 00:20 AWST - Mattermost platform unreachable (observed, not fixed) + +The gateway's Mattermost adapter has been down since at least 2026-10-10. `gateway.log` shows a +5-minute retry cadence with `Mattermost: disconnected` followed by +`Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s` +(attempt 34 at 00:15, attempt 35 at 00:20). + +Probe of the configured endpoint (`MATTERMOST_URL=http://100.98.116.14:8065/`) from CT460: +**100 % ICMP loss and TCP 8065 refuses** - the Mattermost host itself is unreachable from CT460, +not merely an application-level refusal. + +- Cause **not determined**; not investigated further during this audit (no host/VM/proxy probe). +- Impact is limited to the messaging path. No cron job depends on Mattermost, and the 2026-10-10 + cron fleet ran clean. +- Config: plugin block `plugins...mattermost` (`enabled: true`, home channel + `5ipo6wfe3j88xxox1gkta3d5mw`, name `Rhino`); the bot token is env-only and is **not** recorded here. + +**Open item:** re-probe the host; if the host is back but the adapter stays down, restart the +gateway's Mattermost platform. diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index c953db0..ba36e3d 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,8 +3,8 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-10-04 -verified_on: 2026-10-04 +updated: 2026-10-11 +verified_on: 2026-10-11 confidence: high tags: [memory, mnemosyne, backend] sources: [live-probe, mnemosyne-db, config, ops-profile-config] @@ -105,3 +105,23 @@ Primary agent memory store for Hermes. - [[incidents/2026-08-01-mnemosyne-capture-failure]] - [[incidents/2026-08-18-mnemosyne-capture-pipeline]] - [[incidents/2026-07-29-mnemosyne-relay]] + + +--- + +## Re-verified 2026-10-10 (audit window; live probes 2026-10-11 00:14-00:20 AWST) + +- **Gateway load path repaired.** The gateway's `activate_dependencies()` selects one committed uv + env (`795020dbf24343cd9c52df24723dca83`); it was missing `mnemosyne` / `mnemosyne_hermes` and the + native dependencies. Repaired 2026-10-10 21:26-21:34 and confirmed live from 22:01:19 + (`registered (40 tools)` -> `activated`). Detail: + [[incidents/2026-09-28-hermes-managed-runtime-deps]]. +- **"Capture silent since 10-09 15:20" was a false alarm.** It resolved to *no interactive sessions + in the window* - the window was cron-only traffic. What the surrounding hours did capture was + dominated by gateway background-process notifications stored as + `[USER] [Anthony Martin] [IMPORTANT: Background process ...]` rows: recall-visible, low signal. + A capture filter would help; **no fix applied** (carried open item). +- **Coverage script executable again:** `~/.hermes/scripts/check_embedding_coverage.py` now carries + mode `0755`, so the monitor's non-executable finding is resolved. +- **Open, unverified:** 13 orphan rows in `memory_embeddings` (oldest 2026-08-22) reported by a + 2026-10-10 monitor run - neither confirmed nor cleaned by this audit. diff --git a/systems/paseo.md b/systems/paseo.md new file mode 100644 index 0000000..07dce71 --- /dev/null +++ b/systems/paseo.md @@ -0,0 +1,92 @@ +--- +title: Paseo Daemon +type: system +status: active +created: 2026-10-11 +updated: 2026-10-11 +verified_on: 2026-10-11 +review_after: 2026-11-11 +confidence: high +tags: [system, paseo, coding-agent, ct460, systemd] +sources: [live-systemctl-user, live-ps, live-paseo-config, live-session-20261010] +--- + +# Paseo Daemon (CT460) + +Paseo is a **coding-agent orchestrator with a browser terminal workspace**. It runs on +CT460 (`hermes`), hosts terminal sessions against repositories under `/home/hermes`, +and brokers turns to external coding agents — Claude Code, Codex, Cline, OpenCode, and +Hermes itself via ACP. + +- Local: `http://127.0.0.1:6767` +- Tailnet: `https://hermes.kangaroo-eel.ts.net:6767` + +The daemon spawns provider processes on demand; a live example is +`opencode serve --port 42021` running as a child of the unit. + +## State (verified 2026-10-11 00:16 AWST) + +| Item | Value | +|---|---| +| Unit | `paseo-daemon.service` — **user** unit, `enabled` | +| Unit file | `/home/hermes/.config/systemd/user/paseo-daemon.service` | +| State | `active (running)` since **2026-10-10 14:47:28 AWST** | +| Daemon version | **0.11.2** (upgraded from 0.7.0 on 2026-10-10) | +| Main PID | `163127` (`node --disable-warning=DEP0040 /home/hermes/.npm-global/bin/paseo daemon run`) | +| Inner processes | supervisor `163168`, `Paseo Daemon` `163198`, terminal worker, `opencode serve` | +| Listen | `0.0.0.0:6767` | +| Binary used by the unit | `/home/hermes/.npm-global/bin/paseo` | +| Config | `/home/hermes/.paseo/config.json` (mode `0600`) | +| Logs | `/home/hermes/.paseo/-NN-daemon.log` (10 MB rotation) and `journalctl --user -u paseo-daemon` | + +## Maintenance + +`sudo systemctl` will **not** see this unit — it belongs to the user manager: + + systemctl --user status paseo-daemon + systemctl --user restart paseo-daemon + journalctl --user -u paseo-daemon -f + curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:6767/ + +Upgrade: `npm install -g @getpaseo/cli@latest`, then restart the unit. **The daemon keeps +running the old version until the unit restarts.** + +## Why `ExecStart` uses `daemon run`, not `daemon start` + +`paseo daemon start` daemonises and returns immediately, so the foreground process exits +`0`. Under `Type=simple` + `Restart=always` systemd reads that as a clean exit and +restart-loops the service (the flapping observed during the 2026-10-10 repair). +`paseo daemon run` stays in the foreground and is the correct supervised command. + +The unit's `PATH` must include both `/home/hermes/.npm-global/bin` and `~/.local/bin`: +agent CLIs the daemon shells out to are resolved through the *unit's* environment, not an +interactive shell's. + +## Known pitfalls + +- **The `paseo` on `PATH` is not the daemon's version.** `/usr/bin/paseo` reports + **0.7.2** while the unit runs the npm-global **0.11.2**. `paseo --version` therefore + tells you nothing about what the daemon is running — read the unit's `ExecStart`, or + the `daemonVersion` field in the daemon log. +- **A truncated agent binary looks like a provider fault.** A partly-written `cline` + binary (107 KB ELF instead of a full ~3 MB) made that one provider fail while every + other provider worked. Check binary size before chasing provider config; reinstalling + to 3.0.70 fixed it. +- **Hermes as a Paseo provider needs the ACP extra** — `hermes pm install --extra acp` + before the Hermes provider will load (its check reports `Hermes ACP check OK` once + installed). +- **Config edits need a restart.** `baseUrl`/CORS changes in `~/.paseo/config.json` only + take effect after `systemctl --user restart paseo-daemon`. Backups are kept beside the + config (`config.json.bak-20261010-corsfix`, `config.json.bak-20260809-codex-fix`). +- **Cosmetic CLI error while the daemon is healthy (unresolved 2026-10-10).** When a + `daemon.get_status.request` takes ~2.2 s the CLI prints + `daemon detail request to 0.0.0.0:6767 failed` even though `127.0.0.1:6767` returns + HTTP 200. Verify with the health checks above before treating a single such line as an + outage. +- **Pairing payloads are credentials.** The `#offer=…` URL and `~/.paseo/daemon-keypair.json` + carry the daemon key — never paste them into the wiki or a shared channel. + +## Related +- [[systems/hermes-agent]] +- [[systems/terminal-backend]] +- [[infrastructure/network-map]] diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 9d2d29b..65ef39f 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-09 -verified_on: 2026-10-09 -review_after: 2026-10-23 +updated: 2026-10-11 +verified_on: 2026-10-11 +review_after: 2026-10-25 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] @@ -171,3 +171,46 @@ See [[incidents/2026-10-09-memory-recall-relevance]]. 5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open. 6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open. + + +--- + +## Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST) + +**Fleet: 29 jobs - 26 enabled - 3 disabled** (`jobs.json.updated_at 2026-10-11T00:19:44+08:00`). +The count fell from 30 to 29 because **`62619aa6bcb3` (DHL Tracking - Parcel `8052838382`) is gone**: +open item #1 of the 2026-10-09 audit is **CLOSED** (below). Disabled: `d40085da631c` (Important-mail +monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher), `0a7f933d06e2` (AgentMail Rhino Inbox Poll). + +**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` - 10-08 `178 / 2` - +10-09 `163 / 15` - **10-10 `174 completed / 0 failed`** (+1 left `running`). The 2026-10-10 fleet was +wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm. + +**Non-clean `last_status`: 5 jobs, down from 10.** + +| Job | ID | `last_status` | streak | +|---|---|---|---| +| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `error` | 0 | +| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `error` | 0 | +| Daily Marketing Job Radar | `dca8482e4f76` | `error` | 1 | +| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `error` | 1 | +| Uber Eats Monthly Spend Report | `066bb5191280` | `delivery_failed` | 0 | + +The three `streak 0` rows are stale `last_status` fields, not live failure streaks. `1a8e90ea2271` +(stocktake) is back to `last_status: ok`. + +**Incidents: 7 closed during 2026-10-10, 5 open.** The 10-09 storm unwound as each job next ran clean +(`ea79a2` 00:03, `cbfe43` 00:27, `1a8e90ea2271` x2 03:06, `85edf6` 07:04, `08eb86` 08:32, `ecc1fc` +12:05). Still open: `1b23c042e17d` + `dca8482e4f76` (`timeout`, `alerted`, both first seen +2026-10-09) and `0e753e9720fd` + `fad5a86979d4` (`agent`; open since 2026-10-01 / 2026-09-30). + +**Open item #1 CLOSED - the DHL monitor is retired.** Anthony instructed removal again on the evening +of 2026-10-10 and the job is **absent from live `jobs.json`**, verified 2026-10-11 00:19 by direct ID +lookup (no job id contains `62619`; no job name matches dhl/parcel). This is the third "deleted" +claim for this job - the 2026-10-07 and 2026-10-09 reports were both **false** because the check was +the absence of job output rather than a positive lookup. Removals must be verified by ID lookup. + +**Still open:** the Quartz rebuild step in `wiki-maintain.sh` (no `quartz.config.ts`) - skills-index +off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost +platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the +15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).