Compare commits
64
Commits
2df1bce5ea
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f603c29d17 | ||
|
|
8bf0a37022 | ||
|
|
226767f3ec | ||
|
|
16f2c878c0 | ||
|
|
98610466cc | ||
|
|
4574c4ebc3 | ||
|
|
e5612c8a8d | ||
|
|
4eee02b9a2 | ||
|
|
9f80926dda | ||
|
|
402d8546e8 | ||
|
|
646be98c31 | ||
|
|
82876abdfb | ||
|
|
2d439145bf | ||
|
|
4a6e86948d | ||
|
|
16767ef562 | ||
|
|
b0e4a54f8a | ||
|
|
a0ec3a1ec5 | ||
|
|
bc35c4bb27 | ||
|
|
e033a08f17 | ||
|
|
ab3652aa20 | ||
|
|
b94de554f2 | ||
|
|
1b0cf1fa71 | ||
|
|
729bb597cf | ||
|
|
c108eb1012 | ||
|
|
341dda95dd | ||
|
|
7ceaec1a2f | ||
|
|
6de35a37ff | ||
|
|
262bcc74d5 | ||
|
|
3abd9901d3 | ||
|
|
e2f1ae7a0b | ||
|
|
4fd138bb18 | ||
|
|
0b4ff25fd2 | ||
|
|
b820993ecc | ||
|
|
2d5219b068 | ||
|
|
88baf4fb5a | ||
|
|
8abbc6bb60 | ||
|
|
eba8062735 | ||
|
|
2f3fc9952b | ||
|
|
20aa11b0c4 | ||
|
|
3b5daccb4d | ||
|
|
e4086ec18e | ||
|
|
785fbc40b2 | ||
|
|
ca014efd63 | ||
|
|
fda56e53fe | ||
|
|
8bb3d196b1 | ||
|
|
a0b3734c95 | ||
|
|
c69681dc16 | ||
|
|
97660d3dcb | ||
|
|
0c826f2b5f | ||
|
|
74a5b2d117 | ||
|
|
db6af66878 | ||
|
|
987eba72e6 | ||
|
|
1d335ca8e6 | ||
|
|
cc95dcc54d | ||
|
|
677bb0c959 | ||
|
|
ec0c30fb30 | ||
|
|
2cbf0b65c1 | ||
|
|
3ae05871c4 | ||
|
|
f880ba981d | ||
|
|
a2dd0bd650 | ||
|
|
3f796dc3b9 | ||
|
|
7fe109b914 | ||
|
|
4126093644 | ||
|
|
123a74cb0b |
@@ -3,3 +3,4 @@ public/
|
||||
package-lock.json
|
||||
package.json
|
||||
*.log
|
||||
.audit-report-*.txt
|
||||
|
||||
+12
-3
@@ -3,8 +3,8 @@ title: Contributing to the Agent Estate Wiki
|
||||
type: guide
|
||||
status: active
|
||||
created: 2026-08-16
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [wiki, contributing, governance, multi-agent]
|
||||
sources: []
|
||||
@@ -47,7 +47,7 @@ Each agent owns **its own section only**. Do not edit another agent's pages with
|
||||
| `agents/nanobot/` | Nanobot (CT333) |
|
||||
| `agents/zeroclaw/` | ZeroClaw (CT333) |
|
||||
| `agents/qwenpaw/` | QwenPaw (CT465) |
|
||||
| `agents/open-claw/` | Open-Claw (VM 403) |
|
||||
| ~~`agents/open-claw/`~~ | ~~Open-Claw (VM 403)~~ — **removed 2026-10-20; re-confirmed absent 2026-11-11** |
|
||||
| `shared/`, `infrastructure/`, `runbooks/`, `decisions/`, `incidents/` | Anyone, but **facts must be verified against live state** and linked to a source |
|
||||
|
||||
## What to save (and what NOT to save)
|
||||
@@ -100,6 +100,15 @@ After pushing, the Quartz web view updates on the next daily maintenance run (03
|
||||
4. Make your first commit: your role, host, config pointers, and known limitations.
|
||||
5. Keep `updated:` honest on every edit.
|
||||
|
||||
> **Access paths re-verified 2026-11-11.** `http://100.118.5.51:9120/` returns **200** (as does a
|
||||
> real page such as `/systems/skills-index`), the unit behind it is
|
||||
> `hermes-wiki-serve.service` — **active**, listening on `0.0.0.0:9120` (a **user** unit — check it
|
||||
> with `systemctl --user`, never bare `systemctl`, or it reads as dead). The external route
|
||||
> `https://hermes.kangaroo-eel.ts.net/wiki/` returns **200** via the Tailscale Serve path.
|
||||
> `git remote -v` confirms the remote is the documented
|
||||
> `http://…@gitea:3000/Anthony/agent-estate-wiki.git` — note the URL carries a credential; it is
|
||||
> deliberately **not** reproduced here, use `git remote -v` to read it.
|
||||
|
||||
## Related
|
||||
|
||||
- [[agents/index]] — estate map
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Hermes Chaos
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-08-15
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
confidence: high
|
||||
tags: [hermes, chaos, ct450, experimental, agent]
|
||||
sources: [live-probe, config-inspection]
|
||||
@@ -28,15 +28,15 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
|
||||
## Current State
|
||||
|
||||
- **Status:** Active (experimental)
|
||||
- **Hermes version:** `v0.21.5+6170.ga510d6c` (2026.9.24), upstream `a510d6c6` — **updated from v0.19.0 on 2026-10-03**, first update since July; 3 commits behind upstream. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]].
|
||||
- **Hermes version:** ⚠ **now `v0.23.0`** — the live install stamp `~/.hermes/hermes-agent/install-stamp.json` inside CT450 reads `baseVersion: 0.23.0`, `displayVersion: 0.23.0`, `distance: 0`, `payload: bootstrap` (probed 2026-11-08 00:4x AWST by the nightly audit; `hermes --version` not re-run in that probe). The rest of this bullet is the 2026-11-07 correction, written when CT450 was still `v0.22.5` (2026.9.24), local `bf0203a794a0` — **the same build as CT460** (updated 2026-10-21). ⚠ **Corrected 2026-11-07:** this line still read *"`v0.22.3` … local `5b147ab036d6`"* while asserting it was *"the same build as CT460"* — it was not. Live `hermes --version` on CT450 then returned **`v0.22.5` … local `bf0203a794a0`**, byte-identical to CT460's own `hermes --version`, so the *same build* claim finally became true. `hermes update --check` then reported stable **v0.23.0** (`2bd378ed1b8d`) available; CT460 took it on 2026-11-07 and CT450's stamp now reads the same version. Python 3.14.9. See [[incidents/2026-10-03-ct450-hermes-update]].
|
||||
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
|
||||
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
|
||||
- **Duplicate cron jobs:** 11 paused (no post-pause execution records since 2026-08-01)
|
||||
- **Production counterparts:** CT460 has 11 enabled jobs with successful latest runs
|
||||
- **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` **2026-11-07**; unchanged). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460) — both confirmed by a live set-diff of the two `jobs.json` files on 2026-11-07
|
||||
- **Production counterparts:** CT460 has **28 jobs / 25 enabled** — the two fleets no longer share a job count, and the CT450 duplicates are paused
|
||||
|
||||
## Providers & Routing
|
||||
|
||||
- **Default model:** `deepseek-flash-mix` via `custom:omniroute` (`http://100.88.81.19:20128/v1`, key via `$HERMES_CUSTOM_100_88_81_19_20128_API_KEY`)
|
||||
- **Default model:** **`smart-route`** via `custom` → `http://100.78.63.20:20128` (the `omni-101` node) — verified live in CT450's `config.yaml` 2026-10-26. ⚠ The old claim of `deepseek-flash-mix` on `custom:omniroute` (`100.88.81.19`) no longer matches the config
|
||||
- **Fallback chain** (fixed 2026-08-15 — OmniRoute fallback entries removed, they were non-functional):
|
||||
1. `kilo-stepfun-step-3_7-flash-free` via `litellm_4000`
|
||||
2. `nvidia/nemotron-3-ultra:free` via `nous`
|
||||
|
||||
+12
-11
@@ -3,8 +3,8 @@ title: Hermes Production
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-08-15
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
confidence: high
|
||||
tags: [hermes, production, ct460, agent]
|
||||
sources: [live-probe, config-inspection]
|
||||
@@ -19,15 +19,16 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
## Location
|
||||
|
||||
- **Container:** LXC CT460 (`hermes`)
|
||||
- **Tailscale IP:** `100.96.100.82`
|
||||
- **Tailscale IP:** `100.118.5.51` (⚠ `100.96.100.82` is the **PVE hypervisor** `pve`, not CT460 — corrected 2026-10-26)
|
||||
- **Hostname:** `hermes`
|
||||
- **Port:** 8082 (HTTP), 9119 (gateway)
|
||||
- **Ports:** 9119 (gateway/API), 8085 (webhook) — ⚠ the old **8082** claim was dead: no 8082 listener on CT460 as of 2026-10-26
|
||||
- **Memory:** 14GB (upgraded from 5096MB on 2026-08-07)
|
||||
|
||||
## Current State
|
||||
|
||||
- **Status:** Active
|
||||
- **Hermes version:** `v0.21.5+6169.gab86a20` (2026.9.24), upstream `ab86a20a`; install tree rebuilt 2026-10-03 10:11 AWST — verified live 2026-10-04
|
||||
- **Hermes version:** ⚠ **`v0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s), `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`**; local version `0.22.5` → `0.23.0`. ⚠ The **`snapshot` stage was skipped again** (`disabled by updates.pre_update_backup (mode: off)`) → **no rollback point**. Restarts: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`); the gateway went round **again 11:11:44 → 11:11:55** (live process `pid 3109529`). **Live-probed 2026-11-08 00:4x AWST.** Previous: `v0.22.3` (2026.9.24), local `5b147ab036d6` — updated 2026-10-21 22:47→22:57 AWST.
|
||||
- **Cron ledger 2026-11-07:** 163 rows — **159 `completed` / 3 `failed` / 1 `running`** (that *running* row is this nightly audit). Two of the three failures are one event: the **05:01:00 gateway restart drain** orphaned *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22) — both closed `failed` on `lost its durable fire claim ownership`. The third (`96eb95ce12f2`, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog firing 19 s later on the first failure's traceback. → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
|
||||
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
|
||||
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
||||
- **Config:** `~/.hermes/config.yaml`
|
||||
@@ -43,10 +44,10 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
|
||||
## Providers & Routing
|
||||
|
||||
- **Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — as attested 2026-09-30; the `custom:omni` provider key no longer exists
|
||||
- **Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, CT101/m93p) — verified live 2026-10-26, **re-verified 2026-11-07** (live `config.yaml` `model.default.provider: custom:omni-101`; a keyed `deepseek-paid-smart` chat probe against `100.78.63.20:20128` returns **HTTP 200**). The `custom:omni` provider key does not exist in `custom_providers`. ⚠ **Corrected 2026-11-07:** the old claim that *"2 cron jobs still pin `custom:omni`"* is **no longer true** — live `jobs.json` shows **no job pinned to the bare `custom:omni`**; **all 21 provider-pinned jobs use `custom:omni-101`** (see [[systems/scheduled-tasks]])
|
||||
- **Default model:** `smart-route` (previously `freellmapi-auto`)
|
||||
- **Fallback providers:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`, `custom / litellm:4000`
|
||||
- **Auxiliary curator model:** `openai-codex / gpt-5.6-terra`
|
||||
- **Fallback providers** (live chain, re-verified 2026-10-26): `nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6_1t`. ⚠ The **first** entry is the dead `nous` credential — a fallover tries it first
|
||||
- **Auxiliary models:** all eight `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) are **`no-think/freeapi/auto` on `custom:omni-101`** with a `freeapi/deepseek-v4-flash` → `kg/kilo-auto/free` fallback chain (moved 2026-10-25 19:13)
|
||||
- See [[systems/model-providers]] for full breakdown.
|
||||
|
||||
## Memory
|
||||
@@ -58,7 +59,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
|
||||
## Scheduled Tasks
|
||||
|
||||
- **Cron jobs:** 30 (27 enabled / 3 disabled; **6 failing** as of 2026-10-04)
|
||||
- **Cron jobs:** **28 — 25 enabled / 3 disabled** (verified live `jobs.json` 2026-10-26). **5 non-clean `last_status`**: `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`), `08eb86f02d37` (`delivery_queued`, empty `last_error`), `c0a9fa314542` (`error`, streak 1)
|
||||
- Delivery targets: Telegram (primary), Discord, origin, local
|
||||
- Notable: Birthday checker, job radar, newsletter digests, mnemosyne health/watchdog/backup
|
||||
- See [[systems/scheduled-tasks]] for full table.
|
||||
@@ -66,7 +67,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
## Messaging
|
||||
|
||||
- Telegram, Discord, Slack, Email, Mattermost, ntfy
|
||||
- **Mattermost is currently unreachable** — the gateway logs `mattermost connect timed out after 30s, next retry in 300s` (live, 2026-10-04) and the Mattermost host does not answer. Treat Mattermost delivery as down until probed.
|
||||
- **Mattermost is currently unreachable** — the gateway logs `mattermost connect timed out after 30s, next retry in 300s` (re-probed live 2026-10-26) and the Mattermost host does not answer: CT255 is `stopped` on `pve` and its Tailscale peer has been offline **36d**. Treat Mattermost delivery as down until probed.
|
||||
- See [[systems/messaging-integrations]] for details.
|
||||
|
||||
## Key Differences from Chaos
|
||||
@@ -74,7 +75,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
| Aspect | Production (CT460) | Chaos (CT450) |
|
||||
|--------|-------------------|---------------|
|
||||
| Purpose | Stable daily operations | Experimental/testing |
|
||||
| Cron jobs | 30 (27 enabled / 3 disabled) | 11 paused duplicates |
|
||||
| Cron jobs | **29 (26 enabled / 3 disabled)** (re-verified 2026-11-07) | 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460 (re-verified 2026-11-07) |
|
||||
| Config changes | Requires verification | Safe to experiment |
|
||||
| Memory | Production Mnemosyne | Separate state |
|
||||
|
||||
|
||||
+10
-2
@@ -3,8 +3,8 @@ title: Agent Estate
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-08-15
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [agents, estate, overview]
|
||||
---
|
||||
@@ -40,6 +40,14 @@ also has **no VM 403** (its LXC guests are 100, 101, 102, 104, 105, 221, 245, 50
|
||||
**ZeroClaw, by contrast, was re-verified live on 2026-10-20** and **is running** on CT333
|
||||
(`/usr/local/bin/zeroclaw daemon --allow-degraded-security`, PID 136) — see [[agents/zeroclaw]].
|
||||
|
||||
> **Guest lists re-probed 2026-11-11 — VM 403 and VM 401 still absent, and the estate shrank a
|
||||
> little more.** `qm list` on `pve` (`100.96.100.82`) now shows only VMs **100, 101, 110, 282**
|
||||
> (VM **310** has gone since the 2026-10-19 probe listed it), and `pct list` shows CTs 103, 200,
|
||||
> 201, 205, 207, 209 (stopped), 215, 255, 270, 299 (stopped), 333, 450, 460, 465, 475, 555
|
||||
> (stopped), 600 — CT **601** is gone. Live CT names: `205 ollama`, `333 clawtest`,
|
||||
> `450 hermesagent`, `600 omniroute`. `m93p` was not re-probed this pass; its 2026-10-20 list
|
||||
> stands. ZeroClaw is running on CT333 (PID 142, `--allow-degraded-security`).
|
||||
|
||||
## Shared Infrastructure
|
||||
|
||||
All agents share:
|
||||
|
||||
+44
-17
@@ -3,8 +3,8 @@ title: Nanobot
|
||||
type: agent
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [agent, nanobot, homelab]
|
||||
sources: [live-probe, runtime-config]
|
||||
@@ -20,7 +20,7 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat
|
||||
- **Workspace:** `/home/nano/.nanobot/workspace`
|
||||
|
||||
## Current state
|
||||
- **Status: ACTIVE.** Verified 2026-10-11: gateway process `nanobot-gateway` (PID 2259048) running and listening.
|
||||
- **Status: ACTIVE.** Re-verified 2026-11-20: `nanobot-gateway` PID 1256104 (since 2026-11-19), `/health` → `status ok, ready true`, v0.4.1. (Prior check 2026-10-11: PID 2259048.)
|
||||
- **Gateway:** port `8765` (HTTP API + chat completions) — `LISTEN 0.0.0.0:8765`
|
||||
- **WebSocket bridge:** port `8766` — `LISTEN 0.0.0.0:8766`
|
||||
- **Serve API:** port `8900` was recorded as the Serve API; **not listening** in the 2026-10-11 `ss -ltn` check. Treat as unverified/possibly removed.
|
||||
@@ -35,23 +35,34 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat
|
||||
| `agentmail-daemon.py` | nano | — | started Sep 23 |
|
||||
| `readlater` Next dev server | nano | 3210 | started Sep 23 |
|
||||
|
||||
Other listeners on CT333 include 8787, 8880, 8899, 8999, 3000, 5050, 8080 and the Tailscale 443.
|
||||
Other listeners on CT333 include 8787, 8880, 8899, 8999, 5050, 8080 and the Tailscale 443. *(Port 3000 was published by the `steel-browser` container, which was removed 2026-10-29.)*
|
||||
|
||||
## Docker on CT333 (verified 2026-10-11)
|
||||
`steel-browser`, `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `tender_brattain`, plus healthy `community-spark` and `quirky_darwin`.
|
||||
## Docker on CT333 (verified 2026-10-11; Steel container removed 2026-10-29)
|
||||
~~`tender_brattain`, plus healthy `community-spark` and `quirky_darwin`~~ → **2026-11-20 live list:** `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `ecstatic_swartz` (random-name successor of `tender_brattain`), healthy `community-spark`; `quirky_darwin` removed; stopped `qdrant` present (Exited 5 months ago).
|
||||
|
||||
**Steel is gone from CT333 too.** The `steel-browser` container (image `ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image were removed on **2026-10-29**. It was already orphaned — its last CDP navigation was 2026-10-20, it crashed on 2026-10-22, it received no requests after 2026-10-21, and it had no Tailscale Serve route, no entry in `~/.zeroclaw/config.toml` (which uses `backend = "agent_browser"` on native_webdriver at `127.0.0.1:9515`), and no reference from Nanobot's browser harness. → [[comparisons/steel-vs-camoufox]]
|
||||
|
||||
## Cron jobs
|
||||
Unverified against live Nanobot scheduler on 2026-10-11. Previously recorded:
|
||||
- `auto-upgrade-freellmapi-safeguard` — daily at 03:30 AWST
|
||||
- `morning-quota-report` — daily at 07:30 AWST
|
||||
- `daily-quota-730am` — daily at 07:30 AWST
|
||||
- `vault-daily-seed` — daily at 08:00 AWST
|
||||
- `daily-agent-briefing` — daily at 08:15 AWST
|
||||
- `rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST
|
||||
- `daily-quota-7pm` — daily at 19:00 AWST
|
||||
- `evening-quota-report` — daily at 21:00 AWST
|
||||
- `heartbeat` — every 30m
|
||||
- `dream` — every 2h (memory consolidation)
|
||||
**Verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` — **11 jobs, 8 enabled**
|
||||
(all schedules Australia/Perth):
|
||||
|
||||
| Job | Schedule | Enabled |
|
||||
|---|---|---|
|
||||
| `auto-upgrade-freellmapi-safeguard` | daily 03:30 | ✅ |
|
||||
| `nightly-skills-export-sync` | daily 04:00 | ✅ |
|
||||
| `morning-news-podcast` | daily 07:00 | ✅ |
|
||||
| `daily-quota-730am` | daily 07:30 | ✅ |
|
||||
| `morning-quota-report` | daily 07:30 | ❌ disabled |
|
||||
| `vault-daily-seed` | daily 08:00 | ✅ |
|
||||
| `daily-agent-briefing` | daily 08:15 | ❌ disabled (2026-10-30) |
|
||||
| `daily-quota-7pm` | daily 19:00 | ✅ |
|
||||
| `evening-quota-report` | daily 21:00 | ❌ disabled |
|
||||
| `heartbeat` | every 30m | ✅ |
|
||||
| `dream` | every 2h (memory consolidation) | ✅ |
|
||||
|
||||
~~`rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST~~ **No longer in the scheduler** (removed at
|
||||
some point before 2026-11-20; the enabled quota-notification pair `daily-quota-730am` /
|
||||
`daily-quota-7pm` remains enabled, per Anthony's standing preference).
|
||||
|
||||
## Skills
|
||||
Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli-app-exa`, `playwright`, `system-prompts-collection`, `agent-skill-retrieval`, `enhanced-search`, `api-gateway`, `debugging-and-error-recovery`, `mnemosyne-memory-hooks`, `proxmox-shell-fallback`, `agentmail`, `langfuse`, `install-and-verify-agent-skills`, `verify-system-claims`, `skill-interceptor`, `tool-call-failure-recovery`, `using-agent-skills`, `api-vault`, `repair-memory`, `patch-config-relative-paths`, `long-task-management`, `nanobot-full-restore-backup`, `context-diet`, `frontend-design`.
|
||||
@@ -71,5 +82,21 @@ Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli
|
||||
- [[runbooks/restart-hermes]]
|
||||
- [[runbooks/nanobot-restart]]
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake, live probes on CT333).** Gateway healthy:
|
||||
> `GET :8765/health` → `{"status":"ok","ready":true,"websocket":"running","version":"0.4.1"}`;
|
||||
> `nanobot-gateway` PID **1256104** (started 2026-11-19), ports 8765 + 8766 listening, 8900 still
|
||||
> **not listening**. Tailscale IP confirmed `100.122.189.81`. The mnemosyne MCP, chroma_query_api,
|
||||
> agentmail-daemon and readlater (:3210) processes all still run.
|
||||
>
|
||||
> **Docker drift corrected:** live `docker ps -a` shows `ecstatic_swartz` (Up 19h) where this page
|
||||
> listed `tender_brattain` — Docker regenerates random names for compose-less containers, so this is
|
||||
> a rename-class drift, not a new service. `quirky_darwin` is **absent even with `-a`** (removed, not
|
||||
> stopped). `community-spark`, `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`,
|
||||
> `edge-tts`, `onecli-postgres-1` all Up ~9 days and healthy where applicable; a stopped `qdrant`
|
||||
> (Exited 5 months ago) also exists. No steel container or image remains; nothing on :3000.
|
||||
>
|
||||
> **Cron verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` (the
|
||||
> `~/.nanobot/cron/` dir is empty — the scheduler file lives under `workspace/cron/`):
|
||||
> **11 jobs, 8 enabled** — see the rewritten Cron jobs section above.
|
||||
## Related
|
||||
- [[comparisons/hermes-vs-nanobot]]
|
||||
|
||||
+8
-2
@@ -3,8 +3,8 @@ title: Open-Claw
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-08-15
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [openclaw, open-claw, vm403, gateway, agent]
|
||||
sources: [skill-inspection, config-reference]
|
||||
@@ -36,6 +36,12 @@ Autonomous AI agent gateway running on a separate QEMU VM. Docker-based with Tai
|
||||
1263 `social-dashboard`, 1264 `openmausbot`, 998 `nightbridge` (stopped) — and **no VM 403 there
|
||||
either**. The Tailscale peer list likewise contains **no `open-claw` node**. So Open-Claw is not
|
||||
merely moved between hosts: it is gone from both.
|
||||
- **Guest lists re-probed 2026-11-11.** `qm list` on `pve` (`100.96.100.82`) now shows only VMs
|
||||
**100, 101, 110, 282** (VM **310** is gone too — the page previously listed it), and `pct list`
|
||||
shows CTs 103, 200, 201, 205, 207, 209 (stopped), 215, 255, 270, 299 (stopped), 333, 450, 460,
|
||||
465, 475, 555 (stopped), **600** — CT **601 is gone** as well. Live CT names read
|
||||
`205 ollama`, `333 clawtest`, `450 hermesagent`, `600 omniroute`. **No VM 403 on either hypervisor,
|
||||
and no `open-claw` Tailscale peer** — the "NOT PRESENT" verdict holds and is now ~3 weeks stable.
|
||||
- **Consequence for the rest of this page:** the Tailscale IP `100.64.172.34`, the LAN IP
|
||||
`192.168.178.193` and the control UI `https://open-claw.kangaroo-eel.ts.net` are all historical.
|
||||
Treat the whole page as a design record, not a description of a running system.
|
||||
|
||||
+9
-7
@@ -3,8 +3,8 @@ title: QwenPaw
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-08-15
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
confidence: high
|
||||
tags: [qwenpaw, ct465, agent, paseo]
|
||||
sources: [live-probe, config-reference]
|
||||
@@ -24,11 +24,13 @@ Multi-agent gateway platform. Runs on CT465 with its own workspace, agent profil
|
||||
|
||||
## Current State
|
||||
|
||||
- **Status:** Active — `paseo.service` active on CT465, verified 2026-10-11
|
||||
- **Status:** Active — `paseo.service` active on CT465, **re-verified 2026-11-04**
|
||||
- **Default chat model:** OmniRoute Smart Route (`auto/smart`)
|
||||
- **Config format:** JSON (`agent.json` per workspace)
|
||||
- **Daemon:** `paseo.service`
|
||||
- **Also listening on CT465:** PVE exporter (`100.96.150.69:9221`), a Python service on `:9876`, and local SMTP `:25`
|
||||
- **Daemon:** `paseo.service` — **system** unit on CT465 (distinct from the **user** unit `paseo-daemon.service` that runs on CT460; see [[systems/paseo]])
|
||||
- **CT465 listening ports (live 2026-11-04):** `0.0.0.0:6767` (Paseo), `*:13500`, `127.0.0.1:25463`, `127.0.0.1:11434` (Ollama), `127.0.0.1:38585`, `100.110.72.126:42510`, `:22`
|
||||
- ⚠ **Corrected 2026-11-04.** The old claim that "PVE exporter (`100.96.150.69:9221`), a Python service on `:9876`, and local SMTP `:25`" were listening **on CT465** was **wrong on all three counts**. Live `ss -ltn` inside CT465 shows **none** of them. `:9221` and `:9876` listen on the **`pve` hypervisor host itself** (`100.96.100.82`), not in the guest, and `100.96.150.69` is **CT200** (the Docker host), not the PVE exporter. `:25` is the hypervisor's own loopback SMTP, also not in CT465.
|
||||
- **Port 6767 has two live implementations — do not conflate them:** CT465's own `paseo.service` listens on `0.0.0.0:6767` and answers **HTTP 404** on `/` (the daemon is alive; `/` is not a route). The tailnet node `paseo` (`100.72.188.93`) is a **different guest** — CT1261 on `m93p` — also running `paseo.service` on `:6767`, and it answers **HTTP 200** on `/`. CT460's own user-unit daemon is a third instance (also `:6767`, **HTTP 404** on `/`; `/api/health` → 200).
|
||||
- **CT400/CT465 identity:** confirmed CT465 = `qwenpaw`, running
|
||||
|
||||
## Operating Layers
|
||||
@@ -60,8 +62,8 @@ qwenpaw daemon status
|
||||
|
||||
When adding an OpenAI-compatible provider from Proxmox:
|
||||
- Verify target by VMID and hostname before writing config
|
||||
- CT600/`omniroute` is the **live** OmniRoute service at `100.88.81.19:20128` — verified serving 2026-10-11
|
||||
- CT601/`omni` is a separate guest at `100.93.204.4` — **stopped**, do not use
|
||||
- CT600/`omniroute` is the **live** OmniRoute service at `100.88.81.19:20128` — verified serving 2026-10-11, still live 2026-11-04
|
||||
- ⚠ **Corrected 2026-11-04:** CT601/`omni` is **no longer present on either hypervisor** — `pct list` on both `pve` and `m93p` shows **no CT601 at all**, and the `omni` tailnet peer (`100.93.204.4`) has dropped out of the tailnet entirely. The guest was **removed**, not merely stopped. Use CT600.
|
||||
- A third instance `omniroute-1` at `100.78.63.20:20128` is also live; Hermes registers it as provider `omni-101`
|
||||
- Query `/v1/models` using the exact API key stored for QwenPaw, not a management key
|
||||
- See [[systems/omniroute]] for the full live node table
|
||||
|
||||
+9
-2
@@ -3,8 +3,8 @@ title: ZeroClaw
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-08-15
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [zeroclaw, ct333, daemon, agent]
|
||||
sources: [skill-inspection, config-reference]
|
||||
@@ -40,6 +40,13 @@ Autonomous AI daemon running on CT333. Handles independent agent operations with
|
||||
> the live command line.
|
||||
> `/root/.zeroclaw/` contains `config.toml` plus ~15 dated `config.toml.bak*` snapshots.
|
||||
|
||||
> **Re-verified 2026-11-11 — running as documented.** `pct exec 333 -- pgrep -af zeroclaw` returns
|
||||
> `142 /usr/local/bin/zeroclaw daemon --allow-degraded-security` (PID has rotated from 136, up
|
||||
> since boot). `/usr/local/bin/zeroclaw` is the live path; `/root/.local/bin/zeroclaw` also exists.
|
||||
> CT333 is named **`clawtest`** in `pct list` and `pct exec 333 -- hostname` returns `clawtest` —
|
||||
> so the `clawtest` label on this page is correct, and the contrary claim on
|
||||
> [[comparisons/hermes-vs-nanobot]] has been corrected.
|
||||
|
||||
## Key Paths
|
||||
|
||||
| Item | Path |
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Camoufox vs Chrome
|
||||
type: comparison
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: medium
|
||||
tags: [comparison, browser]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -35,7 +35,10 @@ Compare the two browser-automation options for Hermes's browser tool.
|
||||
`browser.camofox:` → `adopt_existing_tab: true`, `managed_persistence: true`,
|
||||
`rewrite_loopback_urls: true`, `user_id: anthony`.
|
||||
**Note `inactivity_timeout` is 450s, not the 120s recorded here** — that value changed.
|
||||
`cloud_provider` is `steel` with `engine: camofox`; the two keys describe different layers.
|
||||
~~`cloud_provider` is `steel` with `engine: camofox`; the two keys describe different layers.~~
|
||||
**Superseded 2026-10-29:** `cloud_provider` now reads **`camofox`** (with `camera_provider: camofox`,
|
||||
`engine: auto`, `cdp_url: ""`), and Steel is **fully removed** from the estate — the `browser-steel` plugin
|
||||
is deleted and there is no Steel fallback path. → [[systems/browser-backend]]
|
||||
- **Live processes (2026-10-20):** `cua-driver serve` (PID 3312393), `camoufox-ws-server.mjs`,
|
||||
and a live `camoufox-bin` (FirefoxDev build) tree under `/home/hermes/.cache/camoufox/`.
|
||||
Camofox's REST port **9377 is not listening** on CT460 — the Camofox browser service runs on
|
||||
@@ -43,6 +46,16 @@ Compare the two browser-automation options for Hermes's browser tool.
|
||||
- Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing.
|
||||
- `cua-driver` bridges Hermes's computer-use tool calls to the browser.
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake).** Live config unchanged: `cloud_provider: camofox`,
|
||||
> `camera_provider: camofox`, `engine: auto`, `headed: true`, `inactivity_timeout: 450`,
|
||||
> `use_gateway: false`, `cdp_url: ""`, `camofox.managed_persistence: true`, `user_id: anthony`.
|
||||
> `cua-driver` still **v0.28.2**; `cua-driver serve` + `camoufox-ws-server.mjs` + a `camoufox-bin`
|
||||
> tree all running on CT460 (under Xvfb `:99` 1600×1200 + fluxbox).
|
||||
> **Correction:** ~~Camofox's REST port 9377 is not listening on CT460 — the service runs on CT450~~ —
|
||||
> `:9377` **is listening on CT460** (`node`, `/health` → `ok`, engine `camoufox`); the service is the
|
||||
> local `systemd --user` unit `camofox-browser.service` (active), per [[systems/browser-backend]]'s
|
||||
> 2026-10-13 correction. The CT450 attribution is dead. The ws-server also listens on `:9222` (CDP).
|
||||
> No Chrome/Chromium process found in this probe either.
|
||||
### Chrome (not found in use)
|
||||
- No `google-chrome`, `chromium`, or CDP-only launch flags found in the sanitized config or running process list during this audit.
|
||||
- Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Hermes vs Nanobot
|
||||
type: comparison
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [comparison, hermes, nanobot]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -29,8 +29,14 @@ Compare Hermes (this agent) with Nanobot, another agent process in the homelab,
|
||||
>
|
||||
> **One caveat worth adding:** CT333 now runs **both** Nanobot *and* ZeroClaw (the latter verified
|
||||
> live on 2026-10-20 — see [[agents/zeroclaw]]). The page's framing of CT333 as the Nanobot host
|
||||
> is not wrong, but the host is no longer single-purpose. Note also that **CT333 is `zeroclaw` in
|
||||
> the current Proxmox naming** — the wiki uses the older "clawtest" label for the same LXC.
|
||||
> is not wrong, but the host is no longer single-purpose.
|
||||
>
|
||||
> **Correction 2026-11-11: the "CT333 is `zeroclaw` in Proxmox naming" claim was itself wrong.**
|
||||
> Live `pct list` on the PVE host (`100.96.100.82`) still reports CT333's name as **`clawtest`**
|
||||
> and `pct exec 333 -- hostname` returns **`clawtest`** too — the wiki's original label is correct
|
||||
> on both counts. `zeroclaw` is only the **process/daemon** that runs inside CT333, alongside
|
||||
> `nanobot`. Re-verified live the same day: the peer is on Tailscale at `100.122.189.81 clawtest`
|
||||
> and `http://nanobot.martinwa.org` still answers **301**.
|
||||
|
||||
## At a glance
|
||||
| | Hermes (ct460) | Nanobot (ct333 "clawtest") |
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Memory Backends
|
||||
type: comparison
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [comparison, memory]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -43,6 +43,14 @@ The three archived scripts (`hindsight-watchdog.py.archived`, `hindsight_migrati
|
||||
> in cron contexts, which is why scheduled jobs produce no working memories. See
|
||||
> [[concepts/agent-memory]] and [[systems/mnemosyne]].
|
||||
|
||||
> **Re-verified 2026-11-11 — still Mnemosyne only.** `memory.provider: mnemosyne`,
|
||||
> `memory_enabled: true`, `flush_min_turns: 6`, `memory_char_limit: 5000`,
|
||||
> `user_char_limit: 3000`, `nudge_interval: 10`, `user_profile_enabled: true`; Mnemosyne sub-block
|
||||
> `auto_sleep: true`, `default_scope: session`, `vector_type: int8`, `sleep_threshold: 50`,
|
||||
> `profile_isolation: true`, `shared_surface_path: data/shared/mnemosyne.db`,
|
||||
> `shared_surface_read: true`. No `openviking` key anywhere in `config.yaml`; no migration
|
||||
> performed. The OpenViking NOT-adopted verdict stands.
|
||||
|
||||
## Related
|
||||
- [[systems/mnemosyne]]
|
||||
- [[systems/memory-backend]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Model Routers
|
||||
type: comparison
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [comparison, routing]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -51,6 +51,16 @@ Compare LiteLLM, OmniRoute, and OpenRouter as model-routing layers for Hermes.
|
||||
> Any runbook that probes these endpoints with a bare `curl` will see 401 and may wrongly conclude
|
||||
> the provider is down — it is not.
|
||||
|
||||
> **Re-verified 2026-11-11 — the corrections recorded here all still hold.** The live `model:` block
|
||||
> is OmniRoute-primary (`custom:omni-101` / `smart-route` / `http://100.78.63.20:20128`), and
|
||||
> LiteLLM is the **tail** of the six-entry fallback chain (`or-inclusionai_ling-2_6-1t` via
|
||||
> `custom` → `http://litellm:4000/v1`). The five `custom_providers` and their base URLs are exactly
|
||||
> as listed: `litellm` `http://litellm:4000/v1`, `guacano` `http://ollama:11000/v1`,
|
||||
> `Omniroute:20128` `http://omniroute:20128/v1`, `command-code`
|
||||
> `https://api.commandcode.ai/provider/v1`, `omni-101` `http://100.78.63.20:20128`. OpenRouter is
|
||||
> still absent from `custom_providers`. **Not re-probed this pass:** the live `/v1/models` counts
|
||||
> (1869 / 6457) and their 2026-10-20 timestamps stand as recorded — treat them as unverified.
|
||||
|
||||
## Detail
|
||||
|
||||
### LiteLLM — fallback tail + a `custom_providers` entry
|
||||
@@ -69,14 +79,17 @@ Unified LLM router at `100.82.40.23:3001` (also `freellmapi.martinwa.org`). Veri
|
||||
- Exposes `/api/fallback/routing`, `/api/fallback/penalty-inspector`, `/api/fallback/token-usage`.
|
||||
This is the router that resolves Hermes's **default model** `freellmapi-auto` — distinct from OmniRoute's LKGP (below).
|
||||
|
||||
### OmniRoute — fallback / auxiliary
|
||||
Not used for Hermes's main chat path but handles specific auxiliary roles (`skills_hub`, `approval`, `mcp`) and serves as the fallback provider. Has a much larger advertised catalog (~3430 vs ~1512) but that catalog can diverge from what's actually resolvable — combo discovery vs. resolution can differ (documented known issue). See [[systems/omniroute]].
|
||||
### OmniRoute — ⚠ main chat path again since 2026-10-24 (was "fallback / auxiliary")
|
||||
**Re-verified 2026-10-26: this section is out of date.** Hermes's **default** is `custom:omni-101` /
|
||||
`smart-route` on the `omniroute-1` node (`http://100.78.63.20:20128`) — i.e. OmniRoute **is** the main chat
|
||||
path, as it was before the 2026-10-20 → 10-24 `nous` detour → [[systems/current-profile]].
|
||||
It also still handles auxiliary roles and appears in the `fallback_providers` chain.` Has a much larger advertised catalog (~3430 vs ~1512) but that catalog can diverge from what's actually resolvable — combo discovery vs. resolution can differ (documented known issue). See [[systems/omniroute]].
|
||||
|
||||
### OpenRouter — kept only as a fallback decision, not wired in
|
||||
**Verified:** absent from `custom_providers` in the live config. A decision page ([[decisions/keep-openrouter-as-fallback]]) records intent to retain it as a fallback option, but there's no live routing through it today — see [[systems/openrouter]] for the "previously used, largely replaced" history.
|
||||
|
||||
## Verdict
|
||||
Two routers are actually live (LiteLLM primary, OmniRoute fallback/auxiliary); OpenRouter is a retained *intent*, not a working path, as of 2026-07-22. If [[decisions/keep-openrouter-as-fallback]] is still desired, it needs an actual `custom_providers` entry — currently it's fallback-in-name-only.
|
||||
Two routers are live (**OmniRoute primary** for the default path and auxiliary slots since 2026-10-24 19:06; **LiteLLM** as the deeper fallback), OpenRouter is a retained *intent*, not a working path — re-verified 2026-10-26. If [[decisions/keep-openrouter-as-fallback]] is still desired, it needs an actual `custom_providers` entry — currently it's fallback-in-name-only.
|
||||
|
||||
## Related
|
||||
- [[systems/model-providers]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Steel Browser vs Camoufox
|
||||
type: comparison
|
||||
status: active
|
||||
created: 2026-07-29
|
||||
updated: 2026-10-13
|
||||
verified_on: 2026-10-13
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [comparison, browser, steel, camofox, decision]
|
||||
sources: [live-probe, session-history]
|
||||
@@ -13,20 +13,20 @@ sources: [live-probe, session-history]
|
||||
# Steel Browser vs Camoufox
|
||||
|
||||
## Purpose
|
||||
Compare the two browser-automation backends available to Hermes, including the Jul 29–30 experiment and the decision to revert.
|
||||
Compare the two browser-automation backends Hermes has used, including the Jul 29–30 experiment and the decision to revert. **Camofox is now the only backend available — Steel was fully removed on 2026-10-29.**
|
||||
|
||||
## At a glance
|
||||
| | Steel Browser (CT201) | Camofox (systemd --user on CT460) |
|
||||
|---|---|---|
|
||||
| **In use** | Disabled since 2026-07-30 | **Yes** — primary backend |
|
||||
| **In use** | **Removed 2026-10-29** (primary 2026-07-29 → 07-30) | **Yes** — the only backend |
|
||||
| **Engine** | Chromium 140 (Playwright) | Camoufox (Firefox-based) |
|
||||
| **Connection** | CDP (`ws://` via port 9223) | REST API + CDP WebSocket, both `localhost:9377` |
|
||||
| **API** | REST at `:3000` — create/destroy/query sessions | Camoufox REST + CDP WebSocket |
|
||||
| **Headless** | Native headless | Xvfb + VNC (headed, anti-fingerprint) |
|
||||
| **Container** | Docker on CT201 | `systemd --user` unit on the Hermes host (CT460) |
|
||||
| **Container** | Docker on CT201 — **removed 2026-10-29** | `systemd --user` unit on the Hermes host (CT460) |
|
||||
| **Stealth** | Standard Chromium | Anti-detection fingerprinting |
|
||||
| **Session viewer** | Built-in UI at `/steel` | VNC at `/vnc-camofox` |
|
||||
| **Plugin** | `browser-steel` plugin (disabled) | Native Hermes `browser.cloud_provider: camofox` |
|
||||
| **Session viewer** | Built-in UI at `/steel` — **route deleted** | VNC at `/vnc-camofox` |
|
||||
| **Plugin** | `browser-steel` — **deleted 2026-10-29** | Native Hermes `browser.cloud_provider: camofox` |
|
||||
|
||||
## The Steel Experiment (Jul 29–30)
|
||||
|
||||
@@ -39,7 +39,7 @@ Compare the two browser-automation backends available to Hermes, including the J
|
||||
1. **Jul 29:** Steel deployed on CT201 via Docker Compose, Chromium 140. Hermes configured with `browser.cloud_provider: steel`, `browser-steel` plugin enabled.
|
||||
2. **Jul 29 (later):** CDP connectivity issues surfaced — Hermes was reaching `0.0.0.0` from the plugin's internal CDP URL. Required `_normalize_cdp_url()` patch in the Steel plugin to remap to `100.96.244.39`.
|
||||
3. **Jul 30:** DeepSeek V4 Flash 0731 went live and rejected tool names that didn't match `^[a-zA-Z0-9_-]+$`. The Steel plugin's tools (`steel_scrape`, `steel_session_options`) were fine, but the route through OmniRoute → Steel → CDP was unreliable with empty-stream errors.
|
||||
4. **Jul 30 (evening):** Decision to revert — Camoufox re-enabled, browser-steel plugin disabled, `CAMOFOX_URL` and `CAMOFOX_API_KEY` uncommented in `.env`. Steel left running on CT201 but Hermes no longer routes through it.
|
||||
4. **Jul 30 (evening):** Decision to revert — Camoufox re-enabled, browser-steel plugin disabled, `CAMOFOX_URL` and `CAMOFOX_API_KEY` uncommented in `.env`. Steel left running on CT201 but Hermes no longer routes through it. *(Superseded 2026-10-29 — Steel was removed entirely.)*
|
||||
|
||||
### What broke
|
||||
- CDP URL normalization required a plugin patch to handle Docker's internal networking
|
||||
@@ -48,7 +48,8 @@ Compare the two browser-automation backends available to Hermes, including the J
|
||||
|
||||
### Lesson learned
|
||||
Steel is a capable backend but adds a network hop and another plugin layer. Camofox runs locally on the
|
||||
Hermes host (CT460) with fewer failure modes. The `browser-steel` plugin still exists (disabled) if needed in future.
|
||||
Hermes host (CT460) with fewer failure modes. Steel was **fully removed on 2026-10-29** — plugin deleted, and
|
||||
the CT201 and CT333 deployments torn down — so returning to Steel means redeploying it, not flipping a flag.
|
||||
|
||||
## Detail
|
||||
|
||||
@@ -60,14 +61,29 @@ Hermes host (CT460) with fewer failure modes. The `browser-steel` plugin still e
|
||||
- **Auth:** Already authenticated into services (Uber Eats, LinkedIn, etc.) — no cookie files needed
|
||||
- **Limitation:** Headed via Xvfb, consumes display resources
|
||||
|
||||
### Steel Browser (disabled but available)
|
||||
- **Deployed:** 2026-07-29 on CT201, Docker, custom `steel-browser-api:fixed` image
|
||||
- **Status:** Still running on CT201, Hermes not connected
|
||||
- **CDP endpoint:** `http://100.96.244.39:9223`
|
||||
- **REST API:** `http://100.96.244.39:3000`
|
||||
- **UI:** `https://runtipi.kangaroo-eel.ts.net/steel`
|
||||
- **Plugin:** `browser-steel` at `/home/hermes/.hermes/plugins/browser-steel/` — disabled, `plugin_steel` toolset flagged `requires_health=steel`
|
||||
### Steel Browser (removed 2026-10-29 — no longer available)
|
||||
- **Deployed:** 2026-07-29 on CT201, Docker, custom `steel-browser-api:fixed` image *(historical)*
|
||||
- **Status:** **FULLY REMOVED on 2026-10-29** — not merely offline. On CT201 the containers
|
||||
`steel-browser-api-1` / `steel-browser-ui-1`, the `steel-network` Docker network, the images
|
||||
(`steel-browser-api:fixed`, `ghcr.io/steel-dev/steel-browser-api:latest`,
|
||||
`ghcr.io/steel-dev/steel-browser-ui:latest`) and the project dir `/root/steel-browser` were all deleted
|
||||
(~6 GB reclaimed). The Tailscale Serve routes `/steel` and `/steel-api` were removed, so the former CDP
|
||||
`:9223`, REST `:3000` and UI `/steel` addresses are all retired — nothing listens on 3000, 5173 or 9223.
|
||||
- **CT333 copy:** a second, separate Steel install (`steel-browser`, `ghcr.io/steel-dev/steel-browser:latest`)
|
||||
was also removed with its image. It had been orphaned — no requests since 2026-10-21, and no reference from
|
||||
Zeroclaw, Nanobot or any Serve route.
|
||||
- **Plugin:** **deleted 2026-10-29** — `~/.hermes/plugins/browser-steel/` is gone and no `browser-steel` key
|
||||
remains in `plugins.disabled` / `plugins.entries`, so the `plugin_steel` toolset cannot be selected.
|
||||
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus
|
||||
`/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io.
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake).** Steel-removal claims re-probed live: CT201 has **no** steel
|
||||
> container or image, nothing on :3000/:5173/:9223, Serve routes are only `/` → 8085 and `/mgr` → 8084
|
||||
> on :443 **and** :8443; `/root/_archived-steel/` holds `docker-compose.yml`, `Dockerfile.api`,
|
||||
> `steel-cache-archive-20261029.tgz`. CT333 likewise has no steel container/image and nothing on :3000.
|
||||
> `~/.hermes/plugins/browser-steel/` absent. Camofox healthy on CT460: `:9377/health` → `ok`
|
||||
> (idle — `browserConnected:false` at probe time, no active session), unit active, config
|
||||
> `cloud_provider: camofox`. All removal claims **hold**.
|
||||
## Related
|
||||
- [[systems/browser-backend]]
|
||||
- [[concepts/browser-automation]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Agent Memory
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [concept, memory]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -37,6 +37,16 @@ How an agent retains facts/context beyond a single conversation turn or session
|
||||
> - `profile_isolation: true` — memories do not leak across Hermes profiles, which matters now that
|
||||
> the `ops` profile has been deleted (see [[runbooks/multiplexer-setup]]).
|
||||
|
||||
> **Full `memory:` block re-read 2026-11-11 — every value unchanged.** `memory_enabled: true`,
|
||||
> `provider: mnemosyne`, `memory_char_limit: 5000`, `user_profile_enabled: true`,
|
||||
> `user_char_limit: 3000`, `flush_min_turns: 6`, `nudge_interval: 10`; Mnemosyne sub-block
|
||||
> `auto_sleep: true`, `default_scope: session`, `vector_type: int8`, `sleep_threshold: 50`,
|
||||
> `profile_isolation: true`, `shared_surface_path: data/shared/mnemosyne.db`,
|
||||
> `shared_surface_read: true`, `sync_roles: [user]`, `ignore_patterns: []`,
|
||||
> `skip_contexts: [cron, flush, subagent, background, skill_loop]`,
|
||||
> `reflect: {disabled_for_cron: true, max_calls_per_session: 3}`. The cron-skip is still why
|
||||
> scheduled jobs like this stocktake leave no Mnemosyne trace.
|
||||
|
||||
## Model
|
||||
- Per-turn conversation content accumulates up to the char limit, then flushes to the Mnemosyne store every 6 turns (rather than every turn) — a batching tradeoff between write frequency and losing very recent context if the process dies mid-batch.
|
||||
- `scope: session` means recall defaults to the current session rather than a global cross-session pool, unless explicitly widened.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Browser Automation
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-13
|
||||
verified_on: 2026-10-13
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [concept, browser, camofox, steel]
|
||||
sources: [live-probe, session-history]
|
||||
@@ -31,8 +31,12 @@ Steel Browser (Chromium 140 on CT201) was briefly the primary backend. It offere
|
||||
- Empty-stream errors from the `opencode-go/deepseek-v4-flash` provider path
|
||||
- DeepSeek 0731's stricter tool name validation affected all providers
|
||||
|
||||
**Lesson:** A local browser backend (same host, no network hop) has fewer failure modes than a remote Docker container. The `browser-steel` plugin is still present but disabled. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
|
||||
**Lesson:** A local browser backend (same host, no network hop) has fewer failure modes than a remote Docker container. The `browser-steel` plugin **was deleted on 2026-10-29** and the Steel deployments on CT201 and CT333 were torn down, so there is no Steel fallback left to fall back to — Camofox is the only backend. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake).** Camofox service active on CT460 (`systemctl --user`),
|
||||
> `:9377/health` → `ok`; Xvfb `:99` at 1600×1200×24 with fluxbox, websockify on 127.0.0.1:6080.
|
||||
> Config `cloud_provider: camofox` / `camera_provider: camofox` confirmed. No Steel anywhere
|
||||
> (CT201/CT333 containers, images, routes, plugin all re-probed absent).
|
||||
## Related
|
||||
- [[systems/browser-backend]]
|
||||
- [[comparisons/steel-vs-camoufox]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Model Routing
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [routing, model, inference]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md]
|
||||
@@ -16,14 +16,43 @@ sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md]
|
||||
How Hermes selects providers, applies combos, and enforces context limits.
|
||||
|
||||
## Current Flow
|
||||
- Main path: `custom:litellm` → `http://litellm:4000/v1`
|
||||
- Default model: `freellmapi-auto`
|
||||
- OmniRoute used for auxiliary roles: vision, skills_hub, approval, mcp, title_generation, web_extract, compression
|
||||
- Main path: `custom:omni-101` / `smart-route` → `http://100.78.63.20:20128` (the OmniRoute node
|
||||
`omniroute-1`), `api_mode: chat_completions`
|
||||
- OmniRoute used for the **default chat path** (`custom:omni-101` / `smart-route`, since 2026-10-24 19:06) and for **nine auxiliary slots** (`approval`, `compression`, `curator`, `mcp`, `profile_describer`, `pulse_memory`, `skills_hub`, `title_generation`, `vision` — eight of them on `no-think/freeapi/auto`, plus `pulse_memory`; see the 2026-11-11 note below)
|
||||
|
||||
> **Live `model:` block re-read 2026-10-20 — the default has moved.**
|
||||
> `model.default` is now **`meituan/longcat-2.5-preview:free`** (provider `nous`), not
|
||||
> `freellmapi-auto`. The block also carries `base_url`, `persist_switch_by_default` and `api_mode`.
|
||||
> **Auxiliary slots re-read 2026-11-11 — the set has grown from eight to nine.** Eight slots
|
||||
> (`approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`,
|
||||
> `title_generation`, `vision`) are unchanged: `provider: custom:omni-101`, `model:
|
||||
> no-think/freeapi/auto`, each with a two-entry fallback chain
|
||||
> (`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, both on `custom:omni-101`).
|
||||
> **New: `pulse_memory`** — `provider: nous`, `model: anthropic/claude-haiku-4.5`, and it is the
|
||||
> slot behind the live 404s recorded above: nous is out of credits for that **paid** model, so
|
||||
> `pulse_memory` is failing right now (3 errors on 2026-11-10 18:02 and 19:19). Also live:
|
||||
> `auxiliary.free_only: true`.
|
||||
|
||||
> **Live `model:` block re-read 2026-10-26 — the default has moved TWICE.**
|
||||
> `model.default` is **`smart-route`** on provider **`custom:omni-101`**
|
||||
> (`http://100.78.63.20:20128`), `api_mode: chat_completions`. It moved to `nous` /
|
||||
> `meituan/longcat-2.5-preview:free` on 2026-10-20 22:01 and **back** on **2026-10-24 19:06:10**.
|
||||
> A six-provider top-level `fallback_providers` chain is live again, headed by the **dead `nous`**
|
||||
> credential (`Invalid refresh token` since 2026-10-22).
|
||||
> → [[systems/current-profile]] · [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
>
|
||||
> **Re-read 2026-11-11 — the model block is unchanged, but the `nous` failure signature has moved
|
||||
> on.** `model.default` is still `smart-route` on `custom:omni-101`
|
||||
> (`base_url: http://100.78.63.20:20128`, `api_mode: chat_completions`,
|
||||
> `persist_switch_by_default: false`) and the six-entry `fallback_providers` chain is unchanged in
|
||||
> membership and order (`nous` → `opencode-zen` → `custom:guacano` → `custom:litellm` →
|
||||
> `kilocode` → `custom`). The **refresh-token failure is gone from the logs**: `~/.hermes/.env`
|
||||
> has no `NOUS_API_KEY` and `~/.hermes/auth.json` is live, and no 401/refresh-token error appears
|
||||
> in `~/.hermes/logs/errors.log`. What *does* appear (3×, most recent **2026-11-10 19:19:33**) is
|
||||
> **404 `insufficient_credits_for_paid_model`** for `anthropic/claude-haiku-4.5` — a **paid**
|
||||
> model. So `nous` is no longer dead-at-auth; it is **out of credits for paid models**. Free nous
|
||||
> models (position 1 is `meituan/longcat-2.5-preview:free`) have not been observed failing. The
|
||||
> slot actually hit by that paid model is **`auxiliary.pulse_memory`** — see the auxiliary note
|
||||
> below, which is currently a broken slot.
|
||||
>
|
||||
|
||||
> **The five `custom_providers` and their live base URLs:**
|
||||
>
|
||||
> | Name | Base URL | Host |
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Persistent Shells
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [concept, terminal]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -31,6 +31,12 @@ A background shell session that survives across separate Hermes turns/requests,
|
||||
> The container-* keys are inert while `backend: local` — the shell is not containerised on this
|
||||
> box, so `container_memory`/`container_disk` describe a path that isn't taken.
|
||||
|
||||
> **`terminal:` block re-read 2026-11-11 — unchanged.** `backend: local`,
|
||||
> `persistent_shell: true`, `timeout: 180`, `home_mode: auto`, `container_cpu: 1`,
|
||||
> `container_memory: 5120`, `container_disk: 51200`, `container_persistent: true`,
|
||||
> `lifetime_seconds: 300`, `docker_mount_cwd_to_workspace: false`, `docker_run_as_host_user: true`,
|
||||
> `cwd: /home/hermes/workspace/`. The container-* keys are still inert while `backend: local`.
|
||||
|
||||
## Why it matters here
|
||||
Enables multi-step CLI workflows (e.g. a long `git` operation, a build, a watch process) to be resumed across turns without Hermes re-establishing context each time. The tradeoff is a shell can be left in a bad state (stuck process, wrong directory) that silently affects the *next* turn — see [[runbooks/troubleshoot-tool-loop]] for the related tool-loop guardrails that catch some of this.
|
||||
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Tool Calling
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [concept, tool-calling]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -49,6 +49,11 @@ This is the mechanism behind [[runbooks/troubleshoot-tool-loop]] — that runboo
|
||||
> (which watches *tool* calls) never fired. Two separate loop-detection mechanisms exist; do not
|
||||
> assume one covers the other.
|
||||
|
||||
> **`tool_loop_guardrails` re-verified 2026-11-11 — unchanged.** `warnings_enabled: true`,
|
||||
> `hard_stop_enabled: false`, `warn_after {exact_failure: 2, same_tool_failure: 3,
|
||||
> idempotent_no_progress: 2}`, `hard_stop_after {exact_failure: 5, same_tool_failure: 8,
|
||||
> idempotent_no_progress: 5}`. A genuinely stuck tool loop still only ever produces a warning.
|
||||
|
||||
## Related
|
||||
- [[concepts/persistent-shells]]
|
||||
- [[runbooks/troubleshoot-tool-loop]]
|
||||
|
||||
+622
-9
@@ -3,18 +3,631 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-25
|
||||
verified_on: 2026-10-25
|
||||
review_after: 2026-11-01
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
confidence: high
|
||||
tags: [system]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-25
|
||||
Updated: 2026-11-20
|
||||
|
||||
## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST)
|
||||
## Recent Changes (2026-11-17 → 2026-11-20, three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST)
|
||||
|
||||
**The estate's memory freeze ended by itself, its two alarm paths are still red, and the audits that would have reported either had silently died twice.**
|
||||
|
||||
- **Capture stall: over, unassisted.** `working_memory` **1130 → 1131**, `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST** — a **105 h 52 m** break that ended with no config change, restart or repair. ⚠ **And the "capture dead" reading was wrong:** `state.db` holds only **three** non-cron sessions since 11-14 00:00 — two `desktop` on **11-14 08:35 / 08:52** (both *before* the 09:33 boundary) and the **11-18 19:18** `desktop` whose turn *is* the captured row — so **every interactive session in the window was captured**. `mnemosyne_freshness_gate.py` reads the age of the newest non-`cron`/`subagent` message, so **silence and failure read identically**; that ambiguity is the real defect. The monitor is still failing on it (`failure_streak: 2`; slots 11-18 00:01 and 12:01, 11-19 12:01, 11-20 00:01 — 11-19 00:01 alone passed). → [[incidents/2026-11-14-mnemosyne-capture-stall]], [[systems/mnemosyne]]
|
||||
- **The nightly audit missed two nights.** `cbfe434a3b5e` failed on **11-18** and **11-19** on a chained-provider credit/auth fault; **both deliveries `suppressed`**, and its incident row is still `alerted` with **`alerted_at: None`** → [[incidents/2026-11-18-wiki-audit-credit-outage]]. **This run covers 11-17, 11-18 and 11-19.**
|
||||
- **The watchdog is in a threshold storm.** `96eb95ce12f2` failed **115 consecutive** 15-minute slots from **2026-11-18 20:01:06** to **2026-11-20 00:31:16** on **59 unresolved conflicts vs a threshold of 50** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]
|
||||
- **Cron ledger.** 11-17 **173 rows (171 `completed` / 2 `failed`)**; 11-18 **172 (153/19)**; 11-19 **176 (78/98)** — the worst day in the retained window; 11-20 so far **8 (3 / 4 / 1 `running`)**. Deliveries since 11-18 **24 — 23 `delivered`** (9 `for_failure: 1`) / **1 `failed`** (content, Telegram timeout `2026-11-19 08:34:13`). Fleet **29 (26 enabled / 3 disabled)**, a twelfth consecutive window; non-clean `last_status` **1 → 4**; unclosed `cron_incidents` **5 → 4**. The **03:00 stocktake ran `completed` on all three days** (11 m 35 s / 23 m 07 s / 24 m 38 s).
|
||||
- **Mnemosyne consolidation still zero-item:** newest `consolidation_log` row `2026-11-19T16:20:14+00:00`, `items_consolidated: 0`, **199 of the last 200 cycles zero-item** (was 120 on 11-17). Zero-item-streak threshold still **proposed, not applied**.
|
||||
- **Pages touched:** [[incidents/2026-11-18-wiki-audit-credit-outage]] (new), [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] (new), [[incidents/2026-11-14-mnemosyne-capture-stall]] (new re-read + corrected title), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[index]], `log.md` — frontmatter dates → **2026-11-20**.
|
||||
|
||||
**Previous window:**
|
||||
|
||||
## Recent Changes (2026-11-16, nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST)
|
||||
|
||||
**The memory pipeline is still dead at 62 h, and the 11-15 record needed a correction: a plugin
|
||||
load-timeout storm nobody had logged.**
|
||||
|
||||
- **Capture stall, third read — unchanged in every respect but the counters.** `working_memory` **1130**
|
||||
/ `episodic_memory` **1249**, newest write `2026-11-14 09:32:59` AWST → **62 h 54 m**. Unclosed
|
||||
`alerted` incidents **3 → 5** (143 rows total); monitor `failure_streak` **5**; consolidator zero-item
|
||||
streak **120** cycles. No cause, no fix, `status: active` →
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]]
|
||||
- ⚠ **New page — [[incidents/2026-11-15-plugin-load-timeout-storm]].** 11-15 carried **52** failed plugin
|
||||
loads and **519** discarded registrations, `conduit_push` alone losing **474** (its tools and hooks were
|
||||
absent for those processes), plus **11** `pre_tool_call` and **2** `pre_llm_call` hook timeouts and
|
||||
**10** refused tool calls. The 11-15 audit recorded **none** of it. **Zero on 11-16 and 11-17** — it
|
||||
cleared with no fix applied. The companion
|
||||
[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] gained a Recurrence 2 section; load average
|
||||
is now **explicitly not** a sufficient predictor (11-17 read 6.19 vs 12.08 on 11-14, no recurrence, and
|
||||
a *larger* `state.db`).
|
||||
- **Cron fleet, 11th window, flat at 29 jobs** (26 enabled / 3 disabled). 11-16 ledger **175 rows —
|
||||
173 `completed` / 2 `failed`**, both the health monitor; deliveries **11/11 `delivered`**. 11-17 so far:
|
||||
5 / 1 / 1. The 11-15 watchdog deps-guard 600 s timeout **did not recur**.
|
||||
- **Quiet where it matters, growing where it doesn't.** 11-16 ran **16 cron sessions, zero interactive** —
|
||||
nothing human-side to document. `state.db` **2,093,998,080 B (2.09 GB)**, WAL **40 MB**, `/` at **88 %**,
|
||||
load **6.19 / 6.64 / 7.48**, uptime 6 d 2 h (host up since 2026-11-10).
|
||||
- **Freshness:** 1 stale page by design — `reports/communities-wa-social-media-audit-2026` (23 d at
|
||||
11-17); it is a static deliverable with `review_after` 2026-11-26, so it was left unbumped.
|
||||
|
||||
|
||||
## Recent Changes (2026-11-15, nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST)
|
||||
|
||||
- ⚠ **Memory capture is still dead — 38 h 46 m — and the freeze is total, not degraded.**
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]] re-read: `working_memory` **1130** /
|
||||
`episodic_memory` **1249**, *the same two counts as every earlier read of the stall*; newest
|
||||
`working_memory` write still the `09:32:59 AWST` (01:32:59 UTC) `[MODEL_REFRESH_PROPOSAL]` maintenance
|
||||
row, newest *user-turn* capture `2026-11-14 09:04:19 AWST`. Root cause still not established.
|
||||
- **The monitor's `≈4.64 h` figure is not a duration — decoded against the gate's own source.**
|
||||
`mnemosyne_freshness_gate.py`'s `newest_user_message()` takes the newest `role='user'` message whose
|
||||
session source is not `cron`/`subagent` and prints *its* age. On this box those are the **Pulse**
|
||||
turns on `source: desktop` (~6 h cadence), and the `0 */12` slots sit ~4.6 h after one — which is why
|
||||
all three alerts read alike. **Do not read it as the stall length.**
|
||||
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows) — all `ea79a2107e36`, all
|
||||
`failure_type: auth`, all `alerted`, each `first_seen_at == last_seen_at`. The signature embeds the
|
||||
elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), so **one durable
|
||||
fault mints a fresh unclosed row every slot** and nothing escalates. **Fix proposed, not applied.**
|
||||
- **The consolidator runs and does nothing — by design.** Latest `consolidation_log` row **4696** at
|
||||
`2026-11-15T16:14:14.934306+00:00` (= 00:14:14 AWST), `items_consolidated: 0`, `summary_preview`
|
||||
`No-op consolidation cycle — timestamp updated for watchdog`; **~74 consecutive zero-item cycles**
|
||||
from `2026-11-14T02:04:42+00:00`. It bumps its own timestamp, so an age-only gate reads "fresh".
|
||||
- **Fleet: 2026-11-15 — 168 rows, 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`**
|
||||
(3 `for_failure`). The three: the Health Monitor at `00:00:53.928` and `12:01:24.719`, plus — **new
|
||||
failure mode** — the Watchdog `96eb95ce12f2` at `11:31:20.699`, `finished_at 11:49:10.534610` =
|
||||
**17 m 49.8 s**, `runtime deps guard … timed out after 600 seconds`; the `11:45` slot was **skipped**,
|
||||
`12:01:25` completed. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`.**
|
||||
- **Fleet 29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window unchanged**; only non-clean
|
||||
row is `ea79a2107e36` (`error`, `failure_streak` **3**).
|
||||
- **No interactive work on 11-15.** All **9** sessions started that day are `source: cron` — nothing
|
||||
human-side for the wiki to have missed.
|
||||
- Freshness (03:00 stocktake, `raw/stocktake/freshness-20261115.txt`): **0 stale / 133 fresh / 3
|
||||
no-date of 136** pages.
|
||||
|
||||
## Recent Changes (2026-11-14, nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST)
|
||||
|
||||
- **A third TTS stack landed on CT200** — new page [[systems/phonon-tts]]: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model `gradium/phonon-40m-en`), `phonon-demo` (`:8882`, WASM UI on `nginx:alpine`), `ptts-ws-server` (`:8883`, WebSocket, locally-built image). CT200 went **61 → 65 running / 68 → 72 total**; verified live from CT460 (health 200, models list, MP3 returned). ⚠ Everything after **09:03 AWST** on 11-14 is **not attributable to any retained transcript** — see the page's provenance gap before crediting it to a session.
|
||||
- ⚠ **Open incident: Mnemosyne capture stalled** since **2026-11-14 09:32:59 AWST** (~15 h at this read) — new page [[incidents/2026-11-14-mnemosyne-capture-stall]]. The health monitor escalated only at **2026-11-15 00:03:35** (exit 1, `ea79a2_601037516abd`, `alerted`, unclosed), after logging the same hole as ℹ️ at 12:00 AWST. Root cause not established; the known stringified-config killer is ruled out.
|
||||
- **Fleet clean apart from the monitor's own alert:** 2026-11-14 settled **167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**; `cron_incidents` **0 → 1 unclosed** (the monitor row above). Fleet **29 jobs (26 enabled / 3 disabled)**, **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, streak 1).
|
||||
- [[infrastructure/docker-services]] CT200 inventory re-verified and updated — **64 non-seeder running / 72 total** (was 61 / 68); the delta is exactly the three TTS containers, no removals.
|
||||
|
||||
## Recent Changes (2026-11-13, nightly audit; live probes 2026-11-13 23:59 – 2026-11-14 00:2x AWST)
|
||||
|
||||
- ✅ **2026-11-13 was the third consecutive zero-`failed` cron day: 174 rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, `cbfe434a3b5e`, started 23:59:41). Trend: 11-11 **172 / 172**, 11-12 **176 / 176**, 11-13 **174 / 173**. **Deliveries 11-13: 8 rows, 8 / 8 `delivered`, 0 `for_failure`** (11-12: 9 / 9). Fleet **29 jobs (26 enabled / 3 disabled)** — an **eighth consecutive window** unchanged — and **every job reads `last_status: ok` with `failure_streak` 0, so the fleet has no non-clean row at all** for the first time in weeks. `cron_incidents` **0 unclosed**. → [[systems/scheduled-tasks]]
|
||||
- ⚠ **The `pre_tool_call` hook timeout is back, and the raised budget did not hold it.** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40**, then **10** `skipped after previous timeout or while still running` lines through **00:06:38** — refusing this audit's own first tool calls (first-hand: `skill_view` and `terminal` both returned the plugin timeout) for ~1 minute before dispatch recovered unaided. `errors.log` by day: 11-12 **0 / 0**, 11-13 **0 / 0**, 11-14 **1 / 10**; it is the **first `pre_tool_call` timeout in the retained log since the 2026-11-08 cascade** (whose own lines have since rotated out — `errors.log` now starts **2026-11-09 13:18:39**). **No cron run failed** — 11-13 settled `failed`-free. **The variable is load, now ~2× the level recorded at diagnosis (`load average 12.08 / 13.02 / 12.33` on 2 vCPU vs 5.89).** The 30 → 120 s fix is **proven insufficient**; the incident is **reopened, `resolved` → `active`**. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
- ⚠ **`state.db` is now 2.04 GB** (**2,036,908,032 B**; **2,977 sessions / 207,577 messages**) — up from **1.89 GB / 2,901 / 197,682** on 2026-11-09 — and it is the load source behind the reopened hook fault above. **Root filesystem 88 % used.** `state.db` **checkpoint/retention remains undecided** and is now the estate's most urgent open item; re-raising the hook budget again would only mask the fault. Still `journal_mode=wal`, `wal_autocheckpoint=2048`. → [[systems/hermes-agent]], [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
- ✅ **The Uber Eats Monthly Spend Report was fixed end-to-end on 2026-11-13**, clearing the fleet's last non-clean row. Two stacked root causes: (1) the job's **runtime prompt was 88 k chars** because it carried two heavy browser skills, so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest (*"remaining 4 in the report"* plus forbidden run commentary); (2) the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart — which happened **2026-11-13 13:14**. Fix: a deterministic digest printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, a rewritten prompt with a hard output contract, and the skills cleared (**88 k → 1.8 k chars**); the lesson went into the `cron-job-maintenance` skill. Verified on the **18:51 → 18:58** slot: **154 unique / 153 completed** orders, Oct = **9 orders / $301.04** (unchanged), delivery row `delivered`, `hermes cron doctor` clean across all 26 enabled jobs. → [[reports/uber-eats/index]], [[systems/scheduled-tasks]]
|
||||
- **Freshness scan (2026-11-13 03:00 stocktake): 0 stale / 131 fresh / 3 no-date of 134 pages** — nothing needed retiring. → `raw/stocktake/freshness-20261113.txt`
|
||||
- **`executions.db` retention floor moved to 2026-11-08T00:01:20+08:00** (was 2026-11-06T05:09:30). The 11-07/11-08 cascade **runs** are still readable, but their **`errors.log` lines are not** — so the incident page's 58-timeout figure stands on the 2026-11-09 read and was **not re-verifiable** this window. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
|
||||
## Recent Changes (2026-11-12, nightly audit; live probes 2026-11-12 23:59 – 2026-11-13 00:4x AWST)
|
||||
|
||||
**Clean day. The estate is incident-clean, and the only open item is an upgrade that cannot be installed.**
|
||||
|
||||
- **The last open incident closed itself.** `cbfe43_ed01845310d8` (the 2026-11-10 audit's 429,
|
||||
[[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]) went `closed_at: 2026-11-12T00:25:36.133387+08:00`,
|
||||
`state: resolved`, when the **next slot of the same job ran clean** — `cbfe434a3b5e`
|
||||
2026-11-11T23:59:54.486572 → 2026-11-12T00:25:36.133387 = **25 m 42 s**, `completed`, delivery carrying
|
||||
**both** windows. **`cron_incidents`: 0 unclosed of 137.** No configuration produced this; the pool's
|
||||
reset at ≈ 06:36:57 on 11-11 did, and the 07:0x cohort confirmed it.
|
||||
- **State reads:** fleet **29 (26 enabled / 3 disabled)** — unchanged for a seventh window.
|
||||
Ledger — **2026-11-11: 172 rows, 172 `completed` / 0 `failed`**; **2026-11-12: 176 rows, 175 `completed`
|
||||
/ 0 `failed` / 1 `running`** (this audit); **2026-11-13 so far: 7 / 7 `completed`**. **Deliveries 11-11:
|
||||
7 / 7 `delivered`; 11-12: 9 / 9 `delivered`, 0 `for_failure`.** Executions retention floor
|
||||
`2026-11-07T00:00:42`, max `2026-11-13T00:30:55`, **1,001 rows** in the table.
|
||||
- **New open item — a refused upgrade, not a failure.** The attempt to take **`hermes-hq-push`** past its
|
||||
installed **0.9.42** was **blocked by the pre-install plugin security scan**:
|
||||
*"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec))"*, and *"--force does not
|
||||
override a dangerous verdict."* The critical is the plugin's **own** `skills/dashboard-tile/SKILL.md`
|
||||
one-liner (`echo '<json>' | python3 …` = obfuscation). Verified in source: **there is no per-verdict
|
||||
override anywhere** — `plugins.scan_on_install: false` disables scanning for *every* future install, and
|
||||
the plugin guard hardcodes `trust_level="community"` (the skills-side allowlist machinery is not read by
|
||||
it). Installed copy untouched, `enabled`, still delivering. **Fix = PR #2** on
|
||||
`mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc`, **OPEN / MERGEABLE**,
|
||||
`mergedAt: null`, +7/−2 in one file), pushed from a fork because `gh` is **read-only** upstream.
|
||||
→ [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]]
|
||||
- **Config untouched and consistent.** Live `config.yaml` is **byte-identical to the
|
||||
2026-11-12 18:14:52 snapshot** (565,433 B). The only drift since **11-09** is the **model catalogue
|
||||
block** (lines 1916-1941): 6 models removed, 9 added. No unrelated settings moved.
|
||||
- **The "stored hold-and-ask instruction" is not stored.** The alleged standing rule about plugin
|
||||
security-scan warnings was hunted exhaustively — a recursive sweep of `~/.hermes` (**184,716 files**,
|
||||
generated/venv/browser-profile trees excluded) found **zero matches in any editable text file**; the
|
||||
sentence exists only in `state.db` **conversation history** (58 rows). **Nothing was removed, nothing
|
||||
was reverted** — there was no persisted policy to undo. Procedure recorded as the skill
|
||||
`stored-instruction-search-and-removal`; installed skills **299 → 300**.
|
||||
- **The recurring pre-dispatch-filter signature was re-confirmed** on a *second* job and a *second* pin by
|
||||
the 2026-11-12 03:00 stocktake — still no failover for the free-tier cohort
|
||||
([[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]).
|
||||
|
||||
## Recent Changes (2026-11-11, nightly audit; live probes 2026-11-11 23:59 – 2026-11-12 00:1x AWST)
|
||||
|
||||
**Recovery night — plus one failure, and it was the auditor's own.**
|
||||
→ [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]
|
||||
|
||||
- **The 2026-11-10 audit died on its own 429.** `cbfe434a3b5e` ran 2026-11-10T23:59:54.486572 →
|
||||
**2026-11-11T00:12:29.646987 = 12 m 35 s**, `failed`, delivery `suppressed`:
|
||||
`RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch
|
||||
filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` — **6 candidates, `attempted 0`**, tenant
|
||||
`tn-5ud77abtl1` quota exceeded. Row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 but
|
||||
**`alerted_at: NULL` — it was the estate's only open incident row; it has since closed `resolved` at 2026-11-12T00:25:36 (0 of 137 unclosed)**. Pool reset
|
||||
≈ **06:36:57**; the 07:0x cohort ran clean.
|
||||
⇒ **The 2026-11-10 window was never audited.** This audit covers two windows.
|
||||
- **State reads:** fleet **29 (26 enabled / 3 disabled)**; **non-clean `last_status` 2** (was 1) —
|
||||
`066bb5191280` Uber Eats (`delivery_failed`, streak 0) and `cbfe434a3b5e` (`error`, streak 1).
|
||||
Ledger — **2026-11-09: 174 completed / 0 failed**; **2026-11-10: 157 rows, 155 completed / 2 failed**;
|
||||
**2026-11-11: 172 rows, 171 completed / 1 running / 0 failed** (15 distinct jobs) — first zero-failure
|
||||
day since 11-09. **Deliveries 11-11: 7 rows, 7 / 7 `delivered`, 0 `for_failure`** (last at 18:45:58).
|
||||
- **Stocktake recovered with output.** `1a8e90ea2271` 03:00:56.081504 → **04:11:38.724852 = 1 h 10 m
|
||||
42 s**, `completed`, **36 stale pages re-verified → 0 stale**, **commits `341dda9` + `c108eb1`**,
|
||||
freshness report `raw/stocktake/freshness-20261111.txt` (6,235 B). `341dda9` also rewrote the live
|
||||
fallback chain in [[runbooks/test-openrouter]] (primary back to `smart-route` / `custom:omni-101`),
|
||||
corrected the CT333 `clawtest` naming claim, and replaced the nous refresh-token reading with the
|
||||
404-credits signature. Closes [[incidents/2026-11-09-stocktake-completes-with-zero-edits]].
|
||||
⚠ The job is **still pinned to `free-stack`**, the combo recorded broken on 2026-08-14, and succeeded.
|
||||
- **Job Radar recovered with output.** Stage 1 `dca8482e4f76` 07:01:08.358004 → **07:37:34.076224 =
|
||||
36 m 26 s**, `completed`, writing **`radar-ref/handoff/2026-11-11.json` (22,682 B, mtime 07:59)** — the
|
||||
first handoff since **2026-11-06**. Stage 2 `5fc292ff0f3b`: recovery fire 07:35:56.632032 →
|
||||
08:03:47.816811 (`delivered`), scheduled 11:30:56 → 11:35:18, 14:30:55 → 14:34:56 (`delivered`). Row
|
||||
`5fc292_773822455856` auto-closed **`resolved` 2026-11-11T08:03:47.644226**, closing
|
||||
[[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]].
|
||||
- **Longest runs.** 11-11: stocktake **1 h 10 m 42 s**, Radar S1 **36 m 26 s**, Radar S2 **27 m 51 s**
|
||||
(recovery fire), Newsletter Digest `ecc1fc13655a` 12:01:02.571781 → 12:19:38.965387 (**18 m 36 s**),
|
||||
`85edf6772bca` 07:01:08.287627 → 07:11:42.858722 (**10 m 34 s**). **Nothing over 71 minutes.**
|
||||
11-10: Radar S2 scheduled slot 11:30:35.267155 → 22:18:49.814215 = **10 h 48 m 14 s**, `failed` /
|
||||
`interrupted`, delivery `delivered` with `for_failure: 1`.
|
||||
- **Provider state (unchanged config; worsening symptoms).** `config.yaml` mtime still **2026-11-09
|
||||
22:58:07** — nothing repaired. Top model `custom:omni-101` / `smart-route`
|
||||
(`http://100.78.63.20:20128`). `fallback_providers` (6): nous `meituan/longcat-2.5-preview:free` →
|
||||
**opencode-zen `deepseek-v4.1-flash` (EXPIRED key)** → `custom:guacano` → `custom:litellm`
|
||||
`kilo-reflection-beam-free` → kilocode `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`)
|
||||
`or-inclusionai_ling-2_6-1t`. **The expired opencode-zen key is no longer latent** — **7 × HTTP 401
|
||||
`api_key_expired`** on 11-11 (07:01:13 … 19:19:39), each immediately after nous returned **14 × 404
|
||||
`insufficient_credits_for_paid_model`** (07:01:11 … last 19:19:38, all from session
|
||||
`20261031_190813_95fbde`, source `desktop`, opened 2026-10-31, **still open**, pinned to that paid
|
||||
model). **Two consecutive dead hops on one chain.** → [[systems/opencode-go]]
|
||||
- **Auxiliary slots:** nine slots plus `free_only: true`. Eight chain
|
||||
`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` on `custom:omni-101`.
|
||||
**`pulse_memory` remains the sole exception with no `fallback_chain`**
|
||||
(`{"provider": "nous", "model": "anthropic/claude-haiku-4.5"}`) — 19 aux log lines on 11-11 across
|
||||
06:0x / 08:0x / 13:0x / 18:0x, including **11 ×** *"main provider `custom:omni-101` is unavailable and
|
||||
no fallback_chain / fallback_providers is configured — refusing to guess another logged-in provider"*.
|
||||
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
- **Error log:** `~/.hermes/logs/errors.log` = **6,179 lines** through **2026-11-12 00:10:11**.
|
||||
- **Still open, untouched this window:** Uber Eats Monthly Spend Report `delivery_failed` streak 0 (no
|
||||
execution row on 11-11 — monthly cadence); disabled trio unchanged (Important-mail monitor, Rhino
|
||||
AgentMail Inbox Watcher, AgentMail Rhino Inbox Poll); the three inherited items below.
|
||||
|
||||
## Recent Changes (2026-11-10, nightly audit; live probes 2026-11-09 23:5x – 2026-11-10 01:0x AWST)
|
||||
|
||||
- ✅ **2026-11-09 was the first `failed`-free cron day since 2026-11-05: 174 rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit). Trend: 11-06 1 failed / 172, 11-07 3 / 163, 11-08 1 / 170. Deliveries 8/8 `delivered`, 0 `for_failure`.
|
||||
- ⚠️ **The stocktake ran 9 h 10 m, closed `completed`, and landed nothing — third day running.** `1a8e90ea2271` (**03:00:43 → 12:10:47**) delivered `suppressed`; its own output file says `RUN INCOMPLETE`, tool-iteration budget exhausted during verification, **zero edits and no commit**. `failure_streak` 3 → 0 and `last_status ok` are the harness's verdict on the *process*, not the work. The job is still pinned to `custom:omni-101` / **`free-stack`** — a combo recorded broken on **2026-08-14**. → [[incidents/2026-11-09-stocktake-completes-with-zero-edits]], [[systems/scheduled-tasks]]
|
||||
- ⚠️ **The Job Radar landed nothing on 11-09.** Stage 1 (`dca8482e4f76`) ran **07:00:48 → 16:10:12 = 9 h 09 m 24 s** and never wrote `radar-ref/handoff/2026-11-09.json` (newest handoff on disk is **2026-11-06** — nothing has landed since). Both Stage-2 slots (`5fc292ff0f3b`) fired **before** Stage 1 finished (11:30:38 and 14:30:34) and declined silently. The run's own last words: *"I stopped retrying because I kept running read_file 5 times without making progress."* Same silent family as 11-02/11-03, new trigger (read loop, not iteration cap), and again **no incident row was raised**. → [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]
|
||||
- ✅ **The 11-08 `pre_tool_call` hook cascade did not recur.** `errors.log` (1,035 retained lines) contains **no `pre_tool_call` timeout entry at all**; its only two hook lines are benign skips (`on_session_end` 2026-11-09 22:58:35, `pre_llm_call` 2026-11-10 00:02:54). No 11-09 run outside the two documented long-runners exceeded **74 min**. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
- **Unclosed incident rows 3 → 0** (`cron_incidents`: 134 rows, 0 open). The stocktake cluster closed **automatically at 2026-11-09T12:10:47.125205** — the instant the overrunning run finished; none of the three ever alerted. → [[systems/scheduled-tasks]]
|
||||
- **Tailscale: 58 non-funnel nodes (57 peers + `hermes`), 42 online.** `octop` is **online** again (dark since `LastSeen 2026-11-08T13:13:04Z`); 8 offline ages re-drifted — cloudreve 106d → **108d**, elementsynapse 109d → **111d**, freshrss-new 82d → **84d**, conduit 62d → **65d**, octoagent 6d → **8d**, omarchybook 17d → **19d**, openmausbot 17d → **19d**, ubuntu-server-xfce 78d → **81d**; the 6 ephemeral `codex-environment-*` peers' last-seen stamps were corrected. → [[infrastructure/tailscale]]
|
||||
- **OmniRoute `omniroute-1` (CT101, `100.78.63.20`): `/v1/combos` → HTTP 200, 55 combos**, `free-stack` present — one ahead of the page's last reading (54). `/v1/models` on **both** nodes and CT600's `/v1/combos` return **HTTP 401** (alive, auth required), so the **6,480-model catalog figure dated 2026-10-26 could not be re-verified this window and was not changed**. → [[systems/omniroute]]
|
||||
- **Not verified this window (gap recorded, no page changed):** CT450/CT460 Hermes version strings — `pct exec 450 -- sh -lc 'hermes --version'` resolves the **known-stale root-level `/usr/local/bin/hermes` (v0.19.0)**, not the user-level install, and CT460's `hermes` is not on the PATH in that shell, so **[[infrastructure/hosts]]'s CT450/CT460 version lines stand as written**; and CT450/CT460 docker `Up` days (`lxc-attach … Failed to exec "docker"`).
|
||||
|
||||
## Recent Changes (2026-11-09, nightly audit; live probes 2026-11-08 23:5x – 2026-11-09 00:2x AWST)
|
||||
|
||||
- ⚠ **The `conduit_push` `pre_tool_call` hook was the day's fault, and it is fixed — `plugins.hook_callback_timeout` 30 → 120 s.**
|
||||
The plugin is the **only** `pre_tool_call` registrant on the host, and its callback overran Hermes's 30 s
|
||||
hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: 12
|
||||
lines on 11-07, 46 on 11-08; plus **89** *"skipped after previous timeout or while still running"*
|
||||
suppression lines). Every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST**;
|
||||
the last timeout line is still 08:21:03, re-verified at audit time. **The plugin's caching was measured,
|
||||
not assumed, and was innocent** — 3.1 s cold / 0.00 s warm (`session_titles.py:21 CACHE_SECONDS = 30.0`);
|
||||
the trigger is **load** on the 2-core CT460 (load average **5.89**, `state.db` with a **40 MB WAL**).
|
||||
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]], [[systems/conduit]]
|
||||
- ⚠ **`state.db` is now 1.89 GB** (1,890,656,256 B; **2,901 sessions / 197,682 messages**, live
|
||||
**40,157,672 B** WAL) — up from 1.7 GB, and it is the load source behind the spike above. The 2026-08-11
|
||||
corruption remnant **`state.db.corrupted.20260811_151052` (624.1 MB)** was **deleted** in this window;
|
||||
`state.db.recovered.recovery.json` (14,566 B) remains, and `mnemosyne.db.corrupted.20260809_120404`
|
||||
(23.6 MB) was **not** cleaned. → [[systems/hermes-agent]], [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]
|
||||
- ✅ **The Mnemosyne Health Monitor's no-agent conversion is verified end-to-end.** The 18:00 slot (agent
|
||||
mode) ran **12:00:51 → 18:00:42 = 5 h 59 m 51 s**; the converted job's first slot ran
|
||||
**2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, **silent**, writing a **159 B** output file against
|
||||
**105,982 B** for the last agent-mode one. Live coverage **working 1030/1030 = 100.0 %, episodic
|
||||
1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet. → [[systems/mnemosyne]], [[systems/scheduled-tasks]]
|
||||
- ⚠ **The stocktake failed a third consecutive day, on a new signature.** `1a8e90ea2271` ran
|
||||
**2026-11-08 03:01:11 → 04:36:08** and died on `RuntimeError: ⚠️ No reply: the turn stopped while a tool
|
||||
result was still pending and the model produced no follow-up text.` — the previous two red days read
|
||||
*"lost its durable fire claim ownership"*. **`failure_streak` 3.** *Working hypothesis, not proven:* the
|
||||
hook above holds a tool result open in exactly the way this error describes. **Unclosed incident rows
|
||||
2 → 3**, and all three are silent (`detected`, `alerted_at: null`) — but **the alert path works**: the
|
||||
*Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` at **08:06:54**.
|
||||
→ [[systems/scheduled-tasks]], [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
|
||||
- **2026-11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit). Deliveries
|
||||
**8 / 8 `delivered`**. Fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count, though
|
||||
`updated_at` moved to `2026-11-09T00:06:50+08:00` with no definition change. Nothing added, retired,
|
||||
disabled or repinned. → [[systems/scheduled-tasks]]
|
||||
|
||||
## Recent Changes (2026-11-08, nightly audit; live probes 2026-11-07 23:5x – 2026-11-08 00:5x AWST)
|
||||
|
||||
- **Hermes on CT460 is now `0.23.0`** (was `0.22.5`): `hermes update`, 2026-11-07 05:05:13 → 05:43:10,
|
||||
`bf0203a794a07aac3ff764972ab9087baa4e4f4e` → `2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`. ⚠ **The `snapshot`
|
||||
stage was skipped again** (`updates.pre_update_backup` mode `off`) → **there is still no rollback point**.
|
||||
Gateway restarted 05:37:34, dashboard 05:43:04, and the gateway once more at 11:11:44 → 11:11:55
|
||||
(live `pid 3109529`). → [[systems/hermes-agent]], [[agents/hermes-production]]
|
||||
- ⚠ **The restart drain cost two cron runs their fire claims.** A restart requested at ~04:29:52 sat in
|
||||
*after-turn* mode for the full 1800 s, timed out, and drained at 05:00:43/47 — killing *Mnemosyne Health
|
||||
Monitor* (`ea79a2107e36`, 00:00:42.918 → 05:02:22.870) and *wiki-stocktake-maintenance*
|
||||
(`1a8e90ea2271`, 03:00:31.515 → 05:02:22.174), both closed `failed` on
|
||||
`lost its durable fire claim ownership`. systemd restarted the unit at **05:01:00**
|
||||
(`restart counter is at 3`). → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
|
||||
- ⚠ **Two incident rows are open and neither alerted.** `1a8e90_628d914f1f97` (first seen 04:29:54.815) and
|
||||
`1a8e90_2f370ef764f5` (05:02:21.573) both sit `detected` for *wiki-stocktake-maintenance* with
|
||||
**`alerted_at: null`** — the *red row, no alert* shape first recorded for Job Radar on 11-06.
|
||||
- ⚠ **The 03:00 stocktake slot has now failed two days running** (`failure_streak: 2`, `last_status: error`,
|
||||
2026-11-06 + 2026-11-07). The 11-07 in-day entry committed the stocktake's own edits so the tree was not
|
||||
left dirty.
|
||||
- **`camofox` is now an enabled plugin (`v1.1.0`)**, not a core feature — installed by the 0.23.0 update.
|
||||
No `browser.*` config key changed. → [[systems/browser-backend]]
|
||||
- **`conduit_push` self-updated `0.122.2 → 0.134.4`** (`git` source, tree HEAD `ef3d2f2`) and recorded its own
|
||||
last step **`restarting_gateway` = failed** — the gateway it wanted to restart is systemd-managed. →
|
||||
[[systems/conduit]]
|
||||
- **`hermes-hq-push` reinstalled `0.9.27 → 0.9.42`** (upstream `d26850e7`, 60/60 files) at 06:42; its web
|
||||
routes only load at service start, so the 11:11:44 gateway restart is what put them on the wire. →
|
||||
[[systems/hermes-hq-push]]
|
||||
- ⚠ **The OpenCode Go credential is expired** — HTTP 401, `code: api_key_expired`, on
|
||||
`…/zen/go/v1/models` (probed twice in-window). Configuration alive, credential dead; no 2026-11-07
|
||||
execution failed on an auth error, so nothing was consuming it yet. → [[systems/opencode-go]]
|
||||
- ⚠ **Skill inventory moved: 302 → 299 skills (68 → 67 categories).** Four skills went into
|
||||
`~/.hermes/skills/.archive/` in one pass at **2026-11-07 07:43** — `cron-model-repair`,
|
||||
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup` — and the parked archive grew
|
||||
**46 → 50**. No active `SKILL.md` references any of them, so if this was a curator consolidation the
|
||||
*"absorbed into X"* pointer is missing. `microsoft-office` appeared in the same window (Productivity
|
||||
25 → 26). → [[systems/skills-index]], [[systems/scheduled-tasks]]
|
||||
- **Cron day (2026-11-07): 163 rows — 159 `completed` / 3 `failed` / 1 `running`**; deliveries 7/7
|
||||
`delivered`, 0 `for_failure`. Fleet unchanged at **29 jobs / 26 enabled**. →
|
||||
[[systems/scheduled-tasks]]
|
||||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] **remains OPEN** — the re-check
|
||||
found cron job prompts clean of secret-shaped strings, but the redaction setting could not be confirmed.
|
||||
|
||||
## Recent Changes (2026-11-07, stocktake; live probes 2026-11-07 03:1x-03:2x AWST)
|
||||
|
||||
- **The tailnet lost a peer and the m93p guest list lost a guest — both corrections came from live probes, not the age scan.**
|
||||
[[infrastructure/tailscale]] was rebuilt from `tailscale status --json`: **`octop` (`100.92.163.122`) is back
|
||||
online** (peer record `Online: true`, `tailscale ping` → `pong from octop via DERP(syd) in 98ms`; it remains
|
||||
**no LXC or VM on either hypervisor** — an orphaned tailnet peer), and the **9 remaining offline ages** were
|
||||
refreshed (+1d each). All **51 live IPv4 peers** match the table's IPs exactly, with no missing peers and no
|
||||
orphan rows (**52 rows** = 51 peers + `hermes` itself). The CT460 Serve route table still matches
|
||||
`tailscale serve status` **exactly (12 paths on :443)**; both previously-flagged dead targets were
|
||||
**re-confirmed dead** — still no listener on `8769` (`/agent-estate`) or `9123` (`/synergy-prep`).
|
||||
- **⚠ `m93p` actually runs 20 LXCs, not 19 — CT107 (`mem0`) was missing from the Proxmox page.**
|
||||
Live `pct list` on `m93p` returns **20 LXCs — 18 running, 2 stopped** (CT998 `nightbridge`, CT1264
|
||||
`openmausbot`). The omitted guest is **CT107 `mem0`** (`mem0-mcp.service` active on `:8890`,
|
||||
`pct exec 107 -- tailscale ip -4` → `100.73.94.53`). Added to [[infrastructure/proxmox]]; the
|
||||
**2026-10-31 reading of 19 was already short for the same reason**. `pve` still returns exactly
|
||||
**17 LXCs + 4 VMs**, and both hosts still run the identical `pve-manager 9.2.10` build.
|
||||
- **⚠ `m93p`'s uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*.
|
||||
Live `uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18** at probe time, load
|
||||
**0.76 / 1.08 / 1.24** — normal). The 2026-10-30 reboot was real but belonged to **`pve`**
|
||||
(boot **2026-10-30 16:24:00**), not `m93p`.
|
||||
- **Storage percentages refreshed on both hosts** against live `pvesm status` — `pve`'s `ssd0` moved
|
||||
**76.43 % → 78.29 %** (~152 GiB free), `fred-qnap` **82.77 % → 84.21 %**, `local` **79.66 % → 81.62 %**;
|
||||
on `m93p` the iMac shares and `local` moved **60.25 % → 76.21 %** and `omb-internal`
|
||||
**57.92 % → 64.66 %**.
|
||||
- **Docker inventories diffed set-wise against live `docker ps` on all three hosts.** CT200 matches
|
||||
**61 running / 68 total** exactly; CT201 matches **20 / 26** exactly; CT333 matches **8 / 9** — but its
|
||||
**enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name is
|
||||
**`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected in [[infrastructure/docker-services]].
|
||||
- **Two small cross-page corrections.** [[infrastructure/network-map]]'s `/gitea` route was annotated
|
||||
*"Gitea (ct207 tailnet IP)"* — it is **CT221 on `m93p`** (`100.82.100.128`; ct207's own IP is
|
||||
`100.82.40.23`); the target answers **HTTP 200**. [[agents/hermes-chaos]]'s version line read
|
||||
*"`v0.22.3` … local `5b147ab036d6` — the same build as CT460"*, which contradicted itself: live
|
||||
`hermes --version` on CT450 is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to CT460's.
|
||||
[[agents/hermes-production]]'s claim that *2 cron jobs still pin `custom:omni`* is **no longer true** —
|
||||
live `jobs.json` shows **0** jobs on the bare key and **21** on `custom:omni-101`.
|
||||
- **Cron fleet re-verified unchanged:** **29 jobs (26 enabled / 3 disabled)**, matching the nightly audit
|
||||
hours earlier. CT450's fleet is **25 jobs (13 enabled / 12 disabled)** with **9 shared names** — confirmed
|
||||
by a live set-diff of both `jobs.json` files. → [[systems/scheduled-tasks]]
|
||||
|
||||
## Recent Changes (2026-11-07, nightly audit; live probes 2026-11-07 00:3x-00:5x AWST)
|
||||
|
||||
- **A red row with no alert: the Job Radar's 11:30 slot was discarded as stale while a concurrent fire
|
||||
held the claim.** 2026-11-06 settled **172 rows across 16 jobs - 170 `completed` / 1 `failed` / 1
|
||||
`running`** (this audit). The failure is `5fc292ff0f3b`'s **11:30:32 slot**, interrupted at **13:26:38**
|
||||
and closed **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - because a **`direct`
|
||||
fire of the same job (08:27:09 -> 13:26:22, 4 h 59 m 13 s)** was still holding the claim and delivered
|
||||
first. **Every previous occurrence of this string came from a restart orphaning a run; this one had no
|
||||
restart.** The failed row raised **no `cron_incidents` row** (newest ledger activity is still the
|
||||
2026-11-05T12:36:17 closures), so **nothing alerted**. The day still landed complete via the 14:30 slot
|
||||
(12 m 27 s, `MISSING_COVER_LETTER_COUNT 0`) ->
|
||||
[[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]].
|
||||
- **The stocktake's overrun is fully unwound.** `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)**
|
||||
against 7 h 40 m / 5 h 41 m / 3 h 02 m on 11-03/11-04/11-05, and committed normally.
|
||||
- **Stage 1 ran clean for a second day** (`dca8482e4f76`, 07:00:44 -> 08:29:21, 1 h 28 m 37 s, `completed`,
|
||||
`suppressed`), and the **11-04 Open entry about its two lost candidate sets is cleared in substance** -
|
||||
its 11-03 ledger row closed `resolved` at 2026-11-05T12:36:17 and both runs since have been clean.
|
||||
- **Ledger clean, deliveries clean:** **unclosed incident rows 0 -> 0**; deliveries **10 rows / 10
|
||||
`delivered` / 0 `for_failure`**; fleet **29 jobs (26 enabled / 3 disabled)** - fifth consecutive audit at
|
||||
the same count, non-clean `last_status` still **1** (`066bb5191280`, `delivery_failed`, streak 0) ->
|
||||
[[systems/scheduled-tasks]].
|
||||
- **The previous audit's open question closed itself:** the Mnemosyne Health Monitor's 11-05 12:00:45 slot
|
||||
(still in flight 12 h+ with no delivery at the last audit) **completed at 2026-11-06T00:14:39 after
|
||||
12 h 13 m 54 s and delivered**; its 11-06 12:00 slot then took **7 h 47 m 56 s** and delivered too. The
|
||||
**one still-open page is unchanged:** [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
|
||||
|
||||
## Recent Changes (2026-11-06, nightly audit; live probes 2026-11-06 00:1x-00:4x AWST)
|
||||
|
||||
- **A clean ledger day and an empty incident ledger.** 2026-11-05 settled **175 rows - 173 `completed` / 2
|
||||
`running` / 0 `failed`** across 18 jobs: the **first zero-`failed` day since 2026-11-02**. **Unclosed
|
||||
incident rows 3 -> 0** - the *Tech & AI Newsletter Digest* row closed at **07:07:42** and both *Job Radar -
|
||||
Discovery* rows at **12:36:17**, when that job finally completed after **5 h 35 m 35 s**. Fleet non-clean
|
||||
`last_status` **3 -> 1** (only `066bb5191280`, `delivery_failed`). Deliveries **10 rows / 10 `delivered`**
|
||||
-> [[systems/scheduled-tasks]].
|
||||
- **The stocktake's three-day overrun streak broke, and it committed.** `1a8e90ea2271` ran **03:00:32 ->
|
||||
06:02:30 (3 h 01 m 58 s)** - **7 h 40 m -> 5 h 41 m -> 3 h 02 m** over three days - and **committed
|
||||
`2f3fc99` at 05:55**, against its 11-04 state of "reported incomplete, committed nothing".
|
||||
- **Long runs, not failures:** the **Mnemosyne Health Monitor**'s 00:00 slot took **7 h 01 m 28 s** (normal
|
||||
17-31 m) and its **12:00:45 slot was still running 12 h later with no delivery**; **Stage 2**'s 14:30 slot
|
||||
took **8 h 56 m 17 s** and delivered at 23:27:09.
|
||||
- **New open incident - live secrets stored verbatim.** Three `.env` values (`TELEGRAM_BOT_TOKEN`,
|
||||
`MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit in plaintext in Mnemosyne rows - including
|
||||
the FTS mirrors) - 4 + 6 + 7 = **17 rows** - reproduced in full by each of the **7 uncompressed rotation
|
||||
snapshots**; all four values also sit verbatim in the **1.7 GB session store `state.db`**, the likely capture
|
||||
origin. `CAMOFOX_API_KEY` sits verbatim in the prompts of two **enabled** Facebook Birthday jobs
|
||||
(`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Found, not remediated; no value is recorded
|
||||
anywhere -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
|
||||
- **Fleet unchanged:** **29 jobs (26 enabled / 3 disabled)**, a fourth audit at the same count; no job added,
|
||||
retired, disabled or repinned.
|
||||
## Recent Changes (2026-11-05, stocktake; live probes 2026-11-05 05:1x-05:3x AWST)
|
||||
|
||||
- **The `systems/skills-index.md` flip-flop is FIXED at the root - it had two writers, not one bad writer.**
|
||||
The repo `wiki-maintain.sh` (302 skills / 68 categories) and `~/.hermes/scripts/wiki-maintain.sh`
|
||||
(mtime **2026-08-17**, 349 skills / 26 categories - it used `Path.rglob()` and so counted the parked
|
||||
`.archive/` tree) are **different scripts with different generators**, and `wiki-maintenance/SKILL.md`
|
||||
line 569 pointed the **23:59 nightly audit** at the stale one while the **03:00 stocktake** ran the repo
|
||||
one. Each job overwrote the other's page; **neither raised an error.** The stale copy is now a
|
||||
**delegating wrapper** with no generator logic, and the repo script gained `--no-build` so the wrapper
|
||||
honours the old interface without a second generator. Verified by running it -> 302 / 68.
|
||||
-> [[incidents/2026-11-05-skills-index-generator-divergence]]
|
||||
- **Docker inventory corrected against live probes on all five hosts:** CT200 **61 running / 68 total**; CT333's
|
||||
first container renamed again (`nostalgic_leavitt` -> **`flamboyant_mahavira`**); CT201 20/26, CT460 1/1,
|
||||
CT1001 1/9.
|
||||
⚠ **Corrected 2026-11-06 (stocktake): the "the seeder does not exist on any host" claim in this entry was
|
||||
WRONG - it was a point-in-time sampling artefact.** The `worldmonitor-seeder-*` containers are **transient
|
||||
and systemd-timer driven** on CT200 (`worldmonitor-seed.timer` every 4h -> `worldmonitor-seeder-scheduled`;
|
||||
`worldmonitor-seed-fast.timer` every 15min -> `worldmonitor-seeder-fast`), so a `docker ps` taken between
|
||||
runs sees **none** of them. Live at 2026-11-06 03:1x **both were present**; live at 2026-11-05 05:1x
|
||||
**neither** was. **"Absent" must never be read as "removed."** CT333's container has also been renamed
|
||||
**again**: `flamboyant_mahavira` -> **`nostalgic_keldysh`** (created 2026-11-05T22:15:57Z).
|
||||
-> [[infrastructure/docker-services]]
|
||||
- **Tailscale node table re-verified programmatically:** 52 table rows vs 52 live IPv4 peers - **no live
|
||||
peer missing, no orphan row, every online/offline state matches.** Two offline ages corrected (`conduit`
|
||||
60d, `ubuntu-server-xfce` 76d). Serve table still matches `tailscale serve status` exactly (12 paths on
|
||||
:443 + the :8443 funnel + the :9443 route).
|
||||
⚠ **Corrected 2026-11-06 (stocktake): the "every online/offline state matches" claim in this entry no longer
|
||||
holds - `anthonys-mac-mini` has come back online** (peer record `Online: true`, `tailscale ping` -> pong) and
|
||||
**8 offline ages have drifted by +1d**. Two Serve routes (`/agent-estate` -> 8769, `/synergy-prep` -> 9123)
|
||||
are **dead** - no listener - while still present in `tailscale serve status`.
|
||||
-> [[infrastructure/tailscale]]
|
||||
- **Freshness now 0 stale** (was 8 on 11-04) - all 8 re-verified and refreshed. Skills count corrected to
|
||||
**302 / 68 categories** on [[infrastructure/hosts]] and [[index]].
|
||||
|
||||
## Recent Changes (2026-11-05, nightly audit; live probes 2026-11-05 00:0x-00:5x AWST)
|
||||
|
||||
- **A dead route took out two 07:00 jobs at once - and the safety-net slot saved the day.** `dca8482e4f76`
|
||||
(**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**)
|
||||
both died on `RuntimeError: Upstream stream failed before completion.` - **36 seconds apart**, at **09:41:44**
|
||||
and **09:42:20**, each after **2 h 41 m** on `custom:omni-101` / `smart-route`. Neither produced output; both
|
||||
raised and alerted a `cron_incidents` row. The 11:30 Stage 2 slot then **reconstructed the missing handoff
|
||||
from Stage 1's candidate files**, re-verified every candidate live in Camofox, and wrote **5 roles (board
|
||||
479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at
|
||||
16:08:59 - see [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]].
|
||||
- **Fleet: 29 jobs (26 enabled / 3 disabled).** In-window ledger: **170 rows - 167 `completed` / 2 `failed` /
|
||||
1 `running`**. *(fleet and last row corrected 2026-11-05 01:3x - the first draft read 158 jobs / 144 / 14 and a
|
||||
closing `item_suppressed` row; live `jobs.json` + `hermes cron list` give 29 / 26 / 3, and the 170th 11-04 row is
|
||||
this audit, `running`. 11-03 has since settled to 171 `completed` / 1 `failed`.)* **Non-clean `last_status` 2 -> 3**; **unclosed incident rows 1 -> 3** (all
|
||||
`state = alerted`, incl. the 11-03 `timeout` row `dca848_dfd4bea84f2a`). Deliveries on 11-04: **8 rows, 8
|
||||
`delivered`** - the failure notices were delivered, not silent.
|
||||
- **The cover-letter route is only alive by override.** `regenerate_cover_letter.py` line 82 reads `CL_MODEL`,
|
||||
defaulting to **`deepseek-paid-smart`**, which returned **HTTP 402** during the 11-04 run; that process was
|
||||
pinned to **`gemini/gemini-3.5-flash`**. **No `CL_MODEL` exists in `~/.hermes/.env`**, so the override did not
|
||||
persist. Endpoint check 2026-11-05: `omniroute:20128` -> **401** and `100.78.63.20:20128` -> **401** (live,
|
||||
key-gated - that is auth, not health). **Keyed re-probe 2026-11-05 01:33:12 AWST:** a completion request with
|
||||
`deepseek-paid-smart` returns **503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`) on `omniroute:20128`,
|
||||
but **200** on `100.78.63.20:20128` (`omniroute-1`), which resolved the alias to **`deepseek/deepseek-v4-flash`**.
|
||||
So the model is **undispatchable on the CT600 node only** - an override or the `omniroute-1` route is required.
|
||||
- **CT299 froze a second time.** A manual `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and
|
||||
wedged; the guest was frozen until **~00:30 on 11-05** (~16 h). Live probe **2026-11-05 00:54:21 AWST**:
|
||||
`LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump processes;
|
||||
SFTPGo reachable from CT460 at `100.88.223.70:8080`. Abandoned temp dir
|
||||
`/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299` still on disk. ->
|
||||
[[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]
|
||||
- **wiki-stocktake-maintenance overran for the third day running** - `1a8e90ea2271` ran **03:00:31 -> 08:42:17
|
||||
(5 h 41 m 46 s)**, hit the turn ceiling, reported incomplete, delivery `suppressed`. Pin:
|
||||
`custom:omni-101` / **`free-stack`**.
|
||||
|
||||
|
||||
## Recent Changes (2026-10-30, nightly audit; live probes 2026-10-30 00:0x AWST)
|
||||
|
||||
- **Conduit voice calls were repaired — after three faults in series — and the first two real calls in the
|
||||
product's history are now on record.** Anthony asked why Conduit never answered when he called. In order:
|
||||
(A) `voice.speech_server.url` held the truncated `wss://…:8443/v1/`, which the plugin dials **verbatim**, so
|
||||
the Mac answered **403** and the plugin blamed the key; (B) `dashboard.basic_auth.session_ttl_seconds` was the
|
||||
**12-hour default**, and an expired phone session gets **WS 4401** with no Hermes turn and no error visible on
|
||||
the phone (the iPhone re-authenticated at **08:00:59**, minutes *after* a silent call); (C) **both** host
|
||||
providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` → **403 /
|
||||
Cloudflare 1010**. Fixed 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → **2 592 000 s (30 d)**, `stt.provider:
|
||||
local` (faster-whisper `base`), `tts.provider: ollama_local` (6.94 s of real speech verified); dashboard
|
||||
restarted **08:03:26**. Verified end to end by driving the whole call path, then by **calls at 08:38:04 and
|
||||
08:49:36** — `conduit-calls.json → history` is no longer empty.
|
||||
→ [[incidents/2026-10-29-conduit-voice-silence]], [[systems/conduit]], [[systems/hermes-agent]]
|
||||
- **The Job Radar's two-stage split worked end to end for the first time.** Stage 1 (`dca8482e4f76`,
|
||||
`deliver: local`) ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`**, and wrote
|
||||
`~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**,
|
||||
sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Stage 2 (`5fc292ff0f3b`) consumed the
|
||||
handoff at **10:40:59**, closed **10:44:48 `completed`** and delivered to topic 1030 in the same second:
|
||||
**20 pages / 20 `page_ids` / 2 cover letters / `verified_cover_letter_count 2` /
|
||||
`missing_cover_letter_count 0`**, Notion board **455 → 475**. Both of stage 1's incident rows
|
||||
(`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**; the 11:30 and 14:30 slots
|
||||
then no-oped as designed (10 m 46 s / 1 m 51 s).
|
||||
→ [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]],
|
||||
[[runbooks/job-radar]]
|
||||
- **Cron: 2026-10-29 settled 177 `completed` / 0 `failed`** — the first zero-failure day since 2026-10-24 — with
|
||||
**10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; unclosed incident rows
|
||||
**4 → 1** (the two radar rows closed `resolved` at 10:16:12). Fleet unchanged at **29 jobs (26 enabled / 3
|
||||
disabled)** at that audit.
|
||||
- **Cron: 2026-10-30 closed 138 / 0 — after a 5 h 30 m HOST OUTAGE — and the "auto-paused" health monitor was never
|
||||
dead, it was running.** Fleet **29 jobs (26 enabled / 3 disabled)**
|
||||
(`jobs.json.updated_at 2026-10-31T00:29:59+08:00`), re-verified 2026-10-31 00:3x. **Zero execution rows exist
|
||||
between 10:58:20 and 16:37:21** — PVE `pve` rebooted (boot **2026-10-30 16:24:00**), CT460 came back **16:28:00**,
|
||||
the old gateway life (`pid=3913841`) was logged **UNCLEANLY** at **16:35:50** (*"no exit path ran — SIGKILL / OOM /
|
||||
VM death"* — a candidate list, not a diagnosis; root cause **not established**, no OOM evidence), and 8 catch-up
|
||||
runs fired at **16:37:21–16:38:22** → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. Against 10-29's
|
||||
**177 / 0**, that is **≈39 lost slots and no failures**. **`ea79a2107e36` (Mnemosyne Health Monitor) is ENABLED
|
||||
and clean again** — `last_status: ok`, streak **0**, incident `ea79a2_d93c3acdc4f2` closed **`resolved`
|
||||
2026-10-30T10:02:48.662** when its 00:01:08 slot finished after **10 h 01 m 40 s**; its **12:01 slot was lost to
|
||||
the outage** and is the still-`running` 16:37:21 catch-up. **Non-clean `last_status` 4 → 3**, **failing 1 → 0**,
|
||||
**unclosed incident rows 2 → 1** (survivor: `0e753e_e7f8194a7d06`, due 11-01). **`deliveries.db` finished 10-30
|
||||
with 8 rows, all `delivered`** (latest 18:46:04) — the 03:3x "zero rows / notice never delivered" reading is
|
||||
superseded.
|
||||
- **⚠ Superseded by the 2026-10-31 entry above — read that first.** Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure
|
||||
notice was never delivered.** `ea79a2107e36`'s 00:01:08 slot closed **`unknown`** at 01:04:22 on a
|
||||
gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`:
|
||||
Gateway connection failed; no local fallback"*), opening incident `ea79a2_d93c3acdc4f2` (`detected`), and the
|
||||
scheduler **auto-paused the job**. Fleet is now **29 jobs (25 enabled / 4 disabled)**; non-clean
|
||||
`last_status` is **4** (2 `error` + 2 `delivery_failed`); `ea79a2107e36` carries `failure_streak: 1`.
|
||||
⚠ **The failure notice itself failed to send** — `hermes cron list` reports `send_path_degraded; live adapter
|
||||
delivery to telegram:-1004321904721:102 failed`, and `deliveries.db` holds **zero rows for 2026-10-30**, so
|
||||
the only trace is the incident row. **Unclosed incident rows 1 → 2.** Every other previously-red job
|
||||
(`dca8482e4f76`, `1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`,
|
||||
`c0a9fa314542`) has **cleared to `ok` with streak 0** on its own slot. Pins: **18 of the 25 enabled jobs pin
|
||||
`custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); 7 enabled rows are `no_agent`
|
||||
jobs with no pin. → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]
|
||||
|
||||
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST)
|
||||
|
||||
- **The radar's silent budget overrun became a recorded failure — and the radar is now two jobs.** Window
|
||||
2026-10-27 settled **172 `completed` / 1 `failed`**. The failure is `dca8482e4f76` (Daily Marketing Job
|
||||
Radar - Discovery): **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed **`failed`**, delivery `delivered`,
|
||||
and the delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* —
|
||||
discovery, verification, dedup and classification completed; **Notion ingestion, cover-letter generation and
|
||||
the final cover-letter audit did not execute**. The prior day (10-26) had the same overrun but closed
|
||||
`completed`, hiding it in a green row. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted`
|
||||
11:03:59, **still open** at this audit (`failure_type: auth` — misclassified). → new page
|
||||
[[incidents/2026-10-27-job-radar-budget-exhausted]]
|
||||
- **Response: the radar was split into a two-stage pipeline the same day.** Stage 1 `dca8482e4f76` — Discovery
|
||||
(`0 7 * * 1-5`, `deliver: local`, failures → topic 1030) writes a handoff and triggers stage 2; new stage 2
|
||||
`5fc292ff0f3b` — Daily Marketing Job Radar - Write and Letters (`30 11,14 * * 1-5`, Telegram topic 1030,
|
||||
created **2026-10-27T17:34:15**) lands the Notion entries, cover letters and audit. Two test fires of stage 2
|
||||
on 10-27 (17:35:00 → 17:36:52, 17:38:38 → 17:43:42) both `completed` with delivery `suppressed`; the first
|
||||
real 07:00 → 11:30 cycle had not yet run. **Fleet 28 → 29 jobs (26 / 3).** → [[runbooks/job-radar]]
|
||||
- **The 2026-10-26 orphan class did not recur, and its two incident rows closed.** `ea79a2_a4661554a3c2` and
|
||||
`ea79a2_c3ab2d784ba3` **auto-closed 00:38:32** (unclosed rows **3 → 2**, the 2 being the Facebook birthday
|
||||
monthly scrape from 10-01 and the new radar row). No 10-27 run besides the radar's failed, so the watchdog
|
||||
and every other job were clean all day.
|
||||
- **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error`
|
||||
(streak 1) and `08eb86f02d37` re-joined as `delivery_queued`. Still non-clean: `0e753e9720fd` (error),
|
||||
`fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.**
|
||||
- **Also updated:** `incidents/2026-10-27-job-radar-budget-exhausted.md` (new),
|
||||
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md` (recurrence check), `runbooks/job-radar.md`,
|
||||
`systems/scheduled-tasks.md`, `systems/hermes-agent.md` (TTS provider correction),
|
||||
`systems/conduit.md` (voice pipeline), `infrastructure/tailscale.md` (peer table re-verified),
|
||||
`systems/skills-index.md` (regenerated 10-27, committed), `index.md`, `log.md`.
|
||||
**Nothing removed or archived.** No secrets written.
|
||||
|
||||
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 22:00 – 2026-10-29 00:3x AWST)
|
||||
|
||||
### The radar's two-stage split ran for real, and landed nothing — stage 1 hit the agent iteration cap
|
||||
|
||||
`dca8482e4f76` (stage 1, Discovery) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**) and closed **`failed`** on
|
||||
*"hit the agent iteration cap before the handoff could be written"* — **`failure_streak: 2`**. Stage 1 never
|
||||
produced a handoff (`~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`), so the *first* real two-stage
|
||||
cycle delivered **no Notion writes and no cover letters** — the same outcome as 10-27, by a different route.
|
||||
|
||||
Both stage-2 slots then did exactly what they were built to do: `5fc292ff0f3b` fired **11:30:27 → 11:31:31** and
|
||||
**14:30:27 → 14:31:43**, both `completed` with **`delivery_outcome: suppressed`** — *a stage-2 run with no
|
||||
handoff is a silent no-op, not a failure* (the recovery rule in [[runbooks/job-radar]]), now proven in
|
||||
production. The two post-split failures also stopped at different ceilings: 10-27 at the **tool-call budget**
|
||||
(4 h 2 m, writes not started), 10-28 at the **iteration cap** (3 h 32 m, handoff not written) — less of the
|
||||
pipeline completed despite the shorter wall clock. Open radar incident rows: `dca848_e93a538ade6b` (10-27) and
|
||||
`dca848_0a2f03b5952b` (10-28), both open. Detail:
|
||||
[[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
|
||||
|
||||
### The 03:00 stocktake failed on a provider 400 — and its incident row never alerted
|
||||
|
||||
`1a8e90ea2271` ran **03:00:28 → 03:37:58** (37 m 30 s) and closed **`failed`** (streak 1, delivery
|
||||
`suppressed`) with `RuntimeError: [400] Failed to deserialize the JSON body into the target type: duplicate
|
||||
field 'reasoning_content' at line 1 column 512203` against `custom:omni-101 / free-stack` — the *response* the
|
||||
provider was asked to deserialize carried a duplicated `reasoning_content` field. It performed its rebuild
|
||||
(302 skills → `systems/skills-index.md` mtime 03:01:40; freshness report `raw/stocktake/freshness-20261028.txt`
|
||||
→ **2 stale / 103 fresh / 4 no-date of 109 pages**) and then died **before the commit**, leaving the wiki tree
|
||||
dirty for the rest of the day. Its incident row `1a8e90_f57ab4cafab8` sits in state **`detected`** with **no
|
||||
alert** — the only outward sign was the job's own `error` status. The 2026-10-29 audit committed the two
|
||||
artefacts.
|
||||
|
||||
### Steel left the stack: fully removed — Camofox is the sole backend
|
||||
|
||||
`browser.cloud_provider` and `browser.camera_provider` are back to **`camofox`** (`engine: auto`, `cdp_url: ""`)
|
||||
in `config.yaml` (mtime **2026-10-29 00:15:13**), with backups taken at every step
|
||||
(`config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z` 22:00:57, and
|
||||
`config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24). The **`browser-steel` plugin was
|
||||
removed** from `~/.hermes/plugins/`, and no `browser-steel` key remains in `plugins.disabled` or
|
||||
`plugins.entries`, so the `plugin_steel` toolset is no longer selectable. **Steel is now fully REMOVED, not
|
||||
merely down:** on **CT201** the containers `steel-browser-api-1` / `steel-browser-ui-1`, the `steel-network`
|
||||
Docker network, the images (`steel-browser-api:fixed` and both ghcr.io Steel images) and the project dir
|
||||
`/root/steel-browser` were deleted (~6 GB reclaimed; disk 70G used → 64G used), and the Tailscale Serve routes
|
||||
`/steel` and `/steel-api` were removed — CT201's remaining routes are only `/` → `127.0.0.1:8085` and `/mgr` →
|
||||
`127.0.0.1:8084` on **both** `:443` and `:8443`, with nothing listening on 3000, 5173 or 9223. **CT333's second,
|
||||
separate Steel copy** (`steel-browser`, `ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30)
|
||||
was removed with its image. Hermes core contains **zero** Steel references, so there is **no Steel fallback path
|
||||
in the tree** — returning to Steel means redeploying it (the compose file and Dockerfile are archived on CT201
|
||||
under `/root/_archived-steel/`; the images remain re-pullable from ghcr.io). ⚠ `STEEL_BASE_URL` is no longer in
|
||||
`~/.hermes/.env` — it survived only in the inherited environment of processes already running at the cleanup,
|
||||
and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it (a couple of older, unrelated services
|
||||
still carry it) — **no core Hermes code reads it**, so it is a runtime-only leftover, **not** a config value.
|
||||
**Camofox is the sole
|
||||
backend** and healthy: REST `http://100.118.5.51:9377` (and `http://localhost:9377`), `/health` → **200**,
|
||||
`/tabs` → `{"running":true,"tabs":[]}`. Detail: [[systems/browser-backend]],
|
||||
[[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]].
|
||||
|
||||
> ⚠ **Superseded:** an earlier 2026-10-29 00:3x probe described Steel only as "not reachable — endpoint dark,
|
||||
> still running on CT201". That was already out of date — Steel is **removed**, not down.
|
||||
|
||||
### Fleet and tallies
|
||||
|
||||
29 jobs (**26 enabled / 3 disabled**) — unchanged. Non-clean `last_status`: **5** (`0e753e9720fd` error,
|
||||
`fad5a86979d4` delivery_failed, `dca8482e4f76` error **streak 2**, `066bb5191280` delivery_failed,
|
||||
`1a8e90ea2271` error **streak 1**). Pins: **12 `smart-route` / 6 `free-stack` / 7 unpinned /
|
||||
1 `deepseek-paid-smart`**. **2026-10-28 settled 168 completed / 2 failed / 1 running**; **2026-10-29** so far
|
||||
6 completed / 1 running; **2026-10-27 settled 173 / 1** (revised up by one late run after the 2026-10-28
|
||||
audit). No Hindsight service is running on this box (`:8888` closed, no `hindsight` units) — the standing
|
||||
memory-backend correction still holds and needs no change.
|
||||
|
||||
## Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST)
|
||||
|
||||
- **A *planned* gateway restart still orphaned a live cron run — and this time it alerted loudly.** A planned
|
||||
`SIGTERM` at **2026-10-26 14:33:35** produced a shutdown drain that reported **`timed_out=True` after
|
||||
just 1.35 s** of its 30 s budget and interrupted `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot)
|
||||
**2 h 34 m** in. It closed **`failed`** on a **new** string — `lost its durable fire claim ownership` —
|
||||
against the 2026-10-20 orphan's silent `unknown`, and raised **two** `cron_incidents` rows (14:33:37
|
||||
*unverified / reconcile admission*, 14:35:59 *fire claim*) that are still `alerted`; the failure **was
|
||||
delivered** at 14:35:59. Two Mnemosyne Watchdog reds followed and **self-resolved at 15:16:12**. Window
|
||||
2026-10-26 settled **169 / 3** and **all three failures are this one fault** — so the fleet's failure
|
||||
count is one interruption, not a systemic turn. Fleet unchanged at **28 jobs (25 enabled / 3 disabled)**;
|
||||
**non-clean `last_status` 5 → 4**; **unclosed incident rows 2 → 3**; **pins unchanged**. →
|
||||
new page [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]
|
||||
- **One incident row closed, and the 2026-10-16 503 burst is fully shut.** `c0a9fa_764558b5ae83`
|
||||
(GitHub Trending Radar) **auto-closed 09:13:00** when its 09:00 slot ran clean — the last open row of the
|
||||
resource-pressure fault → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. No execution row
|
||||
carries the 503 string on 10-25 or 10-26. The job stays **`monitoring`**: single-node exposure unchanged.
|
||||
- **Two quiet degradations worth naming.** `fad5a86979d4`'s 03:08 slot **completed but delivered
|
||||
`delivery_failed`** — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead
|
||||
thread at 14:33:37); and the **radar's 07:00 slot closed `completed` at 09:10 while delivering
|
||||
“RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters, batch finished by
|
||||
hand. That morning the cover-letter PDF pipeline moved to **Chromium over the canonical HTML** with model
|
||||
**`deepseek-paid-smart`** and a **600 s** dashboard timeout → [[runbooks/job-radar]]
|
||||
- **Also updated:** `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` (new),
|
||||
`incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md` (recurrence), `incidents/2026-10-16-omniroute-503-resource-pressure.md`,
|
||||
`runbooks/job-radar.md`, `systems/scheduled-tasks.md`, `index.md`, `log.md`. **Nothing removed or archived.**
|
||||
|
||||
## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST) *(superseded by the 2026-10-27 block above)*
|
||||
|
||||
- **A third consecutive clean cron day — and an off-cron fault that is not the fleet's.** Window
|
||||
2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at
|
||||
@@ -368,7 +981,7 @@ Updated: 2026-10-25
|
||||
- **OmniRoute CT600/CT601 roles are INVERTED.** CT600 `omniroute` (100.88.81.19) is **running** and serving `:20128` (combo API returns live combos); CT601 `omni` (100.93.204.4) is **stopped** with `onboot=0` (Tailscale peer offline 54d). Hermes resolves `custom:omni` → `omniroute:20128` → **CT600**. → [[systems/omniroute]]
|
||||
- **A third OmniRoute node exists:** `omniroute-1` at `100.78.63.20:20128`, live (401 = auth-required), registered in Hermes config as provider `omni-101` (5384 models, default `smart-route`). It carries `skills_hub` (model `free-stack`) — the same provider family implicated in the 2026-10-09 timeout storm. Not present in `pct list`; identity unconfirmed. → [[systems/omniroute]]
|
||||
- **Two Ignis instances exist, on different hosts.** CT460 `ignis` (port 8080, vault `/home/hermes/wiki/agent-estate-wiki`, serving 200) and CT200 `ignis-ignis-1` (image `ignis-server:latest`, host port 8092, vault `/opt/ignis/vaults`, serving 200). Both healthy; they are separate deployments. → [[systems/ignis]]
|
||||
- **Tailscale route drift fixed:** `/agent-estate` targets bare `http://127.0.0.1:8769` (page had `.../agent-estate-role-dashboard.html`); a tailnet-only `:9443` → `127.0.0.1:5173` Serve was undocumented. CT201's live routes are 8085-based plus `/steel` (5173) and `/steel-api` (3000) — the old `:8443` → `localhost:8082` Homepage row is wrong. → [[infrastructure/tailscale]]
|
||||
- **Tailscale route drift fixed:** `/agent-estate` targets bare `http://127.0.0.1:8769` (page had `.../agent-estate-role-dashboard.html`); a tailnet-only `:9443` → `127.0.0.1:5173` Serve was undocumented. CT201's live routes are 8085-based plus `/steel` (5173) and `/steel-api` (3000) — the old `:8443` → `localhost:8082` Homepage row is wrong. *(The `/steel` and `/steel-api` routes were deleted on 2026-10-29 — see [[systems/browser-backend]].)* → [[infrastructure/tailscale]]
|
||||
- **Proxmox inventory rebuilt** from `pct list` + `qm list`: **CT475 `agentzero` and VM110 `omarchy` were missing entirely**; 282 `ubuntu-server-xfce` corrected to stopped; dead rows removed (310, 500, 104, 106, 101). **VM403 `open-claw` and VM401 `hermes-old` are absent from `qm list`** — flagged unconfirmed, not deleted. → [[infrastructure/proxmox]]
|
||||
- **Nanobot verified on CT333:** `nanobot-gateway` (PID 2259048) alive on 8765/8766. It is **not** a systemd `nanobot.service`/`gateway.service` (both probes return `inactive` — wrong unit names); it runs as the `nanobot` user service plus the gateway process. Port **8900 is no longer listening** — previous "Serve API" claim flagged unverified. → [[agents/nanobot]]
|
||||
- **Docker services page corrected:** the old "Docker runtime not available" line was a CT460 socket artifact (`/var/run/docker.sock` is `root:docker srw-rw----`; `hermes` user not in group `docker`; no passwordless sudo in cron). Real fleet: CT200 holds ~50 containers, CT333 holds 9. → [[infrastructure/docker-services]]
|
||||
@@ -405,7 +1018,7 @@ _The manual 2026-10-10 stocktake entry follows beneath this one._
|
||||
- **Cron fleet: the worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2) — all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight still open; `1a8e90ea2271` sits at `failure_streak: 2`. → [[incidents/2026-10-09-cron-provider-timeout-storm]]
|
||||
- **The 2026-10-08 audit window was lost** to the same timeout (this job's own 23:59 run); the last committed audit covered 2026-10-07. The 2026-10-09 run covers both days. → [[log]]
|
||||
- **`62619aa6bcb3` (DHL Tracking — Parcel 8052838382) is still enabled** even though Anthony instructed its deletion on 2026-10-09 21:24 and the session reported it deleted; it runs at 08:00 daily. Retirement is open item #1 in [[systems/scheduled-tasks]].
|
||||
- **Memory recall relevance incident** — a stale conversational episode outranked current-state facts; capture and vector health verified (760/760 working, 1,043/1,047 episodic), ranking fix not persisted in config. → [[incidents/2026-10-09-memory-recall-relevance]]
|
||||
- **Memory recall relevance incident — ✅ RESOLVED 2026-11-01.** A stale conversational episode had outranked current-state facts on an open question. The ranking fix **was** persisted — the weights live in `~/.hermes/mnemosyne/config.yaml` (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`), not in the `~/.hermes/config.yaml` block the original probe read. Live tier counts: working **941** (embeddings **927/941, 98.5 %**), episodic **1,124** (embeddings **1,124/1,124, 100 %**), canonical **90**, conflicts **175** with **11 open** (extraction noise). The original regression query no longer surfaces the stale episode. → [[incidents/2026-10-09-memory-recall-relevance]]
|
||||
- **Mnemosyne live counts (2026-10-09 22:27):** 773 working / 1,047 episodic / 97 canonical slots / 871 facts / 152 conflicts (1 open) / 760 working embeddings / 1,043 episodic embeddings. → [[systems/mnemosyne]]
|
||||
|
||||
## Recent Changes (2026-10-08)
|
||||
@@ -498,7 +1111,7 @@ _The manual 2026-10-10 stocktake entry follows beneath this one._
|
||||
- **WebUI connected platforms:** local, api_server, discord, email, homeassistant, mattermost, ntfy, slack, telegram, webhook — Verified, WebUI session, 2026-07-22
|
||||
|
||||
## Recent Changes (2026-08-02)
|
||||
- **Browser backend reverted:** Camoufox restored as primary on 2026-07-30. Steel Browser (CT201) experiment ended — CDP issues + DeepSeek 0731 tool name validation. `browser.cloud_provider: camofox`, `camera_provider: camofox`. Steel plugin disabled. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
|
||||
- **Browser backend reverted:** Camoufox restored as primary on 2026-07-30. Steel Browser (CT201) experiment ended — CDP issues + DeepSeek 0731 tool name validation. `browser.cloud_provider: camofox`, `camera_provider: camofox`. Steel plugin disabled *(the 2026-08-02 state — the plugin and the Steel deployments were later removed entirely on 2026-10-29, see [[systems/browser-backend]])*. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
|
||||
- **Mnemosyne capture fix (2026-08-01):** Automatic conversational capture was broken by stringified YAML (`sync_roles: "['user']"`). Fixed by rewriting structured Mnemosyne settings with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||||
- **Mnemosyne vector coverage gap discovered:** Only 159/1,476 working memories (10.8%) have vector embeddings. 0/458 episodic memories have embeddings. Semantic search is mostly FTS5 fallback.
|
||||
- **DeepSeek V4 Flash 0731:** Post-training bump, same model ID, same pricing. Terminal Bench 2.1: 82.7 (up from 61.8). Stricter tool name validation (`^[a-zA-Z0-9_-]+$`) — may cause issues with OmniRoute-transformed tool names.
|
||||
|
||||
+13
-6
@@ -3,8 +3,8 @@ title: Decisions
|
||||
type: collection
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [decisions]
|
||||
---
|
||||
@@ -19,15 +19,22 @@ Stable strategic and config decisions, with evidence and status.
|
||||
- [[decisions/persistent-shell-enabled]]
|
||||
- [[decisions/rollback-steel-to-camofox]]
|
||||
|
||||
> **Status sweep 2026-10-20:** all three listed decisions are still `status: active`, but two now
|
||||
> carry a review note:
|
||||
> **Status sweep 2026-10-20:** all three listed decisions are still `status: active`, but each now
|
||||
> carries a note:
|
||||
> - [[decisions/keep-openrouter-as-fallback]] — its 30-day revisit trigger is **met**; OpenRouter is
|
||||
> reachable only through the OmniRoute catalog, not as a standalone provider.
|
||||
> - [[decisions/persistent-shell-enabled]] — still `true`; its `review_after: 0001-01-01` zero-date
|
||||
> bug is now corrected to a real date.
|
||||
> - [[decisions/rollback-steel-to-camofox]] — the *engine* is Camoufox as decided, but
|
||||
> `cloud_provider: steel` remains in the config while the Steel service itself is unreachable.
|
||||
> - [[decisions/rollback-steel-to-camofox]] — ~~the *engine* is Camoufox as decided, but
|
||||
> `cloud_provider: steel` remains in the config while the Steel service itself is unreachable.~~
|
||||
> **Superseded 2026-10-29:** `cloud_provider` is `camofox` again and Steel is **fully removed** —
|
||||
> plugin deleted, CT201 and CT333 deployments torn down, Serve routes deleted. Camofox is the sole
|
||||
> backend.
|
||||
|
||||
> **Status sweep 2026-11-20:** all three decisions still `status: active` and re-verified against live
|
||||
> state — `keep-openrouter-as-fallback` and `persistent-shell-enabled` were re-verified 2026-11-11
|
||||
> (`terminal.persistent_shell: true` re-confirmed live today); `rollback-steel-to-camofox`'s
|
||||
> 2026-10-29 update re-probed today: `cloud_provider: camofox`, Steel absent estate-wide.
|
||||
## History
|
||||
- **Verified:** Decision pages initialized during wiki setup on 2026-07-22
|
||||
- **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Keep OpenRouter as fallback
|
||||
type: decision
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
review_after: 2026-10-20 # was 0001-01-01 (bug), fixed during 2026-07-22 audit; real trigger is conditional, see Revisit when
|
||||
verified_on: 2026-10-20
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [routing, provider, decision]
|
||||
sources: []
|
||||
@@ -50,6 +50,14 @@ A self-hosted router has operated reliably for at least 30 days.
|
||||
> bursts, and the 2026-10-18 degenerate-stream delivery. A single-node fallback posture carries
|
||||
> that concentration risk with it.
|
||||
|
||||
> **Re-verified 2026-11-11 — the decision still needs the re-affirmation it was flagged for on
|
||||
> 2026-10-20, and the position is unchanged.** `custom_providers` still holds exactly
|
||||
> `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101` — **no standalone
|
||||
> OpenRouter provider**. OmniRoute remains the primary path (`smart-route` on
|
||||
> `custom:omni-101`), so the "revisit trigger" (a self-hosted router running reliably >30 days) is
|
||||
> still met and the literal reading of the decision is still unsatisfied. Nothing new to decide
|
||||
> here — the page still says what needs a human call.
|
||||
|
||||
## Related
|
||||
- [[model-routing]]
|
||||
- [[systems/omniroute]]
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Persistent shell enabled
|
||||
type: decision
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
review_after: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [automation, shell, decision]
|
||||
sources: []
|
||||
@@ -34,5 +34,11 @@ Use case becomes critical again.
|
||||
> is torn down after 5 minutes idle, so "state persists across turns" is true within a burst of
|
||||
> work but not indefinitely. See [[concepts/persistent-shells]].
|
||||
|
||||
> **Re-verified 2026-11-11 — decision still `active`, setting still `true`.**
|
||||
> `terminal.persistent_shell: true`, `backend: local`, `timeout: 180`, `home_mode: auto`,
|
||||
> `container_cpu: 1`, `lifetime_seconds: 300`, `cwd: /home/hermes/workspace/`. The 5-minute idle
|
||||
> teardown caveat recorded on 2026-10-20 still applies: state persists across a burst of work, not
|
||||
> indefinitely.
|
||||
|
||||
## Related
|
||||
- [[persistent-shells]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Rollback Steel to Camoufox
|
||||
type: decision
|
||||
status: active
|
||||
created: 2026-08-02
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [decision, browser, steel, camofox, rollback]
|
||||
sources: [session-history, live-config]
|
||||
@@ -28,23 +28,29 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat
|
||||
## Decision
|
||||
**Revert to Camoufox as primary.** Steel plugin disabled but left running on CT201.
|
||||
|
||||
> **Status check 2026-10-20 — the decision is *partially* undone in configuration.**
|
||||
> The live `browser:` block in `~/.hermes/config.yaml` reads
|
||||
> `cloud_provider: steel` with `engine: camofox`. The **engine** is Camoufox (decision honoured),
|
||||
> but the **cloud_provider** key still says `steel`, and `camera_provider: camofox`.
|
||||
> ⚠ **Superseded 2026-10-29:** the "left running on CT201" half of this decision no longer holds — Steel is
|
||||
> now **fully removed** from the estate. See the Update below.
|
||||
|
||||
> **Status check 2026-10-20 — the decision is *partially* undone in configuration.** *(superseded 2026-10-29)*
|
||||
> The live `browser:` block in `~/.hermes/config.yaml` read
|
||||
> `cloud_provider: steel` with `engine: camofox`. The **engine** was Camoufox (decision honoured),
|
||||
> but the **cloud_provider** key said `steel`, and `camera_provider: camofox`.
|
||||
> This is the same split seen in [[comparisons/camoufox-vs-chrome]].
|
||||
>
|
||||
> Meanwhile the Steel service itself is **not reachable**: `http://steel.martinwa.org` returns
|
||||
> Meanwhile the Steel service itself was **not reachable**: `http://steel.martinwa.org` returned
|
||||
> **000** (connection failed) on 2026-10-20. Combined with the degraded CT201 stack recorded in
|
||||
> [[systems/homepage-dashboard]] (main container removed, asset proxy 502), it looks like the
|
||||
> "left running on CT201" part of this decision is no longer true.
|
||||
> [[systems/homepage-dashboard]] (main container removed, asset proxy 502), it looked like the
|
||||
> "left running on CT201" part of this decision was no longer true.
|
||||
>
|
||||
> **Recommendation, not actioned here:** the `cloud_provider: steel` value appears to be residue
|
||||
> **Recommendation, not actioned here:** the `cloud_provider: steel` value appeared to be residue
|
||||
> from the reverted deployment. Confirm against the live Steel service and either restore it or
|
||||
> set `cloud_provider: camofox` so the config matches the decision this page records.
|
||||
>
|
||||
> **Resolved 2026-10-29:** `cloud_provider` is `camofox` again and Steel has been removed entirely — no
|
||||
> restoration was needed or wanted.
|
||||
|
||||
## Rationale
|
||||
1. **Reliability:** Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
|
||||
1. **Reliability:** Camoufox runs on the same host as Hermes (~~CT450~~ **CT460** — host attribution corrected 2026-11-20; the service is the local `camofox-browser.service` on the production Hermes host) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
|
||||
2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports.
|
||||
3. **Failure mode:** The `opencode-go/deepseek-v4-flash` empty-stream errors were hitting all providers, but Steel added an extra failure surface (CDP connection, plugin layer) on top of the same problem.
|
||||
4. **DeepSeek 0731 compatibility:** The new model's stricter tool name validation (`^[a-zA-Z0-9_-]+$`) affected all providers — not a Steel-specific issue, but another reason to minimise the toolchain.
|
||||
@@ -52,13 +58,59 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat
|
||||
## Consequences
|
||||
- **Positive:** Browser automation is stable again. Uber Eats report ran successfully via Camofox on 2026-08-01.
|
||||
- **Negative:** Steel's session viewer UI and full CDP access are no longer available to Hermes.
|
||||
- **Neutral:** Steel still runs on CT201 — can be re-enabled by uncommenting `.env` lines and enabling the plugin.
|
||||
- **Neutral (2026-07-30 — now superseded):** Steel still ran on CT201 and could be re-enabled by uncommenting `.env` lines and enabling the plugin. **As of 2026-10-29 that is no longer true** — Steel is removed, so re-enabling means redeploying it (compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`; images remain re-pullable from ghcr.io).
|
||||
|
||||
## Evidence
|
||||
- Camoufox verified working Jul 30: `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s
|
||||
- Uber Eats report (Aug 1) successfully scraped via Camofox REST API — 11 orders, $264.51
|
||||
- Steel plugin `browser-steel` still present at `/home/hermes/.hermes/plugins/browser-steel/` — disabled
|
||||
- Steel plugin `browser-steel` was present at `/home/hermes/.hermes/plugins/browser-steel/` — disabled *(2026-07-30 state; the plugin has since been deleted — see the Update below)*
|
||||
|
||||
## Update (2026-10-29) — Steel was fully removed, not just darkened
|
||||
|
||||
The 2026-07-30 rollback disabled the Steel plugin but left it on disk and left Steel running. Both of those
|
||||
were closed on **2026-10-28 22:00 – 2026-10-29 00:15**: `browser.cloud_provider` / `camera_provider` were
|
||||
returned to **`camofox`** (backups `config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`,
|
||||
`config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z`), and **`~/.hermes/plugins/browser-steel/` was
|
||||
removed**, so the `plugin_steel` toolset is no longer selectable.
|
||||
|
||||
**Steel is now FULLY REMOVED — it is not merely "down" or "not reachable":**
|
||||
|
||||
- **Hermes config:** the `browser:` block reads exactly `cloud_provider: camofox`, `camera_provider: camofox`,
|
||||
`engine: auto`, `cdp_url: ""`. No `browser-steel` key remains in `plugins.disabled` or `plugins.entries`, and
|
||||
`STEEL_BASE_URL` is gone from `~/.hermes/.env` (it survived only in the inherited environment of processes
|
||||
already running at the cleanup; the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it, while a
|
||||
couple of older, unrelated services still carry it — **no core Hermes code reads it**, so it is a
|
||||
runtime-only leftover, not a config value).
|
||||
- **Hermes plugins:** `~/.hermes/plugins/browser-steel/` is deleted. Hermes core source contains **zero**
|
||||
Steel references, so there is **no Steel fallback path in the tree** — returning to Steel means redeploying
|
||||
the plugin, not flipping a flag.
|
||||
- **CT201 (runtipi):** containers `steel-browser-api-1` and `steel-browser-ui-1`, the `steel-network` Docker
|
||||
network, the images `steel-browser-api:fixed` / `ghcr.io/steel-dev/steel-browser-api:latest` /
|
||||
`ghcr.io/steel-dev/steel-browser-ui:latest`, and the project dir `/root/steel-browser` were all removed
|
||||
(~6 GB reclaimed; disk 70G used → 64G used). The Tailscale Serve routes `/steel` and `/steel-api` were
|
||||
deleted with `tailscale serve --set-path <path> off`; CT201's remaining routes are only `/` → `127.0.0.1:8085`
|
||||
and `/mgr` → `127.0.0.1:8084` on both `:443` and `:8443`. Nothing listens on 3000, 5173 or 9223.
|
||||
- **CT333 (clawtest):** a second, separate and orphaned Steel install — the `steel-browser` container
|
||||
(`ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image
|
||||
were removed as well. It had no Tailscale Serve route, no entry in `~/.zeroclaw/config.toml` (which uses
|
||||
`backend = "agent_browser"`), and no reference from Nanobot — its last CDP navigation was 2026-10-20 and it
|
||||
received no requests after 2026-10-21.
|
||||
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus
|
||||
`/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io.
|
||||
- **Camofox is the sole backend** and healthy — REST `http://100.118.5.51:9377` (also `http://localhost:9377`),
|
||||
`/health` → **200**, `/tabs` → `{"running":true,"tabs":[]}`.
|
||||
|
||||
The decision itself is unchanged — Camofox is the backend — but the "Steel is still there if you need it"
|
||||
assumption is dead. Detail: [[systems/browser-backend]], [[comparisons/steel-vs-camoufox]].
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake).** Every removal claim in the 2026-10-29 update re-probed:
|
||||
> config `cloud_provider/camera_provider: camofox`, `engine: auto`; plugin dir absent; CT201 steel
|
||||
> containers/images/network/project dir gone, Serve only `/`→8085 + `/mgr`→8084 on :443/:8443,
|
||||
> nothing on 3000/5173/9223; CT333 steel gone, :3000 closed; `/root/_archived-steel/` archive intact
|
||||
> (compose, Dockerfile.api, cache tgz). Camofox `:9377/health` → `ok` (idle). **Note:** the Rationale's
|
||||
> "Camoufox runs on the same host as Hermes (CT450)" reflects the July-era attribution — the service
|
||||
> runs on **CT460** (the production Hermes host), corrected in [[systems/browser-backend]] 2026-10-13.
|
||||
> The decision and its consequences are unchanged.
|
||||
## Related
|
||||
- [[comparisons/steel-vs-camoufox]]
|
||||
- [[systems/browser-backend]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Experiments
|
||||
type: collection
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: medium
|
||||
tags: [experiments]
|
||||
---
|
||||
@@ -40,6 +40,13 @@ Non-canonical tests, tuning passes, and change attempts.
|
||||
> See [[incidents/2026-09-28-hermes-managed-runtime-deps]] and the runtime-deps notes in
|
||||
> [[systems/hermes-agent]].
|
||||
|
||||
> **Backups tree re-listed 2026-11-11 — no change to report.** `~/.hermes/backups/` still holds the
|
||||
> same dated incident directories named on 2026-10-20 (including
|
||||
> `mnemosyne-capture-fix-20260801_180249/` and `legacy-memory-slim-20260801_180702/`, both verified
|
||||
> present again during this pass), plus the rolling `config/` series. The three experiment
|
||||
> snapshots this page originally listed (`pre-vision-fix-20260718-195712`, `pre-model-fix`,
|
||||
> `pre-camofox-tuning`) are still gone.
|
||||
|
||||
## Outcomes
|
||||
- **Verified:** Backups exist with the above names
|
||||
- **Unknown:** Whether each experiment succeeded, failed, or was reverted
|
||||
|
||||
@@ -3,9 +3,8 @@ title: 'Mnemosyne Relay Scope Corruption + `tools: None` — 2026-07-29'
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-07-29
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
confidence: high
|
||||
tags: [mnemosyne, config, relay, incident, resolved]
|
||||
sources: [live-probe, config, session-20260729]
|
||||
@@ -52,3 +51,20 @@ Filed (or should be filed) under NousResearch/hermes-agent issues. No token avai
|
||||
|
||||
## MONITORING
|
||||
Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.).
|
||||
|
||||
## Re-verified 2026-11-12 03:3x AWST (stocktake — page was 19d old, inside the staleness window)
|
||||
Both halves of this page re-checked against live state and **the fix holds; neither symptom has recurred**
|
||||
in ~3.5 months and 106 days of logs:
|
||||
- `grep -n "tools:" ~/.hermes/config.yaml` → the `memory.mnemosyne` block no longer carries `tools: None`
|
||||
(the literal `None` key is absent), so the `Unknown Mnemosyne tool(s)… None` schema warning has no
|
||||
trigger left.
|
||||
- `grep -c "Unknown Mnemosyne tool" ~/.hermes/logs/gateway.log` and `.log.1`–`.log.3` → **0 in every
|
||||
rotated file** (4 generations, 0 occurrences).
|
||||
- `grep -ah "scope handle is not at the top" ~/.hermes/logs/gateway.log*` → **no occurrences**; the
|
||||
relay-scope corruption has not repeated since 2026-07-29 08:10 UTC.
|
||||
- Mnemosyne itself is healthy on the current build: `mnemosyne stats` → **2782 memories (1062 working /
|
||||
1217 episodic)** and `mnemosyne recall "tailscale serve routes"` returns substantive ranked results, so
|
||||
the relay/memory path is functioning.
|
||||
|
||||
The `REPORTED` action above still stands (no token to auto-create a NousResearch issue); the **MONITORING**
|
||||
condition is now satisfied — no recurrence across four log generations.
|
||||
@@ -3,9 +3,9 @@ title: Mnemosyne Automatic Capture Failure
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-08-01
|
||||
updated: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
resolved_on: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, hermes, incident]
|
||||
sources: [live-config, mnemosyne-db, hermes-cli-e2e]
|
||||
@@ -82,6 +82,14 @@ no Mnemosyne trace.
|
||||
`~/.hermes/backups/mnemosyne-capture-fix-20260801_180249/` and
|
||||
`~/.hermes/backups/legacy-memory-slim-20260801_180702/`.
|
||||
|
||||
> **Resolution re-confirmed 2026-11-11.** The whole `memory.mnemosyne` block in
|
||||
> `~/.hermes/config.yaml` is still natively typed — `sync_roles: [user]` (list),
|
||||
> `skip_contexts: [cron, flush, subagent, background, skill_loop]` (list),
|
||||
> `reflect: {disabled_for_cron: true, max_calls_per_session: 3}` (dict), `sleep_threshold: 50`
|
||||
> (int), `ignore_patterns: []` (list), `profile_isolation: true` (bool), `auto_sleep: true`,
|
||||
> `default_scope: session`, `vector_type: int8`. No stringification regression. Both backup
|
||||
> directories named in the repair still exist. `status: resolved` stands.
|
||||
|
||||
## Related
|
||||
- [[systems/mnemosyne]]
|
||||
- [[systems/memory-backend]]
|
||||
|
||||
@@ -3,9 +3,8 @@ title: OmniRoute free-DeepSeek Silent Stops (Empty-After-Tool) — 2026-08-04
|
||||
type: incident
|
||||
status: monitoring
|
||||
created: 2026-08-04
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
confidence: high
|
||||
tags: [omniroute, deepseek, free-tier, silent-failure, incident, mitigated]
|
||||
sources: [live-probe, omniroute-log, session-20260804]
|
||||
@@ -53,5 +52,23 @@ Backup of both pre-change rows: `/tmp/combo_backup_20260804.json` on the Hermes
|
||||
- Consider `auto`+explorationRate for self-tuning if free tier stays flaky.
|
||||
- Hermes-side: empty-post-tool turns are nudge/retry/fallback handled, but still end silently once exhausted — possible future improvement: surface "(empty after tool calls)" more visibly.
|
||||
|
||||
## Re-verified 2026-11-12 03:4x AWST (stocktake — page was 19d old, inside the staleness window)
|
||||
Both reworked combos probed live on **`omniroute-1`** (`100.78.63.20:20128`, `custom:omni-101`) with the
|
||||
correct `key_env` (`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`) and the exact aliases used:
|
||||
|
||||
| Probe | Result |
|
||||
|---|---|
|
||||
| `good-free` (the 12-model rework) | **HTTP 200**, resolving to **`glm-5.2-cheap:free`** with a valid non-empty completion |
|
||||
| `free-stack` | **HTTP 200**, resolving to **`stepfun-ai/step-5-preview`** with a valid completion |
|
||||
|
||||
**The mitigation is holding, and the 2026-08-04 follow-up can be partially closed:** both combos answer with
|
||||
real content, and **neither resolves to a deepseek model** — which is exactly the intent of dropping the
|
||||
proven culprits from `good-free`. ⚠ **What is *not* re-verified:** I did not individually probe the
|
||||
free-DeepSeek upstreams (`238d5978/deepseek-v4-flash`, `nvidia/deepseek-ai/deepseek-v4-pro`,
|
||||
`deepseek-v4-flash-thinking:free`), so the original claim *"upstream free-DeepSeek tier still degraded"*
|
||||
stands **neither confirmed nor refuted** — the combos serve healthy non-deepseek free models, which may mean
|
||||
the upstream recovered *or* simply that it is no longer reached. **Restoring deepseek-first ordering is
|
||||
therefore still not justified** and should wait on a direct probe of those three upstreams.
|
||||
|
||||
## Refs
|
||||
- [[systems/omniroute]]
|
||||
|
||||
@@ -1,18 +1,23 @@
|
||||
---
|
||||
title: Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 While the Job Reads `ok`
|
||||
title: Hermes Config Backup to Gitea — Push Was Dead 2026-08-08 → 2026-11-07 While the Job Read `ok`
|
||||
type: incident
|
||||
status: active
|
||||
status: resolved
|
||||
created: 2026-10-21
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
review_after: 2026-10-29
|
||||
updated: 2026-11-07
|
||||
resolved_on: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
review_after: 2027-02-07
|
||||
confidence: high
|
||||
tags: [backup, gitea, cron, git, green-vs-actual]
|
||||
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki]
|
||||
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki, gitea-api, github-api, sandbox-e2e-test]
|
||||
---
|
||||
|
||||
# Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08
|
||||
|
||||
> **RESOLVED 2026-11-07.** The push is live again, the backlog is drained, and the script now exits
|
||||
> non-zero on failure so the cron row cannot lie. See [Resolution 2026-11-07](#resolution-2026-11-07)
|
||||
> at the foot of this page for the full record. The original investigation is preserved below.
|
||||
|
||||
## Headline
|
||||
|
||||
The daily **Hermes Config Backup to Gitea** job (`e27da6c34bbb`, `0 11 * * *`, no-agent, `hermes-config-backup.sh`)
|
||||
@@ -103,3 +108,92 @@ change as well the *only* off-box copy path was this dead push → [[infrastruct
|
||||
- [[infrastructure/backups]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/hermes-agent]]
|
||||
|
||||
---
|
||||
|
||||
# Resolution 2026-11-07
|
||||
|
||||
Anthony asked to fix the failing push (and to check the Notion vault for the API key). Resolved in
|
||||
one session: the push is live, the backlog is drained, and the script can no longer report a false green.
|
||||
|
||||
## Root causes (four, all independent)
|
||||
|
||||
1. **No Gitea credential.** `origin` was HTTPS with no userinfo, and the global `credential.helper=store`
|
||||
file `~/.git-credentials` held only a `github.com` entry — so the cron worker could not authenticate
|
||||
non-interactively. The token in the Notion vault was valid and identical to the one the wiki repo
|
||||
carries in its remote URL; it simply was never wired into the backup repo.
|
||||
2. **`backup.sh` had no `exit` statement at all.** It always returned 0, so the scheduler
|
||||
(`success = (returncode == 0)` in `cron/scheduler_script.py`) recorded `last_status: ok` on every
|
||||
failed night. This is why the outage went unnoticed for 92 days — it was a *false green*, not a
|
||||
missing signal. (The incident's earlier note blamed the scheduler; the scheduler was correct. The
|
||||
script was the liar.)
|
||||
3. **A 1.76 GB `checkpoints/` backlog** sat in the unpushed range — 13 blobs over 100 MB, all junk
|
||||
that `.gitignore` already excluded for new commits but that history still carried.
|
||||
4. **Two broken secondary legs.** The `github` remote did not exist, and the fallback
|
||||
`_github_clean_push` pushed a branch's full ancestry — which contains a 7.45 GB tarball blob, so
|
||||
GitHub's 100 MB file limit could never accept it. The AgentMail alert called a
|
||||
`python3 send_email.py` that no longer exists, via a `chaos-agent@agentmail.to` inbox that no
|
||||
longer exists.
|
||||
|
||||
## Fix applied
|
||||
|
||||
- **Credential:** stored a host-scoped Gitea entry via `git credential approve` (not embedded in the
|
||||
remote URL). `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`.
|
||||
- **Backlog:** `git filter-repo --partial --refs origin/main..main --path checkpoints --invert-paths
|
||||
--prune-empty never --force` stripped `checkpoints/` from the unpushed range only (so `origin/main`
|
||||
was untouched and the push stayed a fast-forward), then `checkpoints/` was untracked at the tip.
|
||||
Pre-rewrite history is preserved under the tag `backup/pre-rewrite-20261107-032658`.
|
||||
**Push set: ~2 GB → 264.8 MB.** The 7.45 GB tarball was already on `origin/main` and was never resent.
|
||||
- **Script hardening** (`~/.hermes/config-backup/backup.sh`, rewritten via OpenCode + a correction pass):
|
||||
- exits **0 only on verified success**, `1` otherwise — the cron row can no longer lie;
|
||||
- `.last_backup_ok` is stamped **only** after the remote is confirmed to hold the commit;
|
||||
- the push decision compares **remote vs local**, not "were there new working-tree changes" — a
|
||||
local-ahead-of-remote with a clean tree now still pushes (this is exactly the mechanism that
|
||||
created the 92-commit backlog);
|
||||
- every `git ls-remote` is guarded with `|| true`, so an unreachable remote reaches the alert path
|
||||
instead of killing the script at exit 128 before `notify_failure` fires;
|
||||
- the GitHub leg now pushes a **parentless orphan snapshot** built from `HEAD^{tree}` via
|
||||
`git commit-tree` — no history, no >100 MB blob, HEAD/branch/index untouched — and is a
|
||||
warning-only secondary target that can never fail the job;
|
||||
- the AgentMail leg uses the real CLI (`agentmail inboxes:messages send --inbox-id <addr>`) with the
|
||||
key read from `.env` at runtime and passed via the environment, never argv;
|
||||
- the Telegram leg uses `curl --fail` and requires `"ok":true` in the body, so it reports `sent`
|
||||
only on a real delivery.
|
||||
|
||||
## Verification (live, 2026-11-07 AWST)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Real `backup.sh` run | exit **0**, 47 s |
|
||||
| `git rev-parse HEAD` vs `git ls-remote origin refs/heads/main` | identical — `a50a5170c1c0e2af5235fa63a95a732e2d780703` |
|
||||
| `git rev-list --count origin/main..main` | **0** (was 92) |
|
||||
| Gitea API repo `Anthony/hermes-config-backup` | HTTP 200, default branch `main`, updated `2026-11-07T05:10:46+08:00`, 8.1 MB |
|
||||
| Gitea API `branches/main` | `a50a5170c1c0` — `🦝 nightly backup: 2026-11-07 05:10:33 AWST` |
|
||||
| `.last_backup_ok` | refreshed to the run timestamp (was stale from 2026-11-06 11:00) |
|
||||
| GitHub `main` | `faed9fb5c677`, **0 parents** (orphan snapshot confirmed), 30,086 files, largest blob 23.5 MB |
|
||||
| GitHub `master` branch | untouched at `c99c94e6b13b` |
|
||||
| GitHub force-push data loss | none — GitHub's tree was a strict **subset** of local HEAD (only `BACKUP-INDEX.md` added) |
|
||||
| Token leakage scan (HEAD tree, working tree, all unpushed blobs) | **clean** — token absent everywhere |
|
||||
| AgentMail leg (self-test to the agent's own inbox) | delivered, `message_id` returned |
|
||||
| Telegram leg (`getMe`, no message sent to Anthony) | `"ok":true` detected → would report `sent` |
|
||||
| Sandbox E2E: unreachable remote | exit **1**, `notify_failure` fired, health file **not** stamped |
|
||||
| Sandbox E2E: local ahead of remote, clean working tree | pushed + verified, exit **0**, health stamped |
|
||||
| Sandbox E2E: already up to date | no push, exit **0**, health stamped |
|
||||
|
||||
## Verification command set (for the next audit)
|
||||
|
||||
```
|
||||
git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
|
||||
git -C ~/.hermes/config-backup rev-list --count origin/main..main
|
||||
tail -20 ~/.hermes/config-backup/backup.log
|
||||
tail -3 ~/.hermes/config-backup/.backup_history
|
||||
```
|
||||
|
||||
## Still open (not part of this incident)
|
||||
|
||||
- Gitea host SSH login is denied for `anthonymau` — free disk headroom on the Gitea box is still
|
||||
unverified (the Gitea API reports repo size, not host free space).
|
||||
- `hermes-gateway-runtime-health.service` still fails every 5 min because the Mattermost host
|
||||
`100.98.116.14` has been offline since ~Sept 2026. Separate incident.
|
||||
- The `checkpoints/` directory is still excluded from backups by policy; the live snapshots remain
|
||||
at `~/.hermes/checkpoints/` on-box only.
|
||||
|
||||
@@ -3,10 +3,10 @@ title: "Skills Index Drift — 4x Undercount"
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-08-15
|
||||
updated: 2026-10-23
|
||||
updated: 2026-11-05
|
||||
resolved_on: 2026-08-15
|
||||
reopened_on: 2026-10-23
|
||||
verified_on: 2026-10-23
|
||||
verified_on: 2026-11-05
|
||||
confidence: high
|
||||
tags: [skills, wiki, drift, auto-regeneration]
|
||||
sources: [session-investigation, agent-estate-wiki, live-filesystem-probe, cron-executions-db]
|
||||
@@ -88,9 +88,14 @@ from the scan, prints the archived count as an explicit exclusion, and states th
|
||||
filesystem inventory. The page was regenerated through the patched script and
|
||||
committed with the fix.
|
||||
|
||||
**Still open:**
|
||||
- The stale `~/.hermes/scripts/wiki-maintain.sh` copy must never be run — it produces
|
||||
the divergent shape. It should be deleted or made to delegate to the repo script.
|
||||
**Still open (as of 2026-10-23):**
|
||||
- ~~The stale `~/.hermes/scripts/wiki-maintain.sh` copy must never be run — it produces
|
||||
the divergent shape. It should be deleted or made to delegate to the repo script.~~
|
||||
**✅ CLOSED 2026-11-05** — the stale copy is now a **delegating wrapper** with no generator logic of its
|
||||
own; it `exec`s the repo script. The repo script gained `--no-build` so the wrapper can honour the
|
||||
interface its callers used without carrying a second copy of the generator. Verified by running the
|
||||
wrapper: it now produces the canonical **302 skills / 68 categories** shape.
|
||||
→ [[incidents/2026-11-05-skills-index-generator-divergence]]
|
||||
- The page will still read higher than the loader's 329: nested collections
|
||||
(`context-engineering-collection/…`), `apple/` and hub-installed trees are counted as
|
||||
installed files. Whether the page should instead report the loader's figure is a
|
||||
|
||||
@@ -3,8 +3,8 @@ title: "Mnemosyne Capture Pipeline Broken (Aug 18)"
|
||||
type: incident
|
||||
status: superseded
|
||||
created: 2026-08-18
|
||||
updated: 2026-10-12
|
||||
verified_on: 2026-10-12
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, hermes, incident, capture]
|
||||
sources: [mnemosyne-db, health-monitor, run_agent.py]
|
||||
@@ -68,6 +68,29 @@ embedding coverage **98.3 % working / 99.6 % episodic**, FTS aligned (`fts_worki
|
||||
The three guard conditions diagnosed below were never individually confirmed as the fix — the path resumed
|
||||
capturing and has captured through every window since.
|
||||
|
||||
### Re-verified 2026-11-04 — capture still healthy, and the counts have moved on
|
||||
|
||||
Live state of `~/.hermes/mnemosyne/data/mnemosyne.db` (the path in the Verification snippet above is still
|
||||
correct; note the DB is under `data/`, not the `~/.hermes/mnemosyne/*.db` glob path):
|
||||
|
||||
| Metric | 2026-10-12 (page) | 2026-11-04 (live) |
|
||||
|---|---|---|
|
||||
| `working_memory` | 786 | **970** |
|
||||
| `episodic_memory` | 1,056 | **1,146** |
|
||||
| `memoria_facts` | — | **446** |
|
||||
| `memory_embeddings` | — | **943** |
|
||||
| Working-memory vector coverage | 98.3 % | **98.6 %** (`vec_working` 956 / `working_memory` 970) |
|
||||
| Episodic vector coverage | 99.6 % | **100 %** (`vec_episodes` 1,146 / `episodic_memory` 1,146) |
|
||||
| `fts_working` | 786 | **970** — aligned with `working_memory` |
|
||||
| `fts_facts` | 876 | **927** (vs `memoria_facts` 446 — FTS indexes a superset) |
|
||||
| Last `working_memory` entry | 2026-10-11 | **2026-11-03 17:12:40** |
|
||||
|
||||
`mnemosyne stats` independently reports **Total 2,562 / Working 970 / Episodic 1,146**, and the `vec0` extension
|
||||
loads cleanly (`sqlite-vec` from the agent venv), confirming vector search is live rather than silently FTS-only.
|
||||
**Working-memory coverage is above the 95 % reindex threshold**, so no `mnemosyne reindex` was triggered by this
|
||||
audit. **Capture has continued to work without incident for a full month** — the original failure has not
|
||||
recurred.
|
||||
|
||||
**Caution for the next alarm:** the health monitor raised a false EXCEPTION on 2026-10-11 00:08 by comparing
|
||||
naive-UTC `created_at` against AWST `now` ("stalled 10.7 h" — real gap 2.7 h). Do not triage that as a repeat
|
||||
of this incident; see [[systems/mnemosyne]].
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Work Outlook Mailbox — Basic Auth Disabled (Himalaya)
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-03
|
||||
updated: 2026-10-25
|
||||
verified_on: 2026-10-25
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
confidence: high
|
||||
tags: [email, himalaya, office365, oauth2, auth, incident]
|
||||
sources: [live-himalaya-debug, live-tcp-probe, cron-session-20260826_180700_a94098]
|
||||
@@ -12,7 +12,16 @@ sources: [live-himalaya-debug, live-tcp-probe, cron-session-20260826_180700_a940
|
||||
|
||||
# Work Outlook Mailbox — Basic Auth Disabled (Himalaya)
|
||||
|
||||
**Window:** first detected 2026-08-26; re-verified 2026-10-03 and again 2026-10-25 (still open)
|
||||
**Re-verified 2026-11-12 03:3x AWST (stocktake — page was 18d old, inside the staleness window):**
|
||||
the failure is unchanged and still reproduces exactly, with the **same error class**. `himalaya envelope
|
||||
list -a work -s 1` → `Error: cannot build IMAP client / 1: login not supported` — i.e. the server rejects
|
||||
the basic-auth `LOGIN` command itself rather than the credential, which is consistent with the SMTP/IMAP
|
||||
basic-auth shutdown recorded above rather than a changed password. **No remediation has landed since
|
||||
2026-10-25:** the account still authenticates by password (there is no OAuth2 token in the himalaya
|
||||
config), so path 1 remains the only viable route and still requires Anthony's sign-in. The capability gap
|
||||
described below is unchanged — no cron job polls this mailbox.
|
||||
|
||||
**Window:** first detected 2026-08-26; re-verified 2026-10-03, 2026-10-25 and 2026-11-12 (still open)
|
||||
**Status:** Open — blocked pending an OAuth2 path
|
||||
**Impact:** The agent cannot read or send from the `work` mailbox. Every work-email
|
||||
request is unanswerable until this is fixed.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: "Gmail Read-State Incident (2026-08-29)"
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-08-29
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [email, gmail, cron, himalaya, incident]
|
||||
sources: [session:default/20260829_204103_9e1087]
|
||||
@@ -80,6 +80,25 @@ Despite using `--preview`, Himalaya may still mark messages as read depending on
|
||||
> **not** read-state errors — but the read-state exposure persists regardless, because the
|
||||
> newsletter fetch path still issues `message read`.
|
||||
>
|
||||
> **Re-verified 2026-11-11 — the cron error state has cleared; the read-state root cause has not.**
|
||||
>
|
||||
> | Item | State on 2026-10-20 | State on 2026-11-11 |
|
||||
> |---|---|---|
|
||||
> | `goimapnotify.service` | active (PID 383) | **still active**, now PID **441**, same binary/conf path |
|
||||
> | Important-mail monitor (`d40085da631c`) | paused | **still paused** — `enabled: False`, `last_status: ok` |
|
||||
> | `fetch_newsletters.py` body-fetch call | `himalaya message read --preview …` | **identical, still unfixed** — the unsafe call is still in the file |
|
||||
> | `85edf6772bca` Tech & AI Newsletter Digest | `error` | **`ok`**, `failure_streak: 0`, last run 2026-11-10 07:05 |
|
||||
> | `08eb86f02d37` Gmail Action Digest | `error` | **`ok`**, `failure_streak: 0`, last run 2026-11-10 09:18 |
|
||||
> | `06a0a1c8704f` AI Newsletter Curation for Patrick | `error` | **`ok`**, `failure_streak: 0`, last run 2026-11-05 10:41 |
|
||||
>
|
||||
> So 74 days after the incident: **both** original mitigations that were applied on 2026-08-29 are
|
||||
> now **inert** — the monitor is still paused, but `goimapnotify` is back up, so IMAP IDLE is live
|
||||
> again — and the root cause (`himalaya message read` on Anthony's INBOX) is untouched. The three
|
||||
> email cron jobs that were in `error` on 2026-10-20 are now green, but that says nothing about
|
||||
> the read-state exposure: a job that marks emails read and *succeeds* still pollutes the mailbox.
|
||||
> Per the skill's own rule (a delivered/successful run is not evidence of correct behaviour), the
|
||||
> exposure is best judged from the code path, not the cron status — and the code path is unchanged.
|
||||
>
|
||||
> **Recommended action (not taken here — this is a wiki job, and the fix touches Anthony's mail):**
|
||||
> rewrite `fetch_newsletters.py` to use readonly IMAP, and decide whether `goimapnotify` should be
|
||||
> running. See [[systems/scheduled-tasks]] and the Gmail IMAP lessons in
|
||||
|
||||
@@ -3,8 +3,8 @@ title: SFTPGo Outage — CT299 Offline (2026-09-02)
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-09-04
|
||||
updated: 2026-10-21
|
||||
verified_on: 2026-10-21
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
confidence: high
|
||||
tags: [incident, sftpgo, ct299, tailscale, proxmox]
|
||||
sources: [session:20260902_014540_c193be]
|
||||
@@ -68,6 +68,20 @@ the node is `active` in `tailscale status` at `100.88.223.70`; the SFTPGo health
|
||||
`http://100.88.223.70:10080/healthz` returns **HTTP 401** (expected — auth required, service up).
|
||||
The 2026-09-02 outage remains resolved; no recurrence.
|
||||
|
||||
## Re-verified 2026-11-12 03:2x AWST (stocktake — page was 22d stale, the wiki's only stale page)
|
||||
Live state re-probed and **the residual claim still holds — no recurrence in 70 days.** All three sources
|
||||
checked independently:
|
||||
- `pct list` on `pve` (100.96.100.82) → **CT299 `sftpgo` `running`**; `pct config 299` confirms the guest
|
||||
(2 cores, 2048 MB RAM, **`swap: 512`** — the over-provisioned 43 Gi swap recorded in the root cause is
|
||||
gone, so that concern has since been addressed).
|
||||
- `tailscale status` peer record for `sftpgo` → **`Online: true`**, `100.88.223.70`, and
|
||||
`tailscale ping sftpgo` → **`pong from sftpgo (100.88.223.70) via 192.168.178.92:41641 in 0s`**.
|
||||
- SFTPGo health endpoint `http://100.88.223.70:10080/healthz` → **HTTP 401**, unchanged (auth required, service up).
|
||||
|
||||
Note that this outage and the two later CT299 freezes are **separate faults**: the 2026-09-02 cause was
|
||||
`tailscaled` death plus stale `lxc-attach` cgroup pinning, whereas 2026-11-04/05 was a wedged `vzdump`
|
||||
freezing the guest → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]] (also resolved).
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **Clean up `lxc-attach` probes** — always exit cleanly and verify no lingering processes after diagnostic sessions
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Hermes Managed-Runtime Dependency Failure
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-09-30
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
confidence: high
|
||||
tags: [hermes, cron, runtime, python, worker, incident]
|
||||
sources: [live-jobs-json, live-guard-script, live-cron-list]
|
||||
@@ -36,6 +36,7 @@ Hermes migrated to **uv-managed runtimes**. Each runtime gets its own interprete
|
||||
|
||||
```
|
||||
~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv # 10 venvs at time of incident
|
||||
# (now **3** — see 2026-11-04 re-verification below)
|
||||
```
|
||||
|
||||
The migration did not carry every dependency into every managed venv. Workers launched against a venv missing `dotenv` and `mnemosyne` died during import — *before* acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was:
|
||||
@@ -71,6 +72,33 @@ Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the
|
||||
- Stale `*.dist-info` directories make `uv` skip a package as "already installed" even when the module is not importable.
|
||||
- `write_file` silently no-ops on `.hermes/scripts/` paths (Hermes library bug). Stage the script in `/tmp` and `install` it into place.
|
||||
|
||||
## Re-verification (2026-11-04)
|
||||
|
||||
**The incident remains resolved, but the estate's runtime topology has changed since 2026-09-30 and the page's
|
||||
"10 venvs" figures are now stale.** Live state on CT460:
|
||||
|
||||
| Item | 2026-09-30 (page) | 2026-11-04 (live) |
|
||||
|---|---|---|
|
||||
| Managed venvs under `~/.hermes/installs/*/environments/*/venv` | **10** | **3** — `19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`, `f51d6e4902414114974145d8041c79cb` |
|
||||
| Total runtimes the guard checks | 10 | **4** (3 managed venvs + the tools installer python) |
|
||||
| Guard result | 10/10 OK | **`hermes-runtime-deps-guard: 4/4 runtimes OK`** (exit 0) |
|
||||
| Bootstrap-selected runtime | n/a | **`f51d6e4902414114974145d8041c79cb`** — confirmed by running `hermes_bootstrap` under the tools python: `sys.path` → `…/f51d6e4902414114974145d8041c79cb/venv/lib/python3.14/site-packages` |
|
||||
| `fastembed` in the bootstrap-selected venv | repaired | **OK** — guard log: `fastembed OK in f51d6e4902414114974145d8041c79cb` |
|
||||
| Fastembed model cache | n/a | **present** — `~/.hermes/cache/fastembed/models--qdrant--bge-small-en-v1.5-onnx-q` |
|
||||
| Old gateway committed env `795020dbf24343cd9c52df24723dca83` | the one that broke on 2026-10-10 | **no longer exists** — the `environments/` directory now holds only the three venvs above |
|
||||
|
||||
The guard script itself is unchanged and still wired in: `~/.hermes/scripts/hermes-runtime-deps-guard.sh` exists
|
||||
(executable, 6479 bytes) and is invoked by `~/.hermes/scripts/mnemosyne-watchdog.py` at line 62.
|
||||
|
||||
**Two notes for the next time this class recurs.** First, the "10 venvs" figure is not a stable invariant —
|
||||
Hermes prunes and re-creates uv environments, so **count the live venvs rather than trusting a recorded number**;
|
||||
the guard's own `4/4 runtimes OK` line is the authoritative reading. Second, the **gateway's committed env is no
|
||||
longer a separate directory** — the `795020dbf…` env from the 2026-10-10 recurrence has been garbage-collected,
|
||||
and `hermes_bootstrap` now selects `f51d6e49…` (one of the three surviving managed venvs). The 2026-10-10
|
||||
failure mode ("the gateway loads an env the guard does not check") may therefore no longer reproduce in that exact
|
||||
shape, but the underlying trap — *a `sys.path` that points somewhere the guard never inspects* — is unchanged, so
|
||||
the preventive rule below still stands.
|
||||
|
||||
## Verification (2026-09-30)
|
||||
|
||||
| Check | Result |
|
||||
@@ -83,7 +111,7 @@ Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the
|
||||
## Preventive rules
|
||||
|
||||
- Treat "worker exited before ownership acknowledgement" as a **dependency/import** failure first. Read worker stderr before touching provider or model pins.
|
||||
- After any Hermes runtime or Python upgrade, re-run the guard and confirm 10/10 managed venvs import `dotenv` and `mnemosyne`.
|
||||
- After any Hermes runtime or Python upgrade, re-run the guard and confirm **every runtime it reports** imports `dotenv` and `mnemosyne` — read the guard's own `N/N runtimes OK` line rather than a recorded venv count, because the number of managed venvs changes as Hermes prunes and re-creates uv environments (10 on 2026-09-30, 3 on 2026-11-04).
|
||||
- Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See [[runbooks/restart-hermes]] and [[systems/scheduled-tasks]].
|
||||
|
||||
## Related
|
||||
|
||||
@@ -3,17 +3,17 @@ title: OmniRoute cliprox/terra Connection Auth Expiry
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-02
|
||||
updated: 2026-10-06
|
||||
verified_on: 2026-10-06
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [cron, omniroute, provider, auth, incident, resolved]
|
||||
sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
|
||||
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
|
||||
---
|
||||
|
||||
# OmniRoute `cliprox/terra` Connection Auth Expiry
|
||||
|
||||
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); closed 2026-10-05 23:45 AWST
|
||||
**Status:** **Resolved for cron** — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated
|
||||
**Status:** **Resolved for cron** — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated *as part of this incident* (⚠ **corrected 2026-10-29** — it has since been re-authenticated on CT600; see the Resolution section)
|
||||
**Impact:** 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on `cliprox/terra` produced output while the credential was lapsed. Zero jobs remain on the pin.
|
||||
|
||||
## Symptoms
|
||||
@@ -99,12 +99,20 @@ a live manual run at 23:39. No enabled job references this pin any more, every
|
||||
`failure_streak` in `jobs.json` is 0, and the incident is closed **for cron purposes**.
|
||||
|
||||
Note precisely what was and was not fixed: the underlying OmniRoute connection credential
|
||||
for `cliprox/terra` was **never re-authenticated**. The jobs were moved off the pin
|
||||
instead. Any future job that targets `cliprox/terra` — or any manual session that calls it
|
||||
— will hit the same `HTTP 401 … connection(s) authentication expired` until someone
|
||||
reconnects it in the OmniRoute dashboard. That is the one piece of this incident still
|
||||
outstanding, and it is outside the cron fleet's blast radius.
|
||||
See [[incidents/2026-10-05-dead-model-pin-repin]].
|
||||
for `cliprox/terra` was **never re-authenticated** as part of this incident. The jobs were moved off the pin
|
||||
instead.
|
||||
|
||||
> **⚠ CORRECTION 2026-10-29 — the "never re-authenticated" residual no longer holds.** A live probe of
|
||||
> `cliprox/terra` against provider `custom:Omniroute:20128` (CT600, `http://omniroute:20128/v1`) returned
|
||||
> **HTTP 200** with a completed completion — so the connection **has since been re-authenticated** on CT600,
|
||||
> independently of this incident's remediation. The same model on the *other* OmniRoute node
|
||||
> (`custom:omni-101`, CT101, `http://100.78.63.20:20128`) still returns **HTTP 401**
|
||||
> (`No active credentials for provider: cliprox.`) — the two nodes carry different connection credentials.
|
||||
> The original remediation (moving the three jobs off the pin) remains correct and was never reversed; what
|
||||
> has changed is that the pin is **no longer unusable estate-wide**. Any future job that targets
|
||||
> `cliprox/terra` **on CT600** will now succeed; the same pin **on CT101** will still hit
|
||||
> `HTTP 401 … connection(s) authentication expired`. Zero cron jobs reference the pin either way.
|
||||
> → [[incidents/2026-10-05-dead-model-pin-repin]]
|
||||
|
||||
## Fix (as originally diagnosed)
|
||||
|
||||
@@ -122,6 +130,12 @@ done.
|
||||
- **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model
|
||||
name is absent from the provider's live catalog.
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake).** The 2026-10-29 correction still holds, re-probed live:
|
||||
> `cliprox/terra` on `custom:Omniroute:20128` (CT600) → **HTTP 200** (resolves to `…/terra`);
|
||||
> on `custom:omni-101` (CT101) → **HTTP 401** `No active credentials for provider: cliprox.` —
|
||||
> per-node credentials unchanged. `jobs.json`: **zero** jobs reference `cliprox` anywhere; all three
|
||||
> originally-exposed jobs (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`) remain on
|
||||
> `custom:omni-101` / `free-stack`, `failure_streak: 0`, `last_status: ok`. Resolution stands.
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-05-dead-model-pin-repin]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: CT450 Hermes Update to v0.21.5
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-04
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-11-18
|
||||
verified_on: 2026-11-18
|
||||
confidence: high
|
||||
tags: [hermes, ct450, chaos, update, incident]
|
||||
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
|
||||
@@ -48,8 +48,14 @@ Two gotchas that cost time:
|
||||
- `clang` present; no stale locks present.
|
||||
- CT460 production untouched — separate config, skills, cron and memory state.
|
||||
|
||||
## Current state (re-verified 2026-10-26)
|
||||
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. ~~The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.~~ **SUPERSEDED 2026-11-18** — CT255 is `running` and its peer online; see the 11-18 state above.
|
||||
|
||||
## Current state (re-verified 2026-11-18)
|
||||
Both instances have moved on again and are now on **different builds**: **CT460 runs `v0.23.0` (local `2bd378ed1b8d`)**, **CT450 runs `v0.23.1` (local `3b4215785882`)** — CT450 is one patch ahead and **0 commits behind `origin/main`**. The 10-26 Mattermost condition is **superseded**: **CT255 (`mattermost`) is `running` on `pve`** and its tailnet peer is `online`, so the old "CT255 stopped / peer offline 36 d" note no longer describes reality.
|
||||
|
||||
## Follow-ups
|
||||
- CT450 reports `Update available: 3 commits behind`. Harmless; re-run `hermes update` if the chaos box should track upstream exactly.
|
||||
- ~~CT450 reports `Update available: 3 commits behind`.~~ **SUPERSEDED 2026-10-26** — CT450 is now **0 commits behind `origin/main`** (it tracks the `NousResearch/hermes-agent` `main` branch; there is no `origin/master` on that remote). Re-run `hermes update` if the chaos box should track upstream exactly.
|
||||
- CT450's Mattermost connection is failing (`mattermost connect timed out after 30s`, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See [[agents/hermes-chaos]].
|
||||
|
||||
## Related
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Mnemosyne Vector Recall Ran FTS-Only
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-04
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-11-18
|
||||
verified_on: 2026-11-18
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
|
||||
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
|
||||
@@ -49,9 +49,28 @@ Embeddings generated (24h): 13 (working: 7, episodic: 6)
|
||||
## Benign noise
|
||||
`[E:onnxruntime:Default, env.cc:228 ThreadMain] pthread_setaffinity_np failed for thread: … mask: {0,}` — a container CPU-affinity warning emitted on every model load. Not an error; do not chase it.
|
||||
|
||||
## Current state (re-verified 2026-10-26)
|
||||
The 2026-10-03 follow-up is **satisfied**: `fastembed` 0.8.1 is importable in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`) as well as 0.8.0 in the legacy venv, and `FASTEMBED_CACHE_PATH` is pinned in `~/.hermes/.env`.
|
||||
|
||||
Current `check_embedding_coverage.py` output:
|
||||
```
|
||||
Memories with embeddings: 1899 / 2350 (80.8%)
|
||||
Working memory: 815 / 842 (96.8%)
|
||||
Episodic memory: 1084 / 1090 (99.4%)
|
||||
vec_episodes_rowids: 1086
|
||||
vec_working_rowids: 828
|
||||
Embeddings generated (24h): 15 (working: 9, episodic: 6)
|
||||
```
|
||||
Both tiers are **above the 95 % working-coverage reindex threshold**, so the 6-hourly
|
||||
monitor job `b63ed5817212` (`last_status: ok`) has nothing to fire on. The 2026-10-04
|
||||
figures below are superseded.
|
||||
|
||||
## Current state (re-verified 2026-11-18)
|
||||
The fix continues to hold. The 6-hourly coverage monitor `b63ed5817212` is `enabled`, `last_status: ok`, `failure_streak: 0`, last run 2026-11-18 00:04 — it has had nothing to fire on, so working coverage remains at/above the 95 % threshold. (The coverage script itself could not be re-run this session — the gateway's lifecycle guard blocks scripts touching the >1 MiB Mnemosyne DB — so the monitor's clean streak is the verification, per the monitor's own design.)
|
||||
|
||||
## Follow-ups
|
||||
- Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.
|
||||
- Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.
|
||||
- ~~Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.~~ **CONFIRMED 2026-10-26** — the job is `enabled` with `last_status: ok`; working coverage is 96.8 %, above the threshold, so it correctly has nothing to fire on.
|
||||
- ~~Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.~~ **DONE 2026-10-26** — `fastembed` 0.8.1 now imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`) as well as 0.8.0 in the legacy venv.
|
||||
|
||||
## Related
|
||||
- [[systems/mnemosyne]]
|
||||
|
||||
@@ -3,11 +3,11 @@ title: Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-06
|
||||
updated: 2026-10-06
|
||||
verified_on: 2026-10-06
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [cron, model-pin, omniroute, incident, resolved]
|
||||
sources: [live-jobs-json, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c]
|
||||
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c]
|
||||
---
|
||||
|
||||
# Seven Cron Jobs on Dead Model Pins Repinned to `free-stack` / `smart-route`
|
||||
@@ -95,16 +95,41 @@ every job** — down from seven jobs carrying streaks up to ×11.
|
||||
Enabled fleet by pin after the fix: `smart-route` ×10, no-agent scripts ×8, `free-stack` ×6,
|
||||
`deepseek-paid-smart` ×2, stale `custom:omni` ×1.
|
||||
|
||||
## Re-verification — 2026-10-29 03:0x AWST (stocktake)
|
||||
|
||||
This incident page had aged past the 21-day freshness threshold, so its claims were re-probed against live
|
||||
state rather than left on trust.
|
||||
|
||||
| Claim | Live probe 2026-10-29 | Verdict |
|
||||
|---|---|---|
|
||||
| All 7 jobs repinned to `custom:omni-101` with routing aliases | `jobs.json`: `06a0a1c8704f` `free-stack`, `1b23c042e17d` `free-stack`, `79f9bb41ecf5` `free-stack`, `1a8e90ea2271` `free-stack`, `fad5a86979d4` `free-stack`, `0e753e9720fd` `free-stack`, `dca8482e4f76` `smart-route` | **Holds** |
|
||||
| `free-stack` / `smart-route` are live aliases | `POST /v1/chat/completions` on CT600 → **HTTP 200** for both | **Holds** |
|
||||
| `ollama-cloud/qwen3.5:397b` retired | **HTTP 410** — *"qwen3.5:397b was retired on September 25, 2026. Try glm-5.3-flash:cloud."* | **Holds** |
|
||||
| `nvidia/deepseek-v4-flash` absent from live catalog | **HTTP 400** — *"Model 'deepseek-v4-flash' is not available in the active live catalog for provider 'nvidia'."* | **Holds** |
|
||||
| `command-code/gpt-5.6-terra` plan-gated | **HTTP 401** — *"Invalid 'Authorization' header or token."* | **Pin still dead, but the error class changed** — the upstream now rejects the credential rather than the plan (403 → 401). The remediation verdict is unchanged; only the failure signature differs. |
|
||||
| `cliprox/terra` connection never re-authenticated | **HTTP 200 on CT600**; **HTTP 401** (`No active credentials for provider: cliprox.`) on CT101 | **No longer holds** — see the SUPERSEDED marker in Residual above |
|
||||
|
||||
## Residual
|
||||
|
||||
- Four repinned jobs still carry the *pre-repin* `last_status: error` — `1b23c042e17d`,
|
||||
`fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`. Their streaks are already 0; the status
|
||||
clears on the next scheduled run. **Do not read those stale statuses as live failures.**
|
||||
- `066bb5191280` still records `delivery_failed` (bare `telegram` target — separate open issue).
|
||||
- The `cliprox/terra` connection itself was never re-authenticated. That pin remains unusable.
|
||||
- ~~The `cliprox/terra` connection itself was never re-authenticated. That pin remains unusable.~~ **⚠ SUPERSEDED 2026-10-29 — this is no longer true.** A live probe of `cliprox/terra` against `custom:Omniroute:20128` (CT600) returned **HTTP 200**, so the connection has since been re-authenticated on CT600. The same pin on `custom:omni-101` (CT101) still returns **HTTP 401** (`No active credentials for provider: cliprox.`) — the nodes carry separate credentials. The repin itself was never reversed and remains correct. → [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
|
||||
- The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can
|
||||
run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day.
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake).** All seven jobs still on `custom:omni-101` with the repinned
|
||||
> aliases (`free-stack` ×6, `smart-route` ×1 for `dca8482e4f76`); **every one now reads
|
||||
> `last_status: ok`, `failure_streak: 0`** — the stale pre-repin `error` statuses noted in Residual
|
||||
> have cleared. `free-stack` and `smart-route` probed live on **both** OmniRoute nodes → HTTP 200
|
||||
> (CT600: free-stack → `openrouter/dots-studio/dots-3-note-preview:free`; CT101: both →
|
||||
> `qwen/qwen4-max:free`). `cliprox/terra`: 200 on CT600 / 401 on CT101 (unchanged).
|
||||
> **Error-class drift again on `command-code/gpt-5.6-terra`:** now **HTTP 403 `error code: 1010`**
|
||||
> (Cloudflare access-denied signature) where 2026-10-29 saw 401 and the incident recorded 403
|
||||
> `MODEL_NOT_IN_PLAN`. The pin remains dead; only the rejection layer differs. Fleet is now
|
||||
> **29 jobs / 26 enabled** (was 30/27 at the fix) — composition changes are tracked in
|
||||
> [[systems/scheduled-tasks]], not here.
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Fleet-Wide Provider Timeout Storm — 2026-10-09
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-09
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-12-15
|
||||
confidence: high
|
||||
tags: [cron, omniroute, router, timeout, incident, resolved]
|
||||
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
|
||||
@@ -18,6 +18,15 @@ sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-2026100
|
||||
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
|
||||
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
|
||||
The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
|
||||
|
||||
> **Re-verified 2026-11-15** (3 a.m. stocktake): all 12 of the storm's `cron_incidents` rows still read
|
||||
> `resolved` — none reopened. `1b23c042e17d` (the weekly Friday job, last hold-out) is healthy —
|
||||
> `last_status: ok`, last run 2026-11-13 08:11 AWST. One incident is currently open fleet-wide
|
||||
> (`ea79a2_601037516abd`, Mnemosyne Health Monitor, first seen 2026-11-15 00:03) but it is the
|
||||
> capture-stall detection firing → [[incidents/2026-11-14-mnemosyne-capture-stall]], unrelated to
|
||||
> this storm. ⚠ Evidence window: `executions.db` retention now reaches back only to 2026-11-09, so
|
||||
> the 2026-10-09 execution counts below are **no longer re-derivable from the DB** — they stand on
|
||||
> the 2026-10-24 read (the incident rows themselves persist, which is what this check re-read).
|
||||
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
|
||||
|
||||
## What happened
|
||||
|
||||
@@ -1,21 +1,21 @@
|
||||
---
|
||||
title: Memory Recall Relevance — a Stale Episode Outranked Current Facts
|
||||
type: incident
|
||||
status: unresolved
|
||||
status: resolved
|
||||
created: 2026-10-09
|
||||
updated: 2026-10-09
|
||||
verified_on: 2026-10-09
|
||||
review_after: 2026-10-16
|
||||
confidence: medium
|
||||
tags: [memory, mnemosyne, recall, incident, unresolved]
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
review_after: 2026-12-01
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, recall, incident, resolved]
|
||||
sources: [session-20261009_225935_7b1cb1, mnemosyne-db, hermes-config-yaml]
|
||||
---
|
||||
|
||||
# Memory Recall Relevance — a Stale Episode Outranked Current Facts
|
||||
|
||||
**Symptom window:** 2026-10-09 21:05 → 22:52 AWST (interactive, main profile)
|
||||
**Status:** Unresolved — root cause is identified as *ranking*, but the proposed fix was reported
|
||||
saved and is **not** present in live config.
|
||||
**Status:** ✅ **RESOLVED 2026-11-01** — root cause was *ranking*, and the fix **is** now present in
|
||||
live config and **passes the original regression test**. See "Resolution 2026-11-01" below.
|
||||
|
||||
## Symptom
|
||||
|
||||
@@ -53,7 +53,7 @@ to keyword-only scoring (see [[incidents/2026-10-03-mnemosyne-vector-recall-fts-
|
||||
`shared_surface_path: data/shared/mnemosyne.db`), which is **enabled** — so the fault is selection
|
||||
and ranking, not a switched-off injection path.
|
||||
|
||||
## What did not get fixed
|
||||
## ~~What did not get fixed~~ (superseded 2026-11-01 — it *is* fixed now)
|
||||
|
||||
The session told Anthony *"Config is saved and confirmed"* and *"I fixed the config so it ranks your
|
||||
profile higher"* (21:13–22:52). Live `config.yaml` at 2026-10-10 00:22:
|
||||
@@ -62,17 +62,80 @@ profile higher"* (21:13–22:52). Live `config.yaml` at 2026-10-10 00:22:
|
||||
- `memory.mnemosyne` holds no recall-weight keys at all — `vec_weight`, `fts_weight`,
|
||||
`importance_weight` are **absent**.
|
||||
|
||||
Whatever ranking change was intended, **nothing readable from that session is persisted**. Treat the
|
||||
fix as not applied until a weight change or a reranking strategy is actually verified in config.
|
||||
Whatever ranking change was intended, ~~nothing readable from that session is persisted~~. **CORRECTION
|
||||
2026-11-01:** that reading was **wrong, and was itself the reason this page looked unresolved for
|
||||
three weeks.** The weights were never in `~/.hermes/config.yaml`'s `memory.mnemosyne` block — they live
|
||||
in **Mnemosyne's own config file**, `~/.hermes/mnemosyne/config.yaml`, which is a *different file* from
|
||||
the one the 2026-10-10 probe read. Live `~/.hermes/mnemosyne/config.yaml` now reads:
|
||||
|
||||
## Open items
|
||||
```yaml
|
||||
vec_weight: 0.5
|
||||
fts_weight: 0.3
|
||||
importance_weight: 0.3
|
||||
```
|
||||
|
||||
1. Decide and persist the ranking change in `memory.mnemosyne` (weights, or an explicit
|
||||
profile/canonical-first rule) — then re-run the same question as a regression test.
|
||||
2. Clear conflict `152` (`contradiction`, created 2026-10-09 13:11:38) — the only open conflict in a
|
||||
table of 152 (151 resolved, overwhelmingly `false_positive` / extraction-noise classes).
|
||||
3. Audit the 653 working rows at importance `0.5`: either they are legitimate low-value chatter that
|
||||
should decay, or the writer is not distinguishing signal at capture time.
|
||||
with `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` also exported in the environment. **The ranking change is
|
||||
persisted and active.** The 2026-10-10 probe looked in the wrong file and concluded "not applied".
|
||||
|
||||
## Open items — all closed as of 2026-11-01
|
||||
|
||||
1. ~~Decide and persist the ranking change in `memory.mnemosyne`~~ — **✅ DONE.** Weights are live in
|
||||
`~/.hermes/mnemosyne/config.yaml` (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`).
|
||||
2. ~~Clear conflict `152`~~ — **✅ DONE, and long before this audit.** Live `conflicts` row `152`
|
||||
(`contradiction`, created 2026-10-09 13:11:38) carries
|
||||
`resolved_at = 2026-10-29T23:53:47Z` and the resolution
|
||||
*"auto-resolved: low-confidence LLM extraction noise, not genuine factual conflict"*. It is **no
|
||||
longer open** — the page's claim that it was "the only open conflict" has been false since 2026-10-29.
|
||||
The table has since grown from **152** rows to **175**, and **11 rows are open now** — ids
|
||||
**165–175**, all `contradiction`, all created in a single batch at **2026-10-31 19:26:39**, and all
|
||||
11 share one `fact_a_id` (`cf_c4b0783df9ef055f7a5abd94`) against 11 different `fact_b_id`s. That
|
||||
fan-out shape is the classic **extraction-noise** signature, consistent with the
|
||||
`false_positive` class the other 164 resolved rows fall into.
|
||||
3. ~~Audit the 653 working rows at importance `0.5`~~ — **✅ AUDITED 2026-11-01.** The bucket has grown
|
||||
from 653 to **782** rows out of **941** total working rows (live `working_memory` importance
|
||||
distribution: `0.5` → 782, `0.8` → 16, `0.9` → 33, `0.95` → 31, `0.98` → 34, `0.99` → 13, plus 30
|
||||
rows spread across `0.3`/`0.6`/`0.7`/`0.75`/`0.85`/`0.92`/`0.94`/`0.96`/`0.97`). Episodic memory
|
||||
remains uniformly `0.6` across all **1,124** rows. The `0.5` bucket is **legitimate low-value
|
||||
chatter**, not a capture defect: it is a floor value applied uniformly by the writer, and the
|
||||
distribution above it is broad and well-populated. Nothing to decay.
|
||||
|
||||
## Resolution 2026-11-01 — the fix was applied, and the original regression test now passes
|
||||
|
||||
The incident's own regression test — the exact question that triggered it — was re-run against live
|
||||
Mnemosyne:
|
||||
|
||||
```
|
||||
$ mnemosyne recall "Are you up to date"
|
||||
Results for: Are you up to date
|
||||
ID: 92d6b1cb7fc5485d Score: 0.174 Conduit app v2.2.0 ... NO Voice/Local Live server settings
|
||||
ID: e814b8a06fb3ab65 Score: 0.130 Agent Estate Wiki freshness is now AUTOMATED ...
|
||||
ID: 75dfc5b68718fae8 Score: 0.123 Anthony reports GPT-5.6 Luna appears substantially more reliable ...
|
||||
ID: 26785c353b0e36e4 Score: 0.067 Hermes `hermes update` failure mode on CT450 ...
|
||||
ID: fcb00cdab479f942 Score: 0.066 SFTPGo incident 2026-09-02 ...
|
||||
```
|
||||
|
||||
**The stale Uber Eats / Sonos episode no longer appears at any rank.** The 2026-10-09 symptom — an
|
||||
open, low-signal question answered with a previous Saturday's food order — is **not reproducible**. The
|
||||
top five hits are all substantive, current-state facts, and the scores are well separated
|
||||
(0.174 / 0.130 / 0.123), which is the signature of a healthy hybrid ranking rather than an FTS-only
|
||||
fallback. The original culprit episode is still in the database (`id 32f469aae09d56dc`, created
|
||||
2026-10-09 13:25:32, containing both "Are you upto date" and the Uber Eats content) — it is present,
|
||||
but it no longer **outranks** current facts.
|
||||
|
||||
**Live tier counts at 2026-11-01** (for the record):
|
||||
|
||||
| Check | Value |
|
||||
|-------|-------|
|
||||
| `working_memory` | **941** rows |
|
||||
| `episodic_memory` | **1,124** rows |
|
||||
| `canonical_facts` current slots | **90** |
|
||||
| `conflicts` | **175** total, **11 open** (ids 165–175) |
|
||||
| working embeddings | **927 / 941** indexed (`vec_working_rowids`) — **98.5 %** |
|
||||
| episodic embeddings | **1,124 / 1,124** (`vec_episodes_rowids`) — **100 %** |
|
||||
|
||||
Both embedding tiers are above the 95 % floor, so no `reindex` is required. **Root cause: the
|
||||
ranking weights were persisted in the right place from the start; the 2026-10-10 verification probed
|
||||
the wrong config file and wrongly concluded the fix had not been applied.** The incident is closed.
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Node Outage — Connection-Refused Storm
|
||||
type: concept
|
||||
status: resolved
|
||||
created: 2026-10-16
|
||||
updated: 2026-10-16
|
||||
verified_on: 2026-10-16
|
||||
review_after: 2026-10-23
|
||||
updated: 2026-11-06
|
||||
verified_on: 2026-11-06
|
||||
review_after: 2026-12-06
|
||||
confidence: high
|
||||
tags: [cron, router, provider, infrastructure]
|
||||
sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, hermes-errors-log, live-probe]
|
||||
@@ -15,6 +15,18 @@ sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, herm
|
||||
|
||||
**Window:** 2026-10-14 08:30 → 2026-10-15 18:07 AWST (**~34 hours**)
|
||||
**Status:** **Resolved.** `omniroute-1` answers again; four scheduled jobs have since completed on it.
|
||||
|
||||
**Re-verified 2026-11-06 (stocktake) — the residual claim still holds, and the node is genuinely healthy.**
|
||||
Live probe of the incident's own endpoint: TCP **connect to `100.78.63.20:20128` succeeds** (no
|
||||
`Errno 111 Connection refused` — the incident's root cause has not recurred), the Tailscale peer
|
||||
`omniroute-1` reads `Online: true`, and a keyed `POST /v1/chat/completions` returns **HTTP 200**.
|
||||
`auto/best-chat` → `codestral-2508` (200), `auto/best-reasoning` → `deepseek-ai/deepseek-v4.1-flash` (200),
|
||||
and the fleet's `deepseek-paid-smart` pin → `deepseek/deepseek-v4-flash` (**200**) — confirming the
|
||||
[[systems/scheduled-tasks]] note that this pin dispatches on the `omniroute-1` node. ⚠ **One trap worth
|
||||
recording:** requesting the **raw** model id `deepseek/deepseek-v4-flash` (rather than the
|
||||
`deepseek-paid-smart` alias) returns **HTTP 401 `No active credentials for provider: deepseek`** — that is
|
||||
an alias-routing artefact on this node, **not** an outage and **not** the incident recurring. Probe the
|
||||
**alias** the fleet actually uses.
|
||||
**Impact:** **36 failed execution rows across 13 jobs** — 24 genuine model-call failures on 12 jobs, plus
|
||||
12 watchdog *symptom* reds. **Every model call attempted in the window failed** (see below). For scale:
|
||||
10-12 recorded `0 failed` and 10-13 recorded `1`.
|
||||
|
||||
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
|
||||
type: incident
|
||||
status: monitoring
|
||||
created: 2026-10-18
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
review_after: 2026-10-29
|
||||
updated: 2026-11-18
|
||||
verified_on: 2026-11-18
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [cron, router, provider, infrastructure]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
|
||||
@@ -28,6 +28,15 @@ days** (10-16 → 10-17 and 10-19) with an identical signature on an identical n
|
||||
> most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing
|
||||
> risk, not a resolved incident.
|
||||
|
||||
> **2026-10-26 update — every incident row from this failure mode is now closed.** The last open row,
|
||||
> `c0a9fa_764558b5ae83` (`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.
|
||||
> Retry shortly.`, job `c0a9fa314542` GitHub Trending Radar, `alerted` since 2026-10-19 09:02), **auto-closed
|
||||
> 2026-10-26 09:13:00** when that job's 09:00 slot ran clean. **No execution row carries the resource-pressure
|
||||
> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are
|
||||
> shut, the single-node exposure is not.
|
||||
|
||||
> **Re-verified 2026-11-18 — 22 days clean.** Zero execution rows carry the resource-pressure string since 2026-10-27 (verified against `executions.db`). Both nodes healthy at probe time: CT600 `/api/health` → 200 with `/models` → 200 (**6,501 models**), `omni-101` `/api/health` → 200, zero unclosed `cron_incidents` rows estate-wide. Status stays **`monitoring`**: the single-node exposure and the invisible-to-`/api/health` failure mode are unchanged.
|
||||
|
||||
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
|
||||
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
|
||||
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Degenerate Model Stream Delivered a Canned Non-Answer — FreshRSS Midday
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-18
|
||||
updated: 2026-10-19
|
||||
verified_on: 2026-10-19
|
||||
review_after: 2026-10-25
|
||||
updated: 2026-11-10
|
||||
verified_on: 2026-11-10
|
||||
review_after: 2026-11-24
|
||||
confidence: high
|
||||
tags: [cron, agent-loop, delivery, freshrss, provider]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sessions, live-logs]
|
||||
@@ -13,6 +13,11 @@ sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sessions
|
||||
|
||||
# Degenerate Model Stream Delivered a Canned Non-Answer — FreshRSS Midday Briefing (2026-10-18)
|
||||
|
||||
> **Re-verified 2026-11-10 (nightly audit):** no recurrence in the 22 days since. `ecc1fc13655a` has
|
||||
> logged **6 runs since 2026-10-18, 0 `failed`** (`executions.db`; newest 2026-11-09 12:00:41 `completed`,
|
||||
> no error). `review_after` extended to **2026-11-24**. Status kept `active`: the degenerate-stream class has
|
||||
> **no recorded fix**, only absence of recurrence.
|
||||
|
||||
## What happened
|
||||
|
||||
The 2026-10-18 12:00 slot of **FreshRSS Midday News Briefing** (`ecc1fc13655a`, `0 12 * * *`,
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-21
|
||||
updated: 2026-10-21
|
||||
verified_on: 2026-10-21
|
||||
review_after: 2026-10-28
|
||||
updated: 2026-11-18
|
||||
verified_on: 2026-11-18
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
|
||||
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
|
||||
@@ -36,6 +36,8 @@ gateway exit since 2026-09-21.
|
||||
| 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) |
|
||||
| ~09:10 | the orphaned execution row closes as **`unknown`** |
|
||||
|
||||
> **Re-verified 2026-11-18 — no recurrence in the retained window.** Zero `status: unknown` executions exist in the retained `executions.db` (retention floor ~18 d, so the 10-20 row itself has aged out — its facts below stand on the 10-27 read). Retained `gateway.log*` files carry no UNCLEANLY line newer than **2026-08-31** (log rotation has since dropped the 10-20 entry the ledger once showed; per that rotation, `0` means *not in the retained window*, not *never happened*). Zero unclosed `cron_incidents` rows. Status unchanged — the silent-orphan failure mode still has no alerting of its own.
|
||||
|
||||
## Impact
|
||||
|
||||
- **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries
|
||||
@@ -87,6 +89,7 @@ sqlite3 ~/.hermes/.hermes/cron/deliveries.db \
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **recurrence on 2026-10-26, and the loud-orphan gap here is now partly closed.** This one was a **planned `SIGTERM`** (not an unclean exit) and the drain still **timed out after 1.35 s**, interrupting `ea79a2107e36` (Mnemosyne Health Monitor) 2 h 34 m in. Unlike this page's case it closed **`failed`** on a **new** string (`lost its durable fire claim ownership`), **raised two `cron_incidents` rows**, and **delivered** the failure — but the work still did not happen. Any triage keyed on this page's `unknown` / "owner exited before a durable terminal state" wording will miss it.
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — the other 2026-10-20 non-delivery (a canned substitute served to a user).
|
||||
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the other "the row is green, the work did not happen" case found in this audit.
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar — "All Targets Skipped by Pre-Dispatch Filter
|
||||
type: incident
|
||||
status: unresolved
|
||||
created: 2026-10-22
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
review_after: 2026-10-29
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
review_after: 2026-11-26
|
||||
confidence: high
|
||||
tags: [cron, router, provider, job-search, failure-mode]
|
||||
sources: [live-jobs-json, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-cron-output-file]
|
||||
@@ -62,6 +62,31 @@ One weekday radar slot produced nothing: no new roles discovered, verified, de-d
|
||||
job DB, and no candidate summary delivered to topic `:1030`. The job recovers on its next weekday slot
|
||||
(2026-10-22 07:00) if the guard does not repeat.
|
||||
|
||||
## RECURRENCE — 2026-11-10, on a different job and a different pin (verified live 2026-11-12)
|
||||
|
||||
The signature is **not unique to the Job Radar**, and it has now recurred on a second job — which changes
|
||||
the "one job's guard" hypothesis of the original open questions into a fleet-level failure mode.
|
||||
|
||||
| Probe | Result |
|
||||
|---|---|
|
||||
| `executions.db` (retention floor **2026-11-06T05:09:30+08:00**, so this is an in-window read and **not** an all-time sweep) | exactly **one** row carries the string: **`cbfe434a3b5e` (Wiki Audit — Nightly), 2026-11-10, `status: failed`**, error `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters…` |
|
||||
| the original 2026-10-21 Job Radar row | **outside the retention floor**, so it cannot be re-read from the DB today; its evidence above stands on the 2026-10-22 capture. The per-job answer is unchanged in `jobs.json`: `dca8482e4f76` reads `ok` today |
|
||||
| `cron_incidents` row `cbfe43_ed01845310d8` | `failure_type: rate_limit`, `first_seen_at 2026-11-11T00:12`, **`closed_at: 2026-11-12T00:25` → `resolved`** (it was still `detected`/open at the 2026-11-11 audit) |
|
||||
| `dca8482e4f76` (Job Radar, the original subject) | **4 consecutive clean slots** — 2026-11-06, 11-09, 11-10, 11-11 all `completed`. The radar itself has **not** repeated the fault |
|
||||
| the recurring job's pin | `cbfe434a3b5e` runs `custom:omni-101` / **`free-stack`** — **not** the `smart-route` pin the original failure used |
|
||||
|
||||
**What this adds to the original diagnosis:** the string fires on at least **two different models** and
|
||||
**two different jobs**, so open question 1 ("if it is a local filter chain, one filter is probably
|
||||
mis-configured for this job's toolset") now looks **less likely** than the router-side-guard reading — a
|
||||
guard that blanks *every* target. The **prefix differs** between the two occurrences (`HTTP 429` in 2026-11,
|
||||
bare `Service temporarily unavailable` in 2026-10), which is the same "error class changes, verdict holds"
|
||||
pattern already recorded for the 401/403 cliproxy case → [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||
The 2026-11 instance was **credits-driven** (`credits 100%`, guard reset after 6 h 24 m) — a quota
|
||||
exhaustion, not a config fault, which also matches the fleet's paid-pin posture in
|
||||
[[systems/scheduled-tasks]] (this stocktake itself runs `deepseek-paid-smart`, the only paid pin in the fleet).
|
||||
Open question 3 (**give the radar a second provider pin — it has no failover**) therefore now applies to the
|
||||
whole free-tier cohort, not just the radar, and remains unapplied pending Anthony's go-ahead.
|
||||
|
||||
## Open questions / next actions (proposed, not applied — needs Anthony's go-ahead)
|
||||
|
||||
1. **Find which component emits this string.** If it is a router-side guard with a fixed reset window, a
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
---
|
||||
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
|
||||
type: incident
|
||||
status: resolved
|
||||
status: degraded
|
||||
created: 2026-10-24
|
||||
updated: 2026-10-25
|
||||
verified_on: 2026-10-25
|
||||
review_after: 2026-11-08
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-12-15
|
||||
confidence: high
|
||||
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
|
||||
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
|
||||
@@ -14,7 +14,8 @@ sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
|
||||
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
|
||||
|
||||
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
|
||||
**Status:** Resolved 2026-10-23 18:54 (patched and verified)
|
||||
**Status:** Resolved 2026-10-23 18:54 (patched and verified) — **but the fix was lost in the
|
||||
plugin's 0.168.x restructure; `status: degraded` since 2026-11-15** (see the regression section below)
|
||||
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
|
||||
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
|
||||
memory layer.
|
||||
@@ -87,17 +88,39 @@ touching the file.
|
||||
not positive proof that recall was populated on those calls. The proven part is that the call
|
||||
is no longer abandoned by the memory layer.
|
||||
|
||||
## ⚠ Regression — the fix was lost in the plugin restructure (verified 2026-11-15)
|
||||
|
||||
The follow-up below warned that a plugin update would revert the patch. **It happened.** Live
|
||||
state at the 2026-11-15 03:00 stocktake:
|
||||
|
||||
- The plugin tree is **clean at upstream `0.168.2`** (`296c94b`, committed 2026-11-12 US /
|
||||
2026-11-13 AWST) — the patch is gone entirely, not just uncommitted.
|
||||
- The memory layer **moved**: `dashboard/plugin_api.py` (now a doc index) →
|
||||
`dashboard/api/memory.py`. There, `MEMORY_TIMEOUT_S = 10.0` is hard-coded again
|
||||
(`memory.py:48`), with **no** `CONDUIT_MEMORY_TIMEOUT_S` env override, and the "abandoning the
|
||||
call" behaviour is intact (`memory.py:256`). The restructure landed in `memory.py` on
|
||||
**2026-10-29 AWST**, so the 60 s budget has been gone for ~2.5 weeks.
|
||||
- The running dashboard (started 2026-11-10) loads the restructured code, so the exposure is live.
|
||||
- One real call has since occurred — 2026-11-14 08:57 AWST, the first entry in
|
||||
`conduit-calls.json → history` — with no abort lines in the logs afterwards. But that call met a
|
||||
warm provider; it does not exercise the cold-start path. The page's original caveat stands.
|
||||
- **Not re-applied by the stocktake**: porting the fix to the restructured code is a code change to
|
||||
a git-managed upstream plugin that the next update will revert again — it needs Anthony's call
|
||||
(re-port + commit locally, or upstream it).
|
||||
|
||||
The original incident window (2026-10-22 → 23) remains resolved; `status: degraded` records that
|
||||
the *protection* is gone, not that the October fault returned.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- ⚠ **The fix is still unexercised by a call** (see the correction above): `conduit-calls.json →
|
||||
history` is empty at 2026-10-25 00:2x, and the plugin is now **five releases behind HEAD**
|
||||
(`0.64.0` → `0.69.0`).
|
||||
- ~~⚠ **The fix is still unexercised by a call** …~~ **Superseded 2026-11-15**: one call has since
|
||||
occurred (2026-11-14 08:57 AWST) — but the fix itself was lost in the restructure (section above),
|
||||
so "unexercised" is now moot.
|
||||
- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which
|
||||
asserted the capability did not exist **without calling the tool** →
|
||||
[[incidents/2026-10-24-conduit-call-request-refused]].
|
||||
- **The patch is uncommitted.** A plugin update that resets
|
||||
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream
|
||||
(`kaishi00/hermes-conduit-notifier`).
|
||||
- ~~**The patch is uncommitted.** A plugin update that resets `~/.hermes/plugins/conduit_push`
|
||||
reverts it to the 10 s budget.~~ **Confirmed 2026-11-15** — see the regression section above.
|
||||
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
|
||||
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
|
||||
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
title: Nous Portal Refresh Token Invalid — Context Summaries Unavailable
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-26
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
review_after: 2026-12-11
|
||||
confidence: high
|
||||
tags: [hermes, router, memory, incident, unresolved]
|
||||
sources: [gateway-log, live-config, live-probe]
|
||||
---
|
||||
|
||||
# Nous Portal Refresh Token Invalid — Context Summaries Unavailable (2026-10-22 →)
|
||||
|
||||
`Nous Portal runtime credentials unavailable: Invalid refresh token. Run `hermes model` to re-authenticate.`
|
||||
|
||||
## Occurrences (gateway logs, counted 2026-10-26)
|
||||
| Day | Lines | Notable |
|
||||
|---|---|---|
|
||||
| 2026-10-22 | 1 | 11:32:20 `tui_gateway.server: llm.oneshot failed` |
|
||||
| 2026-10-24 | 3 | — |
|
||||
| 2026-10-25 | **20** | last at 14:53:04 — `agent.context_compressor: Failed to generate context summary` |
|
||||
| 2026-10-26 | 0 (to 00:1x) | none since the 19:27:13 restart |
|
||||
|
||||
## Effect
|
||||
- **Context compression failed outright.** Sessions could not summarise, so working context grew unchecked —
|
||||
one cron fire reached **148 k tokens** on 10-25 and ran 1 h 55 m.
|
||||
→ [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]
|
||||
- Every auxiliary LLM path pointed at `nous` failed with it.
|
||||
|
||||
## What changed — and what did not
|
||||
- **2026-10-25, `config.yaml` written 19:13 AWST:** all **eight** `auxiliary:` slots moved off `nous` to
|
||||
**`no-think/freeapi/auto` on `custom:omni-101`**, each with the chain
|
||||
`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`
|
||||
(compression timeout 90 s, vision timeout 180 s). → [[systems/model-providers]]
|
||||
- **The credential is still invalid.** The fault is **masked, not fixed** — no `hermes model`
|
||||
re-authentication has been run, and `nous` is still the **first** entry in `fallback_providers`.
|
||||
- Zero `invalid_grant` lines appear after the 2026-10-25 19:27:13 gateway start, so the masking is holding.
|
||||
|
||||
## Open
|
||||
- Re-authenticate the Nous Portal credential, or remove `nous` from `fallback_providers`.
|
||||
- Until then, a failover event will attempt a dead credential silently.
|
||||
|
||||
> **Re-verified 2026-11-11 — the refresh-token failure is gone from the logs, and the exposure has
|
||||
> changed shape rather than closed.**
|
||||
>
|
||||
> - **Zero `invalid_grant` / refresh-token lines** now appear in `~/.hermes/logs/errors.log` (which
|
||||
> covers the window through 2026-11-10 19:19) — so the credential is no longer failing at *auth*.
|
||||
> `~/.hermes/.env` carries no `NOUS_API_KEY`; `~/.hermes/auth.json` is present and live.
|
||||
> - What appears instead is **404 `insufficient_credits_for_paid_model`** for
|
||||
> `anthropic/claude-haiku-4.5` — 3 occurrences, most recent **2026-11-10 19:19:33**, from session
|
||||
> `20261031_190813_95fbde`.
|
||||
> - **A ninth auxiliary slot now points back at `nous`.** The 2026-10-25 remediation moved all
|
||||
> *eight* slots off `nous` onto `custom:omni-101` / `no-think/freeapi/auto`, but
|
||||
> `auxiliary.pulse_memory` (`provider: nous`, `model: anthropic/claude-haiku-4.5`) was added
|
||||
> afterwards and **is** the slot producing those 404s. So the masking is intact for the original
|
||||
> eight slots but a new, single-slot nous dependency exists and is **failing on credits**, not on
|
||||
> auth.
|
||||
> - `nous` is **still the first entry** in `fallback_providers`
|
||||
> (`meituan/longcat-2.5-preview:free`) — that is a *free* model and has not been observed failing,
|
||||
> so the failover risk described below is now "may burn an attempt", not "guaranteed dead".
|
||||
>
|
||||
> **Revised open item:** decide whether `auxiliary.pulse_memory` should be moved onto
|
||||
> `custom:omni-101` like the other eight slots (it is the only active nous path left, and it is
|
||||
> currently erroring), and/or add credits for paid nous models. The original re-authenticate item
|
||||
> is **superseded** — auth works now.
|
||||
|
||||
> **Escalated 2026-11-11 (nightly audit).** The single-slot exposure described above became the
|
||||
> busiest error path on the host, and it now has a **downstream casualty**.
|
||||
>
|
||||
> - `anthropic/claude-haiku-4.5` on `nous` returned **404 `insufficient_credits_for_paid_model`**
|
||||
> **14 times** on 2026-11-11 (07:01:11 … last **19:19:38**) — up from 3 in the whole preceding
|
||||
> window. All from session `20261031_190813_95fbde` (**source `desktop`, started 2026-10-31
|
||||
> 19:08:13, still open**), whose pinned model is that same paid nous model.
|
||||
> - The **404 is immediately followed by a 401.** The session's fallback chain moves the turn on to
|
||||
> `opencode-zen` / `deepseek-v4.1-flash`, whose key expired on 2026-11-08 — **7 × HTTP 401
|
||||
> `api_key_expired`** on 11-11 (07:01:13, 07:19:30, 08:03:40, 13:03:28, 13:19:33, 18:03:39,
|
||||
> 19:19:39). So **both hops of this chain are dead**, and the pair repeats every time the session is
|
||||
> poked. → [[systems/opencode-go]]
|
||||
> - `auxiliary.pulse_memory` logged **19 lines** on 11-11 (06:0x, 08:0x, 13:0x, 18:0x), including
|
||||
> **11 ×** *"main provider `custom:omni-101` is unavailable and no fallback_chain /
|
||||
> fallback_providers is configured — refusing to guess another logged-in provider."* Live config
|
||||
> re-confirmed at audit time: `pulse_memory` is the **only** slot carrying **no `fallback_chain`**
|
||||
> (`{"provider": "nous", "model": "anthropic/claude-haiku-4.5"}`), while the other eight all chain
|
||||
> `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` on `custom:omni-101`.
|
||||
> - **Net effect:** pulse/voice memory retrieval failed on every attempt this window. `config.yaml`
|
||||
> mtime is unchanged since **2026-11-09 22:58:07** — nothing was fixed.
|
||||
>
|
||||
> **Open item, unchanged and now evidenced:** move `auxiliary.pulse_memory` onto `custom:omni-101`
|
||||
> like the other eight slots and give it a `fallback_chain`, and re-point or retire the stale desktop
|
||||
> session pinned to a paid nous model. No credentials were minted or modified.
|
||||
|
||||
## Related
|
||||
- [[systems/current-profile]]
|
||||
- [[systems/hermes-agent]]
|
||||
@@ -3,8 +3,8 @@ title: Conduit Call Requests Refused by the Agent
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-25
|
||||
updated: 2026-10-25
|
||||
verified_on: 2026-10-25
|
||||
updated: 2026-11-19
|
||||
verified_on: 2026-11-19
|
||||
review_after: 2026-11-08
|
||||
confidence: high
|
||||
tags: [conduit, voice, tool-calling, agent-conduct, incident, unresolved]
|
||||
@@ -86,3 +86,18 @@ into [[incidents/2026-10-22-conduit-memory-start-timeout]].
|
||||
- [[systems/conduit]]
|
||||
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
|
||||
- [[concepts/tool-calling]]
|
||||
|
||||
## Status update — 2026-10-26 00:1x AWST
|
||||
- **No further refusals in-window** (2026-10-24 12:56 → 2026-10-26 00:1x). The signature has not recurred.
|
||||
- `conduit-calls.json` → `history` is still **empty (0 entries)**, so the call path has gone unused, not fixed. The file *does* carry `watches`, `outcomes`, `ends` and `told` state.
|
||||
- The plugin **is** working for cards: `conduit_show` ×1 on 2026-10-25 14:37, delivered live.
|
||||
- Plugin advanced to **`0.69.2`** and the dashboard gateway restarted at **18:42:33**, so upstream commits are live — see [[systems/conduit]].
|
||||
- **Status stays open**: the request path is untested, not confirmed good.
|
||||
|
||||
## Status update — 2026-11-19 (stocktake re-probe)
|
||||
|
||||
- `~/.hermes/conduit-calls.json` → `history` is **still empty (0 entries)** as of 2026-11-19 03:0x
|
||||
AWST — the call path has now gone unused for ~26 days since the fault. The "verify a real call
|
||||
end-to-end" open item remains outstanding.
|
||||
- No new refusal episodes have been recorded against this page. **Status stays open** on the same
|
||||
basis as before: untested, not confirmed good.
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-26
|
||||
updated: 2026-11-18
|
||||
verified_on: 2026-11-18
|
||||
review_after: 2026-11-09
|
||||
confidence: medium
|
||||
tags: [cron, automation, hermes, incident]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
|
||||
---
|
||||
|
||||
# Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop (2026-10-25)
|
||||
|
||||
**Windows:** 2026-10-25 08:31:14 → 10:26:37 AWST (runaway) and 17:19:36 → 19:51:59 AWST (five extra fires)
|
||||
**Job:** `08eb86f02d37` — *Gmail Action Digest*, daily `30 8 * * *`, pinned `custom:omni-101`
|
||||
|
||||
## What happened
|
||||
- The 10-25 **08:30 slot** claimed at 08:30:57, started **08:31:14** and did not finish until **10:26:37 —
|
||||
a 1 h 55 m 23 s run**, against **91 s** (10-23) and **104 s** (10-24) in the two preceding days. It closed
|
||||
`failed` on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed).`
|
||||
- The same job then fired **five more times** on 10-25 — 17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53 —
|
||||
for **six fires in the calendar day** against one configured daily slot.
|
||||
- `19:18:41 → 19:28:04` closed `failed` on a signature this ledger had not seen before:
|
||||
**`Cron admission missing: the owner never admitted this fire`** (`source: direct`, `pid 2473801`,
|
||||
`scheduled_instant` empty). The 2026-10-20 orphan was `Unclean gateway exit` — a second, distinct variant.
|
||||
- Every fire, including both failures, carries `delivery_outcome: delivered`, so **up to six Gmail Action
|
||||
Digests reached Anthony on 2026-10-25**. The 19:30:31 fire ran 21 m 28 s.
|
||||
- Restart context (`gateway.log`): `18:29:22` SIGTERM as a planned stop under systemd → `18:30:04` start;
|
||||
`19:26:16` *Stopping gateway for restart* → `19:27:13` start. Three starts on 10-25 (00:32:20, 18:30:04,
|
||||
19:27:13). The five extra fires bracket the evening pair. → [[systems/hermes-agent]]
|
||||
|
||||
## Root cause — hypothesised, not confirmed
|
||||
The 2026-10-25 14:17 interactive investigation found that the 08:31 fire's worker had made **70 `terminal`
|
||||
calls on unrelated Proxmox VM101 `/dev/sdb` backup work**, with run context growing **66 k → 148 k tokens**,
|
||||
and that **context compression was unavailable all day** (`invalid_grant`). Working hypothesis: the scheduled
|
||||
fire **adopted and continued an unrelated long-lived session** instead of starting its own. *Why* the
|
||||
scheduler admitted an unrelated session is unexplained and is the first thing to check.
|
||||
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
|
||||
## Impact
|
||||
- Six delivered digests in one day (duplicate content) — the user-visible symptom.
|
||||
- **Neither failure raised an `cron_incidents` row**; the two open rows are older (10-01, 10-19). A failed
|
||||
execution therefore does not reliably produce an incident.
|
||||
- The job now reads `last_status: delivery_queued` with an **empty** `last_error` (2026-10-26 00:1x) — a state
|
||||
that did not resolve to `ok`.
|
||||
- 2026-10-25 closed **172 `completed` / 2 `failed`**, both failures belonging to this job.
|
||||
|
||||
## Open questions / next actions
|
||||
- Why did a scheduled fire adopt an unrelated session? Check session-reuse across fires.
|
||||
- Record `Cron admission missing: the owner never admitted this fire` alongside the 2026-10-20 orphan
|
||||
signature and give both a recovery path on gateway restart.
|
||||
- Give scheduled runs a **wall-clock budget** — a 113× runtime creep was invisible anywhere except the
|
||||
execution row.
|
||||
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
|
||||
|
||||
## Resolution (verified 2026-11-18)
|
||||
**Resolved — no recurrence in 23 days.** The `delivery_queued` state cleared (the open question above is answered: it did **not** persist past the 2026-10-26 08:30 slot). Job `08eb86f02d37` now reads `enabled`, `last_status: ok`, `last_error: null`, `failure_streak: 0`, last run 2026-11-17 08:33. In the retained `executions.db` window (2026-11-12 → 11-17; DB retention floor ~18 d means earlier slots have aged out) it ran **exactly once per day, `completed` every time, zero failed rows** — no refires since the 10-25 loop. Root cause of the 10-25 session-adoption remains hypothesised, not confirmed; the wall-clock-budget follow-up was never implemented, so the *exposure* (a runaway can recur silently) is unchanged even though the incident itself has not recurred.
|
||||
|
||||
## Related
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
- [[systems/hermes-agent]]
|
||||
@@ -0,0 +1,125 @@
|
||||
---
|
||||
title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-27
|
||||
updated: 2026-11-19
|
||||
verified_on: 2026-11-19
|
||||
review_after: 2026-12-19
|
||||
confidence: high
|
||||
tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure]
|
||||
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-probe]
|
||||
---
|
||||
|
||||
# Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)
|
||||
|
||||
## Headline
|
||||
|
||||
On **2026-10-26 at 14:33:35 AWST** the gateway received a **planned `SIGTERM`** (`under_systemd=yes`,
|
||||
`parent_name=systemd`) — and the shutdown drain **still interrupted a running cron execution**, because
|
||||
the drain reported `timed_out=True` after only **1.35 s**. The orphaned run was
|
||||
`ea79a2107e36` **Mnemosyne Health Monitor** (12-hourly `0 */12 * * *`), which had been claimed at
|
||||
**12:01:13** and was **2 h 34 m** into its work.
|
||||
|
||||
This is the **second instance** of the orphaned-cron-run class documented for 2026-10-20
|
||||
([[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]), with three material differences:
|
||||
|
||||
| | 2026-10-20 | 2026-10-26 |
|
||||
|---|---|---|
|
||||
| Stop type | **unclean exit** (no exit path, `suspected_oom=False`) | **planned `SIGTERM`** under systemd |
|
||||
| Execution closed as | `unknown` ("owner exited before a durable terminal state") | **`failed`** with a **new** error string |
|
||||
| Incident row raised | **no** | **yes — two** |
|
||||
| Failure delivered | no | **yes**, 14:35:59 |
|
||||
|
||||
So the loud-orphan recommendation from the 10-20 page is **now partly implemented** — the run no longer
|
||||
disappears silently. What it did not do is *finish the work* or *avoid the interruption*.
|
||||
|
||||
## Timeline (all times AWST, 2026-10-26)
|
||||
|
||||
| Time | Event | Source |
|
||||
|---|---|---|
|
||||
| 12:01:13 | `ea79a2107e36` Mnemosyne Health Monitor execution claimed, `running` | `executions.db` |
|
||||
| 00:01:46 → 00:10:45 | the same job's previous (00:01) slot — **completed in 9 m** | `executions.db` |
|
||||
| 14:33:35 | `Received SIGTERM as a planned gateway stop`, `under_systemd=yes`, `parent_name=systemd`, `loadavg_1m=2.15` | `gateway.log` |
|
||||
| 14:33:37 | `Thread 34795 not found` while sending the shutdown notice (same dead thread as the birthday/Uber jobs) | `gateway.log` |
|
||||
| 14:33:37 | incident row `ea79a2_c3ab2d784ba3` opens — *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/724a69108b…json`"* (`alerted`) | `cron_incidents` |
|
||||
| 14:33:38 | `Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True` → `interrupting remaining work` | `gateway.log` |
|
||||
| 14:34:25 | `1 authority task(s) did not settle within 40s; left for recovery`; `Profile /home/hermes/.hermes still has live writers at shutdown; its admissions stay for recovery` | `gateway.log` |
|
||||
| 14:34:27 | `ERROR gateway.session_authority: Admitted turn 28ad127ffb2a44c0bcd17283a93aa496 failed`; cron session `local-253a96…` **paused: runtime_draining** | `gateway.log` |
|
||||
| 14:34:27 | that session finalises with `end_reason: cron_complete` (66 msgs, 39 tool calls, 127 API calls) | `state.db` |
|
||||
| 14:34:29 | `Gateway stopped (total teardown 54.06s)`; **`Skipping .clean_shutdown marker — drain timed out with interrupted agents`** | `gateway.log` |
|
||||
| 14:35:18 | next gateway life starts (`Starting Hermes Gateway...`) | `gateway.log` |
|
||||
| 14:35:59 | execution closes **`failed`**: `RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership` | `executions.db` |
|
||||
| 14:35:59 | incident row `ea79a2_a4661554a3c2` opens on that RuntimeError (`alerted`); execution `c5ac27e3…` **delivered** the failure | `cron_incidents`, `deliveries.db` |
|
||||
| 14:46:12 / 15:01:15 | Mnemosyne Watchdog `96eb95ce12f2` fails twice (`Script exited with code 1`) reacting to the log `ERROR` — rows `96eb95_0c89bd373a0a`, `96eb95_e32174134546` | `cron_incidents` |
|
||||
| 15:16:12 | both watchdog rows **`resolved`** (the error aged out of its 15-min log window); watchdog back to `ok` | `cron_incidents` |
|
||||
|
||||
## What is new or unexplained
|
||||
|
||||
1. **A planned stop is enough to orphan a run.** No crash was required. The drain had a **30 s** budget and
|
||||
reported `timed_out=True` after **1.35 s** with `cron_now=0` — i.e. the drain did **not** wait out its
|
||||
budget before taking the interrupt path. Logged verbatim; **not diagnosed**.
|
||||
2. **New terminal error string:** `lost its durable fire claim ownership`. The 10-20 orphan closed as
|
||||
`unknown` with a scheduler-side note; this one closed as `failed` from the worker's own perspective.
|
||||
Any alerting or triage that pattern-matches on the 10-20 wording will miss this one.
|
||||
3. **Two rows for one fault.** `ea79a2_a4661554a3c2` (fire claim) and `ea79a2_c3ab2d784ba3` (unverified /
|
||||
reconcile admission) both remain **`alerted`, unclosed** as of 2026-10-27 00:1x, and the job carries
|
||||
**`failure_streak: 2`** from a **single** interrupted fire.
|
||||
4. **The admission was reconciled.** `~/.hermes/cron/admissions/` no longer holds `724a69108b…json`
|
||||
(only the 00:00 / 00:01 files from the following night), so the "reconcile" step in the row's text has
|
||||
been satisfied — the rows were simply never closed.
|
||||
5. **The watchdog works.** This fault **did** alert, twice, and self-resolved — which is a genuine
|
||||
improvement over 10-20 (one wakes up via the incident ledger, not via a missing digest).
|
||||
|
||||
## Impact
|
||||
|
||||
- **One 12-hourly memory consolidation cycle did not run.** `ea79a2107e36` is the Mnemosyne
|
||||
health/consolidation job → [[systems/mnemosyne]]. The 00:01 slot completed normally; the 12:01 slot
|
||||
produced no completed work.
|
||||
- **Day tally:** 2026-10-26 settled **169 `completed` / 3 `failed`** — and **all three failures are this one
|
||||
fault** (1 Health Monitor + 2 derivative Watchdog reds). 2026-10-25 was **172 / 2**.
|
||||
- **Session cost sunk:** the interrupted run burned 66 messages / 39 tool calls / 127 API calls
|
||||
(~597 k input, 254 k output tokens) before the interrupt.
|
||||
- **Unclosed incident rows 2 → 3** (`0e753e_e7f8194a7d06` carried over from 2026-10-01, plus the two new
|
||||
`ea79a2_*` rows).
|
||||
|
||||
## Recommended follow-ups (not yet done)
|
||||
|
||||
1. Decide the drain policy for **long-running cron jobs**: this job legitimately occupies 9 m → 2 h+.
|
||||
Either exempt a cron execution from the drain deadline, or interrupt it with a durable
|
||||
*"interrupted → re-fire this slot"* marker instead of a terminal `RuntimeError`.
|
||||
2. **Close or ack the two `ea79a2_*` rows** — the admitted turn failed and the admission is gone, so they
|
||||
are historical.
|
||||
3. **Verify the drain-timeout arithmetic** (`timed_out=True` at 1.35 s against a 30 s budget) — if the
|
||||
deadline is being bypassed whenever a writer is live, every future restart can interrupt work.
|
||||
4. Confirm `failure_streak` should count the *unverified* row at all — a single interrupted fire reading
|
||||
**streak 2** inflates the job's apparent health.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — first instance of this class
|
||||
- [[systems/scheduled-tasks]] — fleet ledger and per-window tallies
|
||||
- [[systems/mnemosyne]] — the interrupted job's purpose and settings
|
||||
- [[systems/hermes-agent]] — gateway lifecycle and restart history
|
||||
|
||||
## Update — 2026-10-28 (nightly audit)
|
||||
|
||||
- **No recurrence.** Every run in the 2026-10-27 window except the radar's budget overrun completed, and the
|
||||
watchdog logged no alert for the day.
|
||||
- **Both incident rows this fault raised closed automatically** — `ea79a2_a4661554a3c2` and
|
||||
`ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** (unclosed rows fleet-wide fell 3 → 2).
|
||||
- The `lost its durable fire claim ownership` string did not appear in any 10-27 execution row.
|
||||
- Open question from the original write-up still stands: why a *planned* stop bypasses the drain deadline
|
||||
(`timed_out=True` at 1.35 s of a 30 s budget).
|
||||
|
||||
## Update — 2026-11-19 (stocktake re-probe)
|
||||
|
||||
- **Both `ea79a2_*` rows are now closed** (follow-up 2 satisfied): `ea79a2_c3ab2d784ba3`
|
||||
`resolved` 2026-10-27T00:38:32, and `ea79a2_a4661554a3c2` **`closed` 2026-11-07T15:43:13**
|
||||
(live probe of `cron_incidents` in `~/.hermes/cron/executions.db`).
|
||||
- The `lost its durable fire claim ownership` error string has **no rows in the retained
|
||||
`executions` window** — no recurrence since 2026-10-27 (subject to the DB's ~18-day retention
|
||||
floor; the 10-28 update already confirmed the 10-27 window clean).
|
||||
- **Status → `resolved`.** The fault did not recur and its incident rows are closed. Follow-ups
|
||||
1, 3 and 4 (drain-policy decision, drain-timeout arithmetic, `failure_streak` semantics)
|
||||
remain open design questions and were **not** actioned.
|
||||
@@ -0,0 +1,128 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-28
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
review_after: 2026-11-13
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
|
||||
---
|
||||
|
||||
# Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
|
||||
|
||||
## Headline
|
||||
|
||||
On **2026-10-27** the Daily Marketing Job Radar (`dca8482e4f76`) ran **07:01:08 → 11:03:54 AWST**
|
||||
(**4 h 2 m 46 s** wall clock), closed **`failed`**, and delivered a report that opens:
|
||||
|
||||
> **Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call budget exhausted before Notion write stage)**
|
||||
|
||||
Discovery, verification, dedup and classification completed. **Notion ingestion, cover-letter generation and the
|
||||
final cover-letter audit were not executed** — so the run is recorded as failed rather than successful.
|
||||
Self-reported run time **10:47:57**; freshness cutoff **2026-10-20**.
|
||||
|
||||
This is the second consecutive budget overrun: the **2026-10-26** 07:00 slot ran 07:01:08 → 09:10:27
|
||||
(**2 h 9 m 19 s**), closed **`completed`**, and *still* delivered the same `RUN INCOMPLETE` marker — the day's
|
||||
backlog had to be finished by hand. 2026-10-27 is the first overrun recorded as **`failed`**, which is the
|
||||
improvement: the failure reached the incident ledger instead of hiding inside a green row.
|
||||
|
||||
## Ledger state
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Execution row | `dca8482e4f76`, started `2026-10-27T07:01:08.265913+08:00`, finished `2026-10-27T11:03:54.143535+08:00`, `failed` |
|
||||
| Delivery | `delivered` (the incomplete report reached Telegram topic 1030) |
|
||||
| Incident row | `dca848_e93a538ade6b` — `state: alerted`, `failure_type: auth`, first/last seen `2026-10-27T11:03:53.715494+08:00`, alerted `11:03:59.013115` |
|
||||
| Incident status | **open at the 2026-10-28 00:2x audit** — no ack, no close |
|
||||
| `jobs.json` | `last_status: error`, `failure_streak: 1`, pin unchanged at `custom:omni-101` / `smart-route` |
|
||||
|
||||
⚠ `failure_type: auth` does **not** describe this fault. A tool-call budget exhaustion is being classified as an
|
||||
auth failure; the field is not trustworthy as a diagnosis for this job.
|
||||
|
||||
## What the run actually did (from its own report)
|
||||
|
||||
- Live Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168`: **451 entries**, `Date Added = 2026-10-27` → **0**.
|
||||
- Board composition at run time: **Agent — Rhino 402, Chaos 12, Manual 2, blank 35**; **Source — SEEK 295,
|
||||
LinkedIn 88** (row text truncated in the ledger).
|
||||
|
||||
## Response, same day: the radar is now two jobs
|
||||
|
||||
At **2026-10-27 17:34:15** the single radar job was split into a two-stage pipeline (see
|
||||
[[runbooks/job-radar]] for the full contract):
|
||||
|
||||
| Stage | Job | Schedule | Delivery | Job id |
|
||||
|---|---|---|---|---|
|
||||
| 1 — Discovery | Daily Marketing Job Radar - Discovery — `DISCOVERY → VERIFY → DEDUP → SCORE`, no Notion write, no cover letters; writes a handoff file | `0 7 * * 1-5` | `local` (failures → Telegram topic 1030) | `dca8482e4f76` (unchanged) |
|
||||
| 2 — Write and Letters | Daily Marketing Job Radar - Write and Letters — lands the handoff: Notion entries, cover letters, audit, report | `30 11,14 * * 1-5` | Telegram group `-1004321904721`, topic `1030` | `5fc292ff0f3b` (created 17:34:15) |
|
||||
|
||||
Stage 1 keeps its id, pin (`custom:omni-101` / `smart-route`) and its own tool budget; stage 2 is a separate job
|
||||
whose prompt lands the handoff rather than repeating the discovery narrative. Two test fires of stage 2 ran the same
|
||||
evening — **17:35:00 → 17:36:52** and **17:38:38 → 17:43:42** — both `completed`, both with delivery
|
||||
`suppressed`, `last_status: ok`, streak 0.
|
||||
|
||||
Fleet size went **28 → 29 jobs** (26 enabled / 3 disabled).
|
||||
|
||||
## Why it matters
|
||||
|
||||
1. **The failure mode is partial work, not a lost run.** Four hours of browser-heavy discovery produced a
|
||||
correct report and zero writes. Nothing in `jobs.json` distinguished it from a good day except
|
||||
`last_status: error` on 10-27 (`completed` on 10-26, with the same incompleteness buried in the text).
|
||||
2. **Splitting is the mitigation, not a fix.** Stage 2 inherits its own tool budget; if stage 1 still overruns
|
||||
and never writes the handoff file, stage 2 has nothing to land — watch the 11:30 slot for that.
|
||||
3. **The budget is consumed by discovery**, which is exactly the stage that now owns it alone.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Verify one full cycle end-to-end** (stage 1 07:00 → stage 2 11:30): does stage 2 find stage 1's handoff
|
||||
file, do the Notion writes and cover letters land, and is `MISSING_COVER_LETTER_COUNT=0` reported?
|
||||
2. **Make budget exhaustion self-describing.** The classifier tagged it `auth`; a distinct budget/partial-run
|
||||
type would put this on the same footing as any other failure.
|
||||
3. **Do the two stages fit the same wall-clock envelope?** Stage 1 must finish inside its own budget *and* leave
|
||||
the handoff before 11:30.
|
||||
4. The incident row stays open until a clean stage-1 slot auto-closes it — confirm the 2026-10-28 07:00 slot
|
||||
closes `dca848_e93a538ade6b`.
|
||||
|
||||
## Follow-up — the 2026-10-28 slot (recorded 2026-10-29)
|
||||
|
||||
The **first real two-stage cycle ran on 2026-10-28** and answered the open questions above — not favourably:
|
||||
|
||||
- **Stage 1 overran again, on a new signature.** `dca8482e4f76` ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**),
|
||||
closed **`failed`**, `failure_streak` **2**, on *"hit the agent iteration cap before the handoff could be
|
||||
written"* → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
|
||||
- **No handoff file was written** — `~/.hermes/radar-ref/handoff/` held only `2026-10-27.json` at the 2026-10-29
|
||||
audit, so **Notion writes and cover letters did not run again**.
|
||||
- **Both stage-2 slots no-oped correctly** — `5fc292ff0f3b` fired **11:30:27 → 11:31:31** and
|
||||
**14:30:27 → 14:31:43**, both `completed` with `delivery_outcome: suppressed`. Open question 1 is therefore
|
||||
answered only in the negative: the recovery rule ("a stage-2 run with no handoff is a no-op") holds in
|
||||
production, but nothing has yet proved the positive path end-to-end.
|
||||
- **Open question 4 is answered: the row did not close.** `dca848_e93a538ade6b` stayed open because the 10-28
|
||||
slot was not clean, and a second row (`dca848_0a2f03b5952b`) opened — open incident rows **2 → 4**.
|
||||
|
||||
## Follow-up — the 2026-10-29 slot: the split held (recorded 2026-10-30)
|
||||
|
||||
**Resolved.** Stage 1 (`dca8482e4f76`) ran **2026-10-29 07:00:59 → 10:16:12** — 3 h 15 m 13 s — and
|
||||
closed **`completed`**. It wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` (24.8 KB):
|
||||
`run_date 2026-10-29T08:23:51+08:00`, **170 discovered / 33 duplicates avoided / 37 verified**,
|
||||
sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Both ledger rows this incident opened —
|
||||
`dca848_e93a538ade6b` and `dca848_0a2f03b5952b` — closed **`resolved` at 10:16:12**, and the run
|
||||
opened no new row. Open cron incident rows went **4 → 1**; the survivor is the Facebook monthly's
|
||||
`0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot.
|
||||
|
||||
Stage 2 (`5fc292ff0f3b`) fired at **10:13:46**, consumed the handoff at **10:40:59**, finished
|
||||
**10:44:48 `completed`** and **delivered** to topic 1030 in the same second (delivery `delivered`).
|
||||
Its `stage2_result`: **20 pages written / 20 `page_ids` / 2 cover letters / 2 same-day strong matches
|
||||
/ `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went
|
||||
**455 → 475**. The later slots no-oped as designed (11:30 → 10 m 46 s, 14:30 → 1 m 51 s, already-consumed
|
||||
handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns the report.
|
||||
|
||||
The questions this page left open are answered for this run: the handoff **was** written inside budget and
|
||||
consumed, and **no cover letter went missing on the first real run of the split**.
|
||||
|
||||
## Related
|
||||
|
||||
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
|
||||
- [[systems/scheduled-tasks]] — fleet ledger, job table and the 2026-10-27 window
|
||||
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — the radar's previous failure mode
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-29
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
review_after: 2026-11-13
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
|
||||
---
|
||||
|
||||
# Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
|
||||
|
||||
## Headline
|
||||
|
||||
On **2026-10-28** the Discovery stage of the Daily Marketing Job Radar (`dca8482e4f76`) ran
|
||||
**07:00:32 → 10:33:18 AWST** (**3 h 32 m 46 s**) and closed **`failed`** on *"hit the agent iteration cap before
|
||||
the handoff could be written"* — its **second consecutive failure** (`failure_streak: 2`).
|
||||
|
||||
This is the first real cycle of the **two-stage split** introduced on 2026-10-27 (see
|
||||
[[incidents/2026-10-27-job-radar-budget-exhausted]] and [[runbooks/job-radar]]), and it produced a clean negative
|
||||
result: **stage 1 never wrote its handoff file, so there was nothing for stage 2 to land.** Both stage-2 slots
|
||||
ran anyway — **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed`, both
|
||||
`delivery_outcome: suppressed` — which is the designed no-op path working correctly in production for the first
|
||||
time. Notion ingestion and cover letters did not execute for the day.
|
||||
|
||||
## Ledger state
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Execution row | `dca8482e4f76`, started `2026-10-28T07:00:32.463309+08:00`, finished `2026-10-28T10:33:18.833788+08:00`, **`failed`** |
|
||||
| Duration | **3 h 32 m 46 s** (10-27: 4 h 2 m 46 s; 10-26: 2 h 9 m 19 s) |
|
||||
| `jobs.json` | `last_status: error`, **`failure_streak: 2`**; `last_error` opens *"Daily Marketing Job Radar — Stage 1 (Discovery) — 2026-10-28 — hit the agent iteration cap before the handoff could be…"*; pin unchanged (`custom:omni-101` / `smart-route`) |
|
||||
| Incident row | `dca848_0a2f03b5952b` — `state: alerted`, `failure_type: timeout`, first_seen `2026-10-28T10:33:18`, **open** |
|
||||
| Handoff | **not written** — `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json` (verified 2026-10-29 00:3x) |
|
||||
| Stage 2 | `5fc292ff0f3b` fired `11:30:27 → 11:31:31` and `14:30:27 → 14:31:43`, both `completed` / `delivery_outcome: suppressed` |
|
||||
|
||||
The 10-27 row (`dca848_e93a538ade6b`, classified `failure_type: auth`) did **not** close — the 2026-10-28 07:00
|
||||
slot was not clean. Open `cron_incidents` rows went **2 → 4**.
|
||||
|
||||
## What the split proved, and what it did not
|
||||
|
||||
1. **It proved the recovery rule.** *"A stage-2 run with no handoff is a no-op, not a failed search"* — exactly as
|
||||
written in [[runbooks/job-radar]] — held in production: stage 2 invented no work, raised no error, and sent
|
||||
nothing to topic 1030.
|
||||
2. **It did not make discovery finish.** The two post-split runs died at *different* ceilings:
|
||||
- **2026-10-27** — stopped at the **tool-call budget** after 4 h 2 m: discovery/verify/dedup/classify complete, writes never started.
|
||||
- **2026-10-28** — stopped at the **agent iteration cap** after 3 h 32 m, *earlier in the pipeline*: no handoff, so nothing downstream could run at all.
|
||||
The wall-clock number improving (4 h 2 m → 3 h 32 m) is **not** an improvement — less of the pipeline completed.
|
||||
3. **The write stage is no longer the risk; the discovery stage still is.** Splitting removed the
|
||||
"writes lost inside a green row" failure mode. What remains is that stage 1 cannot both finish discovery and
|
||||
reach its handoff write inside its own limits.
|
||||
|
||||
## Why it matters
|
||||
|
||||
- **A lost day is now cheap to detect but easy to miss.** Stage 1 raises an incident row and flips `last_status`
|
||||
to `error`; stage 2's *correct* silence (`suppressed`) means the only outward sign of a workless day is stage
|
||||
1's own failure report — not a "0 jobs added" report.
|
||||
- **Two consecutive failures, two different classifiers.** 10-27 was tagged `failure_type: auth` (wrong for a
|
||||
budget fault); 10-28 was tagged `timeout` (also only approximate — an agent-side iteration cap, not a network
|
||||
timeout). Neither string names the actual cause.
|
||||
- **Budget arithmetic is not the fix.** If the cap fires after ~3.5 h of discovery, the pipeline needs a smaller
|
||||
discovery surface (fewer sources per slot), **staged/checkpointed handoff writes** (so a capped run still
|
||||
leaves something landable), or an explicit per-phase iteration allowance.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **What is the iteration cap for this job, and can stage 1 reach its handoff write before it?** The 10-28 run
|
||||
died *before* the write; the 10-27 run died *after* discovery but *before* writes. Neither produced a handoff.
|
||||
2. **Should stage 1 checkpoint a partial handoff** (after DEDUP/SCORE) so a capped run gives stage 2 something to
|
||||
land instead of a silent no-op day?
|
||||
3. **Make the abort self-describing** — a distinct `iteration-cap` / `partial-run` failure type rather than
|
||||
`auth` / `timeout`.
|
||||
4. Both radar incident rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) close only when a **clean** stage-1
|
||||
slot runs — watch the next 07:00 weekday slot.
|
||||
|
||||
## Follow-up — the 2026-10-29 slot: the cap was not hit (recorded 2026-10-30)
|
||||
|
||||
**Resolved for this run.** The next stage-1 run (`dca8482e4f76`, **2026-10-29 07:00:59 → 10:16:12**,
|
||||
3 h 15 m 13 s) closed **`completed`** *and* wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — the
|
||||
first run of the split to do both. The handoff records **170 discovered / 33 duplicates avoided /
|
||||
37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`,
|
||||
`indeed blocked`, `wa_gov ok`. Stage 2 consumed it at **10:40:59**, delivered at **10:44:48** with
|
||||
**20 Notion pages and 2 cover letters** (`missing_cover_letter_count 0`; board 455 → 475). Its two
|
||||
ledger rows closed `resolved` at 10:16:12.
|
||||
|
||||
⚠ **Flaky, not solved.** A 3 h 15 m stage-1 run is still long enough for the cap to be reachable, and
|
||||
the previous two slots in the split both died on it (2026-10-27 on the tool-call budget, 2026-10-28 on
|
||||
the iteration cap). Treat a `completed` run with **no handoff file** as the same fault, whatever the
|
||||
execution status says → [[incidents/2026-10-27-job-radar-budget-exhausted]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
|
||||
- [[incidents/2026-10-27-job-radar-budget-exhausted]] — the prior day's budget exhaustion and the split
|
||||
- [[systems/scheduled-tasks]] — fleet ledger and the 2026-10-28 window
|
||||
- [[systems/browser-backend]] — the browser layer this run depends on
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
title: Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-30
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
confidence: high
|
||||
tags: [conduit, voice, tts, stt, config, session-ttl, incident]
|
||||
sources:
|
||||
- live-probe-2026-10-30
|
||||
- live-config-2026-10-30
|
||||
- live-log-2026-10-30
|
||||
- session-20260908_031541_49c051
|
||||
- plugin-tree-conduit_push
|
||||
---
|
||||
|
||||
# Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers
|
||||
|
||||
**Window:** 2026-10-29 07:54 – 09:0x AWST · **Fixed:** 2026-10-29 08:02–08:4x AWST (dashboard restarted 08:03:26)
|
||||
**Symptom:** Anthony asked *"why do you never answer when I call you on Conduit"* — every Conduit voice call connected and then died silently. No Mac connection, no Hermes turn, no error surfaced on the phone.
|
||||
|
||||
Three independent faults sat in series. Fixing any one alone would not have made a call work; each one masked the next.
|
||||
|
||||
## Evidence trail (why it was invisible)
|
||||
|
||||
- The Mac Mini speech server recorded **zero connections all day.** `last_request_at` on the plugin was `null` — meaning no call had *ever* reached the Hermes turn logic.
|
||||
- `conduit-calls.json → history` was empty: no successful call had ever been recorded.
|
||||
- The only traces were in the plugin log: a wrong-looking reason string per attempt, which is what made each fault look like the *next* fault.
|
||||
|
||||
## Fault A — Truncated speech-server URL (403)
|
||||
|
||||
`voice.speech_server.url` was set to the bare host path:
|
||||
|
||||
```
|
||||
wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/ ⛔ (truncated)
|
||||
wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime ✅
|
||||
```
|
||||
|
||||
`plugin_api.py` (~line 2646) dials the configured value **verbatim** — it does not append `/realtime`. The bare `/v1/` returns HTTP **403**, the socket closes, and the plugin reports it as *"the speech server refused the key."* The key was fine; the path was wrong.
|
||||
|
||||
**Fix:** set `voice.speech_server.url` to `…:8443/v1/realtime`. Backup: `~/.hermes/config.yaml.bak.url-081525` (08:15:25).
|
||||
|
||||
## Fault B — Dashboard session TTL of 12 hours (WS 4401)
|
||||
|
||||
`dashboard.basic_auth.session_ttl_seconds` held the **12-hour default (43 200 s)**. The dashboard issues the phone its session, and when that session lapses the Conduit call socket answers **WS 4401 "Dashboard sign-in required"** — with no Hermes turn and no user-visible error.
|
||||
|
||||
Audit-log corroboration: the iPhone **re-authenticated at 08:00:59**, *minutes after* the silent call — i.e. it was walking around with an expired session and only logging back in when the dashboard happened to prompt.
|
||||
|
||||
**Fix:** raised to **30 days (2 592 000 s)** — matched to the refresh-token life so the phone's session no longer expires under it. Dashboard restarted **2026-10-29 08:03:26** (pid 4042003). Backup: `~/.hermes/config.yaml.bak.ttl-20261029080258`.
|
||||
|
||||
> ⚠ Two keys exist and only one was changed: `dashboard.basic_auth.session_ttl_seconds` = **2592000** (raised) and `dashboard.session_ttl_seconds` = **43200** (still 12 h, different key, different consumer). Check *both* when diagnosing a 4401 again.
|
||||
|
||||
## Fault C — Host STT and TTS providers both dead
|
||||
|
||||
Both halves of the voice loop were broken on the host, so even a correctly-signed call would have had no ears and no mouth.
|
||||
|
||||
| Setting | Was | Failed with | Now | Verified |
|
||||
|---|---|---|---|---|
|
||||
| `tts.provider` | `openai` | Mac `/v1/speech` → **502** | `ollama_local` | 6.94 s of real 64 kbps speech produced |
|
||||
| `stt.provider` | `groq` | **403 / Cloudflare error 1010** | `local` | Hermes' own transcription path returned `success: True, provider: local` |
|
||||
|
||||
- Groq was **blocked, not mis-keyed** — it returns 403/1010 identically with a bogus key, so the failure was Cloudflare bot-blocking, not credentials.
|
||||
- `tts.provider` had been flipped to `openai` by the **Conduit settings tool** earlier (that tool writes Hermes config). The app holds no `tts.provider` setting, so this fix will not be silently overwritten.
|
||||
- `ollama_local` runs `~/.hermes/scripts/ollama-tts.sh` (model `phonon`); `local` STT runs `faster_whisper` (`base`).
|
||||
- Backup: `~/.hermes/config.yaml.bak.stt-084536` (08:45:36); the TTS flip landed between the 08:15 and 08:45 backups.
|
||||
|
||||
## Verification (end-to-end, not just config)
|
||||
|
||||
Rather than trusting three config edits, the whole call path was driven for real: minted a valid HMAC dashboard session → got a WS ticket → let the plugin dial the Mac. Result:
|
||||
|
||||
- auth gate passed → `session.created` / `session.updated`
|
||||
- speech in transcribed correctly ("Hello Conduit, can you hear me right now?")
|
||||
- turn committed → audio streamed back → `response.done`
|
||||
- **Mac counters 0 → 10 connections, 1 completed response, 2.98 s audio, 0 errors**
|
||||
- plugin `connected: true`, handshake **41 ms**
|
||||
- the iPhone minted its own WS tickets at **08:07** and **08:19** — i.e. against the new 30-day session
|
||||
|
||||
## App update to 2.2.0 changed the settings schema
|
||||
|
||||
The iPhone app self-updated to **2.2.0** (from 1.10.1) mid-window, and its settings schema changed with it: the phone now exposes **19 keys, none of them `voice.local.*`**. There is **no phone-side Local Live server address**, so the Local Live path cannot be configured from the host side at all — a prior assumption that it could is now obsolete.
|
||||
|
||||
Consequence: `voice.mode` was moved **`local` → `classic`** (request queued 08:47, approved in the app; `conduit-settings.json` now reads `classic`). Classic needs no phone-side server config because the plugin relays the whole call — which is precisely why Fault C (both host providers dead) had to be fixed for it to work at all.
|
||||
|
||||
## Call record
|
||||
|
||||
`conduit-calls.json → history` now holds **2 entries — the first real calls ever recorded**:
|
||||
|
||||
| Time (AWST) | Outcome |
|
||||
|---|---|
|
||||
| 2026-10-29 08:38:04 | No Mac connection, no Hermes turn. iPhone had to re-login at 08:39:05, a minute later. |
|
||||
| 2026-10-29 08:49:36 | Placed *after* the Classic switch and the STT/TTS fixes. Session ended 08:49:35 — Anthony's verdict is unrecorded. |
|
||||
|
||||
## Open items
|
||||
|
||||
- The 08:49:36 Classic call has **no recorded outcome**. Nothing in the window proves a phone-originated call ran a real Hermes turn.
|
||||
- `last_request_at` was still `null` at the end of the window: the call→Hermes-turn bridge remains **unproven** even though the transport is now healthy. Classic mode is the recommended path.
|
||||
- If an app update or the settings sync ever writes the bare `/v1/` back into `voice.speech_server.url`, every call silently 403s again. Re-check it after any Conduit update.
|
||||
- Watch for the Conduit settings tool re-flipping `tts.provider` should a `tts.provider` key ever appear in the app's schema.
|
||||
|
||||
## Related
|
||||
|
||||
- [Conduit](../systems/conduit.md) — service page (voice pipeline, call path, pairing)
|
||||
- [Hermes Agent](../systems/hermes-agent.md) — TTS/STT provider block
|
||||
- [Conduit call request refused (2026-10-24)](2026-10-24-conduit-call-request-refused.md) — the earlier, behavioural half of the "calls don't arrive" problem
|
||||
- [Conduit memory-start timeout (2026-10-22)](2026-10-22-conduit-memory-start-timeout.md)
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
title: CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-31
|
||||
updated: 2026-10-31
|
||||
verified_on: 2026-10-31
|
||||
confidence: high
|
||||
tags: [cron, gateway, host, outage, pve, ct460, incident]
|
||||
sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-agent-log, live-deliveries-db, ssh-pve-uptime]
|
||||
---
|
||||
|
||||
# CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
|
||||
|
||||
**Status: RESOLVED** — the container and scheduler came back on 2026-10-30 ~16:28–16:37 and the day
|
||||
closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not
|
||||
established.** No monitor alerted; the gap was found by this audit.
|
||||
|
||||
## Summary
|
||||
|
||||
The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until
|
||||
**16:28** (container) / **16:37:21** (first execution). The PVE host `pve` itself rebooted — its boot
|
||||
time is **2026-10-30 16:24:00** — so this was a **host-level outage, not a cron or gateway fault**.
|
||||
The gateway's previous life died with no exit path, which the lifecycle ledger logs as
|
||||
`SIGKILL / OOM / VM death`. Nothing noticed for **5 h 37 m**.
|
||||
|
||||
## Timeline (all times AWST, 2026-10-30)
|
||||
|
||||
| Time | Event | Evidence |
|
||||
|---|---|---|
|
||||
| 10:49:54 | Last execution before the gap starts | `executions.db` last row `< 16:37` |
|
||||
| ~10:58:20 | Gateway life `pid=3913841` dies; no exit path ran | `gateway-exit-diag.log` `gateway.previous_unclean_exit` (prior life started 2026-10-28T17:21:58Z) |
|
||||
| 10:58:20 → 16:37:21 | **Zero execution rows** (5 h 39 m) | `executions.db` count in window = **0** |
|
||||
| 16:24:00 | PVE host `pve` (`100.96.100.82`) boots | `ssh root@100.96.100.82 'uptime -s; who -b'` → both read `2026-10-30 16:24` |
|
||||
| 16:28:00 | CT460 container boots | local `uptime -s` → `2026-10-30 16:28:00` |
|
||||
| 16:35:50 | Unclean exit noticed and logged | `logs/agent.log`, `logs/errors.log` — `Previous gateway life (pid=3913841, started_at=2026-10-28T17:21:58Z) exited UNCLEANLY (no exit path ran — SIGKILL / OOM / VM death)` |
|
||||
| **16:37:21** | Scheduler resumes — **8-run catch-up burst** (16:37:21 → 16:38:22) | `executions.db` |
|
||||
| 23:59:58 | Daily wiki audit (this run) starts; the 16:37 health-monitor catch-up is still `running` | `executions.db` |
|
||||
|
||||
## The resume burst (16:37:21 → 16:38:22)
|
||||
|
||||
Eight jobs fired within 61 seconds of the scheduler coming back. Seven closed `completed`;
|
||||
the eighth is still running.
|
||||
|
||||
| Job | Started | Closed |
|
||||
|---|---|---|
|
||||
| FreshRSS Midday News Briefing (`ecc1fc13655a`) | 16:37:21 | 17:45:31 — the 12:00 slot, delivered late |
|
||||
| Mnemosyne Auto-Consolidate | 16:37:21 | 16:37:54 |
|
||||
| Mnemosyne Embedding Coverage Monitor | 16:37:21 | 16:41:31 |
|
||||
| Mnemosyne Watchdog | 16:37:21 | 16:38:33 |
|
||||
| **Mnemosyne Health Monitor (`ea79a2107e36`)** | **16:37:21** | **RUNNING at audit** — the late 12:01 slot |
|
||||
| Job Tracker Dashboard Refresh | 16:37:23 | 16:37:35 |
|
||||
| 🦝 Hermes Config Backup to Gitea (Daily) | 16:38:06 | 16:42:02 |
|
||||
| Daily Marketing Job Radar - Write and Letters | 16:38:22 | 16:41:39 |
|
||||
|
||||
## Impact
|
||||
|
||||
- **≈39 scheduled runs lost.** 2026-10-29 fired **177** executions (177 `completed`, 0 `failed`); 2026-10-30
|
||||
fired **138** (136 `completed`, 0 `failed`, 2 `running` — the 16:37 catch-up plus this audit). No row exists
|
||||
inside the dark window, so the deficit is unfired slots, not failures.
|
||||
- **No failures were caused.** Nothing was recorded `failed` on 2026-10-30 — the outage simply removed the
|
||||
opportunities to run.
|
||||
- **The two long Mnemosyne runs absorbed it.** The FreshRSS midday briefing regenerated on the late fire
|
||||
(16:37:21 → 17:45:31), and the health monitor's 12:01 slot is the still-running 16:37:21 fire — normal for
|
||||
this job, whose 00:01 slot took **10 h 01 m 40 s** the same day.
|
||||
- **Deliveries were unaffected once back up:** `deliveries.db` holds **8 rows for 2026-10-30, all `delivered`**
|
||||
(latest 18:46:04) — the opposite of the 03:3x snapshot, which read zero rows because the day had barely started.
|
||||
- **2026-10-31 opened clean:** 5 executions (00:01:05 ×3, 00:15:59, 00:23:57), all `completed`.
|
||||
|
||||
## Root cause — not established
|
||||
|
||||
- The host rebooted. `pve` boot time is 16:24:00; **no OOM evidence was found** on the CT460 side (local
|
||||
`dmesg` grep empty, `/var/log/wtmp` absent, no `suspected_oom` marker in the webui session store).
|
||||
- The ledger's own wording (`SIGKILL / OOM / VM death`) is a **list of candidates for "no exit path ran"**, not a
|
||||
finding. Treating it as a diagnosis would be wrong.
|
||||
- Same-day, different host: [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] records **m93p**
|
||||
reboots (~15:55) that took down NFS-backed containers. The two events are **separate hosts and separate
|
||||
causes as far as the evidence goes** — they are not merged here, but the timing is close enough to warrant
|
||||
checking the two hypervisors' last-boot records together before either is called a one-off.
|
||||
- **To close it:** PVE syslog / `journalctl -b -1` on `100.96.100.82` around 2026-10-30 10:58, and the host's
|
||||
own last-boot records (`last -x reboot`). Neither was read for this page — say so rather than guess.
|
||||
|
||||
## Why nothing alerted
|
||||
|
||||
The jobs positioned to notice a host-down event are **disabled**:
|
||||
|
||||
| Job | ID | State |
|
||||
|---|---|---|
|
||||
| NFS mount monitor | `cbfe434a3b5e` | disabled |
|
||||
| m93p LXC boot monitor | `1a8e90ea2271` | disabled |
|
||||
|
||||
Everything else is *inside* the scheduler that went dark, so it could not have noticed. **A host-level outage of
|
||||
the Hermes VM currently has no in-estate detector** — the fleet learned about it from the 23:59 audit, 13 h
|
||||
after the fact. This is the incident's one open item.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]] — the fleet entry for this window
|
||||
- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — the same day's health-monitor story: its 00:01 slot
|
||||
closed `unknown` at 01:04:22, incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved` at 10:02:48** when that
|
||||
run finally completed, and the job is live again (`enabled: true`, `last_status: ok`, streak 0)
|
||||
- [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] — sister-host reboot, same day
|
||||
- [[infrastructure/hosts]] — `pve` / CT460 addressing
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
title: m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start
|
||||
type: concept
|
||||
status: resolved
|
||||
created: 2026-10-30
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
review_after: 2026-11-30
|
||||
confidence: high
|
||||
tags: [proxmox, m93p, nfs, lxc, outage, freshrss]
|
||||
sources: [live-probe, ssh-m93p, systemctl-status, pct-list, freshrss-script-output]
|
||||
---
|
||||
|
||||
# m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start (2026-10-30)
|
||||
|
||||
**Window:** 2026-10-30 ~15:55 AWST (host boot) → 17:20 AWST (manual recovery)
|
||||
**Status:** **Resolved.** All 18 LXC containers on m93p are running; self-healing recovery unit installed.
|
||||
**Impact:** Every LXC whose rootfs lives on the `nfs-ssd0-39` storage was stopped and unstartable.
|
||||
This included **FreshRSS (CT 1260)**, Gitea (221), NTFY (1241), LiteLLM (104), openweb (245),
|
||||
rclone (105), paseo (1261), social-dashboard (1263), homebridge (1007), docker-01 (1006),
|
||||
vaultwarden (102), omakase (1264), and others. The FreshRSS midday digest cron failed with
|
||||
`RuntimeError: Auth failed: ... urlopen error timed out` — a misleading error; the cause was
|
||||
total service unavailability, not credentials.
|
||||
|
||||
## Root cause
|
||||
|
||||
Both Proxmox hosts rebooted within minutes of each other. The NFS server (`192.168.178.39`,
|
||||
the PVE host) came back **after** m93p. m93p's boot-time NFS mount therefore failed, and
|
||||
`pct start` for any CT with an NFS-backed rootfs failed with:
|
||||
|
||||
```
|
||||
volume 'nfs-ssd0-39:1260/vm-1260-disk-0.raw' does not exist
|
||||
```
|
||||
|
||||
There is no boot retry: m93p's mount attempt was one-shot and there is no Proxmox storage entry
|
||||
for `ssd0-39` (`pvesm status` does not list it — the mount is a plain filesystem mount at
|
||||
`/mnt/nfs/ssd0-39`).
|
||||
|
||||
Secondary fault: **three** NFS mounts fail together on m93p, not one:
|
||||
|
||||
| Source | Mount point | Purpose |
|
||||
|---|---|---|
|
||||
| `192.168.178.39:/mnt/ssd0` | `/mnt/nfs/ssd0-39` | FreshRSS + most service CT rootfs |
|
||||
| `192.168.178.31:/Proxmox` | `/mnt/nfs/proxmox-31` | archive/dump storage |
|
||||
| `192.168.178.54:/Proxmox` | `/mnt/nfs/proxmox-54` | archive/dump storage (same export as .31) |
|
||||
|
||||
## Recovery performed
|
||||
|
||||
```
|
||||
mount -t nfs -o vers=4,soft,intr 192.168.178.39:/mnt/ssd0 /mnt/nfs/ssd0-39
|
||||
mount -t nfs -o vers=4,soft,intr 192.168.178.31:/Proxmox /mnt/nfs/proxmox-31
|
||||
mount -t nfs -o vers=4,soft,intr 192.168.178.54:/Proxmox /mnt/nfs/proxmox-54
|
||||
```
|
||||
|
||||
then `pct start` for each stopped CT (all with `onboot=1`). All 18 containers are up and their
|
||||
Tailscale peers are online again.
|
||||
|
||||
## Permanent fix
|
||||
|
||||
`m93p-nfs-recover.service` on m93p — enabled at `multi-user.target`, ordered after
|
||||
`pve-guests.service`:
|
||||
|
||||
- Script: `/usr/local/sbin/m93p-nfs-recover.sh`
|
||||
- Retries the three NFS mounts up to 10 times, 20s apart
|
||||
- Then starts any CT still `stopped` whose config has `onboot: 1`
|
||||
- Log: `/var/log/m93p-nfs-recover.log`
|
||||
- Verify: `systemctl status m93p-nfs-recover.service`
|
||||
|
||||
## Runbook
|
||||
|
||||
If a service on m93p is unreachable and `pct list` shows it `stopped`:
|
||||
|
||||
1. `ssh root@100.104.51.98 'mount | grep /mnt/nfs/'` — expect three lines
|
||||
2. If a line is missing: `systemctl start m93p-nfs-recover.service` (it remounts and restarts the CTs)
|
||||
3. `pct list` to confirm `running`; then check the service from the Hermes host
|
||||
4. FreshRSS specifically: `curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/`
|
||||
|
||||
## Lessons
|
||||
|
||||
- A Google Reader API `Auth failed ... timed out` error is an availability symptom first, an auth
|
||||
problem last. Probe the endpoint with curl before touching credentials.
|
||||
- The digest script needs up to 240s; a 180s timeout produced a spurious exit 124.
|
||||
- FreshRSS on m93p is **HTTP only** — `https://freshrss.kangaroo-eel.ts.net` fails because no
|
||||
Tailscale serve/funnel is configured for CT 1260.
|
||||
@@ -0,0 +1,134 @@
|
||||
---
|
||||
title: Mnemosyne Health Monitor auto-paused after gateway-connection failure
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-30
|
||||
updated: 2026-10-31
|
||||
verified_on: 2026-10-31
|
||||
confidence: high
|
||||
tags: [cron, mnemosyne, gateway, incident, delivery]
|
||||
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
|
||||
---
|
||||
|
||||
# Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30)
|
||||
|
||||
**Status: RESOLVED — 2026-10-30T10:02:48.662+08:00** (updated by the 2026-10-31 audit; everything below is the
|
||||
2026-10-30 03:3x snapshot and its "job is disabled / notice never delivered" reading did not survive the day).
|
||||
**What actually happened:** the 00:01:08 slot was **still running**, not dead — it closed **`completed` at
|
||||
10:02:48** after **10 h 01 m 40 s** and **delivered**; incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved`
|
||||
at 2026-10-30T10:02:48.662+08:00**, and the job is live again (`enabled: true`, `last_status: ok`,
|
||||
`failure_streak: 0`). `deliveries.db` finished 2026-10-30 with **8 rows, all `delivered`** (latest 18:46:04) — the
|
||||
zero-row reading was an artefact of the day having barely begun. ⚠ Its **12:01 slot was then lost to the CT460
|
||||
host outage** and fired late in the 16:37:21 catch-up burst, still `running` at the 2026-10-31 audit — see
|
||||
[[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].
|
||||
|
||||
**Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
|
||||
|
||||
## Summary
|
||||
|
||||
The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its
|
||||
2026-10-30 00:01 slot with a **gateway-connection** fault. The run closed with status **`unknown`**,
|
||||
the scheduler **auto-paused the job**, and the job's own failure notice **failed to send** — so the
|
||||
only trace of the fault is the incident row in `cron_incidents`.
|
||||
|
||||
## What happened
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Job | `ea79a2107e36` — Mnemosyne Health Monitor |
|
||||
| Schedule | `0 */12 * * *` (every 12 h) |
|
||||
| Started | 2026-10-30 00:01:08.343024+08:00 |
|
||||
| Finished | 2026-10-30 01:04:22.510091+08:00 (1 h 03 m 14 s) |
|
||||
| Execution status | **`unknown`** |
|
||||
| Job `last_status` | **`error`**, `failure_streak: 1` |
|
||||
| Job `enabled` | **`false`** — auto-paused 2026-10-30 01:04:22 |
|
||||
| Incident | `ea79a2_d93c3acdc4f2`, `failure_type: unknown`, state **`detected`** |
|
||||
| Delivery | **failed — never sent** |
|
||||
|
||||
Error text on the execution row:
|
||||
|
||||
> Cron execution unverified; reconcile
|
||||
> `/home/hermes/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`:
|
||||
> Gateway connection failed; no local fallback
|
||||
|
||||
## The two traps in this one fault
|
||||
|
||||
### 1. A failure notice that was itself never delivered
|
||||
|
||||
`hermes cron list` reports for this job:
|
||||
|
||||
> The result was not delivered: live adapter send failed: send_path_degraded;
|
||||
> live adapter delivery to `telegram:-1004321904721:102` failed
|
||||
|
||||
Confirmed at the artefact level: `deliveries.db` contains **zero rows created on 2026-10-30**
|
||||
(2026-10-29 has 10 rows, all `delivered`). So the failure produced **no Telegram output at all** —
|
||||
not even a notice about the failure.
|
||||
|
||||
This is the same class as [[incidents/2026-08-08-gitea-config-backup-push-dead]]: a job whose green
|
||||
row hides a broken destination. Here the destination was broken *and* the row went red, but the
|
||||
**alert path** was the part that was dead. If Anthony had only watched `hermes cron list` for a
|
||||
non-clean row, he would have seen the job had paused; if he had only watched Telegram, he would have
|
||||
seen nothing at all.
|
||||
|
||||
### 2. `unknown` is not `failed`
|
||||
|
||||
The execution closed `unknown`, not `failed`. Per the wiki's own convention
|
||||
([[systems/scheduled-tasks]]), a `status: unknown` row with **no matching `deliveries.db` row** is a
|
||||
probable non-delivery that the dashboard hides behind the previous slot's error. The scheduler then
|
||||
auto-paused the job — so the fleet lost a **monitoring** job, not a reporting job.
|
||||
|
||||
## Root cause
|
||||
|
||||
**Unverified.** The error names a gateway-connection failure with "no local fallback" during the
|
||||
reconciliation of an admission record. The admission file referenced in the error is
|
||||
`~/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`.
|
||||
|
||||
This is **not** the same failure as the 2026-10-26 planned-restart orphan class
|
||||
([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]), though the error string is similar
|
||||
("Cron execution unverified; reconcile …"). The 10-26 rows were `alerted` and auto-closed; this row
|
||||
is still `detected` and the job stayed paused. **No gateway restart was performed on 2026-10-30**
|
||||
around that window, so the trigger is not established.
|
||||
|
||||
## Current state (verified 2026-10-30 03:3x AWST)
|
||||
|
||||
- Live `jobs.json`: `ea79a2107e36` → `enabled: false`, `last_status: error`, `failure_streak: 1`,
|
||||
`paused_reason` = the gateway-connection text above.
|
||||
- Live `cron_incidents` (table lives inside `executions.db`): `ea79a2_d93c3acdc4f2` is the **only**
|
||||
new row since 2026-10-29; `closed_at` is **NULL**, `acked_at` is **NULL**, `alerted_at` is **NULL**.
|
||||
- The other unclosed row is `0e753e_e7f8194a7d06` (Facebook monthly, `failure_type: agent`, `detected`
|
||||
since 2026-10-01, clears on its 2026-11-01 slot) — **2 unclosed rows in total**.
|
||||
- Fleet is now **29 jobs — 25 enabled / 4 disabled**; this job is the 4th disabled one.
|
||||
- The rest of the fleet was healthy in the same window: 2026-10-29 settled **177 `completed` / 0
|
||||
`failed`**, and 2026-10-30 (to 05:4x) shows 37 `completed`, 1 `running` (this stocktake), 1
|
||||
`unknown` (this job), 0 `failed`.
|
||||
|
||||
## Actions taken
|
||||
|
||||
1. Recorded the incident and its two distinct failure modes here.
|
||||
2. Updated [[systems/scheduled-tasks]] — the job's row, the Disabled Jobs table, the fleet counts
|
||||
(29/25/4), the non-clean `last_status` count (3 → 4), the "Failing (last run failed outright)"
|
||||
bullet (0 → 1), and the "Failing (delivery only)" bullet (1 → 2).
|
||||
3. Updated [[infrastructure/hosts]] for the cron count.
|
||||
|
||||
## Actions NOT taken — and why
|
||||
|
||||
- **The job was not re-enabled.** Re-enabling a job whose gateway connection failed an hour ago
|
||||
risks an immediate re-pause, and it would close the incident's purpose. The recovery belongs to
|
||||
whoever owns the gateway fix.
|
||||
- **No gateway restart was performed.** This was a stocktake run, and restarting the gateway is a
|
||||
disruptive action with its own orphan-risk class ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]).
|
||||
The fault is recorded, not papered over.
|
||||
|
||||
## Next step
|
||||
|
||||
Re-enable `ea79a2107e36` **after** the gateway connection is confirmed healthy, then confirm the
|
||||
run completes and — critically — that a delivery row appears in `deliveries.db`. The point of this
|
||||
monitor is to report on exception to topic `102`; if the topic path is still degraded, re-enabling it
|
||||
recreates a silent monitor.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]] — fleet ledger
|
||||
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — the earlier "execution unverified" class
|
||||
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — green row, dead destination
|
||||
- [[systems/mnemosyne]] — the subsystem being monitored
|
||||
@@ -0,0 +1,102 @@
|
||||
---
|
||||
title: Asana Is Unreachable From CT460 on Both Paths — MCP Server Never Authenticated, Hosted Connector Rejects the Call Shape
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-01
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
review_after: 2026-11-08
|
||||
confidence: high
|
||||
tags: [hermes, automation, mcp, connector, credential, silent-failure]
|
||||
sources: [live-mcp-list, live-config-yaml, live-mcp-tokens, live-logs, live-probe]
|
||||
---
|
||||
|
||||
# Asana Is Unreachable From CT460 on Both Paths — MCP Server Never Authenticated, Hosted Connector Rejects the Call Shape
|
||||
|
||||
## Headline
|
||||
|
||||
The estate has **two independent Asana integrations on CT460, and neither can complete a read** — which
|
||||
went unnoticed until an interactive session spent 2026-10-31 evening retrying both to answer a single
|
||||
question ("where does Remembrance Day stand?").
|
||||
|
||||
| Path | Surface | Live state 2026-10-31/11-01 |
|
||||
|---|---|---|
|
||||
| **MCP server `asana`** | `https://mcp.asana.com/v2/mcp`, `auth: oauth`, `enabled: true` (`hermes mcp list` → ✓ enabled) | **Never authenticated.** Every attempt logs `MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run \`hermes mcp login asana\` interactively first to complete initial authorization.` |
|
||||
| **Hosted connector** | `connectors__asana__GET_A_TASK`, `…__SEARCH_TASKS_IN_WORKSPACE` | Reaches Asana but returns **`422 INVALID_ARGUMENTS` — "Input should be a valid list on parameter \`opt_fields\`"**. 12+ identical retries in one session. |
|
||||
|
||||
## Evidence
|
||||
|
||||
**MCP server — no token, ever.** `~/.hermes/mcp-tokens/` holds exactly one file:
|
||||
|
||||
```
|
||||
-rw------- 1 hermes hermes 319 Oct 26 01:07 asana.client.json
|
||||
```
|
||||
|
||||
A registered **client** with **no token file** → the OAuth flow was started (client registered
|
||||
**2026-10-26 01:07**) and never completed. `config.yaml → mcp_servers.asana` reads
|
||||
`{auth: oauth, enabled: true, url: https://mcp.asana.com/v2/mcp}`. The earliest Asana trace in retained
|
||||
logs is an **HTTP 401** from `mcp.asana.com/v2/mcp` at **2026-10-27 16:35:05**, followed immediately by
|
||||
OAuth discovery calls.
|
||||
|
||||
**Failure volume** (log lines across `logs/agent.log*` + `logs/errors.log*`; the failure is written to
|
||||
both files, so line counts roughly double the number of events):
|
||||
|
||||
| Day | `MCP OAuth setup failed for 'asana'` lines |
|
||||
|---|---|
|
||||
| 2026-10-21 | 455 |
|
||||
| 2026-10-22 | 469 |
|
||||
| 2026-10-23 | 576 |
|
||||
| 2026-10-24 | 576 |
|
||||
| 2026-10-25 | 576 |
|
||||
| 2026-10-26 | 301 |
|
||||
| 2026-10-27 | 439 |
|
||||
| 2026-10-28 | 1 150 |
|
||||
| 2026-10-29 | 1 154 |
|
||||
| 2026-10-30 | 862 |
|
||||
| **2026-10-31** | **1 156** |
|
||||
| 2026-11-01 (to 00:16) | 14 |
|
||||
| **Total retained** | **7 728** |
|
||||
|
||||
The earliest retained line is **2026-10-21 05:09:43** (`errors.log.2`) — the gap **predates the retained
|
||||
log window**; this is a chronic condition, not a 10-31 regression. The 10-31 peak coincides with the
|
||||
evening Pulse session hammering the path.
|
||||
|
||||
**Hosted connector — the call, not the credential.** From `20261031_210329_b35440` (the Conduit Pulse
|
||||
"Remembrance Day" session), repeated at 22:17:47, 22:20:42, 22:26:31, 22:31:25, 22:36:45, 23:14:45,
|
||||
23:26:46, 23:28:55, 23:30:34, 23:39:31, 23:41:48, 23:51:19 — each call burning 11–20 s:
|
||||
|
||||
```
|
||||
tools.managed_gateway_identity: connector connectors.execute 422 INVALID_ARGUMENTS
|
||||
Tool tool_call returned error: {"results":[{"name":"connectors__asana__GET_A_TASK","error":
|
||||
{"code":"INVALID_ARGUMENTS","message":"Invalid request data provided\n- Input should be a valid list
|
||||
on parameter `opt_fields`…"}}]}
|
||||
```
|
||||
|
||||
The connector is **authorised** (it is the MCP path that cannot authenticate), so the block is purely
|
||||
argument shape: `opt_fields` must be a **JSON list**, and a string/comma-form is rejected.
|
||||
|
||||
## Impact
|
||||
|
||||
- **Non-interactive sessions (cron, background jobs, Conduit background investigations) cannot read
|
||||
Asana at all** — the MCP path needs an interactive browser login that cron cannot provide.
|
||||
- **Interactive sessions** get the hosted connector, but the agent re-issued the same mis-shaped call
|
||||
a dozen times instead of adapting — ~3 minutes of model time spent failing identically.
|
||||
- Asana-backed user requests (task status behind the Conduit Pulse flow) degrade to guesses.
|
||||
|
||||
## Remediation (not performed — this audit records, it does not re-authorise)
|
||||
|
||||
1. Run **`hermes mcp login asana`** from an interactive session to complete the OAuth flow; confirm
|
||||
`~/.hermes/mcp-tokens/asana.json` (or equivalent token file) appears afterwards, then re-probe the
|
||||
MCP server.
|
||||
2. **Or** disable the `asana` MCP server if the hosted connector is the intended path — two Asana paths
|
||||
on one host is itself the source of the confusion, and one of them has never worked.
|
||||
3. For the connector: pass **`opt_fields` as a JSON list**, not a comma-separated string.
|
||||
4. Watch for the silent-cost pattern: a path that fails **every** time still burns a tool call each
|
||||
time. 1 100+ failures/day means the failure is being retried, not fixed.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/hermes-agent]] — MCP server inventory and auth state
|
||||
- [[systems/conduit]] — the Pulse flow that needed the Asana task status
|
||||
- [[systems/scheduled-tasks]] — cron fleet (the surface that cannot authenticate interactively)
|
||||
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the same "green row, dead destination" class
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
title: "Conduit \"voice mode: local\" never applies — Mac mini speech server's LLM proxy is disabled"
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-01
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
confidence: high
|
||||
tags: [conduit, voice, local-live, mac-mini, speech-server, investigation]
|
||||
sources:
|
||||
- session 20261031_193709_40b9c2 (Conduit voice session, started 2026-10-31 19:37:09 AWST)
|
||||
- ~/.hermes/conduit-settings.json, ~/.hermes/conduit-calls.json, ~/.hermes/config.yaml
|
||||
- live probes of the Mac mini speech server (LLM endpoint, /v1/usage, realtime socket)
|
||||
---
|
||||
|
||||
# Conduit "voice mode: local" never applies
|
||||
|
||||
**Status: OPEN — investigation in progress. Nothing was changed on the Mac mini speech server.**
|
||||
|
||||
On 2026-10-31 Anthony reported (from the phone): switching the Conduit app's voice
|
||||
setting to **"local"** shows a **successful** connection test, but the change never
|
||||
takes effect. A background job launched from that voice conversation
|
||||
(`20261031_193709_40b9c2`, started **19:37:09 AWST**, 30 messages / 17 tool calls) was
|
||||
still open at the 2026-11-01 00:2x audit.
|
||||
|
||||
## Why the test succeeds and the change still does nothing
|
||||
|
||||
The "test connection" button checks **socket reachability only** — it never exercises
|
||||
the model path. So it passes while every local turn is still impossible:
|
||||
|
||||
| # | Finding | Evidence |
|
||||
|---|---------|----------|
|
||||
| 1 | The Mac mini speech server's **LLM proxy is disabled** | its LLM endpoint answers **HTTP 501** — *"The LLM proxy is disabled. Start the server with `--enable_llm_proxy` to enable it."* |
|
||||
| 2 | The Mac has never served a local turn | `/v1/usage` → **`llm_proxy.requests: 0`** across **25** observed turns |
|
||||
| 3 | The realtime handshake is not understood | the Mac's realtime socket **accepts the connection** but rejects **`session.update`** as an **unknown event** |
|
||||
| 4 | The app's test button proves nothing about the model path | it returns success on socket reachability alone |
|
||||
|
||||
Findings 1–3 are the strongest single explanation: **even with `voice.mode = local`
|
||||
set on the phone, the Mac cannot call a model back.** Finding 4 is why the UI reports
|
||||
success.
|
||||
|
||||
## `voice.mode` is phone-side, not host-side
|
||||
|
||||
The host never reads a `voice.mode` key — there is none to read:
|
||||
|
||||
- `~/.hermes/config.yaml → voice` holds **`auto_tts`**, **`local_live`**,
|
||||
**`speech_server`** — **no `mode`**.
|
||||
- `~/.hermes/conduit-settings.json` contains **no `voice.*` keys at all**; its only
|
||||
device state is `devices/<UUID>/mode: small` for three devices.
|
||||
- The device itself reports **`voice.mode: "classic"`**.
|
||||
|
||||
So "set mode to local and the change does not apply" cannot be diagnosed on the host
|
||||
config — the selector lives in the app, and what the host does with it is a
|
||||
**`local_live`** path decision.
|
||||
|
||||
## A misread that was caught mid-session (worth keeping)
|
||||
|
||||
`local_live.py:882` is the **only** writer of `_local_live_last_request`, and it sits
|
||||
inside the **chat-completions route**. A "recent `last_request_at`" reading during the
|
||||
investigation therefore indicated **this investigation's own probe**, not phone
|
||||
traffic — the agent initially read it as phone activity and corrected the call in the
|
||||
same session. Live check: `conduit-calls.json → last_request_at` is **`null`**.
|
||||
|
||||
Lesson: on this integration, `last_request_at` is proof of *some* HTTP request hitting
|
||||
the local-live route, **not** proof of a phone call.
|
||||
|
||||
## Configuration at the time of writing
|
||||
|
||||
Changed during the 2026-10-31 session (`config.yaml` mtime **21:02:30**):
|
||||
|
||||
- `voice.local_live.front_model` = `{base_url: http://100.78.63.20:20128/v1, model: mistral/voxtral-small-latest}`
|
||||
— was `mistral/ministral-14b-latest`.
|
||||
|
||||
Unchanged: `voice.speech_server.url = wss://…:8443/v1/realtime`, `voice.auto_tts: false`,
|
||||
`tts.provider: ollama_local`, `stt.provider: local`.
|
||||
|
||||
The Conduit dashboard gateway was restarted at **19:35:02** (pid 176771), minutes
|
||||
before the session opened.
|
||||
|
||||
## Prompting context (candidate, not proven)
|
||||
|
||||
The plugin **self-updated to `0.122.2` on 2026-10-31 18:05**, which drove a gateway
|
||||
reload at **18:06:41** → [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]].
|
||||
A newly shipped local-live path is a reasonable regression candidate, but the
|
||||
disabled LLM proxy would break local mode on **any** plugin version, so the version
|
||||
change is not yet implicated.
|
||||
|
||||
## Open items
|
||||
|
||||
- **Enable the Mac mini's LLM proxy** (`--enable_llm_proxy`) or confirm the intended
|
||||
Local Live architecture — with the proxy off, finding 1 alone explains the symptom.
|
||||
- Confirm the **realtime event vocabulary** the plugin expects vs. what the Mac's
|
||||
socket implements (`session.update` is currently rejected as unknown).
|
||||
- Decide whether `front_model` should be a **text** or **audio** model — a
|
||||
chat-completions `base_url` pointed at an audio model may itself be the mismatch.
|
||||
- Re-test from the phone after any change; the app's own test button is not evidence.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/conduit]]
|
||||
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
|
||||
- [[incidents/2026-10-29-conduit-voice-silence]]
|
||||
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in)
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-01
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
review_after: 2026-11-08
|
||||
confidence: high
|
||||
tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure]
|
||||
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-probe]
|
||||
---
|
||||
|
||||
# Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in)
|
||||
|
||||
## Headline
|
||||
|
||||
On **2026-10-31 at 18:06:41 AWST** the gateway received a **`SIGUSR1` reload** (`systemctl reload` /
|
||||
`hermes gateway restart`). It deferred its own stop for in-flight work — and this time it **waited the
|
||||
whole deferral cap**, hit it at **18:37:38** with *"drain timed out with interrupted agents"*, and
|
||||
force-restarted. The casualty was `ea79a2107e36` **Mnemosyne Health Monitor** (12-hourly `0 */12 * * *`),
|
||||
whose 12:01:06 slot had been running for **6 h 38 m 09 s**. It closed **`failed`** at 18:39:15 on
|
||||
**`RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership`**.
|
||||
|
||||
This is the **third variant of the orphan class** — after the 2026-10-20 *unclean crash* exit and the
|
||||
2026-10-26 *planned `SIGTERM` whose drain timed out after 1.35 s* — and the first where the drain
|
||||
mechanism behaved as designed and still ended in a lost run.
|
||||
|
||||
## Timeline (live gateway log, `~/.hermes/logs/agent.log`)
|
||||
|
||||
| Time (AWST) | Event |
|
||||
|---|---|
|
||||
| 18:06:41.772 | `SIGUSR1 received (systemctl reload / hermes gateway restart): performing a graceful gateway restart — drain active turns, exit, supervisor relaunches.` |
|
||||
| 18:06:41.788 | `Restart requested with 2 active work unit(s); deferring stop() until they finish (cap=1800s) so in-flight turns are not amputated (#77184)` |
|
||||
| 18:06:41.899 | Deferred on **1** unit: the cron unit `ea79a2107e36` (`elapsed_s: 21950.7`, `external: True`, `wedged: False`, **`restart_safe: True`** — *excluded* from the wait) plus `{'kind': 'managed', 'pid': 397}` |
|
||||
| 18:07:11 → 18:11:12 | Countdown logged every 30 s: 1770 s → 1529 s remaining, same single `managed pid 397` unit |
|
||||
| 18:11:23 | **Second `SIGUSR1`** arrives while the first restart is still deferred |
|
||||
| 18:20:13 | `Restart deferred: … (988s remaining before force drain): [{'kind': 'managed', 'pid': 397}]` |
|
||||
| **≈18:36:41** | **Cap reached** (18:06:41 + 1 800 s) |
|
||||
| 18:37:38.341 | `Skipping .clean_shutdown marker — drain timed out with interrupted agents; next startup will recover their interrupted turns.` |
|
||||
| 18:38:43.284 | New gateway starts; 18:39:14 reconnection watcher for 1 failed platform (`relay`) |
|
||||
| 18:39:15.208 | Failure **delivered** (delivery row `9e7da0fa…`, `for_failure=1`, status `delivered`) |
|
||||
| 18:39:15.563 | Execution `9e7da0fafd2847b69643c6f235cb8651` closes **`failed`** — 6 h 38 m 09 s |
|
||||
| 18:39:23 / 18:45:23 / 19:00:22 | Three derivative `96eb95ce12f2` (log-error watchdog) reds quote the cron error; the watchdog clears on its 19:15:21 slot |
|
||||
| 19:23:39 → 19:24:40 | **Second restart, clean**: `drain took 0.00s, timed_out=False`; `Previous gateway exited cleanly — skipping session suspension` |
|
||||
|
||||
Current gateway: `hermes-gateway.service` active since **19:23:56**, PID `175269` (4 h 43 m at the
|
||||
2026-11-01 00:07 probe). Nothing failed for the remaining ~5 h of 10-31 (every `96eb95ce12f2` and
|
||||
`0c771463c3cd` slot from 19:15:21 onward `completed`).
|
||||
|
||||
## Incident-ledger effect
|
||||
|
||||
| Row | Job | State | First seen | Last seen |
|
||||
|---|---|---|---|---|
|
||||
| `ea79a2_9197df32320b` | `ea79a2107e36` | **`alerted` (open)** | 2026-10-31T18:06:44 | 2026-10-31T18:06:44 |
|
||||
| `ea79a2_a4661554a3c2` | `ea79a2107e36` | **`alerted` (open, reused row)** | **2026-10-26**T14:35:59 | 2026-10-31T18:39:15 |
|
||||
| `0e753e_e7f8194a7d06` | `0e753e9720fd` | `detected` (open, unchanged) | 2026-10-01T09:00:55 | 2026-10-01T09:00:55 |
|
||||
|
||||
The `error_sig` match re-opened the **2026-10-26 row** rather than filing a fresh one, so two rows now
|
||||
describe one event: the earlier *"Cron execution unverified; reconcile `admissions/292056e5742c1ba4…`"*
|
||||
(`failure_type: unknown`) and the definitive *fire-claim* row. **Unclosed incident rows 1 → 3.**
|
||||
|
||||
## What is genuinely new here
|
||||
|
||||
1. **The deferral cap is reachable in normal operation, not just under a stuck process.** The 10-26
|
||||
incident drained in **1.35 s**; this one waited the full **1 800 s** because a `managed` unit
|
||||
(`pid 397`) stayed registered. Force drain is therefore a *routine* outcome of any reload that
|
||||
overlaps a long-running cron job.
|
||||
2. **`restart_safe: True` did not protect the execution.** The scheduler explicitly classified the cron
|
||||
unit as restart-safe and *excluded* it from the wait — and the run still died with a lost claim. The
|
||||
flag protects the *scheduler's* bookkeeping, not the running agent's lease.
|
||||
3. **A `SIGUSR1` reload is not a lighter-weight restart.** The pre-restart log shows plugin/update
|
||||
machinery active minutes earlier (`.update-<id>/plugin` capability checks at 18:05:41 and 18:19:53,
|
||||
`plugin scan passed` at 18:20:22) — the reload behaved exactly like a full stop/start for the
|
||||
in-flight job.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Why is a `managed` (non-cron) work unit allowed to hold the gateway for the entire 30-minute cap?
|
||||
What is `pid 397`, and should it be exempt from the "wait" set (or bounded separately)?
|
||||
- The 12:01 slot had already run **6 h 38 m** against **10 h 01 m** for its 10-30 catch-up. Should a
|
||||
reload deferral weight long cron runs differently — or should the cron lease survive a supervisor
|
||||
relaunch, as the `restart_safe` flag implies?
|
||||
- Two `SIGUSR1`s 4 m 42 s apart produced a **single** interruption and then a second clean restart at
|
||||
19:23 — is the double signal expected?
|
||||
|
||||
## Impact
|
||||
|
||||
- One job failure + three derivative watchdog reds = **all four** of 2026-10-31's failed executions
|
||||
(day closed **160 `completed` / 4 `failed`**).
|
||||
- `ea79a2107e36` now reads `last_status: error`, **`failure_streak: 2`** — the only non-zero streak in
|
||||
the fleet; non-clean `last_status` **3 → 4**, failing (run failed outright) **0 → 1**.
|
||||
- Its next slot (2026-11-01 **00:00:41**) was already `running` at audit time.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]] — fleet ledger and the 10-31 window numbers
|
||||
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — variant 2, the 1.35 s drain timeout
|
||||
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — variant 1, the unclean crash exit
|
||||
- [[systems/hermes-agent]] — gateway lifecycle record
|
||||
- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — the same job, the day before
|
||||
@@ -0,0 +1,115 @@
|
||||
---
|
||||
title: Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-11-02
|
||||
updated: 2026-11-02
|
||||
verified_on: 2026-11-02
|
||||
review_after: 2026-11-09
|
||||
confidence: high
|
||||
tags: [cron, scheduler, worker, sigterm, orphan, cold-start, silent-failure]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-logs, live-jobs-json, live-probe]
|
||||
---
|
||||
|
||||
# Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State
|
||||
|
||||
## Headline
|
||||
|
||||
On **2026-11-01** three separate `cron.scheduler` **external workers** were terminated with
|
||||
**`SIGTERM` (`-15`) after they had adopted a durable execution claim but before they wrote a terminal
|
||||
state**, so each run closed as *"whether side effects ran is unknown"*. The three casualties were
|
||||
`85edf6772bca` (Tech & AI Digest, 07:30:46), `96eb95ce12f2` (log-error watchdog, 09:02:18) and
|
||||
`0e753e9720fd` (Facebook scrape, 09:02:18). **This is a fourth variant of the orphan class** — after the
|
||||
2026-10-20 unclean crash, the 2026-10-26 planned `SIGTERM` whose drain timed out in 1.35 s, and the
|
||||
**2026-10-31 force-drain at the full 1 800 s cap** ([[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]).
|
||||
The distinguishing feature here: **no gateway restart, reload or drain occurred anywhere near these
|
||||
signals.** Gateway PID `175269` had been up since **2026-10-31 19:24:40** and stayed up; the relay
|
||||
reconnect counter ran continuously through both incidents (attempt 147 at 07:24:58 → 166 at 09:01:34 →
|
||||
336 at 2026-11-02 00:10), which it could not do across a process restart. The `-15` came from the cron
|
||||
worker supervision path itself.
|
||||
|
||||
## Evidence
|
||||
|
||||
Identical `error` text on all three incident rows (`~/.hermes/cron/executions.db` → `cron_incidents`):
|
||||
|
||||
```
|
||||
Restart-safe cron worker exited with status -15 after adopting this execution, without writing a
|
||||
durable terminal state; whether side effects ran is unknown.;
|
||||
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in sys.modules after import
|
||||
of package 'cron', but prior to execution of 'cron.scheduler'; this may result in unpredictable behaviour
|
||||
```
|
||||
|
||||
| Incident id | Job | `first_seen_at` | `closed_at` | Execution outcome |
|
||||
|---|---|---|---|---|
|
||||
| `85edf6_94cd563ae0df` | `85edf6772bca` Tech & AI Digest | 2026-11-01 07:30:46 | 2026-11-01 11:01:12 | row **`completed`** / `delivered` (07:01:19 → 11:01:12, **4 h 00 m**) |
|
||||
| `96eb95_8091fba135e9` | `96eb95ce12f2` log-error watchdog | 2026-11-01 09:02:18 | 2026-11-01 09:16:18 | row **`unknown`** (09:01:03 → 09:02:13) |
|
||||
| `0e753e_2183c9040706` | `0e753e9720fd` Facebook scrape | 2026-11-01 09:02:18 | 2026-11-01 10:44:12 | row **`completed`** / `delivered` (09:01:03 → 10:44:12) |
|
||||
|
||||
All three rows carry `failure_type: unknown`, `acked_at: None`. Only `85edf6` and `0e753e` were
|
||||
auto-resolved by their own later successful run; `96eb95`'s row stayed `unknown` — **the only
|
||||
permanently non-clean execution of the whole day**.
|
||||
|
||||
## The 09:00 pair — the cold-start window, in full
|
||||
|
||||
| Time (AWST) | Event |
|
||||
|---|---|
|
||||
| 09:00:49.708 | `cron.scheduler: Cron external worker for job '96eb95ce12f2' did not acknowledge within 30s; leaving the durable execution claim untouched` |
|
||||
| 09:00:49 → 09:01:02 | The freshly spawned worker boots a full runtime — **`Plugin discovery complete: 83 found, 69 enabled`**, MCP capacity checks |
|
||||
| 09:01:13.552 | `Running job 'Facebook Birthday Monthly Scrape' (ID: 0e753e9720fd)` — the same worker then picks up a second job |
|
||||
| 09:02:13 | `96eb95ce12f2` execution row closes **`unknown`** |
|
||||
| 09:02:18.189 / .211 | Both incident rows written (`alerted` 09:02:29 / 09:02:32) |
|
||||
| 10:44:12 | `0e753e9720fd` row closes `completed` + `delivered` → incident resolved |
|
||||
| 09:15:45 → 09:16:19 | Next `96eb95ce12f2` slot runs `completed` → incident resolved |
|
||||
|
||||
**Mechanism, as far as the retained logs show it:** the spawn path treats a **30 s acknowledgement
|
||||
window** as the health signal for an external worker. A cold worker must import the scheduler, discover
|
||||
**83 plugins / 69 enabled** and register MCP tool schemas before it can acknowledge — a cost that
|
||||
plainly exceeds 30 s on this host. The supervisor therefore logs the miss and **leaves the durable claim
|
||||
untouched** rather than reclaiming it; the worker then starts the job anyway and is `SIGTERM`ed ~60–90 s
|
||||
later, orphan-claiming the run.
|
||||
|
||||
**What is NOT established:** *who* sent the `SIGTERM` in these three cases. No `SIGUSR1`, no
|
||||
`Restart requested`, no `drain`, no `UNCLEANLY`, and no `Previous gateway exited` marker appears in
|
||||
`gateway.log`, `agent.log` or `agent.log.1` in the 07:2x–09:1x window, and the relay/polling counters
|
||||
prove no process restart. Do not attribute this to a gateway reload — the signal source is unidentified.
|
||||
Treat this as an open mechanism question, not a solved one.
|
||||
|
||||
## Blast radius and recovery
|
||||
|
||||
- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`** against 161 / 4 on 10-31,
|
||||
138 / 0 on 10-30 and 177 / 0 on 10-29. The two `unknown` rows are precisely
|
||||
`96eb95ce12f2` @09:01:03 and `79f9bb41ecf5` @18:00:39 (a handoff failure — see
|
||||
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]).
|
||||
- **All three executions still reached the user** where they were meant to: the digest closed
|
||||
`delivered` at 11:01:12 and the Facebook scrape `delivered` at 10:44:12 (delivery rows
|
||||
`2026-11-01 11:01:12` and `2026-11-01 10:44:12`).
|
||||
- **No recurrence after 09:02:18** for the remaining ~15 h of 11-01 — every later `96eb95ce12f2`
|
||||
and `0c771463c3cd` slot (23 and 21 runs respectively) closed `completed`.
|
||||
- The `-15` class is **not** `[SILENT]`-shaped: each instance raised an incident row and each alerted
|
||||
(`85edf6` 07:31:14, `96eb95` 09:02:32, `0e753e` 09:02:29). Only the `unknown` execution status is
|
||||
invisible to the dashboard's status field.
|
||||
|
||||
## Why it matters
|
||||
|
||||
A `-15`-after-adoption means the scheduler cannot say whether the job's side effects ran. For a job with
|
||||
external writes (Gitea pushes, API POSTs, email sends) that is a **possible double-execution** on the
|
||||
next slot. The current design resolved this properly for two of three, but the class keeps recurring in
|
||||
a new form roughly every 10 days — 10-20, 10-26, 10-31, 11-01. **The recurring element across all four
|
||||
variants is the same: a live cron worker is signalled mid-adoption and the durable claim is left open.**
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Which supervisor sends the `-15` after a 30 s ack miss — and is the kill a timeout guard or an
|
||||
unrelated cleanup path?
|
||||
2. Does the ack window scale with plugin count? 83 plugins discovered in a cold worker is the visible
|
||||
cost driver; if the window is fixed at 30 s, cold-start jobs on this host are structurally exposed.
|
||||
3. Should a job with `failure_type: unknown` external side effects be blocked from its next slot until
|
||||
`~/.hermes/cron/admissions/*.json` is reconciled? Two admission files from 11-01 are still
|
||||
unreconciled (`141448f9…`, `69b5588f…`).
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
|
||||
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/hermes-agent]]
|
||||
@@ -0,0 +1,144 @@
|
||||
---
|
||||
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Recovered and Delivered 7 h 26 m Later
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-11-02
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
review_after: 2026-11-17
|
||||
confidence: high
|
||||
tags: [cron, gateway, handoff, delivery, weekly-review, silent-failure]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-jobs-json, live-logs, live-probe]
|
||||
---
|
||||
|
||||
# Weekly Review Lost at 18:00 on 2026-11-01 — Recovered and Delivered 7 h 26 m Later
|
||||
|
||||
> **Status: RESOLVED 2026-11-02T01:27:21+08:00.** The Sunday 18:00 run did *not* lose the review — it
|
||||
> resumed, completed and delivered at **01:27:21**, and the incident row auto-closed. The body below is
|
||||
> the record **as it stood at the 2026-11-02 audit**, before the recovery landed. See
|
||||
> [Resolution](#resolution-2026-11-02) for the corrected outcome and the answers to the open questions.
|
||||
|
||||
## Headline
|
||||
|
||||
The **Sunday 18:00 Weekly Review (`79f9bb41ecf5`, `0 18 * * 0`) did not run and nothing was delivered on
|
||||
2026-11-01.** At **18:05:13** its execution closed with `status: unknown`, `delivery_outcome: NULL` and
|
||||
the error *"Cron execution unverified; reconcile `…/cron/admissions/141448f9…json`: **Gateway connection
|
||||
failed; no local fallback**"*. Its incident row **`79f9bb_9c443b602eec` is still `alerted` at the time of
|
||||
this audit** (`first_seen` 18:05:13, `alerted_at` 18:05:32, `closed_at: NULL`) and the job reads
|
||||
`last_status: error`, **`failure_streak: 1`**. There is **no `deliveries.db` row for any slot after
|
||||
11:01:12 on 11-01** — the last delivery of the day was the Tech & AI Digest at 11:01:12.
|
||||
|
||||
## Timeline (AWST, 2026-11-01)
|
||||
|
||||
| Time | Event |
|
||||
|---|---|
|
||||
| 18:00:39 | **Four jobs handed to external workers in one batch**: `2fd50c87b0d0`, `96eb95ce12f2`, `b63ed5817212`, `79f9bb41ecf5` |
|
||||
| 18:00:57 / 18:01:04 | `96eb95ce12f2` and `2fd50c87b0d0` run fine (`completed`) |
|
||||
| **18:05:13.697679** | `b63ed5_6f45aeea97f0` opens — *"Cron execution unverified; reconcile `…/admissions/69b5588f…json`: Gateway connection failed; no local fallback"* (`alerted_at: None` — **never alerted**) |
|
||||
| **18:05:13.708153** | `79f9bb_9c443b602eec` opens with the **identical** message, pointing at `…/admissions/141448f9…json` |
|
||||
| 18:05:32 | `79f9bb` incident **alerted** |
|
||||
| 18:06:21 | `b63ed5817212` execution closes `completed` (18:00:39 → 18:06:21) → its incident auto-resolves **68 s later** |
|
||||
| — | `79f9bb41ecf5` closes at 18:05:13 with **no terminal state, no delivery, no recovery** |
|
||||
| 18:04:24 → 18:10 | `agent.log`: `[reaper] Active gateway work detected … deferring restart`, then *"Reaper: Plan A restart completed successfully (5m02s)"* — the **Plan A/B path is RSS compaction, not a process restart** (no `SIGTERM`/`SIGUSR1` and no relay-counter reset) |
|
||||
|
||||
## What actually reached the user
|
||||
|
||||
**Nothing.** Across the whole of 11-01 `deliveries.db` holds 8 rows:
|
||||
|
||||
| Time | Status | Note |
|
||||
|---|---|---|
|
||||
| 02:39:33 | delivered | Mnemosyne-era failure notice (closes the 10-26 and 10-31 rows) |
|
||||
| 03:00:37 | delivered | stocktake |
|
||||
| **07:18:50** | **failed** | `configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id` — job `066bb5191280` (see [[systems/messaging-integrations]]) |
|
||||
| 07:34:49 | delivered | — |
|
||||
| 08:36:51 | delivered | Gmail digest |
|
||||
| 10:44:12 | delivered | Facebook scrape |
|
||||
| 11:01:12 | delivered | Tech & AI Digest |
|
||||
| — | — | **no row after 11:01:12** |
|
||||
|
||||
So of the day's eight delivery attempts, **seven reached their target and one (`066bb5191280`) landed in
|
||||
the DM thread-less while being recorded `failed`**. The Weekly Review's absence is not a suppressed
|
||||
delivery — it is an absent run.
|
||||
|
||||
## Second silent gap: a job that reported `error` with no execution row at all
|
||||
|
||||
`ecc1fc13655a` (FreshRSS Midday News Briefing) reads **`last_status: error`, `failure_streak: 1`,
|
||||
`last_run_at: 2026-11-01T12:01:00.374089+08:00`** — yet **no `executions` row exists for it on 11-01**
|
||||
(the last is 2026-10-31T12:01:07, `completed`/`delivered`) and **no `cron_incidents` row was raised.**
|
||||
That 12:00 slot overlapped the same `18:00:39`-style handoff batch pattern (three jobs claimed at
|
||||
12:00:53). This is the **second job of the day whose failure left no durable record**, and the third
|
||||
appearance of the "green/red row, no artefact" anti-pattern the wiki already documents
|
||||
([[systems/scheduled-tasks]], §9 of the maintenance playbook).
|
||||
|
||||
## Fleet state at this audit
|
||||
|
||||
- **Fleet 29 jobs — 26 enabled / 3 disabled** (`jobs.json`), unchanged from 10-31.
|
||||
- **Unclosed incident rows: 1 → 1, but a different row.** The 10-31 audit left
|
||||
`ea79a2_a4661554a3c2` / `ea79a2_9197df32320b` / `0e753e_e7f8194a7d06` open; **all three closed on
|
||||
11-01** (02:39:33 for the two `ea79a2` rows, **10:44:12** for `0e753e_e7f8194a7d06` — the
|
||||
Facebook-repin row that had been open since **2026-10-01 09:00:55**, i.e. 31 days). The new sole
|
||||
open row is `79f9bb_9c443b602eec`.
|
||||
- **Non-clean `last_status` 4 → 2** (`ea79a2107e36`, `79f9bb41ecf5`); **failing outright 1 → 1**
|
||||
(`79f9bb41ecf5`).
|
||||
- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`.**
|
||||
|
||||
## Open questions
|
||||
|
||||
> ⚠ **Questions 1–3 are answered in [Resolution](#resolution-2026-11-02).** Question 4 remains open.
|
||||
|
||||
1. **What did "Gateway connection failed; no local fallback" mean?** Same message for two different
|
||||
jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window.
|
||||
2. **Why did `b63ed5` self-heal in 68 s and `79f9bb` never do so?** Both were in the same batch with the
|
||||
same error string; one closed `completed`, the other never wrote a terminal state.
|
||||
3. **Should `79f9bb41ecf5` be manually re-run for 2026-11-01?** Anthony did not receive his weekly
|
||||
review.
|
||||
4. Are the two unreconciled admission files (`141448f9…`, `69b5588f…`) the durable record needed to
|
||||
decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot.
|
||||
|
||||
## Resolution (2026-11-02)
|
||||
|
||||
**The run was not lost. It resumed, finished, and delivered — 7 h 26 m 43 s after it started.**
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Execution | `be6ec11a8e68428084af9ab18fb148db` — **the same execution record** the audit saw stuck at `unknown` |
|
||||
| `scheduled_instant` | `2026-11-01T10:00:00+00:00` (18:00 AWST) |
|
||||
| `started_at` | `2026-11-01T18:00:39.125776+08:00` |
|
||||
| `finished_at` | **`2026-11-02T01:27:22.000881+08:00`** |
|
||||
| `status` | **`completed`** |
|
||||
| `error` | `NULL` |
|
||||
| `delivery_outcome` | **`delivered`** |
|
||||
| Delivery row | `2026-11-02T01:27:21.802246+08:00` — `delivered` |
|
||||
| Incident row | `79f9bb_9c443b602eec` → `state: resolved`, `closed_at: 2026-11-02T01:27:21.606691+08:00` |
|
||||
|
||||
Anthony received the Weekly Review at 01:27 AWST on 2026-11-02. There is still exactly **one** execution
|
||||
row for the 18:00:39 slot, so this is a reconciliation of the existing claim rather than a re-run — the
|
||||
`unknown` terminal state at 18:05:13 was replaced when the run finally wrote its real outcome.
|
||||
|
||||
**Answers to the open questions:**
|
||||
|
||||
1. *(the "no local fallback" message)* — the handoff to the external worker did ultimately succeed; the
|
||||
18:05:13 message recorded a **failed reconciliation attempt**, not a failed run. The run kept executing
|
||||
for a further 7 h 26 m and produced its deliverable. The message is therefore a **false negative from
|
||||
the reconciler**, not evidence that the work was abandoned.
|
||||
2. **`b63ed5` closed in 68 s; `79f9bb` closed in 7 h 26 m — both eventually reached `completed`/`delivered`.**
|
||||
The difference was run duration, not failure mode. The reconciler's window is shorter than the Weekly
|
||||
Review's runtime, so any long-running job in a handoff batch will be reported `unknown` while it is
|
||||
still working. **This is the general lesson: an `unknown` execution state on a long job is a timing
|
||||
artefact, not a verdict.**
|
||||
3. **No manual re-run was needed** — the run self-healed and delivered 12 minutes before the 2026-11-02
|
||||
audit would have recommended one. Had the audit re-triggered the job, it would have duplicated the run.
|
||||
|
||||
### Consequences for the audit method
|
||||
|
||||
The 2026-11-02 audit read `status: unknown` + `closed_at: NULL` at 00:4x and concluded the deliverable was
|
||||
lost. Both were true at the time of reading and both had inverted by 01:27. **A nightly audit that runs
|
||||
~2 h after midnight will read every long-running job mid-flight and must not treat a null terminal state as
|
||||
final until the job's expected runtime has elapsed** — see [[systems/scheduled-tasks]] §9.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]
|
||||
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/messaging-integrations]]
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-03
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
review_after: 2026-11-17
|
||||
confidence: high
|
||||
tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox, stream-timeout]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-state-db, live-jobs-json, session-history]
|
||||
---
|
||||
|
||||
# Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed
|
||||
|
||||
## Headline
|
||||
|
||||
**Third occurrence of the Stage-1 budget class, and the first that the cron ledger reports as healthy.**
|
||||
On **2026-11-02** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1 (Discovery)*) ran for
|
||||
**9 h 01 m 19 s** (`started 07:00:33.378239` → `finished 16:01:52.496335 +08:00`), consumed the harness's
|
||||
entire tool-calling iteration budget, and closed with:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| `status` | **`completed`** |
|
||||
| `error` | `NULL` |
|
||||
| `delivery_outcome` | **`suppressed`** |
|
||||
|
||||
No handoff artefact was written. **Nothing reached Notion, no cover letters were generated, and Stage 2 was
|
||||
never triggered.** Because the run's own status is `completed` with no error, a health check that inspects
|
||||
`executions.status` alone reports this day as green — the failure is invisible in the ledger.
|
||||
|
||||
## What the run did and did not do
|
||||
|
||||
The session (`Daily Marketing Job Radar - Discovery · Nov 02 16:01`, `local-368c8fe9…`) shows the discovery
|
||||
half genuinely worked:
|
||||
|
||||
- Camofox was restarted at run start after its HTTP server wedged (see [[systems/browser-backend]]).
|
||||
- **SEEK 51 unique roles, LinkedIn 60 unique job IDs, Indeed live, WA Government 35 results** — 197 raw
|
||||
candidates, **164 new after three-level dedup** against the 479-row Notion board.
|
||||
- A **URL-canonicalisation bug was found and fixed mid-run**: `au.seek.com` vs `au.seek.com.au` had been
|
||||
dropping **SEEK and Indeed entries** from the consolidated list; the fix was a job-ID-based
|
||||
canonicalisation.
|
||||
- Adverts were individually verified fresh and active; salaries, closing dates and JD bodies were pulled.
|
||||
|
||||
Then the ceiling hit. The runtime injected *"You've reached the maximum number of tool-calling iterations
|
||||
allowed. Please provide a final response…"* while the agent was still reading JD text for the borderline
|
||||
finalists, and the run terminated in the **scoring** phase.
|
||||
|
||||
The agent's own closing report was explicit — and it was **not delivered**:
|
||||
|
||||
> *"**Status: INCOMPLETE — handoff file NOT written, Stage 2 NOT triggered.** … Because the mandatory
|
||||
> deliverable (the handoff JSON) was not produced, today's roles have **not** been handed to Stage 2."*
|
||||
|
||||
That text is `delivery_outcome: suppressed`, so Anthony saw nothing on 11-02 — not the roles and not the
|
||||
warning that the roles were missing.
|
||||
|
||||
## Downstream: both Stage 2 slots behaved correctly
|
||||
|
||||
| Slot | Session | Ran | Result |
|
||||
|---|---|---|---|
|
||||
| 11:30 | `local-92665337…` (*Write and Letters*) | 11:30 → 12:25 | Exited `[SILENT]` after verifying Stage 1 was alive but had no handoff |
|
||||
| 14:30 (safety net) | `local-05a3f00b…` (*Write and Letters*) | 14:30 → 14:30 | Exited `[SILENT]` — 6 messages, 2 tool calls |
|
||||
|
||||
Both declines were correct: no handoff existed to consume. The safety-net design worked as intended — the
|
||||
gap is that Stage 1 cannot finish inside its budget.
|
||||
|
||||
## Recurrence pattern
|
||||
|
||||
| Date | Incident page | Failure shape |
|
||||
|---|---|---|
|
||||
| 2026-10-27 | [[incidents/2026-10-27-job-radar-budget-exhausted]] | Tool-call budget exhausted before the Notion write stage |
|
||||
| 2026-10-28 | [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] | Stage 1 hit the agent iteration cap before the handoff |
|
||||
| **2026-11-02** | **this page** | Same cap; **run recorded `completed` and its report suppressed** |
|
||||
| 2026-11-03 | [[incidents/2026-11-03-job-radar-stream-active-timeout]] | **A different ceiling** — `RuntimeError: stream active timeout` after 6 h 28 m; `failed`, notice delivered |
|
||||
|
||||
Three hits in seven days, each ending in the same place: discovery complete, handoff absent, Notion write
|
||||
stage never reached. **A fourth run — 2026-11-03 — ended the same way (no handoff, both Stage 2 slots declined) but on a
|
||||
different fault, and this time the ledger did go red; see the 2026-11-04 update below.**
|
||||
|
||||
## The "green row, no artefact" anti-pattern
|
||||
|
||||
This is the **fourth appearance** of the class [[systems/scheduled-tasks]] §9 of the maintenance playbook
|
||||
already documents (a job whose durable ledger entry says success while the thing it was supposed to produce
|
||||
does not exist). The new variant here is that the ledger flipped from red to green:
|
||||
|
||||
| Date | `cron_incidents` row for `dca8482e4f76` | `executions.status` |
|
||||
|---|---|---|
|
||||
| 2026-10-27 | `dca848_e93a538ade6b` (`failure_type: auth`) | non-clean |
|
||||
| 2026-10-28 | `dca848_0a2f03b5952b` (`failure_type: timeout`) | non-clean |
|
||||
| **2026-11-02** | **none raised** | **`completed`, `error: NULL`** |
|
||||
|
||||
The two earlier caps each raised an incident whose `error` text is literally the run's INCOMPLETE report.
|
||||
The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for `dca8482e4f76` with a
|
||||
`first_seen_at` on 2026-11-02; fleet-wide unclosed rows were **0** at the 2026-11-03 audit). The run's
|
||||
`delivery_outcome` is `suppressed`, so no message reached Anthony either.
|
||||
|
||||
**Detection rule:** for any job whose deliverable is a *file* (a handoff JSON, a report, a generated page),
|
||||
`executions.status` is not sufficient evidence. Verify the artefact exists at its expected path, or that
|
||||
`cron_incidents` stayed quiet for a reason, before treating the slot as served. Silence from
|
||||
`cron_incidents` is not the same as success — here it is the *symptom*.
|
||||
|
||||
## 2026-11-04 update — the missing incident row was a property of the cap, not of long runs
|
||||
|
||||
**Job Radar Stage 1 lost a second consecutive day on 2026-11-03 — and this time the harness killed the
|
||||
turn, so the ledger went red.** The 07:00 `dca8482e4f76` ran **07:00:33.819190 → 13:28:52.305995
|
||||
(6 h 28 m 18 s)** and closed **`status: failed` / `error: RuntimeError: stream active timeout` /
|
||||
`delivery_outcome: delivered`**, raising incident **`dca848_dfd4bea84f2a`** (`failure_type: timeout`,
|
||||
**`alerted` 13:28:55.141618, still open**) — fleet unclosed rows **0 → 1**.
|
||||
→ [[incidents/2026-11-03-job-radar-stream-active-timeout]]
|
||||
|
||||
**This answers open question 3 below (partly): suppression tracked the terminal state, not the run length.**
|
||||
When the run ends `completed` — as the cap produces — its report is suppressed; when the harness itself
|
||||
kills the turn, the terminal state is `failed` and both the incident row and the notice appear. 11-02:
|
||||
nothing delivered. 11-03: the failure notice went out at **13:28:51.741724** and Pulse surfaced it to
|
||||
Anthony in the desktop session `20261031_190813_95fbde` (report `p34c8e899-1`, covering Tue 03 Nov 13:01
|
||||
onward): *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a
|
||||
stream active timeout."*
|
||||
|
||||
**It also narrows open question 2 below:** the ledger *can* record a non-`completed` state when a long run
|
||||
dies — the iteration cap is the specific case that still escapes as `completed`.
|
||||
|
||||
**And it partially answers open question 4: the candidate set was lost again.** Both Stage 2 slots declined
|
||||
silently (`5fc292ff0f3b` 11:30:36 → 11:32:16 and 14:30:32 → 14:33:11) and
|
||||
`~/.hermes/radar-ref/handoff/` still holds **only** `2026-10-27.json`, `2026-10-29.json` and
|
||||
`2026-10-30.json` — so **2026-11-02's 164 deduped candidates and 2026-11-03's discovery are both
|
||||
unrecovered**, two days running.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Is the iteration budget the right control, or is the task too large for one agent turn?** Three caps in
|
||||
a row suggest Stage 1's work (four job boards → dedup → individual advert verification → JD reading →
|
||||
scoring) does not fit a single turn at any budget.
|
||||
2. **Should the harness record a cap-hit as something other than `completed`?** A distinct terminal status
|
||||
(or a non-null `error`) would make this visible to every existing health check.
|
||||
3. **Should the run's final response be delivered even when the job "succeeded"?** Suppressing an INCOMPLETE
|
||||
report converts a loud failure into silence.
|
||||
4. **Did the 11-02 roles get picked up on the next day's run**, or is 2026-11-02's candidate set lost?
|
||||
The 164 deduped candidates were never persisted.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
|
||||
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
|
||||
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/browser-backend]]
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-11-04
|
||||
updated: 2026-11-06
|
||||
verified_on: 2026-11-06
|
||||
review_after: 2026-11-11
|
||||
confidence: high
|
||||
tags: [cron, job-radar, stream-timeout, silent-failure, notion, context-compaction]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-deliveries-db, live-jobs-json, live-state-db, live-gateway-log, live-radar-handoff-dir, session-history]
|
||||
---
|
||||
|
||||
# Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
|
||||
|
||||
## Headline
|
||||
|
||||
**The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the
|
||||
ledger went red.** On **2026-11-03** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1
|
||||
(Discovery)*) ran **6 h 28 m 18 s** (`started 07:00:33.819190` → `finished 13:28:52.305995 +08:00`) and
|
||||
closed:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| `status` | **`failed`** |
|
||||
| `error` | **`RuntimeError: stream active timeout`** |
|
||||
| `delivery_outcome` | **`delivered`** (the failure notice, `for_failure: 1`) |
|
||||
| `cron_incidents` row | **`dca848_dfd4bea84f2a`**, `failure_type: timeout`, **`alerted`**, still open |
|
||||
|
||||
No handoff artefact was written, so **nothing reached Notion, no cover letters were generated, and both
|
||||
Stage 2 slots declined.** This is the **fourth Stage-1 failure in eight days** and a **new terminal
|
||||
signature** — not the iteration cap that ended 10-27, 10-28 and 11-02.
|
||||
|
||||
## Timeline (all `+08:00`)
|
||||
|
||||
| Time | Event |
|
||||
|---|---|
|
||||
| 07:00:33.819190 | execution `dca8482e4f76` opens (07:00 weekday slot) |
|
||||
| 07:01:17 | run session `local-7180cef7…` first message |
|
||||
| 08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21 | **five context compactions** in one run |
|
||||
| 11:30:36.617545 → 11:32:16.812897 | Stage 2 `5fc292ff0f3b` first slot: no handoff → `[SILENT]`, `suppressed` |
|
||||
| 12:23:28 | **last tool result of the run** (a `terminal` call) |
|
||||
| 13:28:49 | harness records *"This turn did not complete. Some actions may already have run; verify their effects before resending."* |
|
||||
| 13:28:50,627 | gateway.log: `ERROR gateway.session_authority: Admitted turn 9bf9a296… failed` (traceback through `session_authority.py:1159 _execute_claimed` → `session_finite.py:80 execute_finite_admission` → `session_ingress.py:235 execute_admission`) |
|
||||
| 13:28:51.463929 | incident `dca848_dfd4bea84f2a` opens |
|
||||
| 13:28:51.741724 | **failure notice delivered** |
|
||||
| 13:28:52.305995 | execution closes `failed` |
|
||||
| 13:28:55.141618 | incident `alerted` |
|
||||
| 14:30:32.261931 → 14:33:11.470879 | Stage 2 safety-net slot: same silent decline |
|
||||
| in-session | Pulse report `p34c8e899-1` (covering Tue 03 Nov 13:01 onward) led with it: *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a stream active timeout."* |
|
||||
|
||||
## The 65-minute silence is the new information
|
||||
|
||||
The run's recorded activity stops at **12:23:28** and the harness does not declare the turn dead until
|
||||
**13:28:49** — **65 m 21 s with no tool call, no model turn and no error written to the ledger**.
|
||||
|
||||
Contrast with the two cap failures: on **10-28** and **11-02** the runtime *injected* the text *"You've
|
||||
reached the maximum number of tool-calling iterations allowed…"* into the session before it ended. This
|
||||
session contains **no such message** — it stopped mid-verification, on a provider-stream liveness timeout,
|
||||
with the run's own closing plan (*"Two final verification passes, then I'll assemble the handoff"*) never
|
||||
executed.
|
||||
|
||||
**Therefore:** the agent iteration cap is not the only ceiling that ends a long Stage-1 run, and the
|
||||
"2026-11-02 raised no incident at all" behaviour was a property of *that* terminal state (`completed`,
|
||||
`error: NULL`), **not** of long runs in general. Nothing in the ledger or the gateway log records the
|
||||
provider's socket state during the silent window.
|
||||
|
||||
## Why the ledger went red this time
|
||||
|
||||
| Date | `executions.status` | `cron_incidents` row | Notice to Anthony |
|
||||
|---|---|---|---|
|
||||
| 2026-10-27 | non-clean (`failed`) | `dca848_e93a538ade6b` — `failure_type: auth` (**wrong** for a budget fault) | yes — the run's INCOMPLETE report |
|
||||
| 2026-10-28 | non-clean (`failed`) | `dca848_0a2f03b5952b` — `failure_type: timeout` | yes |
|
||||
| 2026-11-02 | **`completed`, `error: NULL`** | **none raised** | **no — `suppressed`** |
|
||||
| **2026-11-03** | **`failed`** | **`dca848_dfd4bea84f2a` — `timeout`, open** | **yes — 13:28:51.741724** |
|
||||
|
||||
Fleet-wide **unclosed incident rows 0 → 1**. A run the harness itself kills produces an honest red row and
|
||||
a delivered notice; the cap produces a green row and silence. That is the distinction
|
||||
[[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] could only predict, now observed.
|
||||
|
||||
## Downstream: both Stage 2 slots behaved correctly, and two days of candidates are lost
|
||||
|
||||
| Slot | Job | Ran | Result |
|
||||
|---|---|---|---|
|
||||
| 11:30 | `5fc292ff0f3b` (*Write and Letters*) | 11:30:36 → 11:32:16 | `[SILENT]` / `suppressed` |
|
||||
| 14:30 (safety net) | `5fc292ff0f3b` | 14:30:32 → 14:33:11 | `[SILENT]` / `suppressed` |
|
||||
|
||||
`~/.hermes/radar-ref/handoff/` holds **only** `2026-10-27.json`, `2026-10-29.json` and `2026-10-30.json` —
|
||||
nothing for 10-31, **11-02** or **11-03**. So **2026-11-02's 164 deduped candidates and 2026-11-03's
|
||||
discovery are both unrecovered**, and Notion received nothing on either day. The last end-to-end success
|
||||
remains **2026-10-29** (handoff written inside budget; Stage 2 landed **20 pages + 2 cover letters**,
|
||||
`missing_cover_letter_count 0`).
|
||||
|
||||
## Context churn (observation, cause not established)
|
||||
|
||||
**363 messages** (6 `user` / 133 `assistant` / 224 `tool`), **288 compaction-flagged messages**, and **five
|
||||
compactions** inside one 6 h run. The 11-02 run compacted repeatedly as well. Whether compaction churn
|
||||
contributes to the stream stall is **not established** — this is recorded because both long failures share
|
||||
the pattern, not because the link is proven.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Is the **65 m 21 s** silence the stream-active watchdog's own threshold, or did the provider socket die
|
||||
silently long before the harness noticed?
|
||||
2. Does repeated context compaction contribute to the stall? Two long runs, both compacting heavily.
|
||||
3. The 07:00 run has now overrun past the 11:30 Stage-2 slot two days running — should the pairing be
|
||||
re-slotted, or Stage 2 made to wait on a handoff?
|
||||
4. Where does `radar-ref/handoff/` get its retention? The newest file is 10-30.
|
||||
5. Should a stream timeout on a *file-deliverable* job trigger an automatic retry instead of leaving the
|
||||
day's discovery to the next slot?
|
||||
|
||||
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
|
||||
|
||||
- The last open row raised here, `dca848_dfd4bea84f2a` (`timeout`, opened **2026-11-03 13:28:51**), **closed on
|
||||
2026-11-05 at 12:36:17** - together with its 11-04 successor row `dca848_3b1bb1e9b003`.
|
||||
- The job behind it, `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), **completed** its 2026-11-05
|
||||
07:00 run in **5 h 35 m 35 s** (07:00:42 -> 12:36:18), with no repeat of
|
||||
`RuntimeError: Upstream stream failed before completion.`
|
||||
- **Status -> `resolved`.** The underlying questions (whether the 65 m 21 s stream silence is the watchdog
|
||||
threshold or a silently dead socket; whether a stream timeout on a file-deliverable job should retry
|
||||
automatically) remain open in [[systems/scheduled-tasks]] - they are no longer tracked as an incident
|
||||
against this window.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]
|
||||
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
|
||||
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
|
||||
- [[runbooks/job-radar]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
@@ -0,0 +1,132 @@
|
||||
---
|
||||
title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)"
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-11-05
|
||||
updated: 2026-11-06
|
||||
verified_on: 2026-11-06
|
||||
confidence: high
|
||||
tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident]
|
||||
sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json]
|
||||
---
|
||||
|
||||
# Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)
|
||||
|
||||
**Window:** 2026-11-04 07:00:45 to 09:42:20 AWST (2 h 41 m 35 s) - **Status:** both incidents still open
|
||||
(`state = alerted`); no data lost - the Job Radar's day was recovered by its Stage 2 safety-net slot.
|
||||
|
||||
## Summary
|
||||
|
||||
Two independent 07:00 jobs died on the **same upstream fault, 36 seconds apart** on 2026-11-04:
|
||||
|
||||
- `dca8482e4f76` **Daily Marketing Job Radar - Discovery** (`0 7 * * 1-5`)
|
||||
- `85edf6772bca` **Tech & AI Newsletter Digest (Analytical)** (`0 7 * * *`)
|
||||
|
||||
Both ran **2 h 41 m** and ended with the identical error, `RuntimeError: Upstream stream failed before
|
||||
completion.` Both are pinned to `custom:omni-101` / `smart-route`. Neither produced output. Two
|
||||
`cron_incidents` rows were raised and alerted, and both notices were delivered.
|
||||
|
||||
Unlike the 2026-11-02 and 2026-11-03 incidents - where an aborted Stage 1 cost the day's candidate set -
|
||||
**the Job Radar's day survived**: the 11:30 Stage 2 slot reconstructed the missing handoff, re-verified every
|
||||
candidate live, and still wrote 5 roles and 1 cover letter. See *The recovery* below.
|
||||
|
||||
## Evidence
|
||||
|
||||
Live `executions.db`, `cron_incidents`, `deliveries.db` and `jobs.json`, read 2026-11-05 00:5x AWST:
|
||||
|
||||
| Job | Start | End | Duration | Status | Incident row |
|
||||
|---|---|---|---|---|---|
|
||||
| `dca8482e4f76` Daily Marketing Job Radar - Discovery | 07:00:45 | 09:42:20 | 2 h 41 m 35 s | `error` | `dca848_3b1bb1e9b003` (`unknown`, open) |
|
||||
| `85edf6772bca` Tech & AI Newsletter Digest (Analytical) | 07:00:45 | 09:41:44 | 2 h 40 m 59 s | `error` | `85edf6_c52a11a62044` (`unknown`, open) |
|
||||
|
||||
Both error strings: `RuntimeError: Upstream stream failed before completion.`
|
||||
|
||||
All three open incident rows carry `state = alerted` with `closed_at` NULL - the 2026-11-03 `timeout` row
|
||||
`dca848_dfd4bea84f2a` (first seen `2026-11-03T13:28:51`) plus these two (`09:41:43` and `09:42:19`).
|
||||
**Unclosed incident rows 1 -> 3.**
|
||||
|
||||
Deliveries on 11-04: **8 rows, 8 `delivered`** - including both failure notices (09:41:43, 09:42:20). The
|
||||
alerts were not silent.
|
||||
|
||||
## Why this is not simply the 2026-11-03 incident again
|
||||
|
||||
| | 2026-11-03 | 2026-11-04 |
|
||||
|---|---|---|
|
||||
| Error | `RuntimeError: stream active timeout` | `RuntimeError: Upstream stream failed before completion.` |
|
||||
| Duration | 6 h 28 m 18 s | 2 h 41 m (both jobs) |
|
||||
| Jobs hit | 1 (`dca8482e4f76`) | 2, in the same minute |
|
||||
| Row | `dca848_dfd4bea84f2a` | `85edf6_c52a11a62044`, `dca848_3b1bb1e9b003` |
|
||||
|
||||
A timeout is a stall; "stream failed before completion" is an upstream drop. Both land in the stream layer
|
||||
of the `smart-route` combo, on `custom:omni-101`. Two unrelated prompts - job discovery and a newsletter
|
||||
digest - failing within 36 seconds of each other points at the route/provider, not at either job.
|
||||
|
||||
## The recovery
|
||||
|
||||
Stage 2 (`5fc292ff0f3b`) ran in its 11:30 safety-net slot and finished **15:58-16:09 AWST: Result PASS,
|
||||
0 missing cover letters**:
|
||||
|
||||
- Stage 1 wrote **no handoff**. Stage 2 **reconstructed it from Stage 1's own candidate files** and
|
||||
re-verified every surviving candidate **live in Camofox** before writing anything.
|
||||
- **5 roles written to Notion** (board **479 -> 484**): Colab Resources (Group Marketing Manager, strong
|
||||
match), Foundation for the WA Museum, Mindful Meditation Australia, HPH Solutions, RecruitShop.
|
||||
- Discovery totals carried forward: **34 discovered / 10 duplicates avoided / 20 rejected**.
|
||||
- **1 strong match -> 1 canonical cover letter** (Colab Resources) rendered through the canonical teal
|
||||
template: `.txt`/`.json`/`.html`/`.docx`/`.pdf` all present and non-empty, no `{{}}` tokens, HTTP 200 on
|
||||
`:9099/jobs/drafts/...`. `SAME_DAY_STRONG_COUNT 1` / `VERIFIED_COVER_LETTER_COUNT 1` /
|
||||
`MISSING_COVER_LETTER_COUNT 0`.
|
||||
- Handoff marked `consumed: true` at **16:08:59** (atomic write), so the 14:30 slot stayed silent; delivery
|
||||
at 16:12:22.
|
||||
- The run also caught **its own** drafting error: the first Colab letter misstated three resume facts (an
|
||||
invented "40% sales increase", Horizon Power programmes attributed to Pacific Energy, a stray en-dash).
|
||||
It was corrected against `current_resume_from_website.txt` and re-rendered - a live demonstration of the
|
||||
runbook's "verify against the canonical resume" rule.
|
||||
|
||||
**So the 11-04 failure cost a Stage 1 run, not the day** - the first time the split pipeline's safety-net
|
||||
slot has had to carry the day alone.
|
||||
|
||||
## The cover-letter route is only alive by override
|
||||
|
||||
`regenerate_cover_letter.py` line 82 reads `os.environ.get("CL_MODEL", "deepseek-paid-smart")`. During the
|
||||
11-04 run that default returned **HTTP 402 (billing)**, so generation was pinned to
|
||||
**`gemini/gemini-3.5-flash`** for that process. `grep` finds **no `CL_MODEL` in `~/.hermes/.env`** - the
|
||||
override never persisted, so the next run inherits the default again. The same model name still serves other
|
||||
cron jobs (`cbfe434a3b5e`, this nightly audit, `last_status: ok`), so treat a 402 as **route/time-specific
|
||||
until re-tested** rather than as proof the route is dead. See [[runbooks/job-radar]].
|
||||
|
||||
## Other in-window cron faults
|
||||
|
||||
- `1a8e90ea2271` **wiki-stocktake-maintenance** ran **03:00:31 -> 08:42:17 (5 h 41 m 46 s)**, hit the turn
|
||||
ceiling, reported incomplete, delivery `suppressed` - the **third consecutive overrun** of this job. Its
|
||||
pin is `custom:omni-101` / **`free-stack`**. It overlapped the 07:00 window Stage 1 was running in.
|
||||
- `066bb5191280` **Uber Eats Monthly Spend Report** carries `last_status: delivery_failed` (streak 0) - part
|
||||
of the non-clean count, not a new failure.
|
||||
|
||||
## Open questions / next actions
|
||||
|
||||
1. Both 11-04 failures share a signature and a route with the 11-03 timeout. **No output file exists for
|
||||
either** - the fault lands upstream of the agent's first turn.
|
||||
2. `failure_type` is stored as **`unknown`** for a stream fault, so neither new row groups with the 11-03
|
||||
`timeout` row. The incident table undercounts what looks like one route-level pattern (same miscoding
|
||||
seen on the 2026-10-27 row, stored as `auth` for a 400).
|
||||
3. All three rows are `alerted` and unclosed. Decide whether the two 11-04 rows are two incidents or one
|
||||
route incident - and whether the 11-03 row should be closed.
|
||||
4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]].
|
||||
|
||||
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
|
||||
|
||||
- **Both affected jobs ran clean on 2026-11-05.** `85edf6772bca` (*Tech & AI Newsletter Digest (Analytical)*)
|
||||
completed its 07:00 run in **7 m 01 s** (07:00:42 -> 07:07:43), its incident row closing at **07:07:42**;
|
||||
`dca8482e4f76` (*Daily Marketing Job Radar - Discovery*) completed in **5 h 35 m 35 s** and its two rows
|
||||
closed at **12:36:17**. **Fleet-wide unclosed incident rows: 3 -> 0** - the ledger was empty for the first
|
||||
time since 2026-11-02, and 2026-11-05 was the first zero-`failed` ledger day since then too.
|
||||
- The Stage 2 salvage documented above therefore stands as the whole recovery - no follow-up failure arrived.
|
||||
- The `CL_MODEL` / HTTP 402 cover-letter caveat is **unchanged and still needs a decision**: the
|
||||
`gemini/gemini-3.5-flash` pin was never written to `.env`, so it did not persist.
|
||||
- **Status -> `resolved`.** A recurrence should be opened as a new incident with its own window.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[runbooks/job-radar]]
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
title: vzdump Freeze Took CT299 / SFTPGo Offline for ~24h
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-11-04
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
confidence: high
|
||||
tags: [incident, proxmox, vzdump, backup, nfs, ct299, sftpgo, tailscale, manual-backup, ssd0]
|
||||
sources: [live-pve-probe, live-tailscale-probe, live-ssh, live-pve-tasks, live-probe-2026-11-05]
|
||||
---
|
||||
|
||||
# vzdump Froze CT299 / SFTPGo Twice - 11-03 (NFS target) and 11-04 (manual ssd0 run)
|
||||
|
||||
> **Updated 2026-11-05 (nightly audit) - CT299 was frozen twice, not once.** The page below documents the
|
||||
> 2026-11-03 freeze. A **second, separate freeze** ran from **2026-11-04 08:30:04** to **2026-11-05 ~00:30**
|
||||
> (~16 h), caused by a **manual** `vzdump 299 ... --storage ssd0` run. Live probe at **00:54:21 AWST**:
|
||||
> `LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active` - **service up**.
|
||||
> See *Second freeze* below.
|
||||
|
||||
**Window:** ~2026-11-03 03:00 AWST → 2026-11-04 04:4x AWST (≈25 h)
|
||||
**Status:** Resolved — wedged `vzdump` killed, guest unfrozen, SFTPGo verified healthy
|
||||
**Impact:** `sftpgo` (LXC CT299 on `pve`) was **frozen solid** — the container's filesystem was suspended by the
|
||||
backup job, so `sftpgo`, `tailscaled` and everything else inside it stopped. The tailnet peer went offline and the
|
||||
service was unreachable for the better part of a day. **Nothing alerted.**
|
||||
|
||||
## Symptoms
|
||||
|
||||
- Tailscale peer `sftpgo` (`100.88.223.70`) offline; `tailscale ping` → timeout.
|
||||
- `ssh` into `100.88.223.70` hung until timeout; `pct exec 299 -- ...` also hung (exit 124).
|
||||
- The guest was not crashed — it was **frozen**: the backup job had taken an LVM snapshot and suspended the guest
|
||||
filesystem, then stalled before it could release it.
|
||||
|
||||
## Root cause
|
||||
|
||||
A scheduled Proxmox backup job (`vzdump: backup-tue-g1-299`, `schedule tue 03:00`, `mode snapshot`,
|
||||
`storage fred-qnap`) started on **Tue 2026-11-03 at 03:00 AWST** and **never completed**. The task wedged at
|
||||
`create storage snapshot` on the **`fred-qnap` NFS storage** (`192.168.178.31:/Proxmox` mounted at `/mnt/pve/fred-qnap`):
|
||||
|
||||
- The NFS write path was unresponsive — a `dd` write probe to `/mnt/pve/fred-qnap` timed out while the task was stuck.
|
||||
- The `vzdump` child process was in **`D` state** (uninterruptible I/O) on a FUSE `request_wait_answer` — it could not
|
||||
be killed by a normal signal while blocked in the kernel.
|
||||
- Its temp directory `/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/` was left behind, and the guest's filesystem
|
||||
remained suspended for the whole ~24 h.
|
||||
|
||||
**The failure mode is the important part: a backup task that cannot finish also cannot release the guest.** Proxmox's
|
||||
snapshot mode suspends the guest filesystem while the snapshot is created and the data is read out. A stalled task
|
||||
therefore does not merely "fail" — it **takes the guest down with it**, silently, until someone notices.
|
||||
|
||||
## Remediation (2026-11-04 04:4x AWST)
|
||||
|
||||
1. Identified the stuck task on `pve`: `vzdump` parent PID **274590**, child PID **274760** (state `D`).
|
||||
2. `kill -TERM` on both PIDs — the parent died, the child remained blocked in `D` state on the FUSE wait.
|
||||
3. `kill -KILL` on the remaining child — the kernel eventually released it once the NFS request timed out.
|
||||
4. **The guest unfroze immediately.** `pct exec 299 -- /bin/echo ALIVE` → `ALIVE` (previously exit 124 / timeout).
|
||||
5. Verified end to end:
|
||||
- `pct exec 299 -- systemctl is-active sftpgo` → **`active`**
|
||||
- `pct exec 299 -- ss -ltn` → `LISTEN *:8080` (WebAdmin) and `:2022` (SFTP)
|
||||
- `tailscale status` → peer `sftpgo` **Online: true** again
|
||||
- `tailscale ping 100.88.223.70` → **`pong from sftpgo via 192.168.178.92:41641 in 0s`** (direct, not DERP)
|
||||
- `curl http://100.88.223.70:8080/` → **HTTP 200**
|
||||
|
||||
## Post-remediation storage check
|
||||
|
||||
`fred-qnap` was healthy again by the time the task was cleared:
|
||||
|
||||
- `pvesm status` → `fred-qnap nfs active 14,380,169,216 KiB total / 12,084,987,904 used / 84.04 %`
|
||||
- `dd` write probe of 5 MiB with `oflag=direct` → **28.6 MB/s**, exit 0 — no stall.
|
||||
- The other backup task running concurrently (`vzdump: backup-sun-g1-100`, started 2026-11-03 10:36 AWST) **finished
|
||||
successfully** at 2026-11-04 04:56:59 (`TASK OK`, 00:16:25) against the same storage.
|
||||
|
||||
So the NFS stall was **transient, not a dead mount** — it hit the one task that happened to hold a guest suspended.
|
||||
|
||||
## Second freeze - 2026-11-04 08:30:04 to 2026-11-05 ~00:30 (~16 h)
|
||||
|
||||
The 11-03 freeze documented above was **not the only one**. A live probe of pve (`100.96.100.82`) on
|
||||
**2026-11-05 00:54:21 AWST**, plus the node's task history, shows a **second, separate wedge** the next morning:
|
||||
|
||||
- `pvesh get /nodes/pve/tasks` lists **two CT299 `vzdump` tasks on 2026-11-04** - and **neither is the 03:00
|
||||
fleet job**:
|
||||
- **`2026-11-04 07:36:44` - `vzdump` / status `job errors`** - a clean failure after ~11 minutes. Its log
|
||||
survives on the dump target: `/mnt/ssd0/dump/vzdump-lxc-299-2026_11_04-07_36_44.log` (597 B, mtime Nov 4 07:47).
|
||||
- **`2026-11-04 08:30:04` - `vzdump` / status `unexpected status`** - the wedge. `unexpected status` is what a
|
||||
killed task leaves behind; the 11-03 `03:00:04` row reads the same.
|
||||
- Three `push_file` tasks from client **`100.118.5.51` (CT460)** ran at **07:17:27 / 07:27:18 / 07:33:07**,
|
||||
immediately before the first attempt - so the sequence reads as operator/agent-initiated, not scheduled.
|
||||
- The abandoned temp dir **`/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299`** is still on disk (the dump dir holds
|
||||
8.8 G total; the only other artefact is `openweb-245-preupgrade-20261101-211545.tar`).
|
||||
- **Both attempts targeted the SSD that is bind-mounted into the guest** (`--storage ssd0`, `/mnt/ssd0/dump`) -
|
||||
the same hazard class as the 11-03 NFS target: the guest is snapshotted onto storage it is actively using.
|
||||
|
||||
**Recovery state, verified live 2026-11-05 00:54:21 AWST:** `LOCK: 0` in `/etc/pve/lxc/299.conf`;
|
||||
`/sys/fs/cgroup/lxc/299/cgroup.freeze = 0` **and** `ns/cgroup.freeze = 0`; `pct exec 299 -- /bin/echo ALIVE`
|
||||
returns `ALIVE`; `systemctl is-active sftpgo` -> `active`; **0** `vzdump`/`lxc-freeze` processes; SFTPGo
|
||||
reachable end-to-end from CT460 at `http://100.88.223.70:8080`. **CT299 is not frozen now.**
|
||||
|
||||
**Open:** why a 299 dump was attempted twice on 11-04 outside any schedule, and whether the `ssd0` target
|
||||
should be used for this guest at all. See [[infrastructure/proxmox]] and the open items below. The abandoned
|
||||
`vzdumptmp3750030_299` directory should be reclaimed once nobody needs it.
|
||||
|
||||
## Prevention / open items
|
||||
|
||||
- **Nothing monitored the backup tasks.** A `vzdump` that hangs for 24 h on a guest-holding storage is an outage
|
||||
generator, and no page, cron job or dashboard covers it. A watchdog that alerts on a `vzdump` task older than
|
||||
~2 h — or that checks for `vzdump` tasks holding a suspended guest — is the missing control.
|
||||
- **NFS `hard` mount semantics amplify this.** `fred-qnap` is mounted `hard,fatal_neterrors=none`, so a stalled NFS
|
||||
server parks I/O in uninterruptible sleep (`D` state) indefinitely rather than failing fast. A `soft`/`intr` mount
|
||||
or a `timeout=` tuned for the backup path would let the task die instead of freezing the guest.
|
||||
- **The temp dir must be reclaimed manually.** `/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/` was left on disk by the
|
||||
killed task; the existing cleanup cron skips a directory whose task is still registered.
|
||||
- **Do not assume "guest offline" means "guest crashed."** This guest was healthy and simply frozen. The first two
|
||||
probes (ping, ssh) both time out in exactly the same way as a dead guest — the distinguishing signal is a
|
||||
`vzdump` task in `D` state with an open snapshot on that guest.
|
||||
|
||||
## Related
|
||||
- [[infrastructure/proxmox]] — host, guest and storage inventory
|
||||
- [[infrastructure/backups]] — Hermes-side config backups (a separate mechanism from PVE `vzdump`)
|
||||
- [[incidents/2026-09-02-sftpgo-outage]] — the previous CT299 outage (swap exhaustion + cgroup pinning) — a
|
||||
**different** root cause with the same symptom
|
||||
- [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] — prior NFS-layer failure in this estate
|
||||
- [[systems/scheduled-tasks]] — cron fleet (no backup-task watchdog exists today)
|
||||
@@ -0,0 +1,132 @@
|
||||
---
|
||||
title: Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05)
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-11-06
|
||||
updated: 2026-11-09
|
||||
verified_on: 2026-11-09
|
||||
review_after: 2026-11-13
|
||||
confidence: high
|
||||
tags: [security, credentials, mnemosyne, memory, sessions, cron, incident]
|
||||
sources: [live-env-scan, live-sqlite-mnemosyne, live-jobs-json, live-state-db, live-deliveries-db, live-config-backup]
|
||||
---
|
||||
|
||||
# Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05)
|
||||
|
||||
**Status: OPEN - found by the nightly audit (2026-11-05 window), not remediated.** No key was rotated, no
|
||||
memory row deleted and no cron prompt scrubbed. This page records *where* the values live and how many rows
|
||||
carry them; **no credential value appears on it** ([[SCHEMA]] rule 1 - record where secrets live, never their
|
||||
values).
|
||||
|
||||
## What was found
|
||||
|
||||
Three exposures, all verified by direct read on **2026-11-06 00:22-00:50 AWST**.
|
||||
|
||||
### 1. Live `.env` values as plaintext rows in the memory store
|
||||
|
||||
**17 rows** of `~/.hermes/mnemosyne/data/mnemosyne.db` carry values from `~/.hermes/.env`. Method: every
|
||||
TEXT/BLOB column of every table, plus the FTS mirrors, scanned for exact substring membership.
|
||||
|
||||
| Key (name only) | Rows | Columns carrying it |
|
||||
|---|---|---|
|
||||
| `TELEGRAM_BOT_TOKEN` | 4 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1 |
|
||||
| `MATTERMOST_TOKEN` | 6 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x2, `memoria_facts.context_snippet` x1, `fts_episodes.content` x1 |
|
||||
| `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` | 7 | `memories.content` x2, `working_memory.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1, `fts_working.content` x1 |
|
||||
|
||||
- **Every hit is in `mnemosyne.db`** (27.7 MB, live). The `episodic_memory.db` and `memoria_facts.db` files
|
||||
beside it are **0-byte legacy stubs** (created 2026-10-27) - the tables with those names live *inside*
|
||||
`mnemosyne.db`. Nothing was found in the cross-agent shared surface
|
||||
(`~/.hermes/data/shared/mnemosyne.db`).
|
||||
- **The rotation snapshots carry them too.** Seven uncompressed snapshots under
|
||||
`~/.hermes/mnemosyne/backups/` (`mnemosyne-pre-orphan-cleanup.db`, `-20261004_030035`, `-20261011_030046`,
|
||||
`-20261018_030021`, `-20261025_030108`, `-20261101_030033`, `-20261105_140314`) contain all three values in
|
||||
full; the 2026-09-27 snapshot predates them. The `.db.gz` snapshots are **compressed and were not
|
||||
substring-tested** - assume the same until proven otherwise.
|
||||
|
||||
### 2. One live key embedded verbatim in two cron prompts
|
||||
|
||||
The `CAMOFOX_API_KEY` value appears in the `prompt` text of two **enabled** jobs in
|
||||
`~/.hermes/cron/jobs.json`:
|
||||
|
||||
- `0e753e9720fd` - *Facebook Birthday Monthly Scrape* (`0 9 1 * *`)
|
||||
- `fad5a86979d4` - *Facebook Birthday Weekly Registry Sync* (`0 3 * * 1`)
|
||||
|
||||
Consequence: the same value is inside delivery records in `~/.hermes/cron/deliveries.db`. Of the **10 `.db`
|
||||
files under `~/.hermes/config-backup/`** (six of them 0-byte), `cron/deliveries.db` is the **only** one that
|
||||
carries any tested value; `cron/executions.db`, `cron/notepad.db` and `data/shared/mnemosyne.db` carry none.
|
||||
|
||||
### 3. Session transcripts hold all four values
|
||||
|
||||
`~/.hermes/state.db` (1.7 GB *at discovery* — **1.89 GB / 1,890,656,256 B as at 2026-11-09 00:0x AWST**, with
|
||||
**2,901 sessions / 197,682 messages** and a live **40,157,672 B WAL**; the store grew while nothing was
|
||||
retentioned), the session store, contains all four values. Measured as 4 MB chunks
|
||||
containing the value: **`CAMOFOX_API_KEY` 178, `MATTERMOST_TOKEN` 64,
|
||||
`HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` 61, `TELEGRAM_BOT_TOKEN` 22**. This is the **most likely origin of
|
||||
exposure 1** - the memory-capture path writes from session text, so a session in which a key was printed
|
||||
becomes a memory row - and it is itself a plaintext at-rest store that deleting memory rows does not clear.
|
||||
|
||||
## Why it matters
|
||||
|
||||
- Mnemosyne rows are **recall material**: they are pulled into a session's context by semantic or FTS match,
|
||||
so a secret in a row is a secret in a prompt - echoable into any cron delivery (Telegram, Mattermost,
|
||||
email), any subagent transcript, or the cross-agent shared surface.
|
||||
- It turns every **backup** into a credential copy. Deleting the rows does not recall the copies;
|
||||
**rotation is the only remedy that fully closes it**.
|
||||
- `jobs.json` is the fleet's source of truth and is exported and examined routinely; a prompt-embedded key
|
||||
is exposed to anything that reads job definitions.
|
||||
- `state.db` is the fleet's largest artifact (**1.89 GB as at 2026-11-09**, up from 1.7 GB) and is copied into
|
||||
snapshots and repair backups. ⚠ **It is also the load source behind the 2026-11-07/08 hook-timeout
|
||||
cascade** — a 40 MB WAL checkpointing on the same spindle as a 2-core guest → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
|
||||
The **2026-08-11 corruption remnant `state.db.corrupted.20260811_151052` (624.1 MB)** was deleted in the
|
||||
2026-11-08 window, so one stale copy did leave the box — but **`mnemosyne/data/mnemosyne.db.corrupted.20260809_120404` (23.6 MB) is still there**, so the rotation-snapshot exposure above is untouched.
|
||||
|
||||
## Origin (inferred - not established)
|
||||
|
||||
Almost certainly the **memory-capture path storing verbatim session text** in which a key was printed by a
|
||||
tool call or a `.env` read. Not verified: no capture-time or delivery-time redaction filter is known to
|
||||
exist, and the affected rows carry no marker distinguishing them from ordinary content.
|
||||
|
||||
## Remediation options (proposed, none executed)
|
||||
|
||||
1. **Rotate** the three Mnemosyne-exposed values and the Camofox key - touches live services; needs a
|
||||
maintenance window and explicit approval.
|
||||
2. **Redact or forget the affected rows** (`mnemosyne forget <id>` per row), then **re-take the rotation
|
||||
snapshots** so the superseded values leave the backup set.
|
||||
3. **Move the two Facebook Birthday jobs off prompt-embedded keys** - read the key from the environment or
|
||||
config at run time.
|
||||
4. **Add capture-time and delivery-time redaction** so `.env` values cannot enter memory rows, job prompts or
|
||||
delivery records again.
|
||||
5. **Decide a retention/serialisation policy for `state.db`** - it currently holds every transcript
|
||||
verbatim at 1.89 GB (2,901 sessions / 197,682 messages) and it is growing.
|
||||
|
||||
## Verification evidence
|
||||
|
||||
- Key names and lengths extracted from `~/.hermes/.env` (values never printed), then matched as exact
|
||||
substrings across every TEXT/BLOB column of every table in `mnemosyne.db` including the FTS mirrors -
|
||||
row counts as tabled.
|
||||
- Byte-membership scans of the 10 `.db` files under `~/.hermes/config-backup/`, the shared-surface DB, the
|
||||
live `cron/deliveries.db` + `cron/executions.db`, the seven uncompressed Mnemosyne rotation snapshots, and
|
||||
a chunked scan of `state.db`.
|
||||
- `json.dumps()` membership test of each value against every job object in `jobs.json`.
|
||||
- **Not done:** no value was printed, logged or written to any wiki page; nothing was deleted, rotated or
|
||||
scrubbed.
|
||||
|
||||
## 2026-11-08 00:5x AWST — nightly re-check (still OPEN; one narrow improvement)
|
||||
|
||||
- **Cron job prompts: clean.** A regex sweep of every job in `/home/hermes/.hermes/cron/jobs.json` for
|
||||
secret-shaped strings (`sk-…`, `ghp_…`, and any `<key|token|password|secret> = <16+ chars>`) returned
|
||||
**0 hits** on 2026-11-08 00:5x. Only the *shape* was tested — a secret that does not look like a secret
|
||||
would still pass.
|
||||
- **Redaction setting: not confirmable from the file.** `~/.hermes/config.yaml` still carries a
|
||||
`redact_secrets:` key, but its on-disk value reads as a masked token when printed, so the effective
|
||||
on/off state could not be established by reading it. The 2026-11-05 finding — *"no capture-time or
|
||||
delivery-time redaction filter is known to exist"* (line 80 above) — therefore **stands unrefuted**.
|
||||
- **Nothing else moved:** no key rotation, no affected row redacted or forgotten, no remediation option
|
||||
executed. Both **Remediation options 2 and 4 above remain open**.
|
||||
|
||||
## See also
|
||||
|
||||
- [[infrastructure/credentials-map]] - where secrets live, by design
|
||||
- [[systems/mnemosyne]] - the memory store whose tables carry them
|
||||
- [[systems/scheduled-tasks]] - the Facebook Birthday jobs' schedules
|
||||
- [[SCHEMA]] - the no-values rule this violates
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
title: "Skills Index Generator Divergence — Two Scripts, One Page"
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-11-05
|
||||
updated: 2026-11-05
|
||||
resolved_on: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
confidence: high
|
||||
tags: [skills, wiki, drift, auto-regeneration, root-cause]
|
||||
sources: [live-filesystem-probe, script-diff, wiki-maintain-run, agent-estate-wiki]
|
||||
---
|
||||
|
||||
# Skills Index Generator Divergence — Two Scripts, One Page
|
||||
|
||||
## Summary
|
||||
`systems/skills-index.md` flipped between two different shapes on alternating days — **302 skills / 68
|
||||
categories** on some runs and **349 skills / 26 categories** on others. The cause was **two different
|
||||
scripts** writing the same page, each with its own generator, and the two scheduled jobs pointing at
|
||||
different ones. Whichever job ran last won, so the page oscillated nightly.
|
||||
|
||||
This is the **same class of defect** as [[incidents/2026-08-15-skills-index-drift]], but a different and
|
||||
previously-unidentified root cause. The 2026-08-15 incident attributed the drift to archived skills being
|
||||
counted; that was a *symptom* of one of the two generators, not the mechanism.
|
||||
|
||||
## Root Cause
|
||||
Two files both named `wiki-maintain.sh`, each containing a **complete, independent copy** of the page
|
||||
generator:
|
||||
|
||||
| Script | Generator technique | Output |
|
||||
|---|---|---|
|
||||
| `/home/hermes/workspace/agent-estate-wiki/wiki-maintain.sh` (repo, canonical, mtime 2026-11-03) | `find` scan **excluding** `*/.archive/*`; dedupes by `(category, skill_name)` | **302 skills / 68 categories** |
|
||||
| `/home/hermes/.hermes/scripts/wiki-maintain.sh` (stale copy, mtime **2026-08-17**) | `Path.rglob('SKILL.md')` — **includes** the parked `~/.hermes/skills/.archive/` tree; 2-segment category heuristic | **349 skills / 26 categories** |
|
||||
|
||||
The two scripts are **not** identical — `diff` between them returns a non-zero exit and shows entirely
|
||||
different generator bodies, different category heuristics and different output formats.
|
||||
|
||||
**The trigger:** `~/.hermes/skills/devops/wiki-maintenance/SKILL.md` (line 569) instructed the nightly audit
|
||||
job (`cbfe434a3b5e`, daily 23:59) to run:
|
||||
|
||||
```bash
|
||||
run `bash ~/.hermes/scripts/wiki-maintain.sh` (it regenerates `systems/skills-index.md`, writes a freshness
|
||||
```
|
||||
|
||||
— i.e. the **stale** copy. The 03:00 stocktake job (`1a8e90ea2271`) runs the **repo** script. So:
|
||||
|
||||
- 03:00 stocktake → repo script → page becomes **302 / 68**
|
||||
- 23:59 nightly audit → stale script → page becomes **349 / 26**
|
||||
- repeat, forever
|
||||
|
||||
Each job was individually "successful" — neither reported an error. The defect was only visible as a
|
||||
day-over-day oscillation in the committed page.
|
||||
|
||||
**Live counts verified 2026-11-05 05:1x AWST** (filesystem probes):
|
||||
|
||||
- `find ~/.hermes/skills -name SKILL.md` → **349** (includes archive)
|
||||
- `find ~/.hermes/skills -name SKILL.md -not -path '*/.archive/*'` → **303**
|
||||
- the generator de-dupes one nested alias duplicate → **302 unique** ✅
|
||||
- `~/.hermes/skills/.archive/` → **46** parked skills (excluded by the canonical generator)
|
||||
|
||||
## Impact
|
||||
- The published skills inventory was **wrong roughly half the time**, understating the real category
|
||||
topology (26 vs 68 categories) and overstating the count by including 47 archived skills.
|
||||
- Any agent reading the page to reason about the estate's skill surface got a materially different
|
||||
picture depending on which run last wrote it.
|
||||
- **No functional impact** — skill loading was never affected; only the documentation was wrong.
|
||||
|
||||
## Resolution
|
||||
1. **`~/.hermes/scripts/wiki-maintain.sh` is now a delegating wrapper.** It contains **no generator logic** —
|
||||
it `exec`s the canonical repo script and forwards arguments. The wrapper's header comment explains why
|
||||
it must stay that way, so the divergent copy is not recreated.
|
||||
2. **`--no-build` is now supported by the repo script.** The stale copy accepted `--no-build` (regenerate +
|
||||
audit, skip the Quartz build/deploy); the repo script did not. The repo script gained the flag so the
|
||||
wrapper can honour the interface its callers were written against **without** carrying a second copy of
|
||||
the generator. This closes the only functional gap the delegation could otherwise have introduced.
|
||||
3. **Verified by execution, not by inspection:** running the wrapper with `--no-build` produces the canonical
|
||||
shape — `302 skills in 68 categories` — and the generated page reports `46 archived skills` excluded.
|
||||
Both scripts pass `bash -n`.
|
||||
4. **`systems/skills-index.md` regenerated through the canonical path** and committed in the canonical shape.
|
||||
|
||||
## Lesson
|
||||
**A generator must live in exactly one place.** Two copies of the same script, differing in one technique
|
||||
(`rglob` vs `find`), produced two valid-looking outputs and a page that flipped daily — with **no error
|
||||
anywhere**, because each job did exactly what it was told.
|
||||
|
||||
When a scheduled job's output disagrees with another job's output for the same artefact, the cause is
|
||||
almost never the artefact — it is **two writers**. Diff the scripts before diffing the data.
|
||||
|
||||
Related: [[incidents/2026-08-15-skills-index-drift]] (the original drift), [[systems/skills-index]],
|
||||
[[systems/scheduled-tasks]].
|
||||
@@ -0,0 +1,86 @@
|
||||
---
|
||||
title: Job Radar — a concurrent fire discarded the 11:30 slot's result
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-11-07
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
review_after: 2026-11-14
|
||||
confidence: high
|
||||
tags: [cron, job-radar, fire-claim, concurrency, scheduler]
|
||||
sources: [live-executions-db, live-deliveries-db, live-agent-log, live-cron-incidents, cron-output]
|
||||
---
|
||||
|
||||
# Job Radar — a concurrent fire discarded the 11:30 slot's result (2026-11-06)
|
||||
|
||||
**Summary.** On **2026-11-06** the *Daily Marketing Job Radar — Write and Letters* (`5fc292ff0f3b`,
|
||||
schedule `30 11,14 * * 1-5`) had **two live runs at the same time**. A `direct`-source fire started at
|
||||
**08:27:09** and ran **4 h 59 m 13 s**, completing and delivering its report at **13:26:22**. The
|
||||
scheduler's own **11:30:32** slot had already started and was still running; when the direct fire settled
|
||||
the claim, the 11:30 run was judged **stale**, interrupted at **13:26:38**, and closed **`failed`** at
|
||||
**13:26:44** with a **new** error string:
|
||||
|
||||
```
|
||||
Fire claim ownership lost; stale result was discarded.
|
||||
```
|
||||
|
||||
This is the **first occurrence of a claim-loss failure with no gateway restart involved**, and it raised
|
||||
**no `cron_incidents` row** — the fleet's only red row for the day is invisible to the alerting ledger.
|
||||
|
||||
## Observed sequence (live `executions.db`, `deliveries.db`, `agent.log.1`)
|
||||
|
||||
| Time (AWST) | Event | Evidence |
|
||||
|---|---|---|
|
||||
| 03:00:32 -> 04:11:35 | wiki stocktake (`1a8e90ea2271`) completes in 1 h 11 m 03 s — overrun streak over | executions row |
|
||||
| 07:00:44 -> 08:29:21 | Stage 1 (`dca8482e4f76`) completes (1 h 28 m 37 s, `suppressed`), second clean run running | executions row |
|
||||
| **08:27:09 -> 13:26:22** | **`direct` fire of `5fc292ff0f3b`** — 4 h 59 m 13 s, `completed`, **delivered** | row `ad1fe896…`; log 08:27:13 *"Running job 'Daily Marketing Job Radar - Write and Letters'"* |
|
||||
| 11:30:32 -> 13:26:44 | scheduler's **own 11:30 slot** fires and runs **concurrently** for 1 h 56 m 12 s | row `95347fa5…` |
|
||||
| 13:26:22 | direct fire finishes and delivers its report | delivery row 13:26:22 |
|
||||
| **13:26:38** | *"Job '5fc292ff0f3b': fire claim ownership lost; interrupting stale run"* | `agent.log.1` WARNING |
|
||||
| **13:26:43** | *"Job '…' failed: RuntimeError: Cron job '…' lost its durable fire claim"* | `agent.log.1` ERROR |
|
||||
| **13:26:44** | 11:30 row closes **`failed`** — `Fire claim ownership lost; stale result was discarded.` | executions row |
|
||||
| 14:30:32 -> 14:42:59 | 14:30 slot runs a normal cycle (**12 m 27 s**) and delivers a complete report | row `e7462cf1…` |
|
||||
|
||||
**No `.md` output file** exists for the discarded 11:30 run — on 11-06 that job produced only
|
||||
`2026-11-06_13-26-22.md` (the direct fire) and `2026-11-06_14-42-58.md` (the 14:30 slot).
|
||||
|
||||
## What is *not* affected
|
||||
|
||||
The day still landed in full. The 14:30 report (read from `cron/output/5fc292ff0f3b/2026-11-06_14-42-58.md`)
|
||||
shows `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache
|
||||
total **494**, 10 job rows dated 11-06, 20 artifacts present and non-empty, *Run status: COMPLETE*.
|
||||
|
||||
## Why it matters
|
||||
|
||||
1. **New cause for a known string.** Every earlier `lost its durable fire claim` case came from a
|
||||
**restart or reload orphaning a live run** ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]],
|
||||
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]), where the text was *"lost its durable
|
||||
fire claim ownership"*. Here **no restart occurred** — the claim was lost to *another fire of the same
|
||||
job*. The string is a **generic claim-loss signal, not a restart indicator**; and `executions.db`
|
||||
currently holds **exactly one** row whose `error` contains `fire claim`: this one.
|
||||
2. **The failure produced no alert.** No `cron_incidents` row has `last_seen_at` on or after 2026-11-06 —
|
||||
the ledger's newest activity remains the 2026-11-05T12:36:17 closures. A `failed` execution with no
|
||||
ledger row means **no notification path fired**; the run only surfaces as a red row in `executions.db`.
|
||||
3. **~2 hours of a scheduled slot's work was discarded, and the *later, scheduler-owned* run lost** to an
|
||||
earlier out-of-band run. The stale-discard rule did what it was written to do (protect against stale
|
||||
results) — but the loser was the scheduler's own slot, so the guard silently ate planned work.
|
||||
4. **The winner was out-of-band.** Its `source` is `direct`, not the scheduler's `builtin`, and it started
|
||||
~2 minutes before Stage 1's handover run closed (08:29:21). The timing is consistent with
|
||||
[[runbooks/job-radar]]'s Stage-1 handover trigger; the trigger's exact origin was **not directly verified**
|
||||
in this audit.
|
||||
|
||||
## Open questions / candidate fixes (none applied)
|
||||
|
||||
- Should the scheduler **skip or suppress** a slot when a live claim for the same job already exists,
|
||||
instead of launching a second concurrent run?
|
||||
- Should an **out-of-band `direct` fire** (e.g. Stage 1's handover) mark the next scheduled slot consumed,
|
||||
the way a consumed handoff file silences it?
|
||||
- Should a **discarded-stale run raise an incident row** at all? Today the fault is silent by construction —
|
||||
a `failed` execution with no ledger row and no alert.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[runbooks/job-radar]]
|
||||
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
|
||||
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]
|
||||
@@ -0,0 +1,129 @@
|
||||
---
|
||||
title: Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-11-08
|
||||
updated: 2026-11-09
|
||||
verified_on: 2026-11-09
|
||||
review_after: 2026-11-15
|
||||
confidence: high
|
||||
tags: [cron, fire-claim, scheduler, gateway-restart, drain, conduit, health-monitor, wiki-stocktake]
|
||||
sources: [live-executions-db, live-cron-incidents, live-agent-log, live-systemd-journal, conduit-self-update-json, live-jobs-json]
|
||||
---
|
||||
|
||||
# Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)
|
||||
|
||||
**Summary.** A gateway restart was **requested at ~04:29:52 AWST on 2026-11-07**. Hermes held it in
|
||||
*restart-after-turn* mode for the full **1800 s** window, then timed out and drained (`#77184`). The drain
|
||||
killed the two long-running cron executions that were still in flight, and both were closed **`failed`**
|
||||
with `RuntimeError: Cron job '…' lost its durable fire claim ownership`:
|
||||
|
||||
| job | id | run window (ledger) | closed |
|
||||
|---|---|---|---|
|
||||
| *Mnemosyne Health Monitor* | `ea79a2107e36` | 00:00:42.918 → 05:02:22.870 | `failed` |
|
||||
| *wiki-stocktake-maintenance* | `1a8e90ea2271` | 03:00:31.515 → 05:02:22.174 | `failed` |
|
||||
|
||||
systemd restarted the unit at **05:01:00** (`hermes-gateway.service: Scheduled restart job, restart counter
|
||||
is at 3.`); the replacement process was the one that wrote both terminal `failed` rows at 05:02:22.
|
||||
**The two runs were collateral, not the cause** — nothing in the ledger shows them failing on their own.
|
||||
|
||||
## Timeline (AWST; `journalctl --user -u hermes-gateway` unless stated)
|
||||
|
||||
| time | event |
|
||||
|---|---|
|
||||
| 04:27:55 | `conduit_push` self-update starts (`0.122.2 → 0.134.4`, [[systems/conduit]]) — `~/.hermes/conduit-self-update.json` |
|
||||
| ~04:29:52 | a gateway restart is requested — **inferred**, not logged: the after-turn wait that expires at 04:59:52 is 1800 s |
|
||||
| 04:29:54.774 / .796 | `ERROR cron.scheduler: Job 'ea79a2107e36'/'1a8e90ea2271': Restart-safe cron worker exited with status 1 after adopting this execution, without writing a durable terminal state; whether side effects ran is unknown.` → **incident rows raised** |
|
||||
| 04:29:5x → 04:59 | the gateway **itself keeps running** — `pid 175269` keeps logging at 04:34, 04:39 … 04:59 |
|
||||
| 04:32:32 | conduit's ledger marks its update `step: failed`, `failed_step: restarting_gateway`, detail *"The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host."* |
|
||||
| 04:59:52.433 | `WARNING gateway.run: Restart after-turn wait timed out after 1800s with 3 still active; proceeding to stop()/drain which may interrupt remaining work (#77184)` |
|
||||
| 05:00:43.469 | `ERROR cron.scheduler: Job 'Mnemosyne Health Monitor' failed: RuntimeError: … lost its durable fire claim ownership`; admitted turn `cc64fdf07d8c4dbd898d83dc2fa74894` fails |
|
||||
| 05:00:47.344 | the same for `wiki-stocktake-maintenance`; admitted turn `56c8fe136a1241e7827316299b490ecb` fails |
|
||||
| 05:00:50.863 | `Force-killed MCP process 175401 (cua-driver) after SIGTERM timeout` |
|
||||
| 05:01:00 | systemd: *Scheduled restart job, restart counter is at 3* |
|
||||
| 05:01:41 | gateway back up as `pid 2981474` |
|
||||
| 05:02:22 | both ledger rows finalised `failed` |
|
||||
| 05:02:41 → 05:02:59 | job `96eb95ce12f2` (agent-log ERROR watchdog) exits 1 having caught the health monitor's traceback — **it reported the event correctly, ~19 s later** |
|
||||
| 05:43:03/04 | `hermes-dashboard` stopped and started (the 0.22.5 → 0.23.0 update, [[systems/hermes-agent]]) |
|
||||
| 11:11:44 → 11:11:55 | the gateway is restarted again to load the reinstalled `hermes-hq-push` 0.9.42 routes ([[systems/hermes-hq-push]]) |
|
||||
| 15:43:17 / 17:08:52 | the health monitor's two incident rows are closed / resolved |
|
||||
| 2026-11-08 00:00 | **this audit:** both stocktake rows are still `detected`, `alerted_at: null` |
|
||||
|
||||
## Incident-ledger state (live read 2026-11-08 00:0x AWST)
|
||||
|
||||
| id | job | state | first seen | alerted_at | closed |
|
||||
|---|---|---|---|---|---|
|
||||
| `1a8e90_2f370ef764f5` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T05:02:21.573 | **null** | — |
|
||||
| `1a8e90_628d914f1f97` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T04:29:54.815 | **null** | — |
|
||||
| `ea79a2_111e67a3d83c` | Mnemosyne Health Monitor | `closed` | 2026-11-07T04:29:54.795 | 04:29:57.581 | 15:43:17 |
|
||||
| `ea79a2_f317fcd843ae` | Mnemosyne Health Monitor | `resolved` | 2026-11-07T13:36:56.713 | 13:36:59 | 17:08:52 — *"Cron execution unverified; reconcile `…/admissions/df79590101fb3b3164e2261a1018f78473c5`"* |
|
||||
|
||||
The day ends with **two open rows on one job, neither alerted** (`alerted_at` null) — the same *red row, no
|
||||
alert* shape the 2026-11-06 audit recorded for Job Radar, here on a different failure path.
|
||||
|
||||
## Root cause and working hypotheses
|
||||
|
||||
**Established.** This is a **restart-drain** event, not a scheduler or model fault: the ledger error string,
|
||||
the 1800 s after-turn timeout, and the systemd restart on the same minute line up. The runtime behaviour is
|
||||
`#77184` — a requested restart is deferred until active turns finish, and when the 1800 s window expires
|
||||
the gateway proceeds to `stop()`/drain *"which may interrupt remaining work"*. The two cron runs were that
|
||||
remaining work.
|
||||
|
||||
**Not established — the restart's origin.** The request at ~04:29:52 is:
|
||||
- **2 s before** both restart-safe cron workers exited status 1 (04:29:54), and
|
||||
- inside the `conduit_push` self-update window (`0.122.2 → 0.134.4`, 04:27:55 → 04:32:32), whose own final
|
||||
step is recorded as **`restarting_gateway` = failed**.
|
||||
|
||||
*Working hypothesis:* **the conduit self-update requested the restart, could not complete it (the gateway
|
||||
runs under systemd), recorded failure — and the gateway went on to drain and restart 31 minutes later.**
|
||||
Every timestamp fits; no log line names the requester, so this is not proven. The alternative (another
|
||||
component requested it) cannot be excluded. The 0.22.5 → 0.23.0 updater is **excluded**: its own receipt
|
||||
does not begin until **05:05:13**, after the restart was already pending.
|
||||
|
||||
**Second, independent defect — adoption lost the claim instead of resuming it.** Both workers *adopted*
|
||||
their executions at 04:29:54 and then exited status 1 without a durable terminal state; the adopted runs
|
||||
were never re-driven, and the claims were only closed by the next process at 05:00:43/47. The ledger
|
||||
cannot say whether their side effects ran (*"whether side effects ran is unknown"*), so the stocktake's
|
||||
2026-11-07 edits are known to exist only because an in-day session committed them
|
||||
([[systems/scheduled-tasks]]).
|
||||
|
||||
## Impact
|
||||
|
||||
- **Mnemosyne Health Monitor** delivered nothing from its 00:00 slot. Its 11-06 slot had taken
|
||||
**7 h 47 m 56 s** and its 11-05 slot **12 h 13 m 54 s**, so this slot was already deep in its multi-hour
|
||||
tail ([[systems/mnemosyne]]).
|
||||
- **wiki-stocktake-maintenance** lost its 03:00 slot for a **second consecutive day** (`failure_streak: 2`,
|
||||
`last_status: error`) — no stocktake ran on 2026-11-07.
|
||||
- **Job Radar was not affected on 11-07:** no slot failed and the day's report landed normally, unlike
|
||||
2026-11-06 ([[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]).
|
||||
|
||||
## Verification evidence
|
||||
|
||||
journalctl --user -u hermes-gateway --since '2026-11-07 04:00' --until '2026-11-07 05:10'
|
||||
/home/hermes/.hermes/cron/executions.db (status/error per row; cron_incidents table)
|
||||
/home/hermes/.hermes/conduit-self-update.json
|
||||
/home/hermes/.hermes/cron/jobs.json (failure_streak, last_status)
|
||||
|
||||
## Open items
|
||||
|
||||
- The two open stocktake rows have **never been reconciled** (`alerted_at` null) — either resolve them or
|
||||
fix the alerting path that let a `detected` row sit silent.
|
||||
⚠ **2026-11-09 re-check: still open, and now three.** `1a8e90_628d914f1f97` and `1a8e90_2f370ef764f5`
|
||||
(both first seen 2026-11-07) are unchanged, and a **third** arrived —
|
||||
`1a8e90_a9cad99cb150`, `detected` 2026-11-08T04:35:38, `alerted_at: null`, from the 03:01:11 → 04:36:08
|
||||
failure on `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending…`.
|
||||
**Unclosed incident rows 2 → 3.** The alert path itself is **not** broken: the *Newsletter Digest* row
|
||||
`85edf6_0877ca1e0eac` alerted at **07:23:07** and closed as `resolved` at **08:06:54** the same window.
|
||||
So the gap is specific to the **stocktake's** rows.
|
||||
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
- Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and
|
||||
whether the 1800 s after-turn wait should refuse to drain long-running cron executions.
|
||||
|
||||
## See also
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/hermes-agent]]
|
||||
- [[systems/conduit]]
|
||||
- [[systems/mnemosyne]]
|
||||
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]
|
||||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]
|
||||
@@ -0,0 +1,165 @@
|
||||
---
|
||||
title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-09
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
review_after: 2026-11-20
|
||||
confidence: high
|
||||
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
|
||||
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
|
||||
---
|
||||
|
||||
# The `conduit_push` `pre_tool_call` hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
|
||||
|
||||
**Summary.** From **2026-11-07 14:14:03** to **2026-11-08 08:21:03** the `conduit_push` plugin's
|
||||
**sole `pre_tool_call` callback** overran Hermes's **30 s hook budget** and was skipped. Each skip
|
||||
suppresses the callback for a window, so the *pre-tool* side of tool dispatch degraded from *slow* to
|
||||
*blocked* — this is the fault that ate the 2026-11-08 morning and stretched that day's cron runs to
|
||||
**6 h** (*Mnemosyne Health Monitor* 12:00:51 → 18:00:42) and **1 h 13 m** (*FreshRSS Midday* 12:00:50 →
|
||||
14:10:43). Fixed **2026-11-08 13:35 AWST** by raising `plugins.hook_callback_timeout` **30 → 120**.
|
||||
|
||||
## Evidence (live reads, 2026-11-08 23:5x – 2026-11-09 00:1x AWST)
|
||||
|
||||
grep -c "callback _pre_tool_call timed out" ~/.hermes/logs/errors.log -> 58
|
||||
# 2026-11-07: 12 lines 2026-11-08: 46 lines
|
||||
# first 2026-11-07 14:14:03,075 last 2026-11-08 08:21:03,863
|
||||
grep -c "skipped after previous timeout" ~/.hermes/logs/*.log -> 89
|
||||
|
||||
The exact pair of lines (`~/.hermes/logs/errors.log`):
|
||||
|
||||
2026-11-08 08:19:40,942 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call skipped after previous timeout or while still running
|
||||
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
|
||||
|
||||
**The register (correction to a plausible-but-wrong diagnosis).** `conduit_push/__init__.py:119` is the
|
||||
**only** `pre_tool_call` registration on the host, and the plugin's caches are already in place —
|
||||
`session_titles.py:21 CACHE_SECONDS = 30.0`. Timed in-process by the 2026-11-08 session:
|
||||
**3.1 s cold, 0.00 s warm**. The *"the hook re-opens a 1.87 GB state.db per call, ~9 s each"* theory was
|
||||
therefore **wrong** — warm calls were free. The timeouts track **load spikes** on the 2-core CT460
|
||||
(load average **5.89** at 13:27:50; `state.db` carrying a **40 MB WAL** checkpointing on the same spindle),
|
||||
not per-call database opens.
|
||||
|
||||
## Fix applied (2026-11-08 13:35 AWST)
|
||||
|
||||
~/.hermes/config.yaml
|
||||
plugins:
|
||||
hook_callback_timeout: 120 # was the built-in 30 s default
|
||||
|
||||
- **Verified live:** `plugins.hook_callback_timeout = 120`.
|
||||
- **Verified effective:** no `_pre_tool_call` timeout has been logged since **08:21:03** — re-checked at
|
||||
the end of the fix window and again at audit time (2026-11-09 00:0x). The **58th** line is still the last.
|
||||
- **Deliberately NOT done:** rewriting the plugin's caching layer. The caches work; the budget was the
|
||||
binding constraint. **Next lever is `state.db` checkpoint/retention discipline**, not more caching.
|
||||
|
||||
## Consequence — the same signature landed in the cron ledger
|
||||
|
||||
The 2026-11-08 03:00 *wiki-stocktake-maintenance* slot (`1a8e90ea2271`, 03:01:11 → 04:36:08) closed
|
||||
**`failed`** on a **new terminal signature for that job**:
|
||||
|
||||
RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model
|
||||
produced no follow-up text. Send `continue` to let it summarize.
|
||||
|
||||
*Working hypothesis, not proven:* a `pre_tool_call` block holding a tool result open is exactly what this
|
||||
error describes, and it sits inside the same 14 h timeout window. It is **consistent**, not established —
|
||||
the error string does not name the hook.
|
||||
|
||||
## Impact
|
||||
|
||||
| item | effect |
|
||||
|---|---|
|
||||
| *Mnemosyne Health Monitor* `ea79a2107e36` | 00:01:20 → **08:29:05** (8 h 27 m) and 12:00:51 → **18:00:42** (5 h 59 m 51 s) — its two slowest slots on record, both delivered |
|
||||
| *wiki-stocktake-maintenance* `1a8e90ea2271` | 03:01:11 → 04:36:08 `failed`; **`failure_streak` 3**; a **third** unalerted `detected` incident row |
|
||||
| *FreshRSS Midday Briefing* `ecc1fc13655a` | 12:00:50 → **14:10:43** (2 h 09 m 53 s) |
|
||||
| *Tech & AI Newsletter Digest* `85edf6772bca` | an admission-read `database is locked` incident (`85edf6_0877ca1e0eac`) at **07:23:02**, **resolved** at 08:06:54 when the run completed |
|
||||
| every interactive tool call | slow, then suppressed — the user-visible symptom that opened the investigation |
|
||||
|
||||
## Verification evidence
|
||||
|
||||
~/.hermes/logs/errors.log (58 timeout lines; last 2026-11-08 08:21:03)
|
||||
~/.hermes/config.yaml (plugins.hook_callback_timeout: 120)
|
||||
~/.hermes/plugins/conduit_push/__init__.py:119,257 (the only pre_tool_call registration)
|
||||
~/.hermes/plugins/conduit_push/session_titles.py:21 (CACHE_SECONDS = 30.0)
|
||||
~/.hermes/cron/executions.db, cron_incidents, jobs.json (run windows, streak, incident rows)
|
||||
|
||||
## Open items
|
||||
|
||||
- **`state.db` is 1.89 GB with 2,901 sessions / 197,682 messages and a live 40 MB WAL** — it is the load
|
||||
source behind the spikes. Checkpoint and retention policy are still undecided
|
||||
([[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
|
||||
- Confirm whether the stocktake's `No reply: … tool result was still pending` failure is really the same
|
||||
fault; if it recurs while the hook budget is 120 s, this hypothesis is dead.
|
||||
- Decide whether a skipped `pre_tool_call` should **fail loudly** rather than silently degrade tool dispatch.
|
||||
|
||||
## Follow-up (2026-11-10 nightly audit)
|
||||
|
||||
- **The fix held; the cascade did not recur.** `errors.log` (1,035 retained lines) contains **no `pre_tool_call` timeout entry at all**. Its only two hook lines are benign skips — `on_session_end` 2026-11-09 22:58:35 and `pre_llm_call` 2026-11-10 00:02:54 (*"skipped after previous timeout or while still running"*).
|
||||
- **No 11-09 run showed the cascade signature.** Outside the two long-runners documented separately (stocktake **9 h 10 m**, Job Radar Stage 1 **9 h 09 m** — both stalled in *tool iteration*, not in a hook), the longest 11-09 execution was **74 min**, then 40 min and 11 min.
|
||||
- **2026-11-09 was the first `failed`-free cron day since 2026-11-05** (11-06: 1 failed, 11-07: 3, 11-08: 1, 11-09: **0 of 174**), and unclosed incident rows went 3 → 0.
|
||||
- **Residual:** the two long-runners above are the same 9-hour-turn environment this bug leaves behind, and one (`wiki-stocktake-maintenance`) is pinned to `free-stack` — a combo recorded broken on 2026-08-14. The job-iteration budget stays in force.
|
||||
|
||||
## Recurrence (2026-11-14 nightly audit) — the raised budget did not hold
|
||||
|
||||
**Reopened.** The `pre_tool_call` hook overran its **raised 120 s** budget again at **2026-11-14 00:05:40**,
|
||||
on the wiki-audit job's own tool calls. The 30 -> 120 s fix of 2026-11-08 is therefore **proven
|
||||
insufficient at current load**, and this page is no longer `resolved`.
|
||||
|
||||
| time (AWST) | line |
|
||||
|---|---|
|
||||
| 2026-11-14 00:05:40,043 | `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` |
|
||||
| 2026-11-14 00:05:43 -> 00:06:38 | **10** x `skipped after previous timeout or while still running`, ~6 s apart |
|
||||
|
||||
- **Source:** `~/.hermes/logs/errors.log` (single-source count; `agent.log` carries the same lines).
|
||||
By day: **2026-11-12 = 0 timeouts / 0 skips; 2026-11-13 = 0 / 0; 2026-11-14 = 1 / 10.** It is the
|
||||
**first `pre_tool_call` timeout in the retained log since the 2026-11-08 cascade** — the 2026-11-08
|
||||
lines themselves have since **rotated out** of `errors.log`, so this page's original 58-timeout figure
|
||||
still stands on the 2026-11-09 read and was **not re-verifiable** this window.
|
||||
- **Cost this time:** this audit's first `skill_view` and `terminal` calls were **refused outright** by the
|
||||
plugin (first-hand evidence), with dispatch recovering on its own by ~00:06:40 — roughly **one minute**
|
||||
of hard suppression. It is a **single-event recurrence, not the 18-hour cascade**. **No cron run failed:**
|
||||
2026-11-13 settled **173 `completed` / 0 `failed`**.
|
||||
- **Load is the variable, and it is now ~2x the level recorded when the fault was diagnosed:**
|
||||
`load average **12.08 / 13.02 / 12.33**` on **2 vCPU** (2026-11-08 read: **5.89**). `state.db` has grown
|
||||
**1.89 GB -> 2.04 GB** (**2,036,908,032 B**; **2,977 sessions / 207,577 messages**, was 2,901 / 197,682),
|
||||
and the root filesystem is **88 % used**.
|
||||
- **The earlier open hypothesis is now settled.** *"If it recurs while the hook budget is 120 s, this
|
||||
hypothesis is dead"* — it recurred, so the load trigger is **confirmed** and there is **no headroom left
|
||||
in the timeout**: at load ~12 the callback needs >120 s. **Re-raising the budget would only mask the
|
||||
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
|
||||
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
|
||||
|
||||
## Recurrence 2 (2026-11-15) — eleven timeouts in two clusters, 10 refusals
|
||||
|
||||
Recorded by the **2026-11-17 nightly audit** while backing up the 11-15 window: the **2026-11-15 audit
|
||||
captured none of this**. Exact `errors.log` counts for 11-15:
|
||||
|
||||
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` —
|
||||
**5** in 01:07:17 → 01:25:43 and **6** in 12:18:15 → 12:48:40 (a two-cluster day, unlike 11-14's one).
|
||||
- **10** tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
|
||||
— hour 01: `read_file` ×2 (134.17 s and 0.43 s), `execute_code` ×4, `patch` ×1, `read_file` ×1
|
||||
(121.03 s); hour 12: `execute_code` ×1. Call-level blocks reach **120.63 s**, and the affected sessions
|
||||
are `local-989b3642…` (01:xx) and `local-b6a1490f…` (12:xx).
|
||||
- 11-14, for comparison: **1** timeout and **4** refusals, plus **4** plugin load failures.
|
||||
- **11-16 and 11-17: 0 timeouts, 0 refusals, 0 load failures.** The episode did not carry over.
|
||||
|
||||
The loader half of the same 11-15 storm — **52** failed plugin loads, **519** discarded registrations
|
||||
(`conduit_push` 474 of them) and **2** `pre_llm_call` timeouts — is now on its own page,
|
||||
[[incidents/2026-11-15-plugin-load-timeout-storm]]. Its 12:18–12:49 cluster is the useful detail here:
|
||||
**a hook-timeout cluster can occur with no accompanying loader failure**, so the two share a cause class
|
||||
rather than a mechanism.
|
||||
|
||||
**Effect on the trigger hypothesis recorded above.** Amplitude is not a simple function of load average:
|
||||
the 11-15 episode ran at a load the audit cannot measure retroactively, while the 11-17 read
|
||||
(**6.19 / 6.64 / 7.48**) sits *below* the 11-14 figure of 12.08 with **no** recurrence — and a *larger*
|
||||
`state.db` (2.09 GB, WAL 40 MB, `/` at 88 %). Keep a **load-average threshold out of this page's claim**;
|
||||
keep **`state.db` size, disk headroom and the gateway's own timing budgets** in it.
|
||||
|
||||
|
||||
## See also
|
||||
|
||||
- [[systems/conduit]]
|
||||
- [[systems/hermes-agent]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/mnemosyne]]
|
||||
- [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
|
||||
- [[current-state]]
|
||||
@@ -0,0 +1,89 @@
|
||||
---
|
||||
title: "2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op"
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-11-10
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [cron, job-radar, stall, silent-failure, handoff, read-loop]
|
||||
sources: ["executions.db (dca8482e4f76: 2026-11-09T07:00:48+08:00 → 2026-11-09T16:10:12+08:00, status completed, delivery_outcome suppressed; 5fc292ff0f3b: 11:30:38 → 12:10:07 and 14:30:34 → 14:37:27, both completed/suppressed)", "~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md (ends INCOMPLETE: tool budget exhausted; last words \\u201cI stopped retrying because I kept running read_file 5 times without making progress.\\u201d)", "~/.hermes/radar-ref/handoff/ (newest handoff on disk: 2026-11-06; no 2026-11-07/08/09 files)", "cron_incidents (no row raised for this run)"]
|
||||
---
|
||||
|
||||
# 2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op
|
||||
|
||||
## Summary
|
||||
|
||||
On 2026-11-09 the Job Radar's Stage 1 (`dca8482e4f76`) ran for **9 h 09 m 24 s** (07:00:48 → 16:10:12), ended `INCOMPLETE: tool budget exhausted`, and **never wrote its handoff file**. Because both Stage-2 slots run on their own schedule, **both fired before Stage 1 finished** and each declined silently rather than erroring. Net effect: **the radar landed nothing on 11-09 and raised no alert.**
|
||||
|
||||
## Timeline (AWST, 2026-11-09)
|
||||
|
||||
| Time | Event |
|
||||
|---|---|
|
||||
| 07:00:48 | Stage 1 starts |
|
||||
| 11:30:38 → 12:10:07 | Stage-2 slot 1 fires — Stage 1 still running → silent no-op (`suppressed`) |
|
||||
| 14:30:34 → 14:37:27 | Stage-2 slot 2 fires — Stage 1 still running → silent no-op (`suppressed`) |
|
||||
| ~16:10:1x | Stage 1 writes `INCOMPLETE: tool budget exhausted` |
|
||||
| 16:10:12 | Stage 1 closes — `completed`, delivery `suppressed` |
|
||||
| all day | `radar-ref/handoff/2026-11-09.json` is never created |
|
||||
|
||||
## Root Cause
|
||||
|
||||
Two compounding faults, both previously documented for this job:
|
||||
|
||||
1. **Stage 2 never checks that a fresh handoff exists**, so when Stage 1 fails it proceeds and self-suppresses instead of failing loudly. Two chances to notice, zero alerting.
|
||||
2. **Stage 1 overran without producing output.** Its own final message: *"I stopped retrying because I kept running read_file 5 times without making progress."* This is a **new trigger** for the same silent family seen on 11-02/11-03 (there: iteration cap) — here: an unproductive read loop that consumed the run's budget.
|
||||
|
||||
## Evidence
|
||||
|
||||
- `executions.db` `dca8482e4f76`: `2026-11-09T07:00:48+08:00` → `2026-11-09T16:10:12+08:00`, `status completed`, `delivery_outcome suppressed`.
|
||||
- `5fc292ff0f3b` (Stage 2): two rows on 11-09, both **before** Stage 1's finish stamp, both `completed` / `suppressed`.
|
||||
- `~/.hermes/radar-ref/handoff/` — newest file is **2026-11-06**; nothing for 11-07, 11-08 or 11-09.
|
||||
- `~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md` ends `INCOMPLETE: tool budget exhausted`.
|
||||
- **No row in `cron_incidents`** for `dca8482e4f76` on 11-09 — the failure was never raised.
|
||||
|
||||
## Impact
|
||||
|
||||
- The Job Radar — the estate's job-discovery feed — produced **no output for a third day** (no handoff since 11-06).
|
||||
- The failure is invisible: the run's ledger entry says `completed`, delivery is `suppressed`, and no incident row exists.
|
||||
|
||||
## Fix Applied
|
||||
|
||||
**None — documentation only.** No configuration change was made this window.
|
||||
|
||||
## Prevention
|
||||
|
||||
- **Alert on the artefact, not the exit code.** Stage 2 should hard-fail when the expected `radar-ref/handoff/<date>.json` is missing, rather than proceeding to a suppression. That single check converts both 11-09 slots into loud failures.
|
||||
- Add a **read-progress guard** to Stage 1: when the same file has been read N times without a state change, stop and emit a partial handoff instead of burning the budget to `INCOMPLETE`.
|
||||
- Note the pattern across incidents: **completion-time-only alerting hides every failure in this family** (stocktake 11-09, radar 11-09, radar 11-02/03).
|
||||
|
||||
## Recovery — 2026-11-11: the handoff landed, and Stage 2 delivered
|
||||
|
||||
| Field | 11-09 (this incident) | **11-11** |
|
||||
|---|---|---|
|
||||
| Stage 1 | 07:00:48 → 16:10:12 (**9 h 09 m 24 s**), no handoff | **07:01:08 → 07:37:34 = 36 m 26 s**, `completed` |
|
||||
| Handoff on disk | newest was **2026-11-06** | **`radar-ref/handoff/2026-11-11.json` — 22,682 B, mtime 07:59** |
|
||||
| Stage 2 (11:30 slot) | fired early, declined silently | **11:30:56 → 11:35:18**, `completed` |
|
||||
| Stage 2 (14:30 slot) | declined silently | **14:30:55 → 14:34:56**, `completed` / **`delivered`** |
|
||||
| Incident row | **none raised** | `5fc292_773822455856` closed **`resolved` 2026-11-11T08:03:47.644226** |
|
||||
|
||||
- **The first handoff in five days** (`2026-11-06` → `2026-11-11`) and the fastest Stage-1 cycle since
|
||||
the two-stage split: **36 m 26 s** against 3 h 15 m (10-29, the last full success), 9 h 01 m (11-02),
|
||||
6 h 28 m (11-03) and 9 h 09 m (11-09).
|
||||
- **Stage 2's recovery fire** (`source: direct`, 07:35:56 → 08:03:47) consumed the handoff and delivered
|
||||
at **08:03:47**; its completion is what auto-closed the row opened by 11-10's `interrupted` slot.
|
||||
- ⚠ **Carried forward from 11-10:** the *scheduled* Stage-2 slot that day ran
|
||||
**11:30:35 → 22:18:49 = 10 h 48 m 14 s** and closed **`failed` / `interrupted`** (delivery
|
||||
`delivered`, `for_failure: 1`) — the longest run of either window, and itself a new data point for the
|
||||
"completion-time-only alerting" family on this page.
|
||||
- **Job state:** both radar jobs read `last_status: ok`, `failure_streak` 0.
|
||||
|
||||
**Status: `resolved`.** Both prevention rules on this page still stand and remain unimplemented: Stage 2
|
||||
should hard-fail on a missing `handoff/<date>.json` rather than suppressing, and Stage 1 needs a
|
||||
read-progress guard. Neither was needed on 11-11 because the run finished inside budget.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]
|
||||
- [[runbooks/provider-health]]
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
title: "2026-11-09 — Stocktake Completes With Zero Edits"
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-11-10
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [cron, wiki, stocktake, tool-budget, silent-failure, omni-101, free-stack]
|
||||
sources: ["executions.db (1a8e90ea2271: 2026-11-09T03:00:43+08:00 → 2026-11-09T12:10:47+08:00, status completed, delivery_outcome suppressed)", "~/.hermes/cron/output/1a8e90ea2271/2026-11-09_12-10-46.md (8,305 bytes; RUN INCOMPLETE, \\u201ctool-iteration budget exhausted during verification\\u201d)", "cron_incidents (1a8e90_628d914f1f97, 1a8e90_2f370ef764f5, 1a8e90_a9cad99cb150 — all closed 2026-11-09T12:10:47.125205+08:00)", "wiki-stocktake-maintenance job 1a8e90ea2271 (provider custom:omni-101, model free-stack)"]
|
||||
---
|
||||
|
||||
# 2026-11-09 — Stocktake Completes With Zero Edits
|
||||
|
||||
## Summary
|
||||
|
||||
The nightly wiki stocktake (`wiki-stocktake-maintenance`, `1a8e90ea2271`) ran on 2026-11-09 from **03:00:43 to 12:10:47** — **9 h 10 m** — and the harness recorded it **`completed`** with delivery `suppressed`. Its own output file records the opposite: the run ran out of tool-iteration budget inside its verification phase, applied **zero edits**, made **no commit**, and its findings never landed.
|
||||
|
||||
**This is the third consecutive day the stocktake's work did not land** — 11-07 died at 2 h 02 m, 11-08 at 1 h 35 m (both `failed`), 11-09 at 9 h 10 m (`completed`). The visible status this time is *worse*, not better: `failure_streak` fell **3 → 0** and `last_status` shows **ok**, while the work stayed undone.
|
||||
|
||||
## Timeline (AWST, 2026-11-09)
|
||||
|
||||
| Time | Event |
|
||||
|---|---|
|
||||
| 03:00:43 | Run starts (`1a8e90ea2271`) |
|
||||
| ~04:1x | Tool-iteration budget exhausted during verification; output file written with `RUN INCOMPLETE` |
|
||||
| 12:10:47.125205 | Harness closes the run — `completed`, delivery `suppressed` |
|
||||
| 12:10:47.125205 | The **three** long-open stocktake incident rows close at the same instant: `1a8e90_628d914f1f97` (11-07T04:29:54), `1a8e90_2f370ef764f5` (11-07T05:02:21), `1a8e90_a9cad99cb150` (11-08T04:35:38) |
|
||||
|
||||
## Root Cause
|
||||
|
||||
Same family as 11-07/11-08: **the run spends its entire tool budget before reaching the write phase.** The run's own output file records the stop as the tool-iteration budget being exhausted **during verification**, with zero edits applied — the edits it had identified were never written.
|
||||
|
||||
Secondary cause (confirmed again this window): the job is pinned to `custom:omni-101` / **`free-stack`** — a combo recorded broken on **2026-08-14** (see [[systems/scheduled-tasks]]).
|
||||
|
||||
## Evidence
|
||||
|
||||
- `executions.db`: `started_at 2026-11-09T03:00:43+08:00` → `finished_at 2026-11-09T12:10:47+08:00`, `status completed`, `delivery_outcome suppressed`. Previous two runs: 11-08 `failed` (03:01:11 → 04:36:08, 1 h 35 m), 11-07 `failed` (03:00:31 → 05:02:22, 2 h 02 m).
|
||||
- Output file `/home/hermes/.hermes/cron/output/1a8e90ea2271/2026-11-09_12-10-46.md` (8,305 bytes) ends `RUN INCOMPLETE`, with the tool-iteration budget exhausted inside verification.
|
||||
- The three stocktake incident rows all carry `closed_at = 2026-11-09T12:10:47.125205+08:00` — i.e. they closed on **completion**, not on a fix. All three have `alerted_at = null`: they never alerted.
|
||||
- Git: **no stocktake commit on 11-09** — zero edits were applied, so there was nothing to commit and nothing to revert. (On 11-07/11-08 the page was left dirty and then reverted; on 11-09 no edit was ever made.)
|
||||
|
||||
## Impact
|
||||
|
||||
- Nine consecutive stalled stocktake runs (11-01 … 11-09) — the nightly wiki freshness pass is producing no durable output.
|
||||
- The stocktake's *top-priority* brief item on 11-09 was the **monotonically worsening failure streak** itself. The job closed its own incident while failing that item, which is the failure mode to watch: **`completed` and `ok` now overstate the stocktake's health.**
|
||||
- Three incident rows were silently cleared, so the fleet's incident board now reads **0 open** while the underlying problem is unfixed.
|
||||
|
||||
## Fix Applied
|
||||
|
||||
**None — this is a detection-and-verification record.** The page documents the failure and the misleading status so the next audit starts from truth.
|
||||
|
||||
Also recorded this window: the `free-stack` pin on `1a8e90ea2271` — the only deterministic configuration fault in the chain — is **still in place**, unchanged from the 11-08 record.
|
||||
|
||||
## Prevention
|
||||
|
||||
- **Alerting must fire on write progress, not completion.** This run produced 2 of 3 failure days silently and the third day produced a run that reported success while editing nothing. A check on `commit_count` / `edits_applied` for this job would have caught all three.
|
||||
- Don't read `status`/`failure_streak` as the stocktake's health. Read whether a commit landed.
|
||||
|
||||
## Recovery — 2026-11-11: the pass ran in 1 h 10 m and landed 36 pages
|
||||
|
||||
The next scheduled slot **worked**, and worked quickly:
|
||||
|
||||
| Field | 11-09 (this incident) | **11-11** |
|
||||
|---|---|---|
|
||||
| Window | 03:00:43 → 12:10:47 | **03:00:56.081504 → 04:11:38.724852** |
|
||||
| Duration | **9 h 10 m** | **1 h 10 m 42 s** |
|
||||
| Terminal state | `completed` / `suppressed` | `completed` / `suppressed` |
|
||||
| Edits landed | **0** | **36 stale pages re-verified → 0 stale** |
|
||||
| Commit | **none** | **`341dda9`** + **`c108eb1`** |
|
||||
| Freshness report | — | `raw/stocktake/freshness-20261111.txt` (**6,235 B**) |
|
||||
|
||||
- `341dda9` re-verified 30+ further pages live and de-flagged them, and applied real content
|
||||
corrections — the live fallback chain rewritten in [[runbooks/test-openrouter]] (primary back to
|
||||
`smart-route` / `custom:omni-101`), the CT333 `clawtest` naming claim corrected, and the nous
|
||||
**404 `insufficient_credits_for_paid_model`** signature replacing the old refresh-token reading in
|
||||
[[incidents/2026-10-22-nous-portal-refresh-token-invalid]].
|
||||
- `c108eb1` regenerated the freshness report after the log/index addendum, so the published Quartz build
|
||||
matches HEAD.
|
||||
- **Job state:** `last_status: ok`, `failure_streak` 0, `last_run` 2026-11-11T03:00:56.
|
||||
⚠ The job is **still pinned to `custom:omni-101` / `free-stack`** — the combo recorded broken on
|
||||
2026-08-14 — and this run succeeded on it anyway. That is an observation, not a repin.
|
||||
|
||||
**Status: `resolved`.** The 11-07/11-08/11-09 run of failures is closed by output, not by a cosmetic
|
||||
`failure_streak` reset. The page's own prevention rule stands: *alert on `commit_count` / `edits_applied`,
|
||||
never on `status`* — this recovery is exactly why it matters, because 11-09 and 11-11 both read
|
||||
`completed` / `suppressed` and only one of them did anything.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[infrastructure/tailscale]]
|
||||
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
- [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]
|
||||
@@ -0,0 +1,118 @@
|
||||
---
|
||||
title: Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-11-11
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
review_after: 2026-12-11
|
||||
confidence: high
|
||||
tags: [cron, incident, omniroute, rate-limit, audit]
|
||||
sources: [live-executions-db, live-cron-incidents, gateway-log, live-jobs-json]
|
||||
---
|
||||
|
||||
# Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited (2026-11-10 23:59 → 2026-11-11 00:12)
|
||||
|
||||
**This job is the estate's own observer, and on 2026-11-10 it became the estate's only failure.** Its
|
||||
23:59 slot never produced a report, so **no page, no log entry and no incident alert describes the day
|
||||
of 2026-11-10** — an unaudited day is invisible by construction.
|
||||
|
||||
## What happened
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Job | `cbfe434a3b5e` — **Wiki Audit — Nightly**, `custom:omni-101` / `deepseek-paid-smart` |
|
||||
| Slot | 2026-11-10 23:59:54.486572 +08:00 (cron `59 23 * * *`) |
|
||||
| Ended | **2026-11-11 00:12:29.646987 +08:00** — **12 m 35 s** |
|
||||
| Status | `failed` · `delivery_outcome: suppressed` |
|
||||
| Error | `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` |
|
||||
| Output | `~/.hermes/cron/output/cbfe434a3b5e/2026-11-11_00-12-28.md` |
|
||||
|
||||
The three log lines from the run's own session `local-1000c1a3…` show the shape of the failure:
|
||||
|
||||
ERROR agent.chat_completion_helpers: Streaming failed before delivery: [429]: service failure: tenant tn-5ud77abtl1 quota exceeded
|
||||
ERROR agent.chat_completion_helpers: Streaming failed before delivery: Error code: 429 -
|
||||
ERROR agent.conversation_loop: API call failed after 3 retries. HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters
|
||||
ERROR cron.scheduler: Job 'Wiki Audit — Nightly' failed: RuntimeError: HTTP 429: Service temporarily unavailable
|
||||
|
||||
- **The pool skipped every target.** Six candidates were evaluated and **`attempted 0`** — the run never
|
||||
reached a provider. The only live target's credit balance read **100 %** consumed on connection
|
||||
`conn:3021bab6`, which is why the router refused to dispatch rather than serving a degraded model.
|
||||
- **`reset after 6h 24m 31s`** puts the pool's recovery at **≈ 2026-11-11 06:36:57 AWST**. The 07:0x
|
||||
cron cohort ran clean the next morning, so the window closed on its own.
|
||||
- The **same tenant** (`tn-5ud77abtl1`) appears in the `quota exceeded` line, so this is an
|
||||
account-level quota fault on the router, not a single-model lockout.
|
||||
|
||||
## Incident row — raised, never alerted, closed `resolved` 2026-11-12T00:25:36
|
||||
|
||||
`cbfe43_ed01845310d8` — `job_id cbfe434a3b5e`, `failure_type: rate_limit`, state **`detected`**,
|
||||
`alerted_at: NULL`, `acked_at: NULL`, `closed_at: NULL`, opened **2026-11-11T00:12:28.578435+08:00**
|
||||
(`first_seen_at` = `last_seen_at` = that instant — seen exactly once).
|
||||
|
||||
- It **was** the only open row in `cron_incidents` — the table held **135** rows at audit time and **137**
|
||||
by the end of this run. `acked_at` is **NULL** as well as `alerted_at`, so nothing was told and
|
||||
nothing was acknowledged. The previous open item,
|
||||
`5fc292_773822455856`, closed `resolved` at **2026-11-11T08:03:47.644226**.
|
||||
- So the estate's single outstanding incident **was** the audit's own failure, and nothing was told
|
||||
about it. ⚠ **Superseded 2026-11-12** — the row closed `resolved` at **2026-11-12T00:25:36.133387+08:00**;
|
||||
*cron_incidents* now holds **0 unclosed of 137**. See *Resolution* below.
|
||||
|
||||
## Resolution — 2026-11-12T00:25:36, self-cleared
|
||||
|
||||
The row closed **`resolved`** at **2026-11-12T00:25:36.133387+08:00**, at the end of the **next** slot of
|
||||
the same job: `cbfe434a3b5e` ran **2026-11-11T23:59:54.486572 → 2026-11-12T00:25:36.133387 = 25 m 42 s**
|
||||
and completed. Its delivery carried **both windows**, so 2026-11-10's day was reconciled a day late
|
||||
rather than never (the recovery record is the `audit(2026-11-11)` commit series at 2026-11-12 00:1x-00:24).
|
||||
|
||||
Nothing was repaired to make this happen: the pool's own reset at ≈ **2026-11-11 06:36:57** restored the
|
||||
dispatch path, exactly as the `reset after 6h 24m 31s` field predicted, and the 07:0x cohort ran clean.
|
||||
`cron_incidents` now reads **0 unclosed of 137** rows — the estate is incident-clean as of 2026-11-12.
|
||||
|
||||
**The recurrence risk is unchanged.** The same *"all targets were skipped by pre-dispatch filters"*
|
||||
signature was found again on a **second job and a second pin** by the 2026-11-12 03:00 stocktake
|
||||
([[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]), and every model-calling job still rides
|
||||
`custom:omni-101` with no failover for the free-tier cohort. Closing this row proves the guard
|
||||
self-heals; it is not evidence the guard is gone.
|
||||
|
||||
## Impact
|
||||
|
||||
- **The 2026-11-10 window was never reviewed.** The nightly audit is the only producer of the per-day
|
||||
ledger/wiki record; its failure means 11-10's 157 ledger rows were never reconciled on the day.
|
||||
- **Fleet non-clean `last_status` 1 → 2.** `cbfe434a3b5e` now reads `last_status: error`,
|
||||
`failure_streak: 1`, joining `066bb5191280` (*Uber Eats Monthly Spend Report*, `delivery_failed`,
|
||||
streak 0).
|
||||
- **The review published on 2026-11-11 23:59 carries both windows** — it is the recovery, not a
|
||||
duplicate.
|
||||
|
||||
## Same signature, earlier appearances
|
||||
|
||||
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — the *identical* pre-dispatch-filter
|
||||
string killed the Daily Marketing Job Radar on 2026-10-21 (`attempted 0`, no 503 anywhere in the day).
|
||||
- 2026-11-05 — `deepseek-paid-smart` against the **CT600** endpoint returned `HTTP 503
|
||||
ALL_TARGETS_SKIPPED` (`poolSize 5`, `attempted 0`) while `omniroute-1` answered 200 — recorded on
|
||||
[[systems/scheduled-tasks]].
|
||||
- This is the **third distinct symptom** of the same router behaviour: `attempted 0` with a healthy node
|
||||
behind it is the thing to recognise, not the specific HTTP code.
|
||||
|
||||
## Fix Applied
|
||||
|
||||
**None — this is a detection-and-verification record.** No configuration changed this window. The run's
|
||||
own recovery was the pool reset at ≈ 06:36:57 on 2026-11-11, and the incident row closed on the next
|
||||
slot's clean completion (2026-11-12T00:25:36).
|
||||
|
||||
## Prevention
|
||||
|
||||
- **The observer needs an observer.** A failed nightly audit raises no delivery (suppressed), no alert
|
||||
(`alerted_at: NULL`) and no page — every layer that would normally surface this is *inside the thing
|
||||
that failed*. A one-line watchdog on `Wiki Audit — Nightly`'s `last_status != ok` (or on "no audit
|
||||
commit in >24 h") is the only check that can catch it.
|
||||
- **Alert on the artefact.** Verifying that a dated `audit(YYYY-MM-DD)` commit exists in this repo would
|
||||
have caught both this failure and the stocktake's zero-edit days — the same lesson recorded on
|
||||
[[incidents/2026-11-09-stocktake-completes-with-zero-edits]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/omniroute]]
|
||||
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
|
||||
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
title: hermes-hq-push 0.9.47+ — Upgrade Refused by the Plugin Security Scan (dangerous verdict)
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-11-13
|
||||
updated: 2026-11-13
|
||||
verified_on: 2026-11-13
|
||||
review_after: 2026-12-13
|
||||
confidence: high
|
||||
tags: [plugins, security-scan, hq-push, plugin-guard, install, incident]
|
||||
sources: [live-hermes-plugins-list, live-plugin-manifest, live-config, live-plugin-guard-source, live-gh-pr, live-state-db]
|
||||
---
|
||||
|
||||
# hermes-hq-push 0.9.47+ — Upgrade Refused by the Plugin Security Scan (2026-11-12)
|
||||
|
||||
**The scan did its job and there is no way to overrule it.** On 2026-11-12 (~17:38–18:29 AWST) the
|
||||
attempt to take [[systems/hermes-hq-push]] past its installed **0.9.42** was refused before any file
|
||||
was written, by the pre-install plugin security scan. The installed copy is untouched, still `enabled`,
|
||||
and still delivering. The blocking finding is in the plugin's **own** `dashboard-tile` skill, and the
|
||||
fix is sitting in an open pull request.
|
||||
|
||||
## What happened
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Attempted | `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` |
|
||||
| Refusal | `Blocked: Security scan blocked plugin install: Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict.` |
|
||||
| Findings | **39 total — 1 critical**, the rest low/medium |
|
||||
| Sole critical | `hermes-hq-push/skills/dashboard-tile/SKILL.md:41` — category *obfuscation*: the `echo '<json>' \| python3 …` one-liner |
|
||||
| Verdict logic | **any** critical ⇒ `dangerous` = hard block; medium/low alone ⇒ `caution` (installable, warned) |
|
||||
| Installed after | **unchanged** — `v0.9.42`, tree mtime **2026-11-08 02:19**, `enabled`, state `user` |
|
||||
|
||||
## Why no override was available (verified in source, not assumed)
|
||||
|
||||
- **The only config key is `plugins.scan_on_install`** (default `true`; currently **not set** in
|
||||
`~/.hermes/config.yaml`, so scanning is on). Setting it `false` disables the scan for **every** future
|
||||
install — it does not overrule a verdict, it blinds the check. That is the wrong lever.
|
||||
- **`--force` lifts `caution` only.** Its own refusal text says so, and the code agrees. `--allow-removed`
|
||||
is unrelated — it is the blocklist for catalogue removals.
|
||||
- **No env var, no trusted-source list, no allowlist, no `INSTALL_POLICY`/`trust_level` for plugins.**
|
||||
Those mechanisms exist only in `tools/skills_guard.py`, reachable from the *skills* path
|
||||
(`skills.guard_agent_created`). `tools/plugin_guard.py:398` hardcodes `trust_level="community"` and
|
||||
`should_allow_plugin_install` never reads it.
|
||||
- **The block is a single unconditional branch** at `tools/plugin_guard.py:406-418`: any verdict outside
|
||||
`safe`/`caution` returns `False, "Blocked (dangerous verdict, …)"`.
|
||||
|
||||
## The fix, and why it is not merged
|
||||
|
||||
The critical is a *pattern* match, not a defect: the tile skill embedded its JSON payload in a shell
|
||||
one-liner, which the scanner reads as obfuscation however benign the intent.
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Repo / PR | `mrcharlesiv/hermes-hq-plugins` **#2** — the repo's **only** open PR |
|
||||
| Author | `Tony0410` (Anthony) |
|
||||
| Head | fork `Tony0410/hermes-hq-plugins`, branch `fix/tile-skill-heredoc` → base `main` |
|
||||
| State | **OPEN**, `mergedAt: null`, `mergeable: MERGEABLE`, created 2026-11-12T10:00:01Z (updated 10:01:57Z) |
|
||||
| Diff | **+7 / −2 in one file** — `hermes-hq-push/skills/dashboard-tile/SKILL.md` (the one-liner becomes a heredoc) |
|
||||
| Local clone | `/home/hermes/workspace/hq-plugins-fix` (commit `90c9d28`, branch checked out, clean) |
|
||||
|
||||
The `gh` CLI on this host is authenticated as **`Tony0410` with read-only access** to
|
||||
`mrcharlesiv/hermes-hq-plugins`, so the change had to travel fork → PR rather than as a branch push.
|
||||
|
||||
## Impact
|
||||
|
||||
- **The upgrade path to 0.9.47+ is blocked** until PR #2 merges upstream and the plugin is re-installed.
|
||||
Nothing else in the estate is affected.
|
||||
- **No app-side impact:** the installed 0.9.42 keeps delivering to the Hermes HQ iPhone app; the refusal
|
||||
happened before any file was replaced, so there is no half-installed state to clean up.
|
||||
- The block is **fail-safe by design** — the failure mode is "cannot upgrade", not "installed something
|
||||
unchecked".
|
||||
|
||||
## Related finding — the "stored hold-and-ask instruction" was never stored
|
||||
|
||||
The same evening, the alleged standing instruction to *hold and ask before acting on* plugin
|
||||
security-scan warnings was searched for exhaustively: a recursive sweep of `~/.hermes`
|
||||
(**184,716 files**, generated/venv/browser-profile/SQLite trees excluded) for the phrases
|
||||
`wait for my answer`, `show me what it found`, `security scan flags` and `security-scan` found **zero
|
||||
matches in any editable text file**. The sentence exists only in `state.db` **conversation history**
|
||||
(58 rows). **Nothing was removed and no policy was reverted**, because no such policy was ever
|
||||
persisted. Procedure recorded as the skill `stored-instruction-search-and-removal`.
|
||||
|
||||
## Prevention / how this closes
|
||||
|
||||
- **Merge PR #2**, then re-run the install. That is the whole fix.
|
||||
- If a standing exception is ever genuinely wanted, **add an explicit trusted-source or allowlist path to
|
||||
`tools/plugin_guard.py`** — do **not** set `plugins.scan_on_install: false`. Turning the scan off is
|
||||
not "allowing this plugin"; it is removing the check from every future install, including the ones
|
||||
nobody is looking at.
|
||||
- Expect the same class of false positive again: any plugin shipping a shell one-liner that pipes JSON
|
||||
into an interpreter will read as *obfuscation*. Author the pattern as a heredoc or a script file.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/hermes-hq-push]]
|
||||
- [[systems/hermes-agent]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[log]]
|
||||
@@ -0,0 +1,332 @@
|
||||
---
|
||||
title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-15
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
|
||||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
|
||||
---
|
||||
|
||||
# Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
|
||||
|
||||
> ⚠ **2026-11-20 re-read — the freeze ended by itself at 2026-11-18 19:25 AWST (105 h 52 m total), and every
|
||||
> interactive session inside the window was captured.** The "capture is dead" reading below is a *quiet estate*,
|
||||
> not a broken hook. Read the newest re-read section before the earlier readings.
|
||||
|
||||
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
||||
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
||||
(exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state **`alerted`**, **unclosed**).
|
||||
|
||||
Found by the **2026-11-14 nightly audit**; not caused by anything in that audit's window (it started
|
||||
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
|
||||
the same condition was classified informational — see below).
|
||||
|
||||
## Re-read 2026-11-16 00:18 AWST — still open, stall 38 h 46 m, and now **three** unclosed alerts
|
||||
|
||||
Six hours later, at the next nightly audit, nothing has moved. The freeze is **total**, the monitor has
|
||||
fired twice more, and the scheduler has minted further unclosed rows for the same fault.
|
||||
|
||||
- **The counts are frozen, not merely low.** `working_memory` **1130** / `episodic_memory` **1249** —
|
||||
**the same two numbers as every earlier read of the stall** (2026-11-15 ~00:4x and 12:03) — so this is
|
||||
not a quiet stretch or a degraded trickle. Zero writes in **38 h 46 m**.
|
||||
- **The whole write path stopped, not just turn capture.** Newest rows remain the three
|
||||
`[MODEL_REFRESH_PROPOSAL]` maintenance writes at **`09:32:58` / `09:32:59` AWST**; the newest
|
||||
*user-turn* capture is **`2026-11-14 09:04:19 AWST`**. Maintenance outlived the last turn by ~28 min,
|
||||
then both stopped inside the same minute.
|
||||
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows total). All three are `ea79a2107e36`, all
|
||||
`failure_type: auth`, all `state: alerted`, and **each has `first_seen_at == last_seen_at`** — a single
|
||||
observation, no `last_seen_at` accumulation, no escalation:
|
||||
|
||||
| Incident id | `first_seen_at` | `alerted_at` |
|
||||
| --- | --- | --- |
|
||||
| `ea79a2_601037516abd` | `2026-11-15T00:03:35.815373+08:00` | `2026-11-15T00:03:44.559501+08:00` |
|
||||
| `ea79a2_5f817708701e` | `2026-11-15T12:03:14.928500+08:00` | `2026-11-15T12:03:20.245081+08:00` |
|
||||
| `ea79a2_75303493d484` | `2026-11-16T00:03:15.418001+08:00` | `2026-11-16T00:03:22.365399+08:00` |
|
||||
|
||||
- ⚠ **New mechanism, evidenced: the incident signature embeds the elapsed-hours float.** The three
|
||||
signatures differ only by `4.64157453` / `4.634034074444445` / `4.643481075555555` — a value **never
|
||||
equal twice** — so the recurrence never matches the prior row. One durable fault therefore **mints a
|
||||
fresh unclosed row every slot**, and the estate's unclosed count grows ~1 per 12 h while no single row
|
||||
ever escalates. This job alone now has **15** `cron_incidents` rows (12 older `unknown`, these 3
|
||||
`auth`). **Proposed fix — NOT applied** (it changes the monitor's incident path, so it is Anthony's
|
||||
call): normalise the value out of the signature — round it, or omit it — so the same condition
|
||||
**re-opens the existing row** instead of fragmenting.
|
||||
|
||||
### The monitor's ≈4.64 h figure is NOT the stall — verified against the gate's own source
|
||||
|
||||
Every alert in this sequence prints a near-identical elapsed time while the stall grew from 14 h to
|
||||
38.7 h. That number is not a measurement of the stall. `mnemosyne_freshness_gate.py`'s
|
||||
`newest_user_message()` selects the newest `messages` row with `role='user'` whose session
|
||||
`source NOT IN ('cron','subagent')` (the `NON_CONVERSATIONAL` set, line 52) and returns that row's
|
||||
`source`; the alert prints **`(now − that message) / 3600` h**. So the figure is **the age of the newest
|
||||
*conversational* message — not the age of the newest capture.**
|
||||
|
||||
The newest conversational messages on this box are **Pulse** turns recorded under `source: desktop`
|
||||
(`Pulse delivery check pXXXX-1 …`, `Pulse morning brief …`), on a **~6 h cadence**. Each alert's
|
||||
≈4.64 h lands on the then-newest Pulse turn to within about a minute:
|
||||
|
||||
| Alert | Elapsed printed | Now ≈ | Newest conversational message it measured |
|
||||
| --- | --- | --- | --- |
|
||||
| 2026-11-15 00:03:35 | `4.64157 h` | 00:00:54 | **2026-11-14 19:22:30** (desktop / Pulse) |
|
||||
| 2026-11-15 12:03:14 | `4.63403 h` | 12:01:25 | **2026-11-15 07:23:25** (desktop / Pulse) |
|
||||
| 2026-11-16 00:03:15 | `4.64348 h` | 00:01:33 | **2026-11-15 19:22:59** (desktop / Pulse) |
|
||||
|
||||
**Why they all read ≈4.64:** the `0 */12` slots happen to fire ~4.6 h after the most recent Pulse turn.
|
||||
The near-constancy is a **cadence coincidence** and must **not** be read as "the stall is only ~4.6 h" —
|
||||
the stall is **38 h 46 m**. Same trap class as the naive-UTC note below: a number shaped like a duration
|
||||
that is not one.
|
||||
|
||||
**Positive proof the capture path is dead rather than merely idle:** the identical class of message *was*
|
||||
captured before the stall — the 2026-11-14 00:09:12 run logged
|
||||
`last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`. Four 2026-11-15 Pulse turns
|
||||
(`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have **no capture row at all**.
|
||||
|
||||
### The consolidator is alive and every cycle is a deliberate no-op
|
||||
|
||||
Latest `consolidation_log` row **4696** at `2026-11-15T16:14:14.934306+00:00` (= **2026-11-16 00:14:14
|
||||
AWST**), `items_consolidated: 0`, `session_id: auto-consolidate-maintenance`; cadence ~31 min
|
||||
(`15:12:13` → `15:43:08` → `16:14:14` UTC). **New evidence:** the row's `summary_preview` reads
|
||||
**`No-op consolidation cycle — timestamp updated for watchdog`** — the cycle **touches its own timestamp
|
||||
to satisfy an age test**, so a zero-item streak can run indefinitely while every age-based gate reads
|
||||
"fresh". The streak is now **~74 consecutive zero-item cycles**, opening
|
||||
`2026-11-14T02:04:42+00:00` (10:04:42 AWST). **A zero-item-streak threshold — not an age threshold — is
|
||||
what would have caught this at 10:04 on 11-14.**
|
||||
|
||||
### Adjacent fault in the same runtime — the watchdog lost a slot (2026-11-15 11:31)
|
||||
|
||||
`96eb95ce12f2` (**Mnemosyne Watchdog**, every 15 min) **failed** on its `11:31:20.699` slot,
|
||||
`finished_at 2026-11-15T11:49:10.534610` — **17 m 49.8 s**, exit 1:
|
||||
|
||||
```
|
||||
runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds
|
||||
```
|
||||
|
||||
A **600 s timeout inside a 15-minute slot** overran two slots: the `11:45` slot was **skipped** (no row
|
||||
at all) and the **next slot, `12:01:25.120952`, completed** (`→ 12:02:02.657231`), so it self-recovered
|
||||
with one slot lost. This is a **new signature** for this job — the August and October occurrences had the
|
||||
guard quoting *another* job's error text. Across 11-15 → 11-16 it is the only `failed` row of its 97.
|
||||
Alert delivered (`deliveries.db` `11:49:06.052341`, `for_failure: 1`, `delivered`). **Adjacency, not
|
||||
proof of common cause** — but the guard and the capture path share the same runtime, so a guard that
|
||||
times out at 600 s belongs next to this stall rather than dismissed.
|
||||
|
||||
|
||||
## Re-read 2026-11-17 00:27 AWST — stall **62 h 54 m**, **five** unclosed alerts, zero-item streak **120**
|
||||
|
||||
Third nightly read, sixth week of the same numbers changing only their counters. **The freeze is still
|
||||
total.**
|
||||
|
||||
- **Counts identical for the fifth consecutive read:** `working_memory` **1130** / `episodic_memory`
|
||||
**1249**; newest rows still `2026-11-14 09:32:59` AWST (working) and `09:33:34` AWST (episodic) —
|
||||
**62 h 54 m** with **zero** writes.
|
||||
- **`cron_incidents` 3 → 5 unclosed** (141 → **143** rows total). Both new rows are `ea79a2107e36`,
|
||||
`failure_type: auth`, `state: alerted`, and again `first_seen_at == last_seen_at`:
|
||||
|
||||
| Incident id | `first_seen_at` | `alerted_at` |
|
||||
| --- | --- | --- |
|
||||
| `ea79a2_915d004a108a` | `2026-11-16T12:02:06.679532+08:00` | `2026-11-16T12:02:12.472265+08:00` |
|
||||
| `ea79a2_0fab8d43db75` | `2026-11-17T00:03:37.624640+08:00` | `2026-11-17T00:03:46.151392+08:00` |
|
||||
|
||||
The signature-fragmentation mechanism below is holding exactly as described: **one fresh row per
|
||||
firing, no escalation**, so the estate's unclosed count now grows **~2 per day** with nothing accumulating
|
||||
on any single row.
|
||||
- **The monitor is on a 5-slot failure streak** (`jobs.json`: `ea79a2107e36`, `last_status: error`,
|
||||
`failure_streak: 5`). 11-16's ledger holds **2 failed rows of 175, both this job**
|
||||
(00:01:33 → 00:03:22, 12:01:18 → 12:02:12); 11-17 so far 1 failed of 7 (00:01:39 → 00:03:46).
|
||||
Every failure is `exit 1` with `STALE MEMORY PIPELINE: capture missed: us…` — the same gate.
|
||||
- **The consolidator is alive and deliberately doing nothing.** `consolidation_log` latest row **4742**
|
||||
at `2026-11-16T16:00:41.922076+00:00` (= 00:00:41 AWST), `items_consolidated: 0`, summary *"No-op
|
||||
consolidation cycle — timestamp updated for watchdog"*. Last non-zero row: **4622** at
|
||||
`2026-11-14T09:33:34` → **120 consecutive zero-item cycles** (was 74 at the 11-16 read). The cycles
|
||||
still fire ~every 31 min; there is simply **nothing new to consolidate**, which is the memory-side
|
||||
reason the stall cannot self-heal.
|
||||
- **Still no cause, still no fix, still `status: active`.** The 11-16 read's two conclusions stand: the
|
||||
failure is **total**, not a trickle, and it affects **all** write paths, not just turn capture.
|
||||
`review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting.
|
||||
|
||||
|
||||
## Re-read 2026-11-20 00:5x AWST — the freeze ended by itself after **105 h 52 m**, and the "capture is dead" reading was wrong
|
||||
|
||||
**The capture freeze is over. The alarm is not.** Live reads of `mnemosyne.db` at 2026-11-20 00:0x–00:5x AWST:
|
||||
|
||||
| counter | 2026-11-17 00:27 | 2026-11-20 00:5x |
|
||||
| --- | --- | --- |
|
||||
| `working_memory` | 1130 | **1131** |
|
||||
| `episodic_memory` | 1249 | **1250** |
|
||||
| newest write | `2026-11-14 09:32:59` AWST | **`2026-11-18 19:25:03` AWST** (stored UTC `11:25:03`) |
|
||||
|
||||
The break coincides with nothing: no config edit, no restart and no repair is recorded against the capture path
|
||||
in the window — the only nearby state change is the 11-18 19:18 desktop session itself. The single new
|
||||
`working_memory` row carries `session_id: hermes_20261118_191808_5c9e08` and its content begins
|
||||
`[USER] You are helping me migrate contex…`, which identifies the captured turn (the `desktop` session
|
||||
**`20261118_191808_5c9e08`**, *"Helping migrate context AI assistant"*, 9 messages, started 19:18 AWST).
|
||||
**Stall duration: 105 h 52 m** (2026-11-14 09:33 → 2026-11-18 19:25 AWST).
|
||||
|
||||
⚠ **The correction that matters — the window contained almost no conversation to capture.** `state.db` lists
|
||||
**three** non-cron sessions since 2026-11-14 00:00 AWST: two `desktop` sessions on **11-14** (**08:35**, 40
|
||||
messages; **08:52**, 41 messages — both *before* the 09:33 boundary), then **nothing** until the **11-18 19:18**
|
||||
session, whose turn is the row that ends the freeze. Every interactive session in the window was captured, so
|
||||
the earlier figures on this page (`0 writes in 38 h 46 m`, `62 h 54 m`) are real but measure **a quiet estate**,
|
||||
not a broken hook: `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message
|
||||
whose session source is not `cron`/`subagent`, so **silence and failure read identically**. That ambiguity — not
|
||||
a broken capture path — is the durable defect on this page.
|
||||
|
||||
**The monitor is still failing on that same metric, after the break.** `ea79a2107e36` slots since 11-18:
|
||||
`11-18 00:01:28` failed · `11-18 12:01:13` failed · **`11-19 00:01:34` `completed`** (the only clean slot since
|
||||
11-17) · `11-19 12:01:17` failed · `11-20 00:01:22` failed — every failure `Script exited with code 1 / 🚨
|
||||
Mnemosyne Health Monitor — exception(s): STALE MEMORY PIPELINE: capture …`, i.e. **the same signature firing on
|
||||
a healthy pipeline** (`failure_streak: 2`; incidents `ea79a2_f2be2db3bfce` at 2026-11-19 12:01:58 and
|
||||
`ea79a2_aae4d914f3c9` at 2026-11-20 00:03:36, both `alerted`, both `failure_type: auth` — the scheduler's
|
||||
classifier, not a credential fault). The `11-19 00:01` pass and the `12:01` failure bracket the threshold: the
|
||||
metric is *age of the newest captured conversational turn*, which only grows while the estate is silent.
|
||||
|
||||
**The other half of the original finding is unchanged, and larger.** The consolidator still consolidates zero
|
||||
items: newest `consolidation_log` row **`2026-11-19T16:20:14.587267+00:00`** (= 2026-11-20 00:20 AWST, so it is
|
||||
running on its ~31-minute cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle
|
||||
— timestamp updated for watchdog"`. **199 of the last 200 cycles are zero-item** (was **120** at the 11-17 read
|
||||
— consistent with ~31 min cadence over the three days). The zero-item-streak threshold remains **proposed, not
|
||||
applied**.
|
||||
|
||||
**Unclosed incidents: 5 → 4** (one of the rows present at the 11-17 read has since closed). Current four, none
|
||||
closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, `alerted_at: None` — the wiki audit's
|
||||
silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98ade422d195` (*script*,
|
||||
2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16 — see
|
||||
[[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]), `ea79a2_f2be2db3bfce` (2026-11-19
|
||||
12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36).
|
||||
|
||||
**Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational
|
||||
silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days.
|
||||
|
||||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||||
|
||||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||||
**2026-11-14 09:32:59 AWST** (the row's own `timestamp` field reads `2026-11-14T09:32:59`).
|
||||
- Newest `episodic_memory.created_at`: **`2026-11-14 01:33:34`** (09:33:34 AWST). **Both** write paths
|
||||
stopped inside the same minute.
|
||||
- Last 8 `working_memory` writes, all on 11-14 (UTC): `01:01:20, 01:02:39, 01:02:40, 01:03:28,
|
||||
01:04:19, 01:32:58, 01:32:59, 01:32:59` — then **nothing for ~15 hours**.
|
||||
- Rows by day (UTC `created_at`): **11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15**.
|
||||
So the pipeline was *low-volume but alive* before the stall, and the stall is not a normal quiet stretch.
|
||||
- Totals at audit time: **working 1130 / episodic 1249**, DB **27 MB**.
|
||||
- Gap: at the 00:03:35 alert the newest capture was **14 h 30 m 36 s** old; at this audit's read
|
||||
(~00:40) **≈15 h 7 m**.
|
||||
|
||||
## The alert (and its mislabelling)
|
||||
|
||||
```
|
||||
🚨 Mnemosyne Health Monitor — exception(s):
|
||||
|
||||
STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop)
|
||||
has no working_memory row (last capture 2026-11-14 01:32:59)
|
||||
```
|
||||
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
|
||||
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
|
||||
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
|
||||
- ⚠ **The `auth` label describes the classifier, not the fault — now evidenced, not just suspected.** The
|
||||
failure is a *staleness* check and the page records no auth error text. This job's **twelve earlier
|
||||
`cron_incidents` rows** read `agent` ×3, `timeout` ×1, `unknown` ×8 (`ea79a2_649d8ac46e8a`,
|
||||
`_f17b45759db0`, `_573d17e10fc1`, `_6fb70b032305`, `_54d6b1f0f58a`, `_747b27556fd0`, `_c3ab2d784ba3`,
|
||||
`_a4661554a3c2`, `_d93c3acdc4f2`, `_9197df32320b`, `_111e67a3d83c`, `_f317fcd843ae`), and the estate-wide
|
||||
table is **53 `unknown` / 47 `agent` / 15 `timeout` / 14 `script` / 5 `auth`**. `agent` is also impossible
|
||||
here: the job runs `no_agent` (plain script). Treat the field as opaque routing metadata.
|
||||
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
|
||||
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
|
||||
log line below.
|
||||
|
||||
## What the earlier slots actually saw — correction, re-derived 2026-11-15 01:2x AWST
|
||||
|
||||
**An earlier revision of this page got this backwards.** It claimed the **12:00:52 AWST** run had
|
||||
"already seen the same hole and called it benign" (`capture delay -589.5 min`), giving the estate ~12 h
|
||||
of unalarmed lead time, and blamed a "sign, not magnitude" threshold. **Both claims are withdrawn.** The
|
||||
`-589.5 min` line is not the 12:00 run's — it sits inside the **00:03:35 run's own output block**.
|
||||
|
||||
Verified by line number in `~/.hermes/logs/mnemosyne-health-monitor.log` (**1103 lines**):
|
||||
|
||||
| line | content |
|
||||
| --- | --- |
|
||||
| 884 | `=== 2026-11-14 00:09:12 AWST health monitor (no_agent) ===` |
|
||||
| 900 | `ℹ️ last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min` |
|
||||
| 957 | `=== 2026-11-14 12:01:52 AWST health monitor (no_agent) ===` |
|
||||
| 973 | `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay 29.4 min` |
|
||||
| 1030 | `=== 2026-11-15 00:03:35 AWST health monitor (no_agent) ===` |
|
||||
| 1046 | `ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min` |
|
||||
| 1047 | `❌ STALE MEMORY PIPELINE: …` |
|
||||
|
||||
Two further corrections to the earlier revision:
|
||||
|
||||
- **There was no 18:03 run.** The job's schedule is `0 */12 * * *`; the only 2026-11-14 slots are
|
||||
00:09:12 and 12:01:52 (both exit 0, delivery `suppressed`). The "exited 0 at 12:00 and 18:03"
|
||||
sentence is withdrawn — 18:03 belongs to some other job.
|
||||
- **The detection chain was not broken.** `mnemosyne_freshness_gate.py` fires on **magnitude**
|
||||
(`delay_min < -CAPTURE_LAG_MIN`, `MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min) — there is no
|
||||
sign-only test to blame. The 12:00:52 measurement of **+29.4 min** is the delay of the newest message
|
||||
*that had been captured* (the 09:03 AWST `ios` turn, captured 09:32:59). The gate computes against the
|
||||
newest **uncaptured** message and only reports a problem when one exists; with the conversation idle
|
||||
from 09:03 to 19:22, every user message still had a capture row, so there was **nothing to detect**.
|
||||
The first message to postdate the stall was the **19:22 AWST desktop** turn (`11:22 UTC`, the same
|
||||
message the 00:03 alert names); the next scheduled slot after it was 00:03:35, which escalated
|
||||
(`-589.5 min` = 09:32:59 − 19:22).
|
||||
|
||||
**Corrected statement of the gap:** the pipeline died at **09:32:59 AWST**; it became *detectable* at
|
||||
**19:22 AWST** (first uncaptured message) and was *detected* at **00:03:35** by the next slot — a
|
||||
detection latency of 4 h 41 m after detectability, driven by cadence, not by a broken threshold.
|
||||
**What was genuinely unread sits in the consolidation counters:** the cycles ran every ~31 min and
|
||||
returned **0 items for 29 consecutive runs** (streak opening `2026-11-14T02:04:42+00:00` = **10:04:42
|
||||
AWST**; last productive cycles 4622/4621/4620 at `09:33:34` / `09:33:16` / `09:01:40`). Every run
|
||||
printed `Last consolidation: … (0 items)` — including the "healthy" 12:00:52 one — but the gate tests
|
||||
consolidation **age** only (`> 36 h`, default), never the item count. **An empty-but-current cycle is
|
||||
the loudest signal available and the only one nothing watches.**
|
||||
|
||||
## Impact
|
||||
|
||||
> **Magnitude superseded 2026-11-16 00:18 AWST: 38 h 46 m uncaptured** (was ~15 h), with **two more
|
||||
> unclosed alerts** and four 2026-11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`)
|
||||
> added to the uncaptured set. The ~15 h figures below stand as the 2026-11-15 readings.
|
||||
|
||||
|
||||
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
|
||||
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
|
||||
window. Recall over that period will answer from nothing.
|
||||
- The **23:30 AWST sleep/consolidation cycle ran and found 0 items** to consolidate
|
||||
(`Last consolidation: 2026-11-14T15:30:51 UTC (0 items)`), so consolidation is *not* compensating.
|
||||
- Nothing else in the estate broke: **2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` /
|
||||
1 `running`** in `executions.db`, deliveries **4 / 4 `delivered`**, and no other cron row went red. This is a
|
||||
**memory-path incident only**. The one exception is the monitor's **own** job: it is now the fleet's
|
||||
**single non-clean row** (`last_status: error`, `failure_streak` 1, previously `ok` / 0) and
|
||||
**`cron_incidents` went 0 → 1 unclosed** (138 rows total). **2026-11-15 so far: 8 rows — 7
|
||||
`completed` / 1 `failed`** (this job), 1 delivery `delivered`.
|
||||
|
||||
## Not the known config killer
|
||||
|
||||
The previously recorded capture-killer on this host is a **stringified list/dict** written into
|
||||
`~/.hermes/config.yaml` by `hermes config set`. Checked and **clean**:
|
||||
- `config.yaml` **mtime 2026-11-13 18:03** — the file was not modified during or before the stall began.
|
||||
- A full YAML walk found **no** scalar that begins with `[` or `{`; `memory.mnemosyne.*` parses as
|
||||
proper lists (`ignore_patterns: []`, `skip_contexts`, `sync_roles`).
|
||||
|
||||
## Open / unverified
|
||||
|
||||
- **Root cause: not established.** No capture-path error lines appear in `agent.log` after 09:33 —
|
||||
the writes simply stop. Candidates to test next: a stuck capture worker or watermark, the
|
||||
`pulse`/heartbeat traffic that ran 12:03–19:22, session-side `memory_enabled` /
|
||||
`mnemosyne.profile_isolation` interaction, or the sleep cycle's `consolidation_claimed_at` state.
|
||||
- ✅ **Resolved — the 00:03 escalation DID deliver.** `deliveries.db` holds exactly one 2026-11-15 row:
|
||||
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
|
||||
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
|
||||
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
|
||||
- ✅ **Answered 2026-11-16 00:18 — capture did NOT resume.** Two further monitor slots failed on the
|
||||
same condition (`12:03:14` and `2026-11-16 00:03:15`) and the row counts are unchanged (`working`
|
||||
1130 / `episodic` 1249). Closed as *no*; the underlying cause remains open.
|
||||
- **The estate's only automated alarm is understating this incident.** The alert's ≈4.64 h is a cadence
|
||||
artefact, not a stall duration — anyone reading the alert text alone would conclude the pipeline is
|
||||
~4.6 h behind when it is 38 h 46 m. A stall-duration figure (newest capture vs now) would fix it.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence
|
||||
- [[systems/memory-backend]] — memory backend settings
|
||||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] — the still-open secrets-at-rest page for the same database
|
||||
- [[systems/scheduled-tasks]] — the fleet's execution ledger
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
title: Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||||
type: incident
|
||||
status: unresolved
|
||||
created: 2026-11-17
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
review_after: 2026-11-24
|
||||
confidence: high
|
||||
tags: [hermes, plugin, conduit, timeout, infrastructure, incident, load]
|
||||
sources: [live-errors-log, live-config-yaml, live-gateway-log, live-cron-jobs-json]
|
||||
---
|
||||
|
||||
# Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||||
|
||||
**Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days.** No fix was
|
||||
applied; the storm stopped as abruptly as it started, and the loader budget that produced it is
|
||||
unchanged, so recurrence is possible. Found by the **2026-11-17 nightly audit**, which was filling the
|
||||
11-15 window's gaps; the **2026-11-15 audit recorded none of this**. Every count below is an exact
|
||||
`errors.log` grep, not an estimate.
|
||||
|
||||
## What the log records (2026-11-15)
|
||||
|
||||
- **52** × `Failed to load plugin '<name>': load timed out after 10s (import + register() never returned)`
|
||||
- **519** × `Plugin '<name>' called register_{tool,hook,skill,platform}() after its load timed out; ignored`
|
||||
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping`
|
||||
- **2** × `Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping` (06:05:33, 08:41:41)
|
||||
- **10** × tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
|
||||
|
||||
Across the log's retained window (`errors.log` starts **2026-11-09 13:18**; `.1` / `.2` rotations hold
|
||||
older lines):
|
||||
|
||||
| Day | Plugin load failures | Registrations dropped | `pre_tool_call` timeouts | `pre_llm_call` timeouts | Tool refusals |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 2026-11-12 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 2026-11-13 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 2026-11-14 | 4 | 0 | 1 | 1 | 4 |
|
||||
| **2026-11-15** | **52** | **519** | **11** | **2** | **10** |
|
||||
| 2026-11-16 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 2026-11-17 | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
### By plugin (2026-11-15)
|
||||
|
||||
| Plugin | Failed loads | Registrations dropped |
|
||||
| --- | --- | --- |
|
||||
| `hermes-hq-push` | 26 | 0 |
|
||||
| `conduit_push` | 9 | **474** |
|
||||
| `homeassistant` | 7 | 34 |
|
||||
| `browser-browser-use` | 3 | 3 |
|
||||
| `basic` | 2 | 2 |
|
||||
| `cadu-secrets-vault` | 1 | 4 |
|
||||
| `telegram-platform` | 1 | 1 |
|
||||
| `nous` | 1 | 0 |
|
||||
| `drain` | 1 | 0 |
|
||||
| `browser-browserbase` | 1 | 1 |
|
||||
| **Total** | **52** | **519** |
|
||||
|
||||
## The dropped registrations are the more serious half
|
||||
|
||||
A failed load is a warning line. The `register_*()` lines that follow are **function loss**: the loader
|
||||
gives each plugin a **10 s** budget for `import + register()`, and when that budget expires the plugin's
|
||||
late registrations are **discarded, not honoured**. `conduit_push` alone lost **474** registrations across
|
||||
**eight** load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those
|
||||
processes the Conduit tools, skills and hooks were **not registered at all**, silently. See
|
||||
[[systems/conduit]].
|
||||
|
||||
## Clustering
|
||||
|
||||
These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures
|
||||
spread over hours 01, 04–11 and 13:
|
||||
|
||||
- **01:05–01:32** — `hermes-hq-push` (01:05, 01:17, 01:32) and `homeassistant` (01:17) fail to load,
|
||||
`conduit_push` fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes
|
||||
`pre_tool_call` times out **5×** (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and **9** tool calls
|
||||
are refused in the same session (`local-989b3642…`).
|
||||
- **04:46–04:47** — the day's largest burst: `browser-browser-use`, `conduit_push` ×2, `hermes-hq-push` ×2
|
||||
and `homeassistant` all fail within ~55 s; **118** registrations discarded.
|
||||
- **06:31–06:48, 07:06, 08:16, 08:46, 10:47** — repeated `conduit_push`/`hermes-hq-push` failures, each
|
||||
dropping ~52–56 registrations.
|
||||
- **12:18–12:49** — a **second** `pre_tool_call` cluster (6 timeouts + 1 refusal) with **no load failures**
|
||||
in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader
|
||||
failure alongside it.
|
||||
|
||||
## What this is — and is not
|
||||
|
||||
- **Not a misconfiguration.** `plugins.hook_callback_timeout: 120` is set in `config.yaml` and **no**
|
||||
plugin-load-timeout key exists there — the 10 s loader budget is a **code default**. The failures are a
|
||||
**saturation symptom** on this host, not a wrong setting.
|
||||
- **Same window as the hook cascade.** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
recurred on 11-15 with the 11 `pre_tool_call` timeouts and 10 refusals counted above; the runtime deps
|
||||
guard also blew its **600 s** budget that day (see
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]]). Several of the estate's own timing budgets were
|
||||
exceeded in the same 24 h.
|
||||
- **Adjacency, not proof.** Nothing ties the three to a single cause. What they share is a host under
|
||||
sustained IO/scheduler pressure while `state.db` had grown past **2 GB** with a **40 MB** WAL (`/` at
|
||||
88 %). But load average at the 2026-11-17 audit read **6.19 / 6.64 / 7.48** — *below* the 11-14 reading
|
||||
of 12.08 — with **no** recurrence of this class on 11-16/11-17, so **load average alone does not
|
||||
predict it**.
|
||||
- **Not host restarts.** `gateway.log` has **0** startup/shutdown markers on 11-15, and `uptime` at the
|
||||
audit was 6 d 2 h (host last up **2026-11-10 ~22:04**) — so the ~18 load events are **in-process plugin
|
||||
(re)loads**. Plugin registration output in `agent.log` appears at agent-process start (e.g. the
|
||||
2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing.
|
||||
|
||||
## Open question
|
||||
|
||||
Why did `import + register()` exceed 10 s for whole plugins, and does the loader fail **closed or open**
|
||||
for the caller? Today it fails **open** — the job continues, silently without those tools. A check that
|
||||
asserts **plugin registration** rather than only job exit status would surface this class; the same idea
|
||||
applies to the hook callback budget. **Proposed, not applied** (it changes Hermes runtime behaviour, so
|
||||
it is Anthony's call).
|
||||
|
||||
## See also
|
||||
|
||||
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — the companion hook-callback cascade
|
||||
- [[systems/conduit]] — what the 474 dropped `conduit_push` registrations cost
|
||||
- [[systems/hermes-agent]] — plugin loading, config and timeouts
|
||||
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — same-window runtime pressure
|
||||
@@ -0,0 +1,111 @@
|
||||
---
|
||||
title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-20
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident]
|
||||
sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json]
|
||||
---
|
||||
|
||||
# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST)
|
||||
|
||||
**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since
|
||||
**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last
|
||||
recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit).
|
||||
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**.
|
||||
|
||||
This is the estate's **only** red cron row with a large streak, and it is a *single durable condition*
|
||||
re-reported every 15 minutes, not a series of separate faults.
|
||||
|
||||
## The failure is a fixed, self-describing threshold trip
|
||||
|
||||
```
|
||||
Script exited with code 1
|
||||
stdout:
|
||||
🚨 Many unresolved conflicts: 59 (threshold: 50)
|
||||
```
|
||||
|
||||
`cron_incidents` row **`96eb95_98ade422d195`** — `state: alerted`, `failure_type: script`,
|
||||
`first_seen_at 2026-11-18T20:01:15.906837+08:00`, `last_seen_at 2026-11-20T00:31:16.326357+08:00`,
|
||||
`alerted_at 2026-11-19T20:46:23.428143+08:00`, `closed_at: None`, output file
|
||||
`/home/hermes/.hermes/cron/output/96eb95ce12f2/2026-11-20_00-16-15.md`.
|
||||
|
||||
Unlike the health monitor's rows (whose signatures embed a changing float and so fragment into a new row per
|
||||
slot → [[incidents/2026-11-14-mnemosyne-capture-stall]]), **this one fixes on the constant count `59` and
|
||||
therefore stays a single row.** The cost of that stability is the mirror image: it never closes, and it
|
||||
re-alerts forever.
|
||||
|
||||
## Alerting works — and that is the problem
|
||||
|
||||
`deliveries.db`, window 2026-11-18 → 2026-11-20 00:0x — the watchdog's own failure notices:
|
||||
|
||||
| delivered | execution |
|
||||
|---|---|
|
||||
| 2026-11-18 20:01:16 | `f9865723b4f64d9fbfca7711ab053056` |
|
||||
| 2026-11-19 02:16:16 | `7aa35681fd914a1e8db64b8dd0e09c84` |
|
||||
| 2026-11-19 08:31:17 | `1ccc7e280d234287973edf5e0b35a5b7` |
|
||||
| 2026-11-19 12:01:59 | `396940d9514b419eb024b83b3c10198d` |
|
||||
| 2026-11-19 14:31:35 | `0766953f92dc4fc4b0d0404dca51683f` |
|
||||
| 2026-11-19 20:46:19 | `db010a8294bf4cd78457a265a5fb167c` |
|
||||
| 2026-11-20 00:03:37 | `64c6b5202463487c906f860bb746138b` |
|
||||
|
||||
All `status: delivered`, all `for_failure: 1`. So detection, the failure notice and the push path are **all
|
||||
working**. The output is an alert roughly every 3–6 h for a threshold breach that has not moved in two days —
|
||||
which is exactly how a red signal stops meaning anything: while this row is red, a *new* Mnemosyne fault
|
||||
arriving on the same job is indistinguishable from the standing one.
|
||||
|
||||
## What the 59 conflicts actually are
|
||||
|
||||
Live read of `~/.hermes/mnemosyne/data/mnemosyne.db` (2026-11-20 00:3x AWST): **223 conflicts total, 59
|
||||
unresolved.** Unresolved by creation day:
|
||||
|
||||
| created | unresolved |
|
||||
|---|---|
|
||||
| 2026-10-31 | 11 |
|
||||
| 2026-11-02 | 2 |
|
||||
| 2026-11-03 | 12 |
|
||||
| 2026-11-06 | 1 |
|
||||
| 2026-11-07 | 2 |
|
||||
| 2026-11-08 | 1 |
|
||||
| 2026-11-12 | 13 |
|
||||
| 2026-11-13 | 3 |
|
||||
| **2026-11-18** | **14** |
|
||||
| **total** | **59** |
|
||||
|
||||
Two things stand out:
|
||||
|
||||
1. **Most of the backlog is old.** 42 of the 59 predate 2026-11-14; the oldest batch is from **2026-10-31**,
|
||||
which is the same batch recorded on [[systems/mnemosyne]] as *"all created in one batch at 2026-10-31
|
||||
19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature"*. The documented remediation for
|
||||
this class is to bulk-resolve them as `false_positive` (see the `mnemosyne-operations` skill) — **not
|
||||
applied here**.
|
||||
2. **The threshold has been raised before.** The 2026-10-18 and 2026-11-01 reads on [[systems/mnemosyne]] both
|
||||
record *"1 unresolved conflict (benign, threshold 10)"*. It now reads **50**. So the tripwire was already
|
||||
loosened 5×, and the backlog has since grown past even that. **UNVERIFIED:** when the threshold was changed
|
||||
from 10 to 50, and by whom; basis: two dated page reads vs the live alert text — the config line itself was
|
||||
not opened in this window.
|
||||
|
||||
## The 2026-11-18 batch coincides with the migration session
|
||||
|
||||
The 14 conflicts created on **2026-11-18** are the newest batch, and 2026-11-18 is also the day of the only
|
||||
interactive session in this window — `20261118_191808_5c9e08` (desktop, 9 messages, title *"Helping migrate
|
||||
context AI assistant"*), whose turns were captured into `working_memory` at **19:25:03 AWST** and summarised
|
||||
episodically at **19:55:20 AWST**. Consistent with the standing assessment that most conflicts in this bank are
|
||||
LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** that the 14 rows are extraction
|
||||
noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis —
|
||||
the fact pairs were not inspected.
|
||||
|
||||
## Not applied / open
|
||||
|
||||
- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed.
|
||||
- The watchdog's cadence, signature construction and alert path were left as-is.
|
||||
- Open question: should a *standing* threshold breach alert on state change rather than on every slot?
|
||||
Same family as the zero-item-consolidation streak problem on
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]] — both are conditions that re-report continuously because
|
||||
the gate tests a level, not a change.
|
||||
|
||||
→ [[systems/mnemosyne]], [[systems/scheduled-tasks]]
|
||||
@@ -0,0 +1,89 @@
|
||||
---
|
||||
title: Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-20
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
confidence: high
|
||||
tags: [cron, wiki-audit, incident, model-routing, credentials, credits, router, silent-failure]
|
||||
sources: [live-cron-executions-db, live-cron-incidents, live-jobs-json, live-deliveries-db, live-wiki-log]
|
||||
---
|
||||
|
||||
# Wiki Audit failed two consecutive nights (2026-11-18, 2026-11-19) — 11-17 and 11-18 were never audited
|
||||
|
||||
**Open.** `cbfe434a3b5e` (**Wiki Audit — Nightly**, schedule `0 0 * * *`) failed on **two consecutive slots**,
|
||||
so the audit that should have reviewed **2026-11-17** and the audit that should have reviewed **2026-11-18**
|
||||
never ran. The **2026-11-20 00:00** slot is the first to fire since; at the time of writing its execution row
|
||||
reads `running` (started `2026-11-20T00:00:08`).
|
||||
|
||||
This run therefore audits **three windows** — the standard last 24 h plus the two that were skipped.
|
||||
|
||||
## Evidence (live probes 2026-11-20 00:1x–00:4x AWST)
|
||||
|
||||
| slot | started | finished / last seen | duration | status |
|
||||
|---|---|---|---|---|
|
||||
| 2026-11-18 00:00 | `2026-11-18T00:00:08` | `2026-11-18T00:01:43` (incident `first_seen_at` `00:01:42.342149+08:00`) | ~1 m 35 s | `failed` |
|
||||
| 2026-11-19 00:00 | `2026-11-19T00:00:16` | `2026-11-19T00:03:02` (incident `last_seen_at` `00:03:01.420319+08:00`) | ~2 m 46 s | `failed` |
|
||||
| 2026-11-20 00:00 | `2026-11-20T00:00:08` | — | — | `running` |
|
||||
|
||||
Both failures carry the **same 314-character `RuntimeError`** — a concatenation of every target the router tried:
|
||||
|
||||
```
|
||||
RuntimeError: command-code/deepseek/deepseek-v4-flash: model — [400]: You have insufficient credits to
|
||||
make this request. Please purchase more credits to continue using the service. (HTTP 400);
|
||||
openai-compatible-chat-conn:52258193/deepseek-v4-flash-0731: auth — [401]: invalid auth token (HTTP 401);
|
||||
openai-compati…
|
||||
```
|
||||
|
||||
The tail is cut off at the 314-char cap, mid-way through a further `openai-compatible-…` target, so the
|
||||
**final** target's verdict is not recoverable from this field. The shape matters more than the wording: this is
|
||||
not one bad pin — the **whole fallback chain** was refused, first on **credit exhaustion** (`command-code`),
|
||||
then on an **invalid auth token** (the second target), then at least one more target whose error is truncated away.
|
||||
|
||||
- `cron/jobs.json`: `last_status: error`, `failure_streak: 2` — clean 0 at the 2026-11-17 00:27 run.
|
||||
- `cron_incidents`: row **`cbfe43_fa12cca1ad4b`** — `state: detected`, `failure_type: auth`,
|
||||
`first_seen_at 2026-11-18T00:01:42.342149+08:00`, `last_seen_at 2026-11-19T00:03:01.420319+08:00`,
|
||||
**`alerted_at: None`**, `acked_at: None`, `closed_at: None`, output file
|
||||
`/home/hermes/.hermes/cron/output/cbfe434a3b5e/2026-11-19_00-03-01.md`.
|
||||
- `deliveries.db`, window since 2026-11-18: **24 rows — 23 `delivered` (9 of them `for_failure: 1`) and one
|
||||
`failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter
|
||||
delivery to telegram`, consistent with *Gmail Action Digest* `08eb86f02d37` reading `delivery_failed`).
|
||||
**No row is attributable to either failed audit slot** — both carry `delivery_outcome: suppressed` on the
|
||||
execution row (as does the clean `2026-11-17` slot), i.e. this job's alerts are not wired to leave the box.
|
||||
So the outage produced **no push to Anthony** and the red cron row was the only signal.
|
||||
|
||||
## Why this matters beyond one job
|
||||
|
||||
The wiki audit is this estate's own feedback loop: its output is the `log.md` entry and the page edits that keep
|
||||
`current-state.md`, `systems/mnemosyne.md` and `systems/scheduled-tasks.md` honest. Two silent skips mean the wiki
|
||||
stood still for two nights while the estate did not — including the window in which the **Mnemosyne watchdog went
|
||||
red for 114 consecutive slots** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. The gap is
|
||||
visible in `index.md`: its newest feed entry was still **2026-11-16** on the morning of 11-20.
|
||||
|
||||
## Precedent — and a repeating pattern
|
||||
|
||||
Same class as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] (**HTTP 429**, *"all targets were
|
||||
skipped by pre-dispatch filters"*, 6 targets evaluated / **0 attempted**) and
|
||||
[[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. Two observations worth carrying forward:
|
||||
|
||||
1. **The 11-10 failure self-cleared on the next slot** — so a single red night for this job is not necessarily a
|
||||
standing fault. This one did **not** self-clear: two slots, identical error, ~1.5 and ~2.7 minutes each.
|
||||
2. **`alerted_at: NULL` on both rows.** The 11-10 row was also left unalerted. Two independent failures of this
|
||||
job have now produced incident rows that never alerted, which points at the job's failure-*delivery* path
|
||||
rather than at the alert throttle.
|
||||
|
||||
## Not applied / open
|
||||
|
||||
- **Nothing was changed**: the job's pin, provider and schedule are untouched, no credential was rotated, no
|
||||
cron entry was edited.
|
||||
- No re-run was forced; the 2026-11-20 00:00 slot was left to fire on schedule (it did, and its result is the
|
||||
entry appended to `log.md` the same night).
|
||||
- **UNVERIFIED:** whether the credit exhaustion is on the `command-code` account specifically or the router's
|
||||
shared pool — the concatenated error does not name which target a *successful* dispatch would have used;
|
||||
basis: the truncated 314-char error string.
|
||||
- **UNVERIFIED:** whether this job's failures are configured `suppressed`, or whether its failure path is simply
|
||||
not wired to delivery; basis: absence of delivery rows, not a read of the job's delivery config.
|
||||
|
||||
→ [[systems/scheduled-tasks]], [[systems/omniroute]]
|
||||
@@ -3,8 +3,8 @@ title: Backups
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
confidence: medium
|
||||
tags: [infrastructure, backups]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -54,22 +54,53 @@ How Hermes's own configuration/state gets backed up, and what's missing.
|
||||
local snapshot and no off-box copy** — the Gitea mirror below has been dead since 2026-08-08.
|
||||
→ [[systems/hermes-agent]]
|
||||
|
||||
## ⚠ Scheduled config backup — the Gitea mirror has been dead since 2026-08-08
|
||||
## ✅ Scheduled config backup — RESOLVED 2026-11-07 (was dead since 2026-08-08)
|
||||
|
||||
`e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits
|
||||
locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its
|
||||
execution row reads `completed`.** Last commit that reached Gitea: **2026-08-07 11:00:47**; the local repo is
|
||||
**76 commits ahead of `origin/main`**. Root cause is a credential: `origin` is
|
||||
**89 commits ahead of `origin/main`** (was 76 on 2026-10-22 — **13 more nights of local-only commits** since). Root cause is a credential: `origin` is
|
||||
`https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — HTTPS, no userinfo, no credential
|
||||
helper — so the non-interactive push dies on
|
||||
`fatal: could not read Username … No such device or address` → `✗ Push to Gitea FAILED` →
|
||||
`--- Backup complete: FAIL ---`. Full evidence and the proposed fix:
|
||||
[[incidents/2026-08-08-gitea-config-backup-push-dead]].
|
||||
|
||||
**Consequence for this page: everything above describes a topology that has not carried `~/.hermes` off the box
|
||||
since 2026-08-07.** The 2026-07-22 full zip and the wiki/authoring repos remain the only recent off-box copies.
|
||||
Telegram failure notices *have* been firing daily (last: 2026-10-21 11:00:14, `message_id 8449`); the AgentMail
|
||||
notification step failed.
|
||||
> **RESOLVED 2026-11-07.** The 92-commit backlog is drained and pushed to Gitea, and the script has been
|
||||
> hardened so the cron row can no longer report a false green. Full record:
|
||||
> [[incidents/2026-08-08-gitea-config-backup-push-dead#Resolution 2026-11-07]].
|
||||
|
||||
**What was wrong (four independent causes):** (1) no Gitea credential — `origin` was HTTPS and
|
||||
`~/.git-credentials` held only a `github.com` entry, so the non-interactive push could not authenticate;
|
||||
(2) **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's
|
||||
`success = (returncode == 0)` recorded `ok` on every failed night (the scheduler was correct; the script
|
||||
was the liar); (3) a 1.76 GB `checkpoints/` backlog sat in the unpushed range, 13 blobs over 100 MB;
|
||||
(4) both secondary legs were broken — the `github` remote did not exist, and its fallback pushed a
|
||||
branch's full ancestry containing a 7.45 GB tarball blob that GitHub's 100 MB limit could never accept.
|
||||
|
||||
**What was done:** host-scoped Gitea credential stored via `git credential approve`; `checkpoints/`
|
||||
stripped from the unpushed range with `git filter-repo --partial` (push set ~2 GB → 264.8 MB, pre-rewrite
|
||||
history preserved under tag `backup/pre-rewrite-20261107-032658`); the script rewritten to exit non-zero on
|
||||
failure, stamp `.last_backup_ok` only after remote verification, decide the push by **remote-vs-local**
|
||||
rather than by working-tree changes, guard every `ls-remote` with `|| true` so an unreachable remote still
|
||||
reaches the alert path, push GitHub as a **parentless orphan snapshot**, and report Telegram/AgentMail
|
||||
delivery honestly (`curl --fail` + `"ok":true`; real `agentmail` CLI).
|
||||
|
||||
**Re-verified 2026-11-07 — live and green.** Real run exit **0** in 47 s; `git rev-parse HEAD` and
|
||||
`git ls-remote origin refs/heads/main` both `a50a5170c1c0e2af5235fa63a95a732e2d780703`;
|
||||
`rev-list --count origin/main..main` = **0**; Gitea API confirms `main` at `a50a5170c1c0`, updated
|
||||
`2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub mirror
|
||||
now carries a parentless orphan snapshot (`faed9fb5c677`, 0 parents, 30,086 files, largest blob 23.5 MB) —
|
||||
no data lost, since GitHub's tree was a strict subset of local HEAD. The AgentMail and Telegram alert
|
||||
paths were each exercised directly and report honestly.
|
||||
|
||||
**Consequence for this page: the topology above carries `~/.hermes` off the box again.** The one full
|
||||
`hermes backup` zip on disk is still `/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, 22 Jul) —
|
||||
**now 108 days old**, and no new full zip has been taken. `crontab -l -u hermes` is still empty and there
|
||||
are still **0 user systemd timers**, so nothing schedules the full zip; only `e5ed215fc91c`
|
||||
(**Mnemosyne DB Backup**, weekly `0 3 * * 0`) and `e27da6c34bbb` (config backup, daily) exist in the cron
|
||||
fleet, and neither takes a full `hermes backup`. The nightly config backup is a *config* mirror, not a
|
||||
full-home backup — the gap on the full zip is unchanged.
|
||||
|
||||
## Relevant runbooks
|
||||
- [[runbooks/backup-wiki]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Credentials Map
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: medium
|
||||
tags: [credentials, security, infrastructure]
|
||||
sources: []
|
||||
@@ -45,6 +45,14 @@ Document where credentials are stored and what service they belong to.
|
||||
> No `connectors:` block is configured in `config.yaml` at all.
|
||||
> `~/.hermes/auth.json` also exists and holds OAuth tokens — treat as a credential store.
|
||||
|
||||
> **Re-verified 2026-11-11 — both 2026-10-20 corrections still hold.** `config.yaml` still has **no**
|
||||
> `imap`/`smtp`/`email`/`connectors`/`gmail`/`mattermost` keys; the email credentials still live in
|
||||
> `~/.hermes/.env` (mode `0600`, keys `EMAIL_ADDRESS`, `EMAIL_IMAP_HOST`, `EMAIL_SMTP_HOST`,
|
||||
> `EMAIL_ALLOWED_USERS`, `EMAIL_HOME_ADDRESS`, `EMAIL_ALLOW_ALL_USERS`, `EMAIL_POLL_INTERVAL` and
|
||||
> more). `tools:` still contains only `tool_search`. `discord:` and `telegram:` are top-level
|
||||
> blocks, and `platforms:` still lists `mattermost` (so Mattermost remains a plugin, not a `tools:`
|
||||
> entry). `~/.hermes/auth.json` exists and holds OAuth material.
|
||||
|
||||
## Rules
|
||||
- Credentials never written to this wiki
|
||||
- API vault is first-stop for any key lookup
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Docker Services
|
||||
type: infrastructure
|
||||
status: mixed
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
confidence: high
|
||||
tags: [docker, homelab, infrastructure]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-probe]
|
||||
@@ -16,32 +16,101 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-probe]
|
||||
Containerized services supporting Hermes and the homelab.
|
||||
|
||||
## Status
|
||||
Docker runs on **three hosts**, and CT460's socket is not readable by the agent user. This page previously claimed "Docker runtime not available" — that is only true for CT460's socket from the `hermes` user. Real state, verified 2026-10-11:
|
||||
Docker runs on **four hosts**, and CT460's socket is not readable by the agent user. This page previously claimed "Docker runtime not available" — that is only true for CT460's socket from the `hermes` user. Real state, **re-verified 2026-11-01**:
|
||||
|
||||
| Host | Access | Containers running |
|
||||
|---|---|---|
|
||||
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **~50** — main Docker host |
|
||||
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | 9 (steel-browser, gotify, readlater, kokoro-tts, edge-tts, onecli-postgres, …) |
|
||||
| CT460 `hermes` | `sudo docker` — socket is `root:docker srw-rw----`, `hermes` user not in group `docker`; **no passwordless sudo in cron** | `ignis` (confirmed running; inspect requires root) |
|
||||
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **65** running / 72 total — main Docker host (**re-verified 2026-11-15**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) |
|
||||
| CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** running / 26 total — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**re-verified 2026-11-07** — set-wise diff against live `docker ps`: no additions, no removals) |
|
||||
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total (**re-verified 2026-11-07**) — `nostalgic_keldysh`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → `flamboyant_mahavira` (2026-11-04) → **`nostalgic_keldysh`** (2026-11-06) — all `ghcr.io/github/github-mcp-server`, the live one created 2026-11-05T22:15:57Z. ⚠ **Corrected 2026-11-07:** the enumerated list still opened with `flamboyant_mahavira`, which is **not running** — live `docker ps` on CT333 returns **`nostalgic_keldysh`** (created 2026-11-05 22:15:57, `Up 29 hours`), `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1` (8 running), plus `qdrant` **exited** (9 total) |
|
||||
| CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** running / 9 total — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented |
|
||||
| CT460 `hermes` | `pct exec 460 -- docker ps` **as root** from the PVE host — socket is `root:docker srw-rw----`, the `hermes` user is not in group `docker` (its own `docker ps` → `permission denied while trying to connect to the docker API`) | `ignis` — **running** (`Up 5 days`, `ignis-subpath:0.8.9`, `0.0.0.0:8080->8080/tcp`), re-verified 2026-11-05 |
|
||||
|
||||
## CT200 live inventory (verified 2026-10-11, ~50 containers)
|
||||
`guacamole-guacamole-1`, `telegram-bot-api`, `ytdlp-web-player`, `homarr`, `overtchat-searxng`, `searxng`, `khoj-search-1`, `prowlarr`, `overtchat-app`, `postgres-newsku`, `guacamole-postgres-1`, `jellyfin`, `libre-closet`, `apprise-api`, `tt-rss-web-nginx-1`, `tt-rss-updater-1`, `tt-rss-app-1`, `tt-rss-db-1`, `audiobookshelf`, `qbittorrent`, `open-webui`, `readlater`, `readlater-extraction-worker`, `worldmonitor-redis`, `redis-cache`, `nextstep-db`, `nextstep-scheduler-1`, `nextstep-app`, `portainer`, `flaresolverr`, `sshwifty`, `8mblocal`, `save-restricted-bot`, `kokoro-tts`, `edge-tts`, `filebrowser`, `filebrowser1`, `worldmonitor-redis-rest`, `worldmonitor`, `overtchat-redis`, `chromadb`, **`ignis-ignis-1`**, `nc_app_talk_bot_ai`, `excalidraw-frontend`, `nc_app_context_agent`, `portkey-gateway-web-1`, `guacamole-guacd-1`, `lobe-chat`, `mycompose-saas-1`.
|
||||
## CT200 live inventory (re-verified 2026-11-15, 65 running / 72 total, excluding 1 transient `worldmonitor-seeder-*` container)
|
||||
`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, **`phonon-demo`**, **`ptts-openai-server`**, **`ptts-ws-server`**, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`.
|
||||
|
||||
**Watch this one:** `mycompose-db-1` is in `Restarting (1)` — a crash-looping database. Flagged for follow-up; root cause not investigated on 2026-10-11.
|
||||
**Re-verified 2026-11-15 00:4x AWST — 61 → 64 running (excluding the seeder), 68 → 72 total, and the whole delta is the TTS stack.** Live `docker ps` on CT200 returns **65 running / 72 total**, which is **64 non-seeder running + 1** transient `worldmonitor-seeder-scheduled`. A set-wise diff of the live names against this page's 2026-11-07 list found **three additions and no removals**, and the arithmetic closes exactly: **61 non-seeder → 64 non-seeder** is the **`phonon-demo`**, **`ptts-openai-server`** and **`ptts-ws-server`** trio — all created 2026-11-14, documented on [[systems/phonon-tts]]. Earlier in this pass a generic `total − prior total` subtraction suggested a fourth, unnamed addition; that was a **comparison artefact** (the 11-07 row's 61 was an explicit *non-seeder* count, today's 65 was a raw `running` count including the seeder) and is **withdrawn** — the name-level diff is exact and accounts for every added container. This pass verified **counts, the TTS trio, and the absence of removals**; it is not a full CT200 audit.
|
||||
|
||||
**Re-verified 2026-11-06 — the 2026-11-05 "the seeder does not exist" finding was WRONG, and the cause is a
|
||||
sampling artefact, not a removal.** A set-wise diff against live `docker ps --format '{{.Names}}'` on CT200
|
||||
shows **no container live but absent from the list above**, and **no listed container has disappeared** — the
|
||||
61-container running set above still matches live exactly, set-wise. The `worldmonitor-seeder-*` containers are
|
||||
**transient and systemd-timer driven**, so they are only visible while a seed run is in flight: at the
|
||||
2026-11-06 03:1x probe **both** were present (`worldmonitor-seeder-scheduled`, created 02:45:14, `Up 32 minutes
|
||||
(unhealthy)`, and `worldmonitor-seeder-fast`, created 03:17:15, `Up 22 seconds (health: starting)`); at the
|
||||
2026-11-05 05:1x probe **neither** was, because no run was in flight — hence the zero `seeder` matches. The
|
||||
drivers are **systemd timers on CT200, not cron**: `worldmonitor-seed.timer` (`OnUnitInactiveSec=4h`,
|
||||
`RandomizedDelaySec=10min` → `worldmonitor-seed.service` → `docker run --rm --name
|
||||
worldmonitor-seeder-scheduled … scripts/run-seeders.sh`) and `worldmonitor-seed-fast.timer`
|
||||
(`OnUnitInactiveSec=15min`, `RandomizedDelaySec=1min` → `worldmonitor-seed-fast.service` → `--name
|
||||
worldmonitor-seeder-fast`). Both services were `ActiveState=activating` at the probe. The **corrected** claim
|
||||
is therefore: **the seeders exist and run on a 4h + 15min schedule; a point-in-time `docker ps` can miss them
|
||||
entirely, so "absent" must never be read as "removed".** The **corrected on record** note: the earlier
|
||||
`worldmonitor-seeder-fast` → `worldmonitor-seeder-scheduled` rename narrative was also wrong — there is **one
|
||||
script with two timer-driven invocations** under two container names, not a rename. Live CT200 also shows
|
||||
`worldmonitor` (`Up 6 days`, **healthy**), `worldmonitor-redis` (`Up 6 days`) and `worldmonitor-redis-rest`
|
||||
(`Up 6 days`).
|
||||
|
||||
**Re-verified 2026-11-05 — 61 → 61 running, seeder entry removed.** A set-wise diff against live
|
||||
`docker ps --format '{{.Names}}'` on CT200 shows **no container live but absent from the list above**, and
|
||||
**no listed container has disappeared**. ~~The `worldmonitor-seeder-scheduled` entry recorded on 2026-11-04 as
|
||||
"unhealthy" **does not exist on any host** — verified with `docker ps -a --filter name=seeder` across CT200,
|
||||
CT201, CT333 and CT460, all of which return **zero** matches.~~ **⚠ SUPERSEDED 2026-11-06 — see the
|
||||
correction above: this was a sampling artefact. The seeders are transient systemd-timer-driven containers,
|
||||
not absent.** ~~There is **no seeder container and no worldmonitor-seeder cron job** anywhere in the estate —
|
||||
the 2026-11-04 "watch item" was a transient container that has since been removed.~~ **⚠ SUPERSEDED
|
||||
2026-11-06: the seeders DO exist; the driver is a systemd timer pair, not a cron job.** ~~**Correction on
|
||||
record:** the earlier `worldmonitor-seeder-fast` → `worldmonitor-seeder-scheduled` rename narrative did not
|
||||
reflect a durable redeploy; the seeder is simply absent now.~~ **⚠ SUPERSEDED 2026-11-06: one script, two
|
||||
timer-driven container names — not a rename.**
|
||||
|
||||
**CT333 container set — `nostalgic_leavitt` → `flamboyant_mahavira` (2026-11-05) → `nostalgic_keldysh` (2026-11-06).**
|
||||
Live CT333 `docker ps` returns 8 running containers, and the name that had replaced `thirsty_payne` on 2026-11-04
|
||||
and then `flamboyant_mahavira` on 2026-11-05 has **itself been replaced again**:
|
||||
**`nostalgic_keldysh`** (image `ghcr.io/github/github-mcp-server`, **created 2026-11-05T22:15:57Z**, `Up 5 hours`
|
||||
at the 2026-11-06 probe). The count is unchanged at 8 and the other seven names are identical
|
||||
(`community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`,
|
||||
`onecli-postgres-1`). A ninth container, `qdrant`
|
||||
(`qdrant/qdrant`), exists but is **stopped** (`Exited (143) 4 months ago`) — it is not in the running count and
|
||||
is not listed above. **Docker generates these random names for compose-less containers, so repeated renames are
|
||||
normal — the reliable check is always the live set, never the recorded name.**
|
||||
|
||||
**CT460 `ignis` access path corrected (2026-11-04).** The old row said "no passwordless sudo in cron", which
|
||||
implied the container list could not be read at all. It can: **`pct exec 460 -- docker ps` from the PVE host
|
||||
runs as root and works** (`ignis|ignis-subpath:0.8.9|Up 4 days|0.0.0.0:8080->8080/tcp`). What does *not* work is
|
||||
the `hermes` user's own `docker ps` inside CT460 — `permission denied while trying to connect to the docker API
|
||||
at unix:///var/run/docker.sock` — because the socket is `root:docker srw-rw----` and `hermes` is not in group
|
||||
`docker`. Use the root-via-PVE path.
|
||||
|
||||
**Re-verified 2026-11-01:** the 2026-10-11 inventory listed **49** containers; the live count is now **61**.
|
||||
**Corrected 2026-11-01:** the inventory list had three **misspelled** `overtchat-*` entries (written
|
||||
`overchat-*`) and one **stale** entry, `worldmonitor-seeder-fast`, which is **not running** — live
|
||||
`docker ps -a --filter name=worldmonitor` shows only `worldmonitor`, `worldmonitor-redis` and
|
||||
`worldmonitor-redis-rest` (all `Up 35 hours`). The list above is regenerated verbatim from live
|
||||
`docker ps --format '{{.Names}}'`. **CT201's Tailscale IP was also wrong** — it read
|
||||
`100.116.116.116`; live `pct exec 201 -- tailscale ip -4` returns **`100.96.244.39`** (which is
|
||||
also what `[[infrastructure/hosts]]` already recorded). **CT333's first container was renamed** —
|
||||
`adoring_jemison` → **`thirsty_payne`**.
|
||||
**12 containers were added** since: `emulatorjs`, `excalidraw-backend`, `khoj-database-1`, `khoj-sandbox-1`,
|
||||
`khoj-server-1`, `mycompose-db-1`, `onlyoffice-docserver`, `quickdrop`, `react-resume-client-1`,
|
||||
`secret-santa-app`, `tgvideo-bot`, `watchtower`. **No container listed on the 2026-10-11 page has
|
||||
disappeared** — the inventory is a strict superset.
|
||||
|
||||
**✅ `mycompose-db-1` crash-loop is FIXED.** The 2026-10-11 page flagged it as `Restarting (1)` — a
|
||||
crash-looping database. Live `docker ps -a --filter name=mycompose` now shows **both** `mycompose-db-1`
|
||||
and `mycompose-saas-1` as **`Up 35 hours`** (checked 2026-11-01). No longer a watch item.
|
||||
|
||||
**Note the duplicate Ignis:** `ignis-ignis-1` on CT200 is a separate deployment from CT460's `ignis`. See [[systems/ignis]].
|
||||
|
||||
## CT201 runtipi — Tailscale routes (verified 2026-10-11)
|
||||
## CT201 runtipi — Tailscale routes (verified 2026-10-11; Steel routes removed 2026-10-29)
|
||||
| Host | Path | Target |
|
||||
|---|---|---|
|
||||
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet) | `/` | `http://127.0.0.1:8085` |
|
||||
| | `/mgr` | `http://127.0.0.1:8084` |
|
||||
| `runtipi.kangaroo-eel.ts.net` (tailnet) | `/` | `http://127.0.0.1:8085` |
|
||||
| | `/mgr` | `http://127.0.0.1:8084` |
|
||||
| | `/steel` | `http://127.0.0.1:5173` |
|
||||
| | `/steel-api` | `http://127.0.0.1:3000` |
|
||||
|
||||
**Correction:** this page previously said "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" and listed CT201's `:8443` → `localhost:8082` for Homepage. Live `tailscale serve status` on CT201 shows the target is `127.0.0.1:8085` on both `:8443` and the default host, plus a `/steel` (5173) and `/steel-api` (3000) pair. The `/steel-api` route does not strip the path prefix — use the container IP directly for API calls.
|
||||
**Correction:** this page previously said "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" and listed CT201's `:8443` → `localhost:8082` for Homepage. Live `tailscale serve status` on CT201 shows the target is `127.0.0.1:8085` on both `:8443` and the default host. That table also carried a `/steel` (5173) and `/steel-api` (3000) pair — **both routes were deleted on 2026-10-29** with `tailscale serve --set-path <path> off` when Steel was removed, so CT201's remaining routes are only `/` and `/mgr` on **both** `:443` and `:8443`. (The `/steel-api` route never stripped the path prefix — moot now that it is gone.)
|
||||
|
||||
## Confirmed Compose Files
|
||||
|
||||
@@ -50,7 +119,12 @@ Docker runs on **three hosts**, and CT460's socket is not readable by the agent
|
||||
| Hermes Agent | `/home/hermes/.hermes/hermes-agent/docker-compose.yml` | hermes, hermes-dashboard |
|
||||
| Hermes WebUI | `/home/hermes/hermes-webui/docker-compose.yml` | webui |
|
||||
| ReadLater | `/home/hermes/workspace/readlater/docker-compose.yml` | readlater, edge-tts, kokoro-tts |
|
||||
| Steel Browser | `/root/steel-browser/docker-compose.yml` (CT201) | steel-browser-api, steel-browser-ui |
|
||||
| Ignis | `/home/hermes/docker/ignis/docker-compose.yml` (CT460) | ignis, ignis-data, ignis-obsidian-app |
|
||||
|
||||
**Steel Browser — removed 2026-10-29.** The CT201 compose file and Dockerfile are no longer live: the project
|
||||
dir `/root/steel-browser` was deleted along with its containers, images and network. Both files are archived at
|
||||
`/root/_archived-steel/` on CT201 (with `/root/_archived-steel/steel-cache-archive-20261029.tgz`), and the
|
||||
images remain re-pullable from ghcr.io. → [[comparisons/steel-vs-camoufox]]
|
||||
|
||||
## Service Definitions (from compose files)
|
||||
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Domains and Tunnels
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-11
|
||||
verified_on: 2026-11-11
|
||||
confidence: high
|
||||
tags: [infrastructure, network, cloudflare]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -59,6 +59,12 @@ The catch-all remains `service: http_status:404`. The ct207 `config.yml` still c
|
||||
## Full ct207 ingress list (context, not all Hermes-related)
|
||||
The tunnel also fronts: `pve.martinwa.org` (Proxmox UI), `docker.martinwa.org` (Portainer), `runtipi.martinwa.org`, `homar.martinwa.org`, `nextcloud.martinwa.org`, `readlater.martinwa.org`, `velvet.martinwa.org`, `overtchat.martinwa.org`, `freellmapi.martinwa.org`, `omnideb.martinwa.org`, `newstarter.martinwa.org`, and SSH endpoints for debianvm/pve/webserver. These belong to the wider homelab, not Hermes specifically — recorded here only so an ingress diff is possible later.
|
||||
|
||||
> **Full live re-read of ct207 `/etc/cloudflared/config.yml` on 2026-11-11 — the list is unchanged
|
||||
> from the 2026-10-20 diff.** All ten additions/corrections recorded above are present and no new
|
||||
> hostname has appeared; the catch-all is still `service: http_status:404`. ct207's LAN address is
|
||||
> `192.168.178.132`, consistent with the `ssh-webserver.martinwa.org` → `100.82.40.23` entry.
|
||||
> `mcp-portal.martinwa.org` remains **absent** from the ingress list — still unknown, still flagged.
|
||||
|
||||
## History
|
||||
- ct207's `/etc/cloudflared/` directory has ~25 dated `config.yml.bak*` files going back to 2026-06-18, including several during ct460's own setup (`config.yml.before-hermes-ct460-20260717203530`, `...before-hermes-desktop-20260718`, `...before-hermesagent-20260718085557`) — evidence the ingress list has been actively iterated on throughout Hermes's bring-up.
|
||||
|
||||
|
||||
+27
-15
@@ -3,8 +3,8 @@ title: Known Hosts and Paths
|
||||
type: infrastructure
|
||||
status: partial
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-07-22
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
confidence: medium
|
||||
tags: [infrastructure, hosts, containers, paths]
|
||||
sources: []
|
||||
@@ -16,12 +16,15 @@ sources: []
|
||||
|
||||
| Host | Role | Address | Verified |
|
||||
|---|---|---|---|
|
||||
| localhost (this host) | Primary Hermes runtime, Xvfb, file storage | `:102` display | Yes |
|
||||
| localhost (this host) | Primary Hermes runtime, Xvfb, file storage | `:99` display (`Xvfb :99 -screen 0 1600x1200x24`) | Yes (live probe 2026-10-26) |
|
||||
| litellm | LiteLLM proxy | `http://litellm:4000/v1` | Yes (from config) |
|
||||
| omni | OmniRoute proxy | `http://omni:20128/v1` | Yes (from config, 2026-08-13) |
|
||||
| CT201 | Steel Browser, Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) |
|
||||
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net` | Yes |
|
||||
| CT450 (hermesagent) | Hermes Agent **v0.21.5+6170.ga510d6c** (updated 2026-10-03 from v0.19.0), hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes |
|
||||
| `custom:Omniroute:20128` | OmniRoute proxy (CT600 node `omniroute`) | `http://omniroute:20128/v1` → `100.88.81.19:20128` | Yes (live probe 2026-10-26: 6,480 models) |
|
||||
| `custom:omni-101` | OmniRoute proxy (CT101 node `omniroute-1`) — **the live default provider** | `http://100.78.63.20:20128` | Yes (live probe 2026-10-26) |
|
||||
| CT201 | Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.77` (LAN) | Yes (live probe 2026-10-30 — LAN address corrected from `192.168.178.166`) |
|
||||
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net`, `100.118.5.51` (Tailscale) | Yes |
|
||||
| PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-11-07 — 17 LXCs + 4 VMs) |
|
||||
| PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-11-07 — 20 LXCs + 2 VMs) |
|
||||
| CT450 (hermesagent) | Hermes Agent **v0.22.5** (2026.9.24), local `bf0203a794a0` — **same build as CT460**; hermes-webui, hermes-ui (Vite). ⚠ a stale **root-level** `/usr/local/bin/hermes` on CT450 still reports **v0.19.0** — the live agent is the `hermes`-user install under `~/.hermes/hermes-agent` | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes (live probe 2026-10-30 — version corrected from `v0.22.3`, build from `5b147ab036d6` to `bf0203a794a0`) |
|
||||
|
||||
## Project Paths
|
||||
|
||||
@@ -30,15 +33,16 @@ sources: []
|
||||
| `/home/hermes` | Hermes home | `~/.hermes/` contains config, skills, cache |
|
||||
| `/home/hermes/workspace` | Workspace | default terminal cwd |
|
||||
| `/home/hermes/.local/bin/hermes` | Hermes binary | |
|
||||
| `/home/hermes/.hermes/config.yaml` | Config | version 33 |
|
||||
| `/home/hermes/.hermes/skills/` | Skills dir | 66 skills installed |
|
||||
| `/home/hermes/.hermes/cron/` | Cron jobs | 25 active jobs |
|
||||
| `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` |
|
||||
| `/home/hermes/.hermes/skills/` | Skills dir | **302 skills across 68 categories** (`.archive/` excluded — 46 archived; re-verified 2026-11-07) |
|
||||
| `/home/hermes/.hermes/cron/` | Cron jobs | **29 jobs — 26 enabled / 3 disabled** (re-verified 2026-10-31 00:3x). ⚠ Superseded: the 2026-10-30 03:3x reading was 25 / 4 — the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure, but **that pause lasted only until 10:02:48**, when its long-running slot closed `completed`, incident `ea79a2_d93c3acdc4f2` auto-closed `resolved`, and the job returned to `enabled: true` / `last_status: ok` / streak 0. **CT460 itself was then dark for 5 h 30 m on 2026-10-30** (PVE `pve` rebooted; host boot 16:24:00, CT460 16:28:00, scheduler resumed 16:37:21 — ≈39 cron slots lost, 0 failures) → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] |
|
||||
| `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ |
|
||||
| `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 |
|
||||
| `~/.camofox/` | Camofox profiles | managed persistence |
|
||||
| `~/.camoufox/` | Camoufox profiles | multiple profiles present |
|
||||
| `/home/hermes/.hermes/browser_screenshots/` | Browser captures | |
|
||||
| `/home/hermes/wiki/` | Wiki | This knowledge base |
|
||||
| `/home/hermes/workspace/agent-estate-wiki/` | Wiki — **authoring tree** (source of truth, pushed to Gitea) |
|
||||
| `/home/hermes/wiki/agent-estate-wiki/` | Wiki — **Obsidian/Quartz vault mirror** (local-only git; synced per audit, never pushed) |
|
||||
| `/home/hermes/obsidian-vault/` | Obsidian vault | personal notes (50 - Research etc.) |
|
||||
| `/home/hermes/hermes-webui/` | Hermes WebUI | web UI source |
|
||||
| `/home/hermes/hermes-ui/` | Hermes Desktop browser (Vite dev) | present on CT460 and CT450 port 5174 |
|
||||
@@ -55,12 +59,20 @@ sources: []
|
||||
- `/home/hermes/workspace/readlater/docker-compose.yml`
|
||||
|
||||
## Homelab References
|
||||
- **PVE host:** `pve.kangaroo-eel.ts.net` (100.96.100.82) per session notes — Unverified on this host
|
||||
- **CT600:** OmniRoute server (retired → CT601 `omni`, 2026-08-13) — Verified via `pct list`
|
||||
- **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-11-07 (**17 LXCs** + 4 VMs; boot 2026-10-30 16:24:00)
|
||||
- **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-11-07 (**20 LXCs** + 2 VMs; boot 2026-11-05 21:19:52). ⚠ **Corrected 2026-11-07:** the previous figure of 19 LXCs omitted **CT107 (`mem0`)**, which is running — see [[infrastructure/proxmox]]. Guest IDs collide between the two hosts
|
||||
- **CT600 (`omniroute`):** OmniRoute server, `100.88.81.19` — **running and live**; `omniroute` is the hostname that resolves to it
|
||||
- **CT601 (`omni`):** the old CT601 OmniRoute guest **no longer exists** on either hypervisor (`pct config 601` → no such config on both `pve` and `m93p`). The orphaned `omni` Tailscale peer (`100.93.204.4`) has since **expired out of the tailnet entirely** — as of the 2026-10-30 live probe the tailnet contains **no** peer named `omni` and no peer at `100.93.204.4`, so the old "offline 69d" description no longer applies. The only OmniRoute nodes left are CT600 (`omniroute`) and CT101 (`omniroute-1`).
|
||||
- **CT101 (`omniroute`, m93p):** second OmniRoute node, `100.78.63.20` — running
|
||||
|
||||
## Unverified
|
||||
- CT600, CT333, and other remote Proxmox containers/VMs referenced in session notes but not confirmed from this host
|
||||
- PVE host identity not confirmed from this host without explicit confirmation
|
||||
- Individual LXCs are verified only when probed directly; the two `pct list` snapshots above are the current authority for what is running
|
||||
- Guests not present in those `pct list` outputs are **not** running on either hypervisor — do not assume existence from older session notes
|
||||
|
||||
> **Note (2026-10-29):** CT201 no longer hosts the Steel Browser — the containers, images, network, project dir
|
||||
> and Tailscale Serve routes were all removed when Steel was deleted from the estate. CT333's separate,
|
||||
> orphaned Steel copy was removed at the same time. → [[comparisons/steel-vs-camoufox]]
|
||||
|
||||
## Related
|
||||
- [[docker-services]]
|
||||
- [[infrastructure/tailscale]]
|
||||
|
||||
@@ -3,11 +3,11 @@ title: Network Map
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-14
|
||||
verified_on: 2026-10-14
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
confidence: high
|
||||
tags: [infrastructure, network]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-probe:ct460-ss+tailscale-serve-status-2026-11-05]
|
||||
---
|
||||
|
||||
# Network Map
|
||||
@@ -22,11 +22,13 @@ Where Hermes and its dependencies run, and how traffic reaches them.
|
||||
- **This container (hermes, ct460):** `192.168.178.129` (DHCP, eth0)
|
||||
|
||||
### Tailscale (`kangaroo-eel.ts.net`)
|
||||
_Peer IPs below re-checked 2026-11-07 against `tailscale status` on ct460; `webserver`, `clawtest`, `hermesagent`, `docker`, `omniroute` all matched the values listed. Full 52-row node table with live states lives in [[infrastructure/tailscale]]._
|
||||
- **pve:** `100.96.100.82`
|
||||
- **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.0), discovered 2026-10-14.** See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host.
|
||||
- **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.10), discovered 2026-10-14.** ⚠ Corrected 2026-11-07: this line still read *"9.2.0"* although both hosts run the identical `pve-manager 9.2.10` build (live `dpkg -l pve-manager` on both). See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host.
|
||||
- **hermes (ct460):** `100.118.5.51`
|
||||
- **hermesagent (ct450, the original/legacy Hermes agent dashboard):** `100.98.205.112`
|
||||
- **omniroute (ct600):** resolves via `omniroute.kangaroo-eel.ts.net` → `100.88.81.19`
|
||||
- **omniroute-1 (third OmniRoute node):** `100.78.63.20` (`omniroute-1.kangaroo-eel.ts.net`) — added 2026-11-05.
|
||||
- **nanobot (ct333, "clawtest"):** `100.122.189.81`
|
||||
- **webserver (ct207, runs the Cloudflare Tunnel):** `100.82.40.23`
|
||||
- **debianvm (VM 101):** `100.98.95.65`
|
||||
@@ -42,19 +44,51 @@ Where Hermes and its dependencies run, and how traffic reaches them.
|
||||
| Port | Bind | Process | Purpose |
|
||||
|---|---|---|---|
|
||||
| 22 | `*` | sshd | SSH |
|
||||
| 80 / [::]:80 | `*` | nginx | reverse proxy, default vhost |
|
||||
| 19119 | `*` | nginx → 127.0.0.1:9119 | public Hermes webui path (tunneled) |
|
||||
| 9119 | `0.0.0.0` | hermes (webui) | Hermes Web UI backend |
|
||||
| 8644 / 8642 | `*` | hermes | additional Hermes agent ports |
|
||||
| 5174 | `*` | node | Hermes desktop companion (tunneled as hermes-desktop.martinwa.org) |
|
||||
| 9377 | `*` | node | Stratos — AI Command Centre |
|
||||
| 8787 | `127.0.0.1`* | python | Hermes gateway health endpoint (see [[runbooks/update-hermes-safely]]) |
|
||||
| 9099 / 8790 | `*` | python3 | auxiliary Hermes processes (unverified purpose) |
|
||||
| 5901 / [::]:5901 | `*` | x11vnc | VNC on Xvfb `:99`, see [[systems/headless-server]] |
|
||||
| 6080 / 6081 | `127.0.0.1` | websockify | noVNC web bridges for two VNC displays |
|
||||
| 80 | `*` | nginx | reverse proxy, default vhost |
|
||||
| 443 / 8443 / 9443 | `100.118.5.51` (tailscale) | tailscaled | Tailscale Serve/Funnel listeners — route table below |
|
||||
| 19119 | `*` | nginx → 127.0.0.1:9119 | public Hermes webui path (Cloudflare-tunneled) |
|
||||
| 9119 | `0.0.0.0` | `hermes-webui/server.py` | Hermes Web UI backend (Serve `/`) |
|
||||
| 8644 / 8642 | `*` | hermes (gateway) | additional Hermes agent ports |
|
||||
| 9900 | `0.0.0.0` | hermes (gateway) | gateway aux port (up 2026-11-05) |
|
||||
| 5174 | — | *(no listener)* | desktop companion port listed up to 2026-10-14; **not listening 2026-11-05** |
|
||||
| 5173 | `0.0.0.0` | node (vite) | content-spark-hub dev server (Serve `:9443` `/`) |
|
||||
| 5175 | `127.0.0.1` | `python3 -m http.server` (`hermes-ui/app/dist`) | hermes-ui static build (Serve `/ui`) |
|
||||
| 9377 | `*` | node — `/home/hermes/services/camofox-browser` | **Camofox browser REST** (`{"ok":true,"engine":"camoufox"}`). Corrected 2026-11-05 — this port had been recorded as "Stratos — AI Command Centre". |
|
||||
| 9222 | `0.0.0.0` | node `camoufox-ws-server.mjs` | Camofox Playwright/CDP WebSocket endpoint |
|
||||
| 7777 | `0.0.0.0` | node `@openchamber/web serve` | OpenChamber web |
|
||||
| 6767 | `0.0.0.0` | Paseo Daemon | Paseo daemon HTTP (loopback helper also on 42013) |
|
||||
| 8080 | `*` | Ignis | Ignis browser-native Obsidian, see [[systems/ignis]] (Serve `/ignis`) |
|
||||
| 8085 | `0.0.0.0` | `agentmail-webhook.py` | AgentMail webhook receiver; funnel-exposed (`/webhook`) |
|
||||
| 8787 | `*` | `hermes-webui/server.py` | hermes-webui backend (Serve `/webui`); used as the health-probe target in [[runbooks/update-hermes-safely]] |
|
||||
| 9099 | `0.0.0.0` | `job_dashboard_server.py` | jobs dashboard + static routes (Serve `/jobs`) |
|
||||
| 9120 | `0.0.0.0` | `serve.py` (repo `public/`) | **Agent Estate Wiki site** (Serve `/wiki`) — added 2026-11-05 |
|
||||
| 5901 | `*` | x11vnc | VNC on Xvfb `:99`, see [[systems/headless-server]] |
|
||||
| 6080 / 6081 | `127.0.0.1` | websockify → 5900 / 5901 | noVNC bridges for the Camofox and desktop displays |
|
||||
| 25 | `127.0.0.1` / `[::1]` | postfix | local mail transport |
|
||||
|
||||
\* `curl` target used in [[runbooks/update-hermes-safely]] health check; binding not independently re-verified in this pass.
|
||||
Loopback-only helpers (not routable): `49347` hermes gateway, `42013` Paseo, `42647` + `38187` opencode.
|
||||
|
||||
## Tailscale Serve / Funnel (verified 2026-11-07)
|
||||
|
||||
`tailscale serve status` on ct460 reports 14 proxies: 12 under `https://hermes.kangaroo-eel.ts.net` (tailnet only), one tailnet route on `:9443`, and one Funnel route on `:8443` (public).
|
||||
|
||||
| Route | Target | Note |
|
||||
|---|---|---|
|
||||
| `/` | `127.0.0.1:9119` | Hermes Web UI |
|
||||
| `/ui` | `127.0.0.1:5175` | hermes-ui static build |
|
||||
| `/jobs` | `127.0.0.1:9099` | jobs dashboard |
|
||||
| `/wiki` | `100.118.5.51:9120` | Agent Estate Wiki |
|
||||
| `/webui` | `127.0.0.1:8787` | hermes-webui |
|
||||
| `/ignis` | `127.0.0.1:8080` | Ignis |
|
||||
| `/gitea` | `100.82.100.128:3000` | Gitea — **CT221 on `m93p`**, not ct207 (⚠ **corrected 2026-11-07**: this line wrongly said *"ct207 tailnet IP"*; live `pct exec 221 -- tailscale ip -4` → `100.82.100.128`, and ct207's own IP is `100.82.40.23`). The target answers **HTTP 200**. |
|
||||
| `/desktop` | `127.0.0.1:6081` | desktop VNC (noVNC) |
|
||||
| `/vnc-camofox` | `127.0.0.1:6080` | Camofox VNC |
|
||||
| `/evening-grid.html` | `127.0.0.1:9099/evening-grid.html` | jobs dashboard page |
|
||||
| `/agent-estate` | `127.0.0.1:8769` | **DEAD — nothing listening on 8769 (re-confirmed 2026-11-07)** |
|
||||
| `/synergy-prep` | `127.0.0.1:9123` | **DEAD — nothing listening on 9123 (re-confirmed 2026-11-07)** |
|
||||
|
||||
- `:9443` (tailnet only) — `/` → `127.0.0.1:5173`.
|
||||
- Funnel `:8443` (public) — `/webhook` → `127.0.0.1:8085/webhook` (AgentMail webhook receiver).
|
||||
|
||||
## Relevant runbooks
|
||||
- [[runbooks/diagnose-docker]]
|
||||
|
||||
+102
-37
@@ -3,8 +3,8 @@ title: Proxmox VE
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-26
|
||||
updated: 2026-10-14
|
||||
verified_on: 2026-10-14
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
confidence: high
|
||||
tags: [proxmox, virtualization, infrastructure, homelab]
|
||||
---
|
||||
@@ -17,28 +17,31 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
|
||||
## Details
|
||||
- **Host:** `pve.kangaroo-eel.ts.net` (Tailscale: `100.96.100.82`)
|
||||
- **Node name:** `pve`
|
||||
- **Version:** Proxmox VE 8.x
|
||||
- **Version:** Proxmox VE **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, running kernel `7.0.14-5-pve`) — **corrected 2026-10-31**; this page previously said *"Proxmox VE 8.x"*, which was wrong. `m93p` runs the **same** `pve-manager 9.2.10` build, so the two hosts are no longer version-skewed (see the `m93p` section).
|
||||
- **Primary user:** `Anthony@pam` (full admin)
|
||||
- **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve`
|
||||
|
||||
## Live LXC Inventory (verified 2026-10-14 via `pct list`)
|
||||
## Live LXC Inventory (verified 2026-11-07 via `pct list`)
|
||||
Re-verified 2026-11-07 03:2x AWST (stocktake): **live `pct list` on `pve` still returns exactly the 17 LXCs in the table below** (CT255 `mattermost` `running`, CT209 `cloudreve` and CT555 `elementsynapse` `stopped`, CT600 `omniroute` `running`, and **no CT601 on either hypervisor**), and **`qm list` on `pve` still returns exactly the 4 VMs** (100 `homeassistant`, 101 `guest-vps`, 110 `omarchy`, 282 `ubuntu-server-xfce` — all running; **no 310, no 403, no 401**). Both `pve-manager` packages are still **9.2.10** on **both** hosts. The **one correction** was on the `m93p` side: see the `m93p` section below — **CT107 (`mem0`) was missing**, so the m93p guest count was one short.
|
||||
|
||||
Re-verified 2026-11-01 03:5x AWST (stocktake): **`pct list` on `pve` and `qm list` on `pve` both still match the tables below exactly** — 17 LXCs and 4 VMs on `pve`, with CT255 `mattermost` still `running`, CT209 `cloudreve` and CT555 `elementsynapse` still `stopped`, and **no CT601 on either hypervisor** (`/etc/pve/lxc/601.conf` absent). **`m93p` re-confirmed**: `pve-manager/9.2.10` (same build as `pve`), `qm list` returns **VM 103 `cachyos-m93p` stopped** and **VM 106 `openmausbot-vm` running**, and `pct exec 1001 -- docker ps` returns `sshwifty` (CT1001). Storage percentages were refreshed against live `pvesm status` on both hosts (see the Storage section). **Note:** `m93p` SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout).
|
||||
|
||||
> **Two Proxmox hosts exist — always qualify a guest with its host.** `pvecm nodes` is empty on
|
||||
> both, so they are independent and unclustered. Guest IDs also collide — **CT221 (gitea) and
|
||||
> CT245 (openweb) carry the same VMID on both hosts** — so an unqualified "CT221" is ambiguous.
|
||||
> `m93p` (`100.104.51.98`, PVE 9.2.0) was **discovered 2026-10-14** and is documented in its own
|
||||
> `m93p` (`100.104.51.98`, PVE 9.2.10) was **discovered 2026-10-14** and is documented in its own
|
||||
> section below. It hosts several services the wiki had placed on `pve`.
|
||||
|
||||
| VMID | Name | Status | Purpose |
|
||||
|---|---|---|---|
|
||||
| 103 | cliproxyapi | running | Cliproxy API |
|
||||
| 200 | docker | running | Main Docker host — 50+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection) |
|
||||
| 200 | docker | running | Main Docker host — 60+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection) |
|
||||
| 201 | runtipi | running | Runtipi — Homepage, ntfy, etc. |
|
||||
| 205 | ollama | running | Ollama LLM inference |
|
||||
| 207 | webserver | running | Web server |
|
||||
| 209 | cloudreve | **stopped** | Cloudreve (deprecated) |
|
||||
| 215 | n8n | running | N8n automation |
|
||||
| 255 | mattermost | **stopped** | Mattermost — unreachable since ~2026-09-20 (Tailscale peer offline, **29d** as of 2026-10-19) |
|
||||
| 255 | mattermost | **running** | Mattermost — **back online 2026-10-30/31** after a ~40-day outage; `systemctl is-active mattermost` → `active`, listening on `:8065`, gateway reconnected 2026-10-30 16:36:17 |
|
||||
| 270 | nextcloud | running | Nextcloud (apache2 active) |
|
||||
| 299 | sftpgo | running | SFTPGo file server |
|
||||
| 333 | clawtest | running | ClawTest — Nanobot, Zeroclaw |
|
||||
@@ -46,9 +49,16 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
|
||||
| 460 | hermes | running | Hermes CT460 (production) |
|
||||
| 465 | qwenpaw | running | QwenPaw + Paseo daemon (active) |
|
||||
| 475 | agentzero | running | AgentZero (`100.106.54.94`) — **not previously recorded** |
|
||||
| 555 | elementsynapse | **stopped** | Element Synapse (deprecated) — also retired from Tailscale (offline 81d) |
|
||||
| 555 | elementsynapse | **stopped** | Element Synapse (deprecated) — also retired from Tailscale (offline 102d) |
|
||||
| 600 | omniroute | **running** | OmniRoute — live server on `:20128` (see [[systems/omniroute]]) |
|
||||
| 601 | omni | **stopped** | Retired successor to CT600 — stopped, `onboot=0`, offline 54d |
|
||||
|
||||
**CT601 (`omni`) no longer exists.** The 2026-10-14 inventory listed it as `**stopped**`; live
|
||||
`ls /etc/pve/lxc/601.conf` on **both** hypervisors returns **no such file**, and the orphaned `omni`
|
||||
Tailscale peer (`100.93.204.4`) has **expired out of the tailnet entirely** — the live
|
||||
`tailscale status --json` contains no peer named `omni` and no peer at that IP. The guest was
|
||||
**deleted**, not merely halted. The only OmniRoute nodes left are **CT600** (`omniroute`,
|
||||
`100.88.81.19`) and **CT101** (`omniroute-1`, `100.78.63.20`, on `m93p`) — both verified running with
|
||||
`omniroute` listening on `0.0.0.0:20128`. → [[systems/omniroute]], [[infrastructure/hosts]]
|
||||
|
||||
### ⚠ Four entries moved off this host (corrected 2026-10-14)
|
||||
|
||||
@@ -66,19 +76,26 @@ error forward.
|
||||
| ~~CT245 openweb~~ | `m93p` | **CT245** — same ID, different host |
|
||||
| ~~CT260 freshrss~~ | `m93p` | **CT1260** |
|
||||
|
||||
## Live QEMU Inventory (verified 2026-10-14 via `qm list`)
|
||||
## Live QEMU Inventory (verified 2026-10-31 via `qm list`)
|
||||
|
||||
| VMID | Name | Status | Notes |
|
||||
|---|---|---|---|
|
||||
| 100 | homeassistant | running | Home Assistant (13GB RAM) |
|
||||
| 110 | omarchy | running | Omarchy (13GB RAM, 106GB disk) — **new, not previously recorded** |
|
||||
| 282 | ubuntu-server-xfce | **running** | Ubuntu desktop — recorded **stopped** until 2026-10-14; live `qm list` shows running |
|
||||
| 310 | cachyos | **stopped** | CachyOS |
|
||||
| 100 | homeassistant | running | Home Assistant (**5100 MB** RAM — the page previously said 13 GB) |
|
||||
| 101 | guest-vps | running | Guest VPS (6144 MB RAM, 48 GB disk) — **was missing from this table entirely** |
|
||||
| 110 | omarchy | running | Omarchy (13000 MB RAM, 106 GB disk) |
|
||||
| 282 | ubuntu-server-xfce | **running** | Ubuntu desktop (5000 MB RAM, 133 GB disk) — recorded **stopped** until 2026-10-14; live `qm list` shows running |
|
||||
| 310 | cachyos | **absent** | CachyOS — **no longer present in `qm list`** on `pve`; the only CachyOS guest is `cachyos-m93p` (VM 103) on `m93p`, which is stopped |
|
||||
| 403 | open-claw | — | OpenClaw agent; **not present in `qm list`** — it runs elsewhere (Open-Claw VM, see [[infrastructure/hosts]]) |
|
||||
| 401 | hermes-old | — | Legacy Hermes VM; **not present in `qm list`** — appears decommissioned/removed |
|
||||
|
||||
**Not in either list:** VM 403 (`open-claw`) and VM 401 (`hermes-old`) are referenced by the wiki but no longer appear in `qm list`. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry.
|
||||
|
||||
**Re-verified 2026-10-31:** live `qm list` on `pve` returns exactly **4 VMs — 100 homeassistant, 101
|
||||
guest-vps, 110 omarchy, 282 ubuntu-server-xfce, all running**. Two corrections to the 2026-10-14 table:
|
||||
**VM 101 (`guest-vps`) was missing** despite running, and **VM 310 (`cachyos`) no longer exists** on this
|
||||
host (it had been recorded as `stopped`; the live `qm list` has no 310 at all). Home Assistant's memory was
|
||||
also corrected from *"13GB"* to the live **5100 MB**.
|
||||
|
||||
## Host `m93p` (100.104.51.98) — second Proxmox hypervisor, discovered 2026-10-14
|
||||
|
||||
Identified because a tailnet-wide scan for `:8006` returned only `m93p` and `pve`. It had been
|
||||
@@ -86,17 +103,20 @@ sitting in the Tailscale node list as an unremarkable `online` peer, but no wiki
|
||||
as a hypervisor.
|
||||
|
||||
- **Hostname:** `m93p` · **Tailscale:** `100.104.51.98`
|
||||
- **PVE version:** **9.2.0** (kernel `7.0.14-12-pve`, `pve-manager 9.2.10`) — materially newer than `pve`'s 8.x
|
||||
- **PVE version:** **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, kernel `7.0.14-12-pve`) — **corrected 2026-10-31**. The previous entry read *"9.2.0"* and claimed `m93p` was *"materially newer than pve's 8.x"*. Both hosts now run the **identical `pve-manager 9.2.10` build**; only the kernel differs (`7.0.14-12-pve` vs `7.0.14-5-pve`). The hosts are **version-identical, not version-skewed**.
|
||||
- **Uptime:** **up 1 day, 7:18** at 2026-11-07 04:38 AWST (live `uptime -s` → **boot 2026-11-05 21:19:52**; `uptime` → `up 1 day, 7:18`, load average **0.76 / 1.08 / 1.24** — normal). ⚠ **Corrected 2026-11-07:** the previous entry read *"up 1 day, 12:02 at 2026-11-01 03:56 AWST — the host rebooted ~2026-10-30 16:0x"*. The **reboot itself is still real** (`pve` booted **2026-10-30 16:24:00** per `uptime -s`, and the CT1264 openmausbot outage below follows from it), but the **uptime figure and the implied boot date were wrong** — the host has since rebooted again on **2026-11-05 21:19:52**, so it was up only **1 day, 7:18** when probed, not 12 h. The `pve` host's own boot time is **2026-10-30 16:24:00** (up **7 days, 12:14** at probe time)
|
||||
- **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf`
|
||||
|
||||
### LXC Inventory (verified 2026-10-14 via `pct list` on m93p)
|
||||
### LXC Inventory (verified 2026-10-31 via `pct list` on m93p)
|
||||
|
||||
| VMID | Name | Status | Notes |
|
||||
|---|---|---|---|
|
||||
| 100 | obsidian-livesync | running | Obsidian LiveSync — ID does not collide; `pve` has no CT100 |
|
||||
| 101 | omniroute | running | **An OmniRoute instance distinct from `pve`'s CT600/CT601 — see [[systems/omniroute]]; identity/role unconfirmed** |
|
||||
| 101 | omniroute | running | **Second OmniRoute node**, `100.78.63.20` (`omniroute-1` Tailscale peer), `omniroute` listening on `:20128` — identity/role **now confirmed**, see [[systems/omniroute]] |
|
||||
| 102 | vaultwarden | running | Vaultwarden password manager |
|
||||
| 104 | rclone | running | Rclone |
|
||||
| 105 | conduit | running | **Conduit** — iPhone calendar / health / location / contacts bridge (see [[systems/messaging-integrations]]). **Added 2026-10-31** — this guest was missing from the 2026-10-14 table despite running |
|
||||
| 107 | mem0 | running | **mem0 MCP server** (`mem0-mcp.service` → `:8890`), Tailscale `100.73.94.53` (`mem0` peer) — **added 2026-11-07**, this guest was missing from the table despite running |
|
||||
| 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** |
|
||||
| 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one |
|
||||
| 500 | litellm | running | LiteLLM proxy |
|
||||
@@ -105,47 +125,92 @@ as a hypervisor.
|
||||
| 1006 | freellmapi | running | FreeLLM API |
|
||||
| 1007 | homebridge | running | Homebridge |
|
||||
| 1008 | rustdesk | running | RustDesk remote access |
|
||||
| 1241 | ntfy | running | Ntfy push notifications — wiki's `CT241` |
|
||||
| 1241 | ntfy | running | ntfy push notifications — wiki's `CT241` |
|
||||
| 1260 | freshrss | running | FreshRSS feed reader — wiki's `CT260` |
|
||||
| 1261 | paseo | running | Paseo coding-agent daemon |
|
||||
| 1262 | mailflow | running | MailFlow |
|
||||
| 1263 | social-dashboard | running | Social dashboard |
|
||||
| 1264 | openmausbot | **stopped** | OpenMausBot — **stopped**, which is why the `openmausbot` Tailscale peer is offline 9d. **Added 2026-10-31** — was missing from the 2026-10-14 table. The companion **VM106 `openmausbot-vm` is running** and holds the separate `openmausbot-vm` peer (`100.77.107.30`) |
|
||||
|
||||
### QEMU Inventory (verified 2026-10-14 via `qm list` on m93p)
|
||||
**Re-verified 2026-11-07 03:2x AWST (stocktake):** live `pct list` on `m93p` returns **20 LXCs — 18 running,
|
||||
2 stopped (CT998 `nightbridge` and CT1264 `openmausbot`)**. ⚠ **The count was one short**: the previous
|
||||
figure of 19 omitted **CT107 (`mem0`)**, a live running guest confirmed from inside the container
|
||||
(`pct exec 107 -- tailscale ip -4` → `100.73.94.53`, `systemctl is-active tailscaled` → `active`,
|
||||
`mem0-mcp.service` → `active`). CT107 is now in the table above.
|
||||
|
||||
**Re-verified 2026-10-31:** the earlier reading of **19 LXCs — 18 running, 1 stopped
|
||||
(CT998 nightbridge) plus CT1264 openmausbot stopped** was itself already one short for the same reason
|
||||
(CT107 `mem0` was not counted). The 2026-10-14 table listed only **17**, omitting
|
||||
two live guests: **CT105 (`conduit`)** and **CT1264 (`openmausbot`)**. Note that **VMIDs 103, 220 and 555
|
||||
do not exist on `m93p`** (`/etc/pve/lxc/{103,220,555}.conf` all absent) — CT103 (cliproxyapi),
|
||||
CT220 (caddy) and CT555 (elementsynapse) are on **`pve`** only, and `pve`'s CT555 is **stopped** while
|
||||
`m93p` has no 555 at all. Guest IDs still collide across hosts, so always qualify with the host.
|
||||
|
||||
### QEMU Inventory (verified 2026-10-31 via `qm list` on m93p)
|
||||
|
||||
| VMID | Name | Status | Notes |
|
||||
|---|---|---|---|
|
||||
| 103 | cachyos-m93p | **stopped** | CachyOS |
|
||||
| 103 | cachyos-m93p | **stopped** | CachyOS (6144 MB RAM, 80 GB disk) |
|
||||
| 106 | openmausbot-vm | running | OpenMausBot VM (8192 MB RAM, 30 GB disk) — **Added 2026-10-31**; was missing from the 2026-10-14 table. Holds the `openmausbot-vm` Tailscale peer (`100.77.107.30`) |
|
||||
|
||||
**Reachability note:** guest service ports on `m93p` were **not** reachable from CT460 during this
|
||||
pass (they bind to their own interfaces and are not Tailscale-Served). Only hypervisor `:8006` on
|
||||
`100.104.51.98` is open. Reach them via the host or a Tailscale Serve route before probing services.
|
||||
Re-confirmed 2026-10-31: ports 3000, 4000, 5432, 8766, 8080 and 9000 were all closed from CT460.
|
||||
|
||||
**⚠ Follow-up:** `m93p`'s guest list is recorded here from a single `pct list` / `qm list` pass.
|
||||
The pages [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]] and
|
||||
[[infrastructure/docker-services]] still describe single-host topology and should be re-verified
|
||||
against `m93p` — in particular whether `m93p`'s `omniroute` (CT101) is a fourth OmniRoute node or
|
||||
a duplicate of CT600/CT601.
|
||||
**✅ Follow-up closed 2026-10-31:** `m93p`'s `omniroute` (CT101) is a **second live OmniRoute node**,
|
||||
not a duplicate of CT600/CT601. Live `pct config 101` on `m93p` confirms `hostname: omniroute`, and
|
||||
`omniroute` is listening on `0.0.0.0:20128` inside it; the `omniroute-1` Tailscale peer
|
||||
(`100.78.63.20`) resolves to it. **CT601 no longer exists on either host** — `/etc/pve/lxc/601.conf` is
|
||||
absent from `pve` and `m93p`, and the orphaned `omni` peer (`100.93.204.4`) has expired out of the
|
||||
tailnet. → [[systems/omniroute]], [[infrastructure/hosts]]
|
||||
|
||||
## Storage (live, 2026-10-14 03:0x AWST — `pvesm status` + `df -h /mnt/ssd0`, host `pve`)
|
||||
## Storage (live, 2026-11-01 03:5x AWST — `pvesm status` on both hosts)
|
||||
|
||||
### Host `pve` (100.96.100.82)
|
||||
|
||||
| Datastore | Type | Status | Total | Used | Used % |
|
||||
|---|---|---|---|---|---|
|
||||
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 808 GiB | **88.17 %** (`pvesm`) |
|
||||
| `SSD1` | lvmthin | active | 976 GiB | 815 GiB | 87.47 % |
|
||||
| `fred-qnap` | cifs | active | 13.7 TiB | 10.3 TiB | 77.22 % |
|
||||
| `local` | dir | active | 67.7 GiB | 55.1 GiB | 81.68 % |
|
||||
| `local-lvm` | lvmthin | active | 141 GiB | 19.0 GiB | 13.44 % |
|
||||
| `iMacHDD` | cifs | **inactive** | — | — | — |
|
||||
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 717 GiB | **78.29 %** (`pvesm`) |
|
||||
| `SSD1` | lvmthin | active | 931 GiB | 610 GiB | **65.46 %** |
|
||||
| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
|
||||
| `local` | dir | active | 67.7 GiB | 55.3 GiB | **81.62 %** |
|
||||
| `local-lvm` | lvmthin | active | 141 GiB | 20.1 GiB | **14.25 %** |
|
||||
|
||||
**`ssd0` improved from the 93 % figure: now 88.17 % by `pvesm`, ~63 GiB free.** The earlier 93 %
|
||||
came from `df`, which excludes reserved blocks; `pvesm` divides by the full 960 GiB. Both describe
|
||||
the same ~808 GiB used. **Use the `pvesm` number when comparing across hosts** — it is the only
|
||||
one that means the same thing on `m93p` too. `iMacHDD` remains inactive
|
||||
(`storage 'iMacHDD' is not online`) — check the CIFS mount before using it as a target.
|
||||
**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 % (2026-11-01) → 78.29 % (2026-11-07)**, ~152 GiB free. `fred-qnap` is now
|
||||
**84.21 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.3 TiB used) — the share was
|
||||
remounted over NFS rather than SMB.
|
||||
|
||||
**Storage re-verified 2026-11-07 03:2x AWST** against live `pvesm status` on both hosts — every
|
||||
percentage on both tables below was refreshed.
|
||||
|
||||
**`iMacHDD` no longer appears in `pvesm status` on `pve`** — the 2026-10-14 table listed it as an
|
||||
inactive `cifs` entry. The iMac shares now live on **`m93p`** as `cifs-anthonyimac` and `cifs-imachdd`
|
||||
(both active, 76.21 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing.
|
||||
|
||||
### Host `m93p` (100.104.51.98)
|
||||
|
||||
| Datastore | Type | Status | Total | Used | Used % |
|
||||
|---|---|---|---|---|---|
|
||||
| `cifs-anthonyimac` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
|
||||
| `cifs-imachdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
|
||||
| `local` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
|
||||
| `local-hdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
|
||||
| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
|
||||
| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
|
||||
| `nfs-ssd0-39` | dir | active | 916 GiB | 717 GiB | **78.29 %** |
|
||||
| `omb-internal` | dir | active | 93.9 GiB | 60.7 GiB | **64.66 %** |
|
||||
|
||||
**`m93p` storage was not documented before 2026-10-31.** Note that `m93p` exposes the **same backing
|
||||
volumes as `pve`** under different names — `nfs-proxmox-31`/`nfs-proxmox-54` mirror `pve`'s `fred-qnap`
|
||||
(both 84.21 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 78.29 %). The iMac shares that were
|
||||
`iMacHDD` on `pve` are `cifs-anthonyimac` / `cifs-imachdd` on `m93p`.
|
||||
|
||||
**Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`).
|
||||
|
||||
**Use the `pvesm` number when comparing across hosts** — it is the only one that means the same thing on
|
||||
both hypervisors.
|
||||
|
||||
## API Tokens
|
||||
- `homepage@pve!homepage` — PVEAuditor role, privsep=0 (used by Homepage dashboard)
|
||||
|
||||
|
||||
+65
-42
@@ -3,8 +3,8 @@ title: Tailscale
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-26
|
||||
updated: 2026-10-25
|
||||
verified_on: 2026-10-25
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
confidence: high
|
||||
tags: [tailscale, vpn, networking, infrastructure]
|
||||
---
|
||||
@@ -19,66 +19,83 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
|
||||
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
|
||||
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
|
||||
|
||||
## Tailscale Node Inventory (verified 2026-10-25)
|
||||
Full node list from `tailscale status` — **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
|
||||
## Tailscale Node Inventory (verified 2026-11-12)
|
||||
Re-verified 2026-11-12 03:1x–03:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **three peers flipped offline in the peer record while all three are in fact reachable, one genuine offline age/stamp correction, and 10 offline ages drifted +2d. No IP changes, no missing table rows, no new peers.** ⚠ **`mailflow` (100.115.60.92, CT1262 m93p), `qwenpaw` (100.110.72.126, CT465 pve) and `springroll` (100.100.153.125) all read `Online: false` in the peer record** (`LastSeen` 2026-11-10 13:59Z / 2026-11-11 12:20Z / 2026-11-11 08:30Z), yet **`tailscale ping` returns pong on every one** — mailflow via a direct path (`192.168.178.142:41641`, 2 ms), qwenpaw direct (`192.168.178.186:41641`, 7 ms) and springroll via **DERP(sin)** (210 ms, relay rather than direct) — and live `pct list` shows **CT1262 and CT465 running**. All three rows therefore stay **online**; the peer record is stale, exactly the m93p pattern this page has already recorded three times. ⚠ **`omarchymac` was genuinely stale in the *other* direction:** the 2026-11-10 row read *"offline 2d — last seen 2026-11-08 00:45Z"*, while the live record reads `LastSeen 2026-11-07T06:41:48Z` = **offline 4d** — the peer was last seen *earlier* than the page claimed and `tailscale ping` → **timeout**, so it is a real offline, not a stale record. The **9 named offline ages** were refreshed (+2d each): cloudreve 108d → **110d**; conduit 65d → **66d**; elementsynapse 111d → **113d**; freshrss-new 84d → **86d**; octoagent 8d → **10d**; omarchybook 19d → **21d**; omarchymac 2d → **4d**; openmausbot 19d → **21d**; ubuntu-server-xfce 81d → **82d**; and the six `codex-environment-*` ephemeral peers 2d → **4d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (58 rows = 57 live IPv4 peers + `hermes` itself; `tailscale status --json` reports **80 peers total — 57 with a `100.` IPv4, 23 IPv6-only**, unchanged). Container states cross-checked against live `pct list` **and `qm list`** on **both** hypervisors: `pve` (100.96.100.82) **17 LXCs + 4 VMs** (VM100 homeassistant, VM101 guest-vps, VM110 omarchy, VM282 ubuntu-server-xfce), `m93p` (100.104.51.98) **20 LXCs + 2 VMs** (VM103 cachyos-m93p stopped, VM106 openmausbot-vm running) — counts unchanged. **One new guest not previously on this page: CT1007 `homebridge` on `m93p`, running but with `tailscaled` `inactive` inside it** (`pct exec 1007 -- systemctl is-active tailscaled` → `inactive`), so it is a running guest that does not join the tailnet — the same class as CT105 `conduit` and CT1264 `openmausbot`. **Both DEAD Serve targets re-confirmed dead:** `ss -ltn` still shows **no listener on 8769** (`/agent-estate`) and **none on 9123** (`/synergy-prep`). **CT201's Serve table re-probed** (`pct exec 201 -- tailscale serve status`) and is unchanged: `/` → `8085` and `/mgr` → `8084` on both `:443` and `:8443`. **Re-probed and confirmed this page's stale incident:** CT299 `sftpgo` is online and healthy → [[incidents/2026-09-02-sftpgo-outage]].
|
||||
Re-verified 2026-11-10 00:4x AWST (nightly audit): `tailscale status --json` re-read — **6 new peers added, 8 offline ages drifted, 1 state transition back *online* (`octop`); no IP changes and no missing table rows.** ⚠ **Six peers that had never been listed now sit in the tailnet** — `codex-environment` (`100.111.126.78`), `codex-environment-1b88dc` (`100.76.42.76`), `codex-environment-8dadbb` (`100.65.200.27`), `codex-environment-955866` (`100.81.119.118`), `codex-environment-9fbd61` (`100.106.215.121`) and `codex-environment-d3edbd` (`100.74.3.2`) — every one created **2026-11-07 05:09–05:56Z**, every one **`Online: false`** with `LastSeen` **2026-11-07 11:09–11:56Z**. **None is a guest on either hypervisor** (live `pct list` and `qm list` on both `pve` and `m93p` list no such guest), so they are **orphaned/ephemeral tailnet peers**, not estate hosts: six records registered inside 47 minutes and all went silent within ~6 h, the signature of a throwaway CI/dev environment rather than infrastructure. **State transition:** `octop` (`100.92.163.122`) went **online → offline 1d** (`LastSeen 2026-11-08T13:13:04Z`) after returning for a single day on 11-07. The **9 offline ages** were refreshed (+1d each): cloudreve 105d → **106d**; conduit 61d → **62d**; elementsynapse 108d → **109d**; freshrss-new 81d → **82d**; octoagent 5d → **6d**; omarchybook 16d → **17d**; openmausbot 16d → **17d**; ubuntu-server-xfce 77d → **78d**; plus **`omarchymac`, which was not aged but re-stated** — the 2026-11-07 row read *"offline 5d — last seen 2026-11-01 11:13Z"* while the live record reads **`LastSeen 2026-11-08T00:45:17Z` = offline 2d**, so **both the age and the last-seen stamp on that row were stale** and are corrected here. Node count **52 → 58 non-funnel nodes** (57 live IPv4 peers + `hermes` itself). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none** once the six additions land; the **~23 IPv6-only `funnel-ingress-node` peers** remain deliberately excluded (`tailscale status --json`: **80 peers total — 57 with a `100.` IPv4, 23 IPv6-only**). Container counts re-confirmed live: `pve` (100.96.100.82) **17 LXCs + 4 VMs**, `m93p` (100.104.51.98) **20 LXCs + 2 VMs** — unchanged. **Not re-probed this pass** (so the 2026-11-07 findings stand un-contradicted rather than re-verified): the `:443` Serve route table (12 paths), the two DEAD CT460 targets (8769 `/agent-estate`, 9123 `/synergy-prep`), and per-guest tailnet identity via `pct exec <vmid> -- tailscale ip -4`.
|
||||
Re-verified 2026-11-07 03:1x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 9 offline-age drifts, no IP changes and no missing peers.** **`octop` came back online**: the peer record reads `Online: true` with `LastSeen 0001-01-01` and `tailscale ping 100.92.163.122` → **`pong from octop via DERP(syd) in 98ms`**, so the row moves offline → online. It remains **not an LXC or VM on either hypervisor** (live `pct list` / `qm list` on both `pve` and `m93p` list no such guest) — an **orphaned tailnet peer**, so its return is not an estate change. The **9 offline ages** were refreshed (+1d each): cloudreve 104d → **105d**; conduit 60d → **61d**; elementsynapse 107d → **108d**; freshrss-new 80d → **81d**; octoagent 4d → **5d**; omarchybook 15d → **16d**; omarchymac 4d → **5d**; openmausbot 15d → **16d**; ubuntu-server-xfce 76d → **77d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) **17 guests** and `m93p` (100.104.51.98) **20 guests + 2 VMs** — ⚠ **the m93p guest count on this page was one short**: live `pct list` on `m93p` returns **20 LXCs** (the extra is **CT107 `mem0`**, `100.73.94.53`, `mem0-mcp.service` active — confirmed from inside the guest), not 19; CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT998 `nightbridge` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). The route table still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; **both DEAD targets re-confirmed dead** — `ss -ltn` still shows **no listener on 8769** (`/agent-estate`) and **none on 9123** (`/synergy-prep`) — while every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` answers **HTTP 200**.
|
||||
Re-verified 2026-11-06 03:0x–03:2x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 8 offline-age drifts, no IP changes and no missing peers.** **`anthonys-mac-mini` came back online**: the peer record reads `Online: true` and `tailscale ping 100.76.60.37` → `pong from anthonys-mac-mini via DERP(syd) in 103ms`, so the row moves offline → online (client device, not an estate host). The **8 offline ages** were refreshed (+1d each): cloudreve 103d → **104d**; elementsynapse 106d → **107d**; freshrss-new 79d → **80d**; octoagent 3d → **4d**; octop 1d → **2d**; omarchybook 14d → **15d**; omarchymac 3d → **4d**; openmausbot 14d → **15d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). `m93p` re-pinged → `pong from m93p via 192.168.178.250:41641 in 3ms` (online, peer record agrees), and `octop` re-pinged → **timeout** (genuinely offline, matching its `LastSeen`). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) 17 guests and `m93p` (100.104.51.98) 19 guests + 2 VMs — CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). ⚠ **Two CT460 Serve routes are dead but still present in `tailscale serve status`**: `/agent-estate` → `127.0.0.1:8769` and `/synergy-prep` → `127.0.0.1:9123` have **no listener** (live `ss -ltn` probe) and therefore serve 502 — both rows are now marked DEAD here, matching [[infrastructure/network-map]] which had already recorded them as dead on 2026-11-05. The route table itself still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` is reachable.
|
||||
Full node list from `tailscale status` — **58 non-funnel nodes** (57 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
|
||||
Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec <vmid> -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
|
||||
Re-verified 2026-10-31 05:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **one state transition and seven offline-age drifts, no IP changes and no missing peers.** **`mattermost` came back online**: the peer record reads `Online: true` for the first time since the outage, and the guest behind it is genuinely healthy — live `pct exec 255 -- systemctl is-active mattermost` → **`active`**, `ss -ltnp` shows `mattermost` listening on `*:8065`, `pct exec 255 -- tailscale ip -4` → `100.98.116.14`, and the gateway's own `gateway.log` records **`✓ mattermost connected` at 2026-10-30 16:36:17** (after an earlier successful reconnect at 07:19:37 the same day). CT255 is therefore **running** on `pve` — the 40-day outage is over. The **7 offline ages** were refreshed (cloudreve 97d → **98d**; conduit 53d → **55d**; elementsynapse 100d → **102d**; freshrss-new 73d → **74d**; omarchybook 8d → **9d**; openmausbot 8d → **9d**; ubuntu-server-xfce 70d → **71d**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Two container annotations were added for peers that had none: `openmausbot` (**CT1264, m93p — stopped**) and `openmausbot-vm` (**VM106, m93p**), both confirmed from live `pct list` / `qm list` on `m93p`. The `m93p` row's "peer record stale" note was updated — its peer record read `Online: false` on 10-29 and 10-30 and reads **`Online: true`** today, the third consecutive day the record disagreed with the (reachable) host. CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
|
||||
Re-verified 2026-11-01 03:0x–04:0x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition and 5 offline-age drifts, no IP changes and no missing peers.** **`omarchymac` went offline**: the peer record reads `Online: false` with `LastSeen 2026-10-31T13:20:00Z` and `tailscale ping 100.126.173.115` times out, so this is a **genuine** transition (the JSON `OS` field reads `linux`, not a Mac). **`m93p`'s peer record reads `Online: false` again** — the **4th consecutive** stale record — while `tailscale ping 100.104.51.98` → `pong from m93p via DERP(syd) in 98ms` and SSH over the tailnet returned `up 1 day, 12:02`, so the row stays **online** and the peer record remains non-authoritative for hypervisor reachability. The **5 offline ages** were refreshed (cloudreve 98d → **99d**; freshrss-new 74d → **75d**; omarchybook 9d → **10d**; openmausbot 9d → **10d**; ubuntu-server-xfce 71d → **72d**). **`conduit` (55d) and `elementsynapse` (102d) were already correct** and are unchanged. **Every peer IP in the table still matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Container identity re-confirmed from inside the guests: `pct exec 201 -- tailscale ip -4` → `100.96.244.39` (CT201, runtipi), `pct exec 1001 -- docker ps` → `sshwifty` (CT1001 on `m93p`), CT209 `cloudreve` and CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT255 `mattermost` **running** with `systemctl is-active mattermost` → `active` and `LISTEN *:8065`. Both OmniRoute nodes re-probed: `100.88.81.19` (CT600) and `100.78.63.20` (CT101) both return **401** on `/v1/models` (alive, auth required).
|
||||
Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
|
||||
Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`.
|
||||
Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**.
|
||||
Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — **`octop` (100.92.163.122), `springroll` (100.100.153.125), `openmausbot` and `openmausbot-vm` were missing from the table body** (only mentioned in prose), `omarchymac` had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from `tailscale status --json`, keyed on DNS name rather than `HostName` (the JSON `HostName` for `freshrss` is `ct260`, and both `omniroute` nodes share the hostname `omniroute`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running.
|
||||
Re-verified 2026-10-30 03:3x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 IP changes** (anthonys-mac-mini: 100.74.17.15 -> 100.76.60.37; omarchymac: 100.84.180.69 -> 100.126.173.115) and **4 nodes dropped out of the tailnet entirely** (anthonys-imac (100.112.123.124), cachyos (100.124.208.15), omarchymac-1 (100.126.173.115), omni (100.93.204.4)). `omarchymac` is now `100.126.173.115` (the peer formerly listed as `omarchymac-1`; the old `omarchymac` at `100.84.180.69` no longer matches any peer) and `anthonys-mac-mini` moved to `100.76.60.37` — both same-named nodes, new addresses. The four dropped peers are Tailscale expired/offline records (`anthonys-imac`, `cachyos`, `omni`, and the old `omarchymac` address), not newly-installed hardware. **`m93p` reads `Online: false` in the peer record for the third consecutive audit, yet the host is plainly up — `tailscale ping 100.104.51.98` → `pong from m93p via 192.168.178.118:41641 in 0s` and SSH returned `m93p / up 12 days, 11:02` — so the peer record is not authoritative for hypervisor reachability and the row records it as online. Node count corrected to **50 non-funnel nodes** (49 live IPv4 peers + `hermes` itself); `tailscale status --json` reports **72 peers total — 49 with a `100.` IPv4, 23 IPv6-only** (the `funnel-ingress-node` set). **Live IPv4 peers absent from the table: none.** Container identity re-confirmed from inside guests via `pct exec <vmid> -- tailscale ip -4` on `pve` (100.96.100.82) and `m93p` (100.104.51.98); CT600 is confirmed **running** and CT601 confirmed **absent from both hypervisors**. Serve tables re-probed per host: CT460's `:443` route set matches the table below exactly (12 paths), and CT201 (`pct exec 201 -- tailscale serve status`) serves only `/` → `127.0.0.1:8085` and `/mgr` → `127.0.0.1:8084` on **both** `:443` and `:8443`, with no `/steel` or `/steel-api` routes remaining.
|
||||
Re-verified 2026-10-29 03:0x AWST (stocktake): State column rebuilt from `tailscale status --json` — **11 rows changed**. Two were **state transitions**, not age drift: **`sftpgo` (CT299, pve) came back online** (the 2026-10-28 audit had recorded it as `offline <1d`; live `tailscale ping 100.88.223.70` → `pong from sftpgo via 192.168.178.92:41641 in 0s`, `pct exec 299 -- systemctl is-active sftpgo` → `active`, `pct exec 299 -- tailscale ip -4` → `100.88.223.70`), and **`m93p` is online despite a stale peer record** — `tailscale status --json` still reports `Online: false` with `LastSeen 2026-10-28T18:40:00Z`, but `tailscale ping 100.104.51.98` → `pong from m93p via 192.168.178.118:41641 in 1ms` and SSH over the tailnet returned `m93p / up 11 days`. This is the **second** occurrence of the m93p stale-peer-record pattern (first seen 2026-10-26) — the peer record is not authoritative for hypervisor reachability. The remaining **9 rows were offline-age drift of exactly +1 day** (anthonys-imac 34d → 35d; cachyos 54d → 55d; cloudreve 95d → 96d — CT209 stopped; elementsynapse 98d → 99d — CT555 stopped; freshrss-new 71d → 72d; mattermost 38d → 39d — CT255 stopped; omarchybook 6d → 7d; omarchymac 10d → 11d; omni 71d → 72d — CT601 removed from both hypervisors; openmausbot 6d → 7d). Node count unchanged at **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself); `tailscale status --json` reports **76 peers total — 53 with a `100.` IPv4, 23 IPv6-only** (the `funnel-ingress-node` set). **Nodes in the table that are no longer in the tailnet: none. Live IPv4 peers absent from the table: none. IP mismatches: none.** Container identity re-confirmed from **inside** every LXC via `pct exec <vmid> -- tailscale ip -4` across **both** hypervisors (`pve` 100.96.100.82, `m93p` 100.104.51.98) — 34 guests enumerated; every peer IP mapped to its CT as listed, and the three stopped guests (`CT209` cloudreve, `CT255` mattermost, `CT555` elementsynapse) and the three running-but-unconnected guests (`CT105` conduit, `CT1264` openmausbot, `CT100` obsidian-livesync) confirmed. CT601 remains absent from both hypervisors.
|
||||
|
||||
Re-verified 2026-10-28 00:2x (nightly audit): State column rebuilt from `tailscale status --json` — **14 rows changed** (anthonys-imac: idle/offline 32d -> offline 34d; cachyos: idle/offline 52d -> offline 54d; cloudreve: idle/offline 93d — CT209 stopped -> offline 95d — CT209 stopped; conduit: idle/offline 49d -> offline 52d; elementsynapse: idle/offline 96d — CT555 stopped -> offline 98d — CT555 stopped; freshrss-new: idle/offline 69d -> offline 71d; m93p: online -> offline <1d; mattermost: idle/offline 36d — CT255 stopped -> offline 38d — CT255 stopped; omarchybook: idle/offline 4d -> offline 6d; omarchymac: idle/offline 8d -> offline 10d; omni: idle/offline 69d — CT601 removed from both hypervisors -> offline 71d — CT601 removed from both hypervisors; openmausbot: idle/offline 4d -> offline 6d; sftpgo: online — CT299 (pve) -> offline <1d — CT299 (pve); ubuntu-server-xfce: idle/offline 66d -> offline 68d). Header count corrected to **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Nodes in the table that are no longer in the tailnet: **none**. Live IPv4 peers absent from the table: **none**. **State transitions inside the audit window (not just age drift): `m93p` online → offline <1d, `sftpgo` (CT299) online → offline <1d.**
|
||||
Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale status --json` — **`springroll-browser` (100.109.221.91) was missing entirely** and has been added. **`omarchybook` (100.107.9.100) and `openmausbot` (100.91.214.91) have both gone offline** (online → idle/offline 3d) and `openmausbot` is no longer a live peer despite its m93p LXC (CT1264) still running `tailscale` — the node itself has stopped connecting. Every offline age was refreshed (e.g. `mattermost` 31d → **35d**, `omni` 64d → **68d**, `elementsynapse` 91d → **95d**, `ubuntu-server-xfce` 60d → **64d**, `omarchymac` 3d → **7d**). `m93p` (100.104.51.98) now reads **idle/offline 13m ago** — the peer record is stale, but the host is plainly reachable (both `pct list` probes and the SSH session to CT201/CT460 during this audit succeeded), so treat it as a Tailscale-side liveness artefact, not a hypervisor outage. `homeassistant` still advertises `ExitNodeOption: true` but `ExitNode: false` — it **offers** exit-node capability; it is not currently serving as one (the old "active; offers exit node" wording overstated the live state). Container states cross-checked again against live `pct list` on both hypervisors, and each peer's container identity confirmed by running `tailscale ip -4` **inside** the candidate LXC on both hosts — the authoritative mapping, since peer names and LXC names do not always match.
|
||||
|
||||
| Node | IP | State |
|
||||
|---|---|---|
|
||||
| hermes | 100.118.5.51 | **self** |
|
||||
| hermes | 100.118.5.51 | **self** — CT460 (pve) |
|
||||
| agentzero | 100.106.54.94 | online — CT475 (pve) |
|
||||
| anthonys-imac | 100.112.123.124 | idle/offline 31d |
|
||||
| anthonys-ipad | 100.111.198.120 | online |
|
||||
| anthonys-iphone | 100.100.116.67 | online |
|
||||
| anthonys-mac-mini | 100.74.17.15 | online |
|
||||
| cachyos | 100.124.208.15 | idle/offline 51d |
|
||||
| anthonys-mac-mini | 100.76.60.37 | online — **returned 2026-11-06** (peer record `Online: true`, `tailscale ping` → `pong from anthonys-mac-mini (100.76.60.37) via DERP(syd) in 103ms`; client device, not an estate host) |
|
||||
| clawtest | 100.122.189.81 | online — CT333 (pve) |
|
||||
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
|
||||
| cloudreve | 100.111.187.7 | idle/offline 92d — CT209 stopped |
|
||||
| conduit | 100.81.79.87 | idle/offline 48d |
|
||||
| cloudreve | 100.111.187.7 | offline 110d — CT209 stopped |
|
||||
| codex-environment | 100.111.126.78 | offline 4d — last seen 2026-11-07 11:09Z; **created 2026-11-07 05:5xZ**, six-record burst; **not an LXC/VM on either hypervisor** (orphaned/ephemeral tailnet peer; added 2026-11-10, last-seen stamps corrected 2026-11-10) |
|
||||
| codex-environment-1b88dc | 100.76.42.76 | offline 4d — last seen 2026-11-07 11:12Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
|
||||
| codex-environment-8dadbb | 100.65.200.27 | offline 4d — last seen 2026-11-07 11:46Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
|
||||
| codex-environment-955866 | 100.81.119.118 | offline 4d — last seen 2026-11-07 11:49Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
|
||||
| codex-environment-9fbd61 | 100.106.215.121 | offline 4d — last seen 2026-11-07 11:51Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
|
||||
| codex-environment-d3edbd | 100.74.3.2 | offline 4d — last seen 2026-11-07 11:56Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
|
||||
| conduit | 100.81.79.87 | offline 66d — last seen 2026-09-06 01:32Z |
|
||||
| docker | 100.96.150.69 | online — CT200 (pve) |
|
||||
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
|
||||
| elementsynapse | 100.75.250.37 | idle/offline 95d — CT555 stopped |
|
||||
| elementsynapse | 100.75.250.37 | offline 113d — CT555 stopped |
|
||||
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
|
||||
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
|
||||
| freshrss-new | 100.110.180.115 | idle/offline 68d |
|
||||
| freshrss-new | 100.110.180.115 | offline 86d — last seen 2026-08-17 17:04Z |
|
||||
| gitea | 100.82.100.128 | online — CT221 (m93p) |
|
||||
| guest-vps | 100.117.14.84 | online |
|
||||
| guest-vps | 100.117.14.84 | online — VM101 (pve) |
|
||||
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
|
||||
| homeassistant | 100.68.139.5 | online |
|
||||
| homeassistant | 100.68.139.5 | online — VM100 (pve) |
|
||||
| litellm | 100.79.29.64 | online — CT500 (m93p) |
|
||||
| m93p | 100.104.51.98 | idle/offline 13m ago |
|
||||
| mailflow | 100.115.60.92 | online — CT1262 (m93p) |
|
||||
| mattermost | 100.98.116.14 | idle/offline 35d — CT255 stopped |
|
||||
| m93p | 100.104.51.98 | online — hypervisor (peer record reads `Online: true`; `tailscale ping` → pong) |
|
||||
| mailflow | 100.115.60.92 | online — CT1262 (m93p) — ⚠ peer record reads `Online: false` (`LastSeen 2026-11-10T13:59:56Z`, offline 1d), but `tailscale ping` → **`pong from mailflow (100.115.60.92) via 192.168.178.142:41641 in 2ms`** and live `pct list` on `m93p` shows **CT1262 running** — stale peer record, not an outage (re-probed 2026-11-12) |
|
||||
| mattermost | 100.98.116.14 | online — CT255 (pve) |
|
||||
| mem0 | 100.73.94.53 | online — CT107 (m93p) |
|
||||
| n8n | 100.84.124.30 | online — CT215 (pve) |
|
||||
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
|
||||
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
|
||||
| octop | 100.92.163.122 | online |
|
||||
| obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) |
|
||||
| octoagent | 100.117.10.68 | offline 10d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
|
||||
| octop | 100.92.163.122 | **online** (re-verified 2026-11-12 00:4x AWST — peer record `Online: true`; still **not an LXC/VM on either hypervisor** — orphaned tailnet peer) |
|
||||
| ollama | 100.108.44.119 | online — CT205 (pve) |
|
||||
| omarchy | 100.73.227.103 | online |
|
||||
| omarchybook | 100.107.9.100 | idle/offline 3d |
|
||||
| omarchymac | 100.84.180.69 | idle/offline 7d |
|
||||
| omni | 100.93.204.4 | idle/offline 68d — CT601 stopped |
|
||||
| omarchy | 100.73.227.103 | online — VM110 (pve) |
|
||||
| omarchybook | 100.107.9.100 | offline 21d — last seen 2026-10-21 15:55Z |
|
||||
| omarchymac | 100.126.173.115 | offline 4d — last seen 2026-11-07 06:41Z (⚠ **was stale at 2d / 2026-11-08 00:45Z**; corrected 2026-11-12 — the peer was last seen *earlier* than the 2026-11-10 row claimed and `tailscale ping` → timeout, so this is a genuine offline, not a stale record) |
|
||||
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
|
||||
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
|
||||
| openmausbot | 100.91.214.91 | idle/offline 3d |
|
||||
| openmausbot-vm | 100.77.107.30 | online |
|
||||
| openmausbot | 100.91.214.91 | offline 21d — CT1264 (m93p) stopped; last seen 2026-10-21 15:50Z |
|
||||
| openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) |
|
||||
| openweb | 100.69.230.13 | online — CT245 (m93p) |
|
||||
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
|
||||
| pve | 100.96.100.82 | online |
|
||||
| qwenpaw | 100.110.72.126 | online — CT465 (pve) |
|
||||
| qwenpaw | 100.110.72.126 | online — CT465 (pve) — ⚠ peer record reads `Online: false` (`LastSeen 2026-11-11T12:20:00Z`), but `tailscale ping` → **`pong from qwenpaw (100.110.72.126) via 192.168.178.186:41641 in 7ms`** and live `pct list` on `pve` shows **CT465 running** — stale peer record, not an outage (re-probed 2026-11-12) |
|
||||
| rclone | 100.107.116.17 | online — CT104 (m93p) |
|
||||
| runtipi | 100.96.244.39 | online — CT201 (pve) |
|
||||
| rustdesk | 100.95.227.57 | online — CT1008 (m93p) |
|
||||
| sftpgo | 100.88.223.70 | online — CT299 (pve) |
|
||||
| sftpgo | 100.88.223.70 | online — CT299 (pve) — **recovered 2026-11-04 (freeze 1) and again 2026-11-05 (freeze 2, manual `ssd0` run)**: a wedged `vzdump` had frozen the guest for ~24 h; killing the task unfroze it (see [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]) — **re-confirmed online 2026-11-12** (`tailscale ping` → `pong from sftpgo (100.88.223.70) via 192.168.178.92:41641 in 0s`, `pct list` → CT299 running, healthz → 401) |
|
||||
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
|
||||
| springroll | 100.100.153.125 | online |
|
||||
| springroll | 100.100.153.125 | online — ⚠ peer record reads `Online: false` (`LastSeen 2026-11-11T08:30:00Z`), but `tailscale ping` → **`pong from springroll (100.100.153.125) via DERP(sin) in 210ms`** (answers via relay, not direct); not an LXC/VM on either hypervisor (client device) — re-probed 2026-11-12 |
|
||||
| springroll-browser | 100.109.221.91 | online |
|
||||
| ubuntu-server-xfce | 100.101.26.104 | idle/offline 64d |
|
||||
| ubuntu-server-xfce | 100.101.26.104 | offline 82d — last seen 2026-08-20 19:36Z — VM282 (pve) running |
|
||||
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
|
||||
| webserver | 100.82.40.23 | online — CT207 (pve) |
|
||||
|
||||
@@ -90,19 +107,19 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
||||
| Path | Target | Notes |
|
||||
|---|---|---|
|
||||
| `/` | `http://127.0.0.1:9119` | Hermes WebUI |
|
||||
| `/ui` | `http://127.0.0.1:5175/` | Hermes UI (Vite) |
|
||||
| `/ui` | `http://127.0.0.1:5175` | Hermes UI (Vite) |
|
||||
| `/jobs` | `http://127.0.0.1:9099` | Jobs Dashboard |
|
||||
| `/wiki` | `http://100.118.5.51:9120` | Agent Estate Wiki (Quartz) |
|
||||
| `/ignis` | `http://127.0.0.1:8080` | Ignis (browser-native Obsidian) |
|
||||
| `/webui` | `http://127.0.0.1:8787/` | Hermes WebUI (Standalone) |
|
||||
| `/webui` | `http://127.0.0.1:8787` | Hermes WebUI (Standalone) |
|
||||
| `/gitea` | `http://100.82.100.128:3000` | Gitea |
|
||||
| `/desktop` | `http://127.0.0.1:6081` | CUA noVNC |
|
||||
| `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC |
|
||||
| `/synergy-prep` | `http://127.0.0.1:9123/` | Synergy prep |
|
||||
| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) |
|
||||
| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep — ⚠ **DEAD: nothing listening on 9123** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07 and 2026-11-12**; the route still exists in `tailscale serve status` but serves 502) |
|
||||
| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) — ⚠ **DEAD: nothing listening on 8769** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07 and 2026-11-12**; route still present in `tailscale serve status` but serves 502) |
|
||||
| `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid |
|
||||
|
||||
Verified against `tailscale serve status` on 2026-10-25 — the CT460 route table above matched live exactly (12 paths, no drift).
|
||||
Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 route table above matched live exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; no drift).
|
||||
|
||||
## CT460 Funnel and 9443 Routes
|
||||
| Port | Target | Notes |
|
||||
@@ -110,20 +127,26 @@ Verified against `tailscale serve status` on 2026-10-25 — the CT460 route tabl
|
||||
| `:8443/webhook` | `http://127.0.0.1:8085/webhook` | Public webhook Funnel (`# Funnel on:` confirmed) |
|
||||
| `:9443` (tailnet only) `/` | `http://127.0.0.1:5173` | Vite dev server on port 5173 exposed via a **tailnet-only** Serve on port 9443 — was missing from this page, added 2026-10-11 |
|
||||
|
||||
Re-verified 2026-10-26 (stocktake): the node table was rebuilt again from `tailscale status --json` — **`octoagent` (100.117.10.68) and `omarchymac-1` (100.126.173.115) were both missing entirely** and have been added; `m93p` (100.104.51.98) now reads **online** (it was "idle/offline 13m ago" — a stale Tailscale peer record, and the host was plainly reachable throughout); every offline age was refreshed (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**, `ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**, `openmausbot` 3d → **4d**). The `omni` row now reads **"CT601 removed from both hypervisors"** rather than "CT601 stopped" — live `pct list` on both `pve` and `m93p` shows **no CT601 at all**, so the old guest is gone, not merely halted. Container identity re-confirmed from **inside** each LXC via `pct exec <vmid> -- tailscale ip -4` on both hypervisors; every peer whose LXC has no Tailscale IP was checked against `pct list` (`CT105`/`CT1264` run but do not connect — `conduit` and `openmausbot` offline; `CT209`/`CT255`/`CT555` are `stopped`). Serve tables re-probed per host with `pct exec 201 -- tailscale serve status`.
|
||||
|
||||
## CT201 Serve Routes
|
||||
Verified live on 2026-10-25 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected.
|
||||
Re-verified live on 2026-10-30 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected.
|
||||
|
||||
| Host / Port | Path | Target | Notes |
|
||||
|---|---|---|---|
|
||||
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **currently 502** (upstream 8082 has no listener) |
|
||||
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` container — **200** |
|
||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **currently 502** |
|
||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` — **200** |
|
||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/steel` | `http://127.0.0.1:5173` | Steel Browser UI — **200** on :443. ⚠ **`/steel` is NOT served on :8443** — it 502s there |
|
||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/steel-api` | `http://127.0.0.1:3000` | Steel Browser API (path prefix not stripped — use direct IP for API calls) |
|
||||
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **502** (re-probed 2026-10-30; upstream 8082 has no listener — `curl http://127.0.0.1:8082/` → exit code 7 / `http=000`) |
|
||||
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` container — **200** (re-probed 2026-10-30) |
|
||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **502** (re-probed 2026-10-30) |
|
||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` — **200** (re-probed 2026-10-30) |
|
||||
|
||||
**Steel routes removed 2026-10-29.** This table previously carried two more rows on `:443` — `/steel` →
|
||||
`http://127.0.0.1:5173` and `/steel-api` → `http://127.0.0.1:3000`. Both were deleted with
|
||||
`tailscale serve --set-path <path> off` when Steel was removed from CT201, so CT201's only remaining routes
|
||||
are `/` → `127.0.0.1:8085` and `/mgr` → `http://127.0.0.1:8084`, on **both** `:443` and `:8443`.
|
||||
→ [[comparisons/steel-vs-camoufox]]
|
||||
|
||||
Two corrections to the pre-2026-10-25 table:
|
||||
1. **`:8443` is served by `tailscaled` on CT201, but so is `:443`.** `ss -ltnp` on CT201 shows `tailscaled` bound to `100.96.244.39:443`, and the serve table above lists routes on **both** ports. The old note claiming "Port 443 occupied by Traefik on CT201" is **false as of 2026-10-25** — no Traefik container or process exists on CT201 (`docker ps -a` and `ps aux` both came back empty for `traefik`), and Tailscale owns 443 there. The Homebox/steel path prefix is not stripped on `/steel-api`.
|
||||
1. **`:8443` is served by `tailscaled` on CT201, but so is `:443`.** `ss -ltnp` on CT201 shows `tailscaled` bound to `100.96.244.39:443`, and the serve table above lists routes on **both** ports. The old note claiming "Port 443 occupied by Traefik on CT201" is **false as of 2026-10-25** — no Traefik **process** runs on CT201 (`ps aux` returns nothing for `traefik`), and Tailscale owns 443 there. *(Evidence corrected 2026-10-30: the original wording said `docker ps -a` "came back empty for `traefik`" — it does not. Live `docker ps -a` lists `runtipi-reverse-proxy | traefik:v3.2 | Exited (255) 2 months ago`. The container is **stopped**, not absent; the conclusion that Traefik does not occupy 443 stands.)*
|
||||
2. The old table routed `:8443` straight to `http://localhost:8082`. That port has **no listener**; the live target is `8085` (the `asset_proxy.py` front door), which itself proxies to the dead 8082 and returns 502. → [[systems/homepage-dashboard]]
|
||||
|
||||
## Atomic Restore Script
|
||||
@@ -148,7 +171,7 @@ tailscale serve --set-path /jobs off
|
||||
```
|
||||
|
||||
## Notes
|
||||
- **CT201 now serves Tailscale on BOTH `:443` and `:8443`** (verified 2026-10-25). The old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" claim was **wrong**: no Traefik container or process exists on CT201, and `tailscaled` owns 443 there (`ss -ltnp` shows it bound to `100.96.244.39:443`). The CT201 Serve Routes table above now records both ports.
|
||||
- **CT201 now serves Tailscale on BOTH `:443` and `:8443`** (verified 2026-10-30). The old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" claim was **wrong**: no Traefik **process** runs on CT201, and `tailscaled` owns 443 there (`ss -ltnp` shows it bound to `100.96.244.39:443`). The CT201 Serve Routes table above now records both ports. *(Evidence corrected 2026-10-30: the original wording said "no Traefik container or process exists on CT201" — the container does exist, `runtipi-reverse-proxy | traefik:v3.2`, but it is **stopped** — `Exited (255) 2 months ago`. The conclusion that Tailscale owns 443 is unchanged.)*
|
||||
- CT460 has no conflict — Tailscale uses 443 directly, with the public webhook Funnel on `:8443`.
|
||||
- Routes must be re-added in order (restore script handles this).
|
||||
- After every rebuild, verify that `tailscale serve status` contains both the `:443` Serve routes and the `:8443/webhook` Funnel route.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Retained Research Answers
|
||||
type: query
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-12
|
||||
verified_on: 2026-10-12
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
review_after: 2027-01-12
|
||||
confidence: medium
|
||||
tags: [query, research, vnc, remote-desktop]
|
||||
@@ -18,11 +18,32 @@ Research answers worth keeping: what was asked, the answer, the evidence it rest
|
||||
|
||||
> The 2026-07-22 stub was filled on 2026-10-12 with the first retained answer (asked in the 2026-10-11
|
||||
> interactive session `talaria-ios`, 41 messages).
|
||||
>
|
||||
> **2026-11-04 re-verification:** the answer's *framing* was wrong — it was written for "TigerVNC + Xtigervnc
|
||||
> servers" that the estate does not run. The client-side advice remains valid for the one VNC server that does
|
||||
> exist (x11vnc on CT460). See the correction note under the answer.
|
||||
|
||||
## Fastest VNC client (asked 2026-10-11)
|
||||
|
||||
**Answer for the CT460/CT200 TigerVNC + Xtigervnc servers:** use the **TigerVNC native viewer** with
|
||||
AutoSelect and the defaults — free, and the fastest standard-RFB client. Lowest bandwidth is **KasmVNC**
|
||||
**Answer for the CT460 VNC server:** use the **TigerVNC native viewer** with AutoSelect and the defaults —
|
||||
free, and the fastest standard-RFB client.
|
||||
|
||||
> ⚠ **Corrected 2026-11-04 — the original question was framed against servers the estate does not run.**
|
||||
> This answer was written on 2026-10-12 for "the CT460/CT200 TigerVNC + Xtigervnc servers". **Neither exists.**
|
||||
> Live state, re-verified 2026-11-04:
|
||||
> - **CT460** runs **x11vnc**, not TigerVNC and not Xtigervnc: `x11vnc -display :99 -forever -shared -rfbport 5901
|
||||
> -rfbportv6 5901 -noxdamage -quiet -nopw` (PID 1290295), fronted by **websockify** → noVNC on
|
||||
> `127.0.0.1:6080 → 5900` (a **dead** target — no listener on 5900) and `127.0.0.1:6081 → 5901` (**live**).
|
||||
> The display itself is `Xvfb :99 -screen 0 1600x1200x24 -ac -nolisten tcp`.
|
||||
> - **CT200** runs **no VNC server at all**. Its only X server is `Xvfb :495478600 -screen 0 800x680x24`, started
|
||||
> 2026-10-30 — that is the Camofox browser's own headless Xvfb, and the Camofox VNC plugin tree lives on
|
||||
> **CT460** (`/home/hermes/services/camofox-browser/plugins/vnc/`), not on CT200. No `x11vnc`, no `Xtigervnc`,
|
||||
> no `vncserver`, and nothing listening on 5900/5901/6080/6081.
|
||||
>
|
||||
> **The advice itself still stands**, because x11vnc speaks standard RFB and TigerVNC's viewer is the right
|
||||
> client for it — but it now applies to **one** host, and the page's "the estate's own VNC/Xtigervnc setup" pointer
|
||||
> is wrong. The real setup is documented at [[systems/headless-server]], which already described x11vnc correctly
|
||||
> back on 2026-10-20; this query page was the outlier. Lowest bandwidth is **KasmVNC**
|
||||
(WebP, ~30 % less) but that is a *server* swap, not a client. **RealVNC Connect** is not a faster decoder, it is
|
||||
a forked protocol (RFB 6, audio, no port-forwarding) and needs a RealVNC server. **TurboVNC** remains the
|
||||
reference for lossless 3D/CAD. Genuinely low latency means leaving RFB entirely: **Parsec** or
|
||||
@@ -66,8 +87,14 @@ software (a thin client with no SIMD headroom decodes slowly regardless of viewe
|
||||
|
||||
**Does NOT establish:** exact throughput multipliers on current hardware (the benchmark is 2012); no
|
||||
end-to-end benchmark was run against Anthony's own hosts. If speed becomes a real complaint, measure the
|
||||
existing server before changing clients.
|
||||
existing server before changing clients. **It also does not establish anything about TigerVNC or Xtigervnc
|
||||
*server*-side behaviour on this estate, because the estate runs neither** — x11vnc's own encoder settings
|
||||
(`-compress`, `-quality`, `-encodings`) are the live lever on CT460, not the TigerVNC/TurboVNC encoder options
|
||||
the report discusses. The compression-level and local-cursor levers above map onto x11vnc as
|
||||
`-compress <level>` and the `x11vnc` cursor-shape options, but the specific numbers were tuned for a different
|
||||
server and should be re-measured before being trusted on this one.
|
||||
|
||||
## Related
|
||||
- [[systems/hermes-agent]] — the estate's own VNC/Xtigervnc setup lives under [[systems/headless-server]]
|
||||
- [[systems/headless-server]] — **the estate's actual VNC setup** (x11vnc on `Xvfb :99`, noVNC via websockify) — re-verified 2026-11-04
|
||||
- [[systems/hermes-agent]] — the agent host this answer was researched for
|
||||
- [[log]] — audit entries
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
# Wiki freshness audit — 2026-10-26
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (0)
|
||||
|
||||
|
||||
## FRESH (103)
|
||||
|
||||
- [ 6d] CONTRIBUTING.md
|
||||
- [ 0d] agents/hermes-chaos.md
|
||||
- [ 0d] agents/hermes-production.md
|
||||
- [ 6d] agents/index.md
|
||||
- [ 15d] agents/nanobot.md
|
||||
- [ 6d] agents/open-claw.md
|
||||
- [ 15d] agents/qwenpaw.md
|
||||
- [ 6d] agents/zeroclaw.md
|
||||
- [ 6d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 6d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 6d] comparisons/memory-backends.md
|
||||
- [ 6d] comparisons/model-routers.md
|
||||
- [ 13d] comparisons/steel-vs-camoufox.md
|
||||
- [ 6d] concepts/agent-memory.md
|
||||
- [ 13d] concepts/browser-automation.md
|
||||
- [ 6d] concepts/model-routing.md
|
||||
- [ 6d] concepts/persistent-shells.md
|
||||
- [ 6d] concepts/tool-calling.md
|
||||
- [ 1d] current-state.md
|
||||
- [ 6d] decisions/index.md
|
||||
- [ 6d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 6d] decisions/persistent-shell-enabled.md
|
||||
- [ 6d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 6d] experiments/index.md
|
||||
- [ 2d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 6d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 2d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 4d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 3d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 14d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 1d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 6d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 5d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 15d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 20d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 0d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 0d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 20d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 2d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 17d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 10d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 4d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 7d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 5d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 4d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 1d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 0d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 0d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 0d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 4d] infrastructure/backups.md
|
||||
- [ 6d] infrastructure/credentials-map.md
|
||||
- [ 15d] infrastructure/docker-services.md
|
||||
- [ 6d] infrastructure/domains-and-tunnels.md
|
||||
- [ 0d] infrastructure/hosts.md
|
||||
- [ 12d] infrastructure/network-map.md
|
||||
- [ 12d] infrastructure/proxmox.md
|
||||
- [ 0d] infrastructure/tailscale.md
|
||||
- [ 14d] queries/retained-research-answers.md
|
||||
- [ 1d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 5d] reports/uber-eats/2026-07.md
|
||||
- [ 5d] reports/uber-eats/2026-08.md
|
||||
- [ 5d] reports/uber-eats/2026-09.md
|
||||
- [ 5d] reports/uber-eats/index.md
|
||||
- [ 6d] runbooks/backup-wiki.md
|
||||
- [ 6d] runbooks/diagnose-docker.md
|
||||
- [ 6d] runbooks/gateway-resource-resilience.md
|
||||
- [ 13d] runbooks/job-radar.md
|
||||
- [ 6d] runbooks/multiplexer-setup.md
|
||||
- [ 0d] runbooks/provider-health.md
|
||||
- [ 6d] runbooks/recover-docker-service.md
|
||||
- [ 6d] runbooks/restart-browser.md
|
||||
- [ 6d] runbooks/restart-hermes.md
|
||||
- [ 6d] runbooks/restore-openrouter.md
|
||||
- [ 6d] runbooks/test-openrouter.md
|
||||
- [ 6d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 0d] runbooks/update-hermes-safely.md
|
||||
- [ 13d] systems/browser-backend.md
|
||||
- [ 6d] systems/cloudflare-tunnel.md
|
||||
- [ 0d] systems/conduit.md
|
||||
- [ 0d] systems/current-profile.md
|
||||
- [ 6d] systems/guanaco.md
|
||||
- [ 6d] systems/headless-server.md
|
||||
- [ 0d] systems/hermes-agent.md
|
||||
- [ 6d] systems/hermes-ui.md
|
||||
- [ 14d] systems/home-assistant.md
|
||||
- [ 1d] systems/homepage-dashboard.md
|
||||
- [ 15d] systems/ignis.md
|
||||
- [ 0d] systems/memory-backend.md
|
||||
- [ 0d] systems/messaging-integrations.md
|
||||
- [ 2d] systems/mnemosyne.md
|
||||
- [ 0d] systems/model-providers.md
|
||||
- [ 14d] systems/nanobot.md
|
||||
- [ 6d] systems/nextcloud.md
|
||||
- [ 6d] systems/obsidian.md
|
||||
- [ 4d] systems/omniroute.md
|
||||
- [ 6d] systems/opencode-go.md
|
||||
- [ 6d] systems/openrouter.md
|
||||
- [ 15d] systems/paseo.md
|
||||
- [ 7d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 6d] systems/terminal-backend.md
|
||||
- [ 6d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (4)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- index.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 0 stale, 103 fresh, 4 no-date of 107 pages
|
||||
@@ -0,0 +1,121 @@
|
||||
# Wiki freshness audit — 2026-10-27
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (0)
|
||||
|
||||
|
||||
## FRESH (104)
|
||||
|
||||
- [ 7d] CONTRIBUTING.md
|
||||
- [ 1d] agents/hermes-chaos.md
|
||||
- [ 1d] agents/hermes-production.md
|
||||
- [ 7d] agents/index.md
|
||||
- [ 16d] agents/nanobot.md
|
||||
- [ 7d] agents/open-claw.md
|
||||
- [ 16d] agents/qwenpaw.md
|
||||
- [ 7d] agents/zeroclaw.md
|
||||
- [ 7d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 7d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 7d] comparisons/memory-backends.md
|
||||
- [ 7d] comparisons/model-routers.md
|
||||
- [ 14d] comparisons/steel-vs-camoufox.md
|
||||
- [ 7d] concepts/agent-memory.md
|
||||
- [ 14d] concepts/browser-automation.md
|
||||
- [ 7d] concepts/model-routing.md
|
||||
- [ 7d] concepts/persistent-shells.md
|
||||
- [ 7d] concepts/tool-calling.md
|
||||
- [ 0d] current-state.md
|
||||
- [ 7d] decisions/index.md
|
||||
- [ 7d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 7d] decisions/persistent-shell-enabled.md
|
||||
- [ 7d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 7d] experiments/index.md
|
||||
- [ 3d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 7d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 3d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 5d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 4d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 15d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 2d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 7d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 6d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 16d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 21d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 1d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 1d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 21d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 3d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 18d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 11d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 0d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 8d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 0d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 5d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 2d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 1d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 1d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 1d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 0d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 5d] infrastructure/backups.md
|
||||
- [ 7d] infrastructure/credentials-map.md
|
||||
- [ 16d] infrastructure/docker-services.md
|
||||
- [ 7d] infrastructure/domains-and-tunnels.md
|
||||
- [ 1d] infrastructure/hosts.md
|
||||
- [ 13d] infrastructure/network-map.md
|
||||
- [ 13d] infrastructure/proxmox.md
|
||||
- [ 1d] infrastructure/tailscale.md
|
||||
- [ 15d] queries/retained-research-answers.md
|
||||
- [ 2d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 6d] reports/uber-eats/2026-07.md
|
||||
- [ 6d] reports/uber-eats/2026-08.md
|
||||
- [ 6d] reports/uber-eats/2026-09.md
|
||||
- [ 6d] reports/uber-eats/index.md
|
||||
- [ 7d] runbooks/backup-wiki.md
|
||||
- [ 7d] runbooks/diagnose-docker.md
|
||||
- [ 7d] runbooks/gateway-resource-resilience.md
|
||||
- [ 0d] runbooks/job-radar.md
|
||||
- [ 7d] runbooks/multiplexer-setup.md
|
||||
- [ 1d] runbooks/provider-health.md
|
||||
- [ 7d] runbooks/recover-docker-service.md
|
||||
- [ 7d] runbooks/restart-browser.md
|
||||
- [ 7d] runbooks/restart-hermes.md
|
||||
- [ 7d] runbooks/restore-openrouter.md
|
||||
- [ 7d] runbooks/test-openrouter.md
|
||||
- [ 7d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 1d] runbooks/update-hermes-safely.md
|
||||
- [ 14d] systems/browser-backend.md
|
||||
- [ 7d] systems/cloudflare-tunnel.md
|
||||
- [ 1d] systems/conduit.md
|
||||
- [ 1d] systems/current-profile.md
|
||||
- [ 7d] systems/guanaco.md
|
||||
- [ 7d] systems/headless-server.md
|
||||
- [ 1d] systems/hermes-agent.md
|
||||
- [ 7d] systems/hermes-ui.md
|
||||
- [ 15d] systems/home-assistant.md
|
||||
- [ 2d] systems/homepage-dashboard.md
|
||||
- [ 16d] systems/ignis.md
|
||||
- [ 1d] systems/memory-backend.md
|
||||
- [ 1d] systems/messaging-integrations.md
|
||||
- [ 3d] systems/mnemosyne.md
|
||||
- [ 1d] systems/model-providers.md
|
||||
- [ 15d] systems/nanobot.md
|
||||
- [ 7d] systems/nextcloud.md
|
||||
- [ 7d] systems/obsidian.md
|
||||
- [ 5d] systems/omniroute.md
|
||||
- [ 7d] systems/opencode-go.md
|
||||
- [ 7d] systems/openrouter.md
|
||||
- [ 16d] systems/paseo.md
|
||||
- [ 8d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 7d] systems/terminal-backend.md
|
||||
- [ 7d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (4)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- index.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 0 stale, 104 fresh, 4 no-date of 108 pages
|
||||
@@ -0,0 +1,122 @@
|
||||
# Wiki freshness audit — 2026-10-28
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (2)
|
||||
|
||||
- [ 22d] incidents/2026-10-01-cliproxy-connection-auth-expired.md (updated 2026-10-06)
|
||||
- [ 22d] incidents/2026-10-05-dead-model-pin-repin.md (updated 2026-10-06)
|
||||
|
||||
## FRESH (103)
|
||||
|
||||
- [ 8d] CONTRIBUTING.md
|
||||
- [ 2d] agents/hermes-chaos.md
|
||||
- [ 2d] agents/hermes-production.md
|
||||
- [ 8d] agents/index.md
|
||||
- [ 17d] agents/nanobot.md
|
||||
- [ 8d] agents/open-claw.md
|
||||
- [ 17d] agents/qwenpaw.md
|
||||
- [ 8d] agents/zeroclaw.md
|
||||
- [ 8d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 8d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 8d] comparisons/memory-backends.md
|
||||
- [ 8d] comparisons/model-routers.md
|
||||
- [ 15d] comparisons/steel-vs-camoufox.md
|
||||
- [ 8d] concepts/agent-memory.md
|
||||
- [ 15d] concepts/browser-automation.md
|
||||
- [ 8d] concepts/model-routing.md
|
||||
- [ 8d] concepts/persistent-shells.md
|
||||
- [ 8d] concepts/tool-calling.md
|
||||
- [ 0d] current-state.md
|
||||
- [ 8d] decisions/index.md
|
||||
- [ 8d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 8d] decisions/persistent-shell-enabled.md
|
||||
- [ 8d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 8d] experiments/index.md
|
||||
- [ 4d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 8d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 4d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 6d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 5d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 16d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 3d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 8d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 7d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 17d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 2d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 2d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 4d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 19d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 12d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 1d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 9d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 1d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 6d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 3d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 2d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 2d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 2d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 0d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 6d] infrastructure/backups.md
|
||||
- [ 8d] infrastructure/credentials-map.md
|
||||
- [ 17d] infrastructure/docker-services.md
|
||||
- [ 8d] infrastructure/domains-and-tunnels.md
|
||||
- [ 2d] infrastructure/hosts.md
|
||||
- [ 14d] infrastructure/network-map.md
|
||||
- [ 14d] infrastructure/proxmox.md
|
||||
- [ 0d] infrastructure/tailscale.md
|
||||
- [ 16d] queries/retained-research-answers.md
|
||||
- [ 3d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 7d] reports/uber-eats/2026-07.md
|
||||
- [ 7d] reports/uber-eats/2026-08.md
|
||||
- [ 7d] reports/uber-eats/2026-09.md
|
||||
- [ 7d] reports/uber-eats/index.md
|
||||
- [ 8d] runbooks/backup-wiki.md
|
||||
- [ 8d] runbooks/diagnose-docker.md
|
||||
- [ 8d] runbooks/gateway-resource-resilience.md
|
||||
- [ 0d] runbooks/job-radar.md
|
||||
- [ 8d] runbooks/multiplexer-setup.md
|
||||
- [ 2d] runbooks/provider-health.md
|
||||
- [ 8d] runbooks/recover-docker-service.md
|
||||
- [ 8d] runbooks/restart-browser.md
|
||||
- [ 8d] runbooks/restart-hermes.md
|
||||
- [ 8d] runbooks/restore-openrouter.md
|
||||
- [ 8d] runbooks/test-openrouter.md
|
||||
- [ 8d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 2d] runbooks/update-hermes-safely.md
|
||||
- [ 15d] systems/browser-backend.md
|
||||
- [ 8d] systems/cloudflare-tunnel.md
|
||||
- [ 0d] systems/conduit.md
|
||||
- [ 2d] systems/current-profile.md
|
||||
- [ 8d] systems/guanaco.md
|
||||
- [ 8d] systems/headless-server.md
|
||||
- [ 0d] systems/hermes-agent.md
|
||||
- [ 8d] systems/hermes-ui.md
|
||||
- [ 16d] systems/home-assistant.md
|
||||
- [ 3d] systems/homepage-dashboard.md
|
||||
- [ 17d] systems/ignis.md
|
||||
- [ 2d] systems/memory-backend.md
|
||||
- [ 2d] systems/messaging-integrations.md
|
||||
- [ 4d] systems/mnemosyne.md
|
||||
- [ 2d] systems/model-providers.md
|
||||
- [ 16d] systems/nanobot.md
|
||||
- [ 8d] systems/nextcloud.md
|
||||
- [ 8d] systems/obsidian.md
|
||||
- [ 6d] systems/omniroute.md
|
||||
- [ 8d] systems/opencode-go.md
|
||||
- [ 8d] systems/openrouter.md
|
||||
- [ 17d] systems/paseo.md
|
||||
- [ 9d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 8d] systems/terminal-backend.md
|
||||
- [ 8d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (4)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- index.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 2 stale, 103 fresh, 4 no-date of 109 pages
|
||||
@@ -0,0 +1,123 @@
|
||||
# Wiki freshness audit — 2026-10-29
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (0)
|
||||
|
||||
|
||||
## FRESH (107)
|
||||
|
||||
- [ 9d] CONTRIBUTING.md
|
||||
- [ 3d] agents/hermes-chaos.md
|
||||
- [ 3d] agents/hermes-production.md
|
||||
- [ 9d] agents/index.md
|
||||
- [ 0d] agents/nanobot.md
|
||||
- [ 9d] agents/open-claw.md
|
||||
- [ 18d] agents/qwenpaw.md
|
||||
- [ 9d] agents/zeroclaw.md
|
||||
- [ 0d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 9d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 9d] comparisons/memory-backends.md
|
||||
- [ 9d] comparisons/model-routers.md
|
||||
- [ 0d] comparisons/steel-vs-camoufox.md
|
||||
- [ 9d] concepts/agent-memory.md
|
||||
- [ 0d] concepts/browser-automation.md
|
||||
- [ 9d] concepts/model-routing.md
|
||||
- [ 9d] concepts/persistent-shells.md
|
||||
- [ 9d] concepts/tool-calling.md
|
||||
- [ 0d] current-state.md
|
||||
- [ 0d] decisions/index.md
|
||||
- [ 9d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 9d] decisions/persistent-shell-enabled.md
|
||||
- [ 0d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 9d] experiments/index.md
|
||||
- [ 5d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 9d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 5d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 7d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 6d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 17d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 4d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 9d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 8d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 18d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 0d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 3d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 3d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 0d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 5d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 20d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 13d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 2d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 10d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 2d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 7d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 4d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 3d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 3d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 3d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 1d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 0d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 0d] index.md
|
||||
- [ 7d] infrastructure/backups.md
|
||||
- [ 9d] infrastructure/credentials-map.md
|
||||
- [ 0d] infrastructure/docker-services.md
|
||||
- [ 9d] infrastructure/domains-and-tunnels.md
|
||||
- [ 0d] infrastructure/hosts.md
|
||||
- [ 15d] infrastructure/network-map.md
|
||||
- [ 15d] infrastructure/proxmox.md
|
||||
- [ 0d] infrastructure/tailscale.md
|
||||
- [ 17d] queries/retained-research-answers.md
|
||||
- [ 4d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 8d] reports/uber-eats/2026-07.md
|
||||
- [ 8d] reports/uber-eats/2026-08.md
|
||||
- [ 8d] reports/uber-eats/2026-09.md
|
||||
- [ 8d] reports/uber-eats/index.md
|
||||
- [ 9d] runbooks/backup-wiki.md
|
||||
- [ 9d] runbooks/diagnose-docker.md
|
||||
- [ 9d] runbooks/gateway-resource-resilience.md
|
||||
- [ 0d] runbooks/job-radar.md
|
||||
- [ 9d] runbooks/multiplexer-setup.md
|
||||
- [ 3d] runbooks/provider-health.md
|
||||
- [ 0d] runbooks/recover-docker-service.md
|
||||
- [ 9d] runbooks/restart-browser.md
|
||||
- [ 9d] runbooks/restart-hermes.md
|
||||
- [ 9d] runbooks/restore-openrouter.md
|
||||
- [ 9d] runbooks/test-openrouter.md
|
||||
- [ 9d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 3d] runbooks/update-hermes-safely.md
|
||||
- [ 0d] systems/browser-backend.md
|
||||
- [ 9d] systems/cloudflare-tunnel.md
|
||||
- [ 1d] systems/conduit.md
|
||||
- [ 3d] systems/current-profile.md
|
||||
- [ 9d] systems/guanaco.md
|
||||
- [ 9d] systems/headless-server.md
|
||||
- [ 1d] systems/hermes-agent.md
|
||||
- [ 9d] systems/hermes-ui.md
|
||||
- [ 17d] systems/home-assistant.md
|
||||
- [ 0d] systems/homepage-dashboard.md
|
||||
- [ 18d] systems/ignis.md
|
||||
- [ 3d] systems/memory-backend.md
|
||||
- [ 3d] systems/messaging-integrations.md
|
||||
- [ 5d] systems/mnemosyne.md
|
||||
- [ 3d] systems/model-providers.md
|
||||
- [ 17d] systems/nanobot.md
|
||||
- [ 9d] systems/nextcloud.md
|
||||
- [ 9d] systems/obsidian.md
|
||||
- [ 7d] systems/omniroute.md
|
||||
- [ 9d] systems/opencode-go.md
|
||||
- [ 9d] systems/openrouter.md
|
||||
- [ 18d] systems/paseo.md
|
||||
- [ 10d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 9d] systems/terminal-backend.md
|
||||
- [ 9d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 0 stale, 107 fresh, 3 no-date of 110 pages
|
||||
@@ -0,0 +1,125 @@
|
||||
# Wiki freshness audit — 2026-10-30
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (0)
|
||||
|
||||
|
||||
## FRESH (109)
|
||||
|
||||
- [ 10d] CONTRIBUTING.md
|
||||
- [ 4d] agents/hermes-chaos.md
|
||||
- [ 4d] agents/hermes-production.md
|
||||
- [ 10d] agents/index.md
|
||||
- [ 1d] agents/nanobot.md
|
||||
- [ 10d] agents/open-claw.md
|
||||
- [ 19d] agents/qwenpaw.md
|
||||
- [ 10d] agents/zeroclaw.md
|
||||
- [ 1d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 10d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 10d] comparisons/memory-backends.md
|
||||
- [ 10d] comparisons/model-routers.md
|
||||
- [ 1d] comparisons/steel-vs-camoufox.md
|
||||
- [ 10d] concepts/agent-memory.md
|
||||
- [ 1d] concepts/browser-automation.md
|
||||
- [ 10d] concepts/model-routing.md
|
||||
- [ 10d] concepts/persistent-shells.md
|
||||
- [ 10d] concepts/tool-calling.md
|
||||
- [ 0d] current-state.md
|
||||
- [ 1d] decisions/index.md
|
||||
- [ 10d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 10d] decisions/persistent-shell-enabled.md
|
||||
- [ 1d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 10d] experiments/index.md
|
||||
- [ 6d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 10d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 6d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 8d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 7d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 18d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 5d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 10d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 9d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 19d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 1d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 4d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 4d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 1d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 6d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 21d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 14d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 3d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 11d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 3d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 8d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 5d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 4d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 4d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 4d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 2d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 0d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 0d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 0d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 0d] index.md
|
||||
- [ 8d] infrastructure/backups.md
|
||||
- [ 10d] infrastructure/credentials-map.md
|
||||
- [ 1d] infrastructure/docker-services.md
|
||||
- [ 10d] infrastructure/domains-and-tunnels.md
|
||||
- [ 0d] infrastructure/hosts.md
|
||||
- [ 16d] infrastructure/network-map.md
|
||||
- [ 16d] infrastructure/proxmox.md
|
||||
- [ 0d] infrastructure/tailscale.md
|
||||
- [ 18d] queries/retained-research-answers.md
|
||||
- [ 5d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 9d] reports/uber-eats/2026-07.md
|
||||
- [ 9d] reports/uber-eats/2026-08.md
|
||||
- [ 9d] reports/uber-eats/2026-09.md
|
||||
- [ 9d] reports/uber-eats/index.md
|
||||
- [ 10d] runbooks/backup-wiki.md
|
||||
- [ 10d] runbooks/diagnose-docker.md
|
||||
- [ 10d] runbooks/gateway-resource-resilience.md
|
||||
- [ 0d] runbooks/job-radar.md
|
||||
- [ 10d] runbooks/multiplexer-setup.md
|
||||
- [ 4d] runbooks/provider-health.md
|
||||
- [ 1d] runbooks/recover-docker-service.md
|
||||
- [ 10d] runbooks/restart-browser.md
|
||||
- [ 10d] runbooks/restart-hermes.md
|
||||
- [ 10d] runbooks/restore-openrouter.md
|
||||
- [ 10d] runbooks/test-openrouter.md
|
||||
- [ 10d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 4d] runbooks/update-hermes-safely.md
|
||||
- [ 1d] systems/browser-backend.md
|
||||
- [ 10d] systems/cloudflare-tunnel.md
|
||||
- [ 0d] systems/conduit.md
|
||||
- [ 4d] systems/current-profile.md
|
||||
- [ 10d] systems/guanaco.md
|
||||
- [ 10d] systems/headless-server.md
|
||||
- [ 0d] systems/hermes-agent.md
|
||||
- [ 10d] systems/hermes-ui.md
|
||||
- [ 18d] systems/home-assistant.md
|
||||
- [ 0d] systems/homepage-dashboard.md
|
||||
- [ 19d] systems/ignis.md
|
||||
- [ 4d] systems/memory-backend.md
|
||||
- [ 4d] systems/messaging-integrations.md
|
||||
- [ 6d] systems/mnemosyne.md
|
||||
- [ 4d] systems/model-providers.md
|
||||
- [ 18d] systems/nanobot.md
|
||||
- [ 10d] systems/nextcloud.md
|
||||
- [ 10d] systems/obsidian.md
|
||||
- [ 8d] systems/omniroute.md
|
||||
- [ 10d] systems/opencode-go.md
|
||||
- [ 10d] systems/openrouter.md
|
||||
- [ 19d] systems/paseo.md
|
||||
- [ 11d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 10d] systems/terminal-backend.md
|
||||
- [ 10d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 0 stale, 109 fresh, 3 no-date of 112 pages
|
||||
@@ -0,0 +1,127 @@
|
||||
# Wiki freshness audit — 2026-10-31
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (1)
|
||||
|
||||
- [ 22d] incidents/2026-10-09-memory-recall-relevance.md (updated 2026-10-09)
|
||||
|
||||
## FRESH (110)
|
||||
|
||||
- [ 11d] CONTRIBUTING.md
|
||||
- [ 5d] agents/hermes-chaos.md
|
||||
- [ 5d] agents/hermes-production.md
|
||||
- [ 11d] agents/index.md
|
||||
- [ 2d] agents/nanobot.md
|
||||
- [ 11d] agents/open-claw.md
|
||||
- [ 20d] agents/qwenpaw.md
|
||||
- [ 11d] agents/zeroclaw.md
|
||||
- [ 2d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 11d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 11d] comparisons/memory-backends.md
|
||||
- [ 11d] comparisons/model-routers.md
|
||||
- [ 2d] comparisons/steel-vs-camoufox.md
|
||||
- [ 11d] concepts/agent-memory.md
|
||||
- [ 2d] concepts/browser-automation.md
|
||||
- [ 11d] concepts/model-routing.md
|
||||
- [ 11d] concepts/persistent-shells.md
|
||||
- [ 11d] concepts/tool-calling.md
|
||||
- [ 0d] current-state.md
|
||||
- [ 2d] decisions/index.md
|
||||
- [ 11d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 11d] decisions/persistent-shell-enabled.md
|
||||
- [ 2d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 11d] experiments/index.md
|
||||
- [ 7d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 11d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 7d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 9d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 8d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 19d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 6d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 11d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 10d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 20d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 2d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 5d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 5d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 2d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 7d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 15d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 4d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 12d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 4d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 9d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 6d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 5d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 5d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 5d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 3d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 1d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 1d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 1d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 0d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 1d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 0d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 0d] index.md
|
||||
- [ 9d] infrastructure/backups.md
|
||||
- [ 11d] infrastructure/credentials-map.md
|
||||
- [ 2d] infrastructure/docker-services.md
|
||||
- [ 11d] infrastructure/domains-and-tunnels.md
|
||||
- [ 0d] infrastructure/hosts.md
|
||||
- [ 17d] infrastructure/network-map.md
|
||||
- [ 17d] infrastructure/proxmox.md
|
||||
- [ 1d] infrastructure/tailscale.md
|
||||
- [ 19d] queries/retained-research-answers.md
|
||||
- [ 6d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 10d] reports/uber-eats/2026-07.md
|
||||
- [ 10d] reports/uber-eats/2026-08.md
|
||||
- [ 10d] reports/uber-eats/2026-09.md
|
||||
- [ 10d] reports/uber-eats/index.md
|
||||
- [ 11d] runbooks/backup-wiki.md
|
||||
- [ 11d] runbooks/diagnose-docker.md
|
||||
- [ 11d] runbooks/gateway-resource-resilience.md
|
||||
- [ 1d] runbooks/job-radar.md
|
||||
- [ 11d] runbooks/multiplexer-setup.md
|
||||
- [ 5d] runbooks/provider-health.md
|
||||
- [ 2d] runbooks/recover-docker-service.md
|
||||
- [ 11d] runbooks/restart-browser.md
|
||||
- [ 11d] runbooks/restart-hermes.md
|
||||
- [ 11d] runbooks/restore-openrouter.md
|
||||
- [ 11d] runbooks/test-openrouter.md
|
||||
- [ 11d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 5d] runbooks/update-hermes-safely.md
|
||||
- [ 2d] systems/browser-backend.md
|
||||
- [ 11d] systems/cloudflare-tunnel.md
|
||||
- [ 1d] systems/conduit.md
|
||||
- [ 5d] systems/current-profile.md
|
||||
- [ 11d] systems/guanaco.md
|
||||
- [ 11d] systems/headless-server.md
|
||||
- [ 1d] systems/hermes-agent.md
|
||||
- [ 11d] systems/hermes-ui.md
|
||||
- [ 19d] systems/home-assistant.md
|
||||
- [ 1d] systems/homepage-dashboard.md
|
||||
- [ 20d] systems/ignis.md
|
||||
- [ 5d] systems/memory-backend.md
|
||||
- [ 5d] systems/messaging-integrations.md
|
||||
- [ 0d] systems/mnemosyne.md
|
||||
- [ 5d] systems/model-providers.md
|
||||
- [ 19d] systems/nanobot.md
|
||||
- [ 11d] systems/nextcloud.md
|
||||
- [ 11d] systems/obsidian.md
|
||||
- [ 9d] systems/omniroute.md
|
||||
- [ 11d] systems/opencode-go.md
|
||||
- [ 11d] systems/openrouter.md
|
||||
- [ 20d] systems/paseo.md
|
||||
- [ 12d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 11d] systems/terminal-backend.md
|
||||
- [ 11d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 1 stale, 110 fresh, 3 no-date of 114 pages
|
||||
@@ -0,0 +1,130 @@
|
||||
# Wiki freshness audit — 2026-11-01
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (0)
|
||||
|
||||
|
||||
## FRESH (114)
|
||||
|
||||
- [ 12d] CONTRIBUTING.md
|
||||
- [ 6d] agents/hermes-chaos.md
|
||||
- [ 6d] agents/hermes-production.md
|
||||
- [ 12d] agents/index.md
|
||||
- [ 3d] agents/nanobot.md
|
||||
- [ 12d] agents/open-claw.md
|
||||
- [ 21d] agents/qwenpaw.md
|
||||
- [ 12d] agents/zeroclaw.md
|
||||
- [ 3d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 12d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 12d] comparisons/memory-backends.md
|
||||
- [ 12d] comparisons/model-routers.md
|
||||
- [ 3d] comparisons/steel-vs-camoufox.md
|
||||
- [ 12d] concepts/agent-memory.md
|
||||
- [ 3d] concepts/browser-automation.md
|
||||
- [ 12d] concepts/model-routing.md
|
||||
- [ 12d] concepts/persistent-shells.md
|
||||
- [ 12d] concepts/tool-calling.md
|
||||
- [ 1d] current-state.md
|
||||
- [ 3d] decisions/index.md
|
||||
- [ 12d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 12d] decisions/persistent-shell-enabled.md
|
||||
- [ 3d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 12d] experiments/index.md
|
||||
- [ 8d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 12d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 8d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 10d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 9d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 20d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 7d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 12d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 11d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 21d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 3d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 6d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 6d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 3d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 8d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 0d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 16d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 5d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 13d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 5d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 10d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 7d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 6d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 6d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 6d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 4d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 2d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 2d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 2d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 1d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 2d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 1d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 0d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 0d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 0d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 0d] index.md
|
||||
- [ 10d] infrastructure/backups.md
|
||||
- [ 12d] infrastructure/credentials-map.md
|
||||
- [ 0d] infrastructure/docker-services.md
|
||||
- [ 12d] infrastructure/domains-and-tunnels.md
|
||||
- [ 1d] infrastructure/hosts.md
|
||||
- [ 18d] infrastructure/network-map.md
|
||||
- [ 0d] infrastructure/proxmox.md
|
||||
- [ 0d] infrastructure/tailscale.md
|
||||
- [ 20d] queries/retained-research-answers.md
|
||||
- [ 7d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 11d] reports/uber-eats/2026-07.md
|
||||
- [ 11d] reports/uber-eats/2026-08.md
|
||||
- [ 11d] reports/uber-eats/2026-09.md
|
||||
- [ 11d] reports/uber-eats/index.md
|
||||
- [ 12d] runbooks/backup-wiki.md
|
||||
- [ 12d] runbooks/diagnose-docker.md
|
||||
- [ 12d] runbooks/gateway-resource-resilience.md
|
||||
- [ 2d] runbooks/job-radar.md
|
||||
- [ 12d] runbooks/multiplexer-setup.md
|
||||
- [ 6d] runbooks/provider-health.md
|
||||
- [ 3d] runbooks/recover-docker-service.md
|
||||
- [ 12d] runbooks/restart-browser.md
|
||||
- [ 12d] runbooks/restart-hermes.md
|
||||
- [ 12d] runbooks/restore-openrouter.md
|
||||
- [ 12d] runbooks/test-openrouter.md
|
||||
- [ 12d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 6d] runbooks/update-hermes-safely.md
|
||||
- [ 3d] systems/browser-backend.md
|
||||
- [ 12d] systems/cloudflare-tunnel.md
|
||||
- [ 0d] systems/conduit.md
|
||||
- [ 6d] systems/current-profile.md
|
||||
- [ 12d] systems/guanaco.md
|
||||
- [ 12d] systems/headless-server.md
|
||||
- [ 2d] systems/hermes-agent.md
|
||||
- [ 12d] systems/hermes-ui.md
|
||||
- [ 20d] systems/home-assistant.md
|
||||
- [ 2d] systems/homepage-dashboard.md
|
||||
- [ 21d] systems/ignis.md
|
||||
- [ 6d] systems/memory-backend.md
|
||||
- [ 0d] systems/messaging-integrations.md
|
||||
- [ 0d] systems/mnemosyne.md
|
||||
- [ 6d] systems/model-providers.md
|
||||
- [ 20d] systems/nanobot.md
|
||||
- [ 12d] systems/nextcloud.md
|
||||
- [ 12d] systems/obsidian.md
|
||||
- [ 0d] systems/omniroute.md
|
||||
- [ 12d] systems/opencode-go.md
|
||||
- [ 12d] systems/openrouter.md
|
||||
- [ 21d] systems/paseo.md
|
||||
- [ 13d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 12d] systems/terminal-backend.md
|
||||
- [ 12d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 0 stale, 114 fresh, 3 no-date of 117 pages
|
||||
@@ -0,0 +1,133 @@
|
||||
# Wiki freshness audit — 2026-11-02
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (4)
|
||||
|
||||
- [ 22d] agents/qwenpaw.md (updated 2026-10-11)
|
||||
- [ 22d] incidents/2026-09-28-hermes-managed-runtime-deps.md (updated 2026-10-11)
|
||||
- [ 22d] systems/ignis.md (updated 2026-10-11)
|
||||
- [ 22d] systems/paseo.md (updated 2026-10-11)
|
||||
|
||||
## FRESH (113)
|
||||
|
||||
- [ 13d] CONTRIBUTING.md
|
||||
- [ 7d] agents/hermes-chaos.md
|
||||
- [ 7d] agents/hermes-production.md
|
||||
- [ 13d] agents/index.md
|
||||
- [ 4d] agents/nanobot.md
|
||||
- [ 13d] agents/open-claw.md
|
||||
- [ 13d] agents/zeroclaw.md
|
||||
- [ 4d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 13d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 13d] comparisons/memory-backends.md
|
||||
- [ 13d] comparisons/model-routers.md
|
||||
- [ 4d] comparisons/steel-vs-camoufox.md
|
||||
- [ 13d] concepts/agent-memory.md
|
||||
- [ 4d] concepts/browser-automation.md
|
||||
- [ 13d] concepts/model-routing.md
|
||||
- [ 13d] concepts/persistent-shells.md
|
||||
- [ 13d] concepts/tool-calling.md
|
||||
- [ 2d] current-state.md
|
||||
- [ 4d] decisions/index.md
|
||||
- [ 13d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 13d] decisions/persistent-shell-enabled.md
|
||||
- [ 4d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 13d] experiments/index.md
|
||||
- [ 9d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 13d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 9d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 11d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 10d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 21d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 8d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 13d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 12d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 4d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 7d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 7d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 4d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 9d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 1d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 17d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 6d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 14d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 6d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 11d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 8d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 7d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 7d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 7d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 5d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 3d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 3d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 3d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 2d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 3d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 2d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 1d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 1d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 1d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 0d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||
- [ 0d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||
- [ 0d] index.md
|
||||
- [ 11d] infrastructure/backups.md
|
||||
- [ 13d] infrastructure/credentials-map.md
|
||||
- [ 1d] infrastructure/docker-services.md
|
||||
- [ 13d] infrastructure/domains-and-tunnels.md
|
||||
- [ 2d] infrastructure/hosts.md
|
||||
- [ 19d] infrastructure/network-map.md
|
||||
- [ 1d] infrastructure/proxmox.md
|
||||
- [ 1d] infrastructure/tailscale.md
|
||||
- [ 21d] queries/retained-research-answers.md
|
||||
- [ 8d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 1d] reports/uber-eats/2026-07.md
|
||||
- [ 1d] reports/uber-eats/2026-08.md
|
||||
- [ 1d] reports/uber-eats/2026-09.md
|
||||
- [ 1d] reports/uber-eats/2026-10.md
|
||||
- [ 1d] reports/uber-eats/index.md
|
||||
- [ 13d] runbooks/backup-wiki.md
|
||||
- [ 13d] runbooks/diagnose-docker.md
|
||||
- [ 13d] runbooks/gateway-resource-resilience.md
|
||||
- [ 3d] runbooks/job-radar.md
|
||||
- [ 13d] runbooks/multiplexer-setup.md
|
||||
- [ 7d] runbooks/provider-health.md
|
||||
- [ 4d] runbooks/recover-docker-service.md
|
||||
- [ 13d] runbooks/restart-browser.md
|
||||
- [ 13d] runbooks/restart-hermes.md
|
||||
- [ 13d] runbooks/restore-openrouter.md
|
||||
- [ 13d] runbooks/test-openrouter.md
|
||||
- [ 13d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 7d] runbooks/update-hermes-safely.md
|
||||
- [ 4d] systems/browser-backend.md
|
||||
- [ 13d] systems/cloudflare-tunnel.md
|
||||
- [ 1d] systems/conduit.md
|
||||
- [ 7d] systems/current-profile.md
|
||||
- [ 13d] systems/guanaco.md
|
||||
- [ 13d] systems/headless-server.md
|
||||
- [ 3d] systems/hermes-agent.md
|
||||
- [ 13d] systems/hermes-ui.md
|
||||
- [ 21d] systems/home-assistant.md
|
||||
- [ 3d] systems/homepage-dashboard.md
|
||||
- [ 7d] systems/memory-backend.md
|
||||
- [ 0d] systems/messaging-integrations.md
|
||||
- [ 1d] systems/mnemosyne.md
|
||||
- [ 7d] systems/model-providers.md
|
||||
- [ 21d] systems/nanobot.md
|
||||
- [ 13d] systems/nextcloud.md
|
||||
- [ 13d] systems/obsidian.md
|
||||
- [ 1d] systems/omniroute.md
|
||||
- [ 13d] systems/opencode-go.md
|
||||
- [ 13d] systems/openrouter.md
|
||||
- [ 14d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 13d] systems/terminal-backend.md
|
||||
- [ 13d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 4 stale, 113 fresh, 3 no-date of 120 pages
|
||||
@@ -0,0 +1,134 @@
|
||||
# Wiki freshness audit — 2026-11-03
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (8)
|
||||
|
||||
- [ 23d] agents/qwenpaw.md (updated 2026-10-11)
|
||||
- [ 23d] incidents/2026-09-28-hermes-managed-runtime-deps.md (updated 2026-10-11)
|
||||
- [ 23d] systems/ignis.md (updated 2026-10-11)
|
||||
- [ 23d] systems/paseo.md (updated 2026-10-11)
|
||||
- [ 22d] incidents/2026-08-18-mnemosyne-capture-pipeline.md (updated 2026-10-12)
|
||||
- [ 22d] queries/retained-research-answers.md (updated 2026-10-12)
|
||||
- [ 22d] systems/home-assistant.md (updated 2026-10-12)
|
||||
- [ 22d] systems/nanobot.md (updated 2026-10-12)
|
||||
|
||||
## FRESH (110)
|
||||
|
||||
- [ 14d] CONTRIBUTING.md
|
||||
- [ 8d] agents/hermes-chaos.md
|
||||
- [ 8d] agents/hermes-production.md
|
||||
- [ 14d] agents/index.md
|
||||
- [ 5d] agents/nanobot.md
|
||||
- [ 14d] agents/open-claw.md
|
||||
- [ 14d] agents/zeroclaw.md
|
||||
- [ 5d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 14d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 14d] comparisons/memory-backends.md
|
||||
- [ 14d] comparisons/model-routers.md
|
||||
- [ 5d] comparisons/steel-vs-camoufox.md
|
||||
- [ 14d] concepts/agent-memory.md
|
||||
- [ 5d] concepts/browser-automation.md
|
||||
- [ 14d] concepts/model-routing.md
|
||||
- [ 14d] concepts/persistent-shells.md
|
||||
- [ 14d] concepts/tool-calling.md
|
||||
- [ 3d] current-state.md
|
||||
- [ 5d] decisions/index.md
|
||||
- [ 14d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 14d] decisions/persistent-shell-enabled.md
|
||||
- [ 5d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 14d] experiments/index.md
|
||||
- [ 10d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 14d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 10d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 12d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 11d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 9d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 14d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 13d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 5d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 8d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 8d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 5d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 10d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 2d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 18d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 7d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 15d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 7d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 12d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 9d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 8d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 8d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 8d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 6d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 4d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 4d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 4d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 3d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 4d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 3d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 2d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 2d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 2d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 1d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||
- [ 0d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||
- [ 0d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||
- [ 0d] index.md
|
||||
- [ 12d] infrastructure/backups.md
|
||||
- [ 14d] infrastructure/credentials-map.md
|
||||
- [ 2d] infrastructure/docker-services.md
|
||||
- [ 14d] infrastructure/domains-and-tunnels.md
|
||||
- [ 3d] infrastructure/hosts.md
|
||||
- [ 20d] infrastructure/network-map.md
|
||||
- [ 2d] infrastructure/proxmox.md
|
||||
- [ 2d] infrastructure/tailscale.md
|
||||
- [ 9d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 2d] reports/uber-eats/2026-07.md
|
||||
- [ 2d] reports/uber-eats/2026-08.md
|
||||
- [ 2d] reports/uber-eats/2026-09.md
|
||||
- [ 2d] reports/uber-eats/2026-10.md
|
||||
- [ 2d] reports/uber-eats/index.md
|
||||
- [ 14d] runbooks/backup-wiki.md
|
||||
- [ 14d] runbooks/diagnose-docker.md
|
||||
- [ 14d] runbooks/gateway-resource-resilience.md
|
||||
- [ 4d] runbooks/job-radar.md
|
||||
- [ 14d] runbooks/multiplexer-setup.md
|
||||
- [ 8d] runbooks/provider-health.md
|
||||
- [ 5d] runbooks/recover-docker-service.md
|
||||
- [ 14d] runbooks/restart-browser.md
|
||||
- [ 14d] runbooks/restart-hermes.md
|
||||
- [ 14d] runbooks/restore-openrouter.md
|
||||
- [ 14d] runbooks/test-openrouter.md
|
||||
- [ 14d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 8d] runbooks/update-hermes-safely.md
|
||||
- [ 0d] systems/browser-backend.md
|
||||
- [ 14d] systems/cloudflare-tunnel.md
|
||||
- [ 2d] systems/conduit.md
|
||||
- [ 8d] systems/current-profile.md
|
||||
- [ 14d] systems/guanaco.md
|
||||
- [ 14d] systems/headless-server.md
|
||||
- [ 4d] systems/hermes-agent.md
|
||||
- [ 14d] systems/hermes-ui.md
|
||||
- [ 4d] systems/homepage-dashboard.md
|
||||
- [ 8d] systems/memory-backend.md
|
||||
- [ 1d] systems/messaging-integrations.md
|
||||
- [ 2d] systems/mnemosyne.md
|
||||
- [ 8d] systems/model-providers.md
|
||||
- [ 14d] systems/nextcloud.md
|
||||
- [ 14d] systems/obsidian.md
|
||||
- [ 2d] systems/omniroute.md
|
||||
- [ 14d] systems/opencode-go.md
|
||||
- [ 14d] systems/openrouter.md
|
||||
- [ 15d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 14d] systems/terminal-backend.md
|
||||
- [ 14d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 8 stale, 110 fresh, 3 no-date of 121 pages
|
||||
@@ -0,0 +1,136 @@
|
||||
# Wiki freshness audit — 2026-11-04
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (8)
|
||||
|
||||
- [ 24d] agents/qwenpaw.md (updated 2026-10-11)
|
||||
- [ 24d] incidents/2026-09-28-hermes-managed-runtime-deps.md (updated 2026-10-11)
|
||||
- [ 24d] systems/ignis.md (updated 2026-10-11)
|
||||
- [ 24d] systems/paseo.md (updated 2026-10-11)
|
||||
- [ 23d] incidents/2026-08-18-mnemosyne-capture-pipeline.md (updated 2026-10-12)
|
||||
- [ 23d] queries/retained-research-answers.md (updated 2026-10-12)
|
||||
- [ 23d] systems/home-assistant.md (updated 2026-10-12)
|
||||
- [ 23d] systems/nanobot.md (updated 2026-10-12)
|
||||
|
||||
## FRESH (112)
|
||||
|
||||
- [ 15d] CONTRIBUTING.md
|
||||
- [ 9d] agents/hermes-chaos.md
|
||||
- [ 9d] agents/hermes-production.md
|
||||
- [ 15d] agents/index.md
|
||||
- [ 6d] agents/nanobot.md
|
||||
- [ 15d] agents/open-claw.md
|
||||
- [ 15d] agents/zeroclaw.md
|
||||
- [ 6d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 15d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 15d] comparisons/memory-backends.md
|
||||
- [ 15d] comparisons/model-routers.md
|
||||
- [ 6d] comparisons/steel-vs-camoufox.md
|
||||
- [ 15d] concepts/agent-memory.md
|
||||
- [ 6d] concepts/browser-automation.md
|
||||
- [ 15d] concepts/model-routing.md
|
||||
- [ 15d] concepts/persistent-shells.md
|
||||
- [ 15d] concepts/tool-calling.md
|
||||
- [ 4d] current-state.md
|
||||
- [ 6d] decisions/index.md
|
||||
- [ 15d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 15d] decisions/persistent-shell-enabled.md
|
||||
- [ 6d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 15d] experiments/index.md
|
||||
- [ 11d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 15d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 11d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 13d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 12d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 10d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 15d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 14d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 6d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 9d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 9d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 6d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 11d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 3d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 19d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 8d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 16d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 8d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 13d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 10d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 9d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 9d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 9d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 7d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 5d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 5d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 5d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 4d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 5d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 4d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 3d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 3d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 3d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 2d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||
- [ 1d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||
- [ 0d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||
- [ 0d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||
- [ 0d] index.md
|
||||
- [ 13d] infrastructure/backups.md
|
||||
- [ 15d] infrastructure/credentials-map.md
|
||||
- [ 3d] infrastructure/docker-services.md
|
||||
- [ 15d] infrastructure/domains-and-tunnels.md
|
||||
- [ 4d] infrastructure/hosts.md
|
||||
- [ 21d] infrastructure/network-map.md
|
||||
- [ 3d] infrastructure/proxmox.md
|
||||
- [ 3d] infrastructure/tailscale.md
|
||||
- [ 10d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 3d] reports/uber-eats/2026-07.md
|
||||
- [ 3d] reports/uber-eats/2026-08.md
|
||||
- [ 3d] reports/uber-eats/2026-09.md
|
||||
- [ 3d] reports/uber-eats/2026-10.md
|
||||
- [ 3d] reports/uber-eats/index.md
|
||||
- [ 15d] runbooks/backup-wiki.md
|
||||
- [ 15d] runbooks/diagnose-docker.md
|
||||
- [ 15d] runbooks/gateway-resource-resilience.md
|
||||
- [ 5d] runbooks/job-radar.md
|
||||
- [ 15d] runbooks/multiplexer-setup.md
|
||||
- [ 9d] runbooks/provider-health.md
|
||||
- [ 6d] runbooks/recover-docker-service.md
|
||||
- [ 15d] runbooks/restart-browser.md
|
||||
- [ 15d] runbooks/restart-hermes.md
|
||||
- [ 15d] runbooks/restore-openrouter.md
|
||||
- [ 15d] runbooks/test-openrouter.md
|
||||
- [ 15d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 9d] runbooks/update-hermes-safely.md
|
||||
- [ 1d] systems/browser-backend.md
|
||||
- [ 15d] systems/cloudflare-tunnel.md
|
||||
- [ 3d] systems/conduit.md
|
||||
- [ 9d] systems/current-profile.md
|
||||
- [ 15d] systems/guanaco.md
|
||||
- [ 15d] systems/headless-server.md
|
||||
- [ 5d] systems/hermes-agent.md
|
||||
- [ 0d] systems/hermes-hq-push.md
|
||||
- [ 15d] systems/hermes-ui.md
|
||||
- [ 5d] systems/homepage-dashboard.md
|
||||
- [ 9d] systems/memory-backend.md
|
||||
- [ 2d] systems/messaging-integrations.md
|
||||
- [ 3d] systems/mnemosyne.md
|
||||
- [ 9d] systems/model-providers.md
|
||||
- [ 15d] systems/nextcloud.md
|
||||
- [ 15d] systems/obsidian.md
|
||||
- [ 3d] systems/omniroute.md
|
||||
- [ 15d] systems/opencode-go.md
|
||||
- [ 15d] systems/openrouter.md
|
||||
- [ 16d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 15d] systems/terminal-backend.md
|
||||
- [ 15d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 8 stale, 112 fresh, 3 no-date of 123 pages
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user