Stocktake 2026-10-30: 0 stale by age, but 5 pages drifted from live state

Ran wiki-maintain.sh (skills-index regenerated: 302 skills / 69 categories,
unchanged; freshness report 0 stale / 109 fresh / 3 no-date of 112 pages;
Quartz rebuilt and deployed).

Freshness is not accuracy. A live-vs-page diff found real drift on five pages.
All corrections verified against live sources first.

- infrastructure/tailscale: node table rebuilt from tailscale status --json.
  50 non-funnel nodes (49 IPv4 peers + hermes), not 54. 2 IP changes, 4 peers
  expired out of the tailnet. Zero IP mismatches, zero missing/extra rows on
  re-verification. m93p records online (peer record chronically stale, third
  occurrence -- ping + SSH confirm). CT460 :443 route set (12 paths) and
  CT201 :443/:8443 tables re-probed and match.
- systems/scheduled-tasks: fleet is 29 jobs (25 enabled / 4 disabled), not
  28/25/3. Mnemosyne Health Monitor ea79a2107e36 was auto-paused 2026-10-30
  01:04:22 after its 00:01:08 slot closed "unknown" on a gateway-connection
  failure. Non-clean last_status is 4, not 3. "Failing (delivery only)" is 2.
  Seven job rows carried stale error claims for jobs since cleared to ok --
  each marked superseded with a dated correction, not rewritten (SCHEMA rule 9).
- NEW incidents/2026-10-30-mnemosyne-health-monitor-paused: the job's own
  failure notice was never delivered (deliveries.db has zero rows for
  2026-10-30), so the incident row is the only trace. Job not re-enabled and
  gateway not restarted by this run; root cause recorded as unverified.
- infrastructure/hosts: CT201 LAN is 192.168.178.77 (was .166); CT450 Hermes
  is v0.22.5 / bf0203a794a0, same build as CT460 (page said v0.22.3 /
  5b147ab036d6); cron count 29/25/4; orphaned "omni" peer has expired out of
  the tailnet entirely. Skills count of 302 left alone -- it is correct.
- systems/homepage-dashboard: live app count is 123 (was 114); grid is a
  responsive repeat(auto-fill, minmax(230px, 1fr)), not a fixed 5-column grid;
  tab bar is built from the service groups (renderTabs iterates state.groups),
  so the settings.yaml layout block and its tab:/columns: keys are inert --
  there is no "Overview" tab; saved theme is Boarding Pass / light, not dark
  amber; manager.py runs inside the homepage-manager container; and the "no
  Traefik container" evidence was wrong -- the container exists but is
  stopped (Exited (255) 2 months ago). The conclusion that Traefik does not
  occupy 443 stands. Same evidence error corrected in tailscale.md.

Also corrected two of my own first-pass errors before committing: the
homepage tab list (groups vs tabs) and the CT450 version (a stale root-level
/usr/local/bin/hermes reports v0.19.0; the live agent is the hermes-user
install at v0.22.5).

No secrets written -- provider names, key names and HTTP status codes only.
This commit is contained in:
Tony0410
2026-10-30 05:41:11 +08:00
parent 1d335ca8e6
commit 987eba72e6
10 changed files with 448 additions and 73 deletions
+17 -5
View File
@@ -39,11 +39,23 @@ Updated: 2026-10-30
then no-oped as designed (10 m 46 s / 1 m 51 s). then no-oped as designed (10 m 46 s / 1 m 51 s).
→ [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]],
[[runbooks/job-radar]] [[runbooks/job-radar]]
- **Cron: 2026-10-29 settled 176 `completed` / 0 `failed`** — the first zero-failure day since 2026-10-24 — with - **Cron: 2026-10-29 settled 177 `completed` / 0 `failed`** — the first zero-failure day since 2026-10-24 — with
**10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; every `failure_streak` **10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; unclosed incident rows
reads **0**; unclosed incident rows **4 → 1**. Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**, pins **4 → 1** (the two radar rows closed `resolved` at 10:16:12). Fleet unchanged at **29 jobs (26 enabled / 3
unchanged (19 of the 26 enabled jobs on `custom:omni-101`: 12 `smart-route`, 6 `free-stack`, 1 explicit; 7 disabled)** at that audit.
unpinned on the global default). - **Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure
notice was never delivered.** `ea79a2107e36`'s 00:01:08 slot closed **`unknown`** at 01:04:22 on a
gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`:
Gateway connection failed; no local fallback"*), opening incident `ea79a2_d93c3acdc4f2` (`detected`), and the
scheduler **auto-paused the job**. Fleet is now **29 jobs (25 enabled / 4 disabled)**; non-clean
`last_status` is **4** (2 `error` + 2 `delivery_failed`); `ea79a2107e36` carries `failure_streak: 1`.
⚠ **The failure notice itself failed to send** — `hermes cron list` reports `send_path_degraded; live adapter
delivery to telegram:-1004321904721:102 failed`, and `deliveries.db` holds **zero rows for 2026-10-30**, so
the only trace is the incident row. **Unclosed incident rows 1 → 2.** Every other previously-red job
(`dca8482e4f76`, `1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`,
`c0a9fa314542`) has **cleared to `ok` with streak 0** on its own slot. Pins: **18 of the 25 enabled jobs pin
`custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); 7 enabled rows are `no_agent`
jobs with no pin. → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST) ## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST)
@@ -0,0 +1,124 @@
---
title: Mnemosyne Health Monitor auto-paused after gateway-connection failure
type: incident
status: active
created: 2026-10-30
updated: 2026-10-30
verified_on: 2026-10-30
confidence: high
tags: [cron, mnemosyne, gateway, incident, delivery]
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
---
# Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30)
**Status: ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
## Summary
The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its
2026-10-30 00:01 slot with a **gateway-connection** fault. The run closed with status **`unknown`**,
the scheduler **auto-paused the job**, and the job's own failure notice **failed to send** — so the
only trace of the fault is the incident row in `cron_incidents`.
## What happened
| Field | Value |
|---|---|
| Job | `ea79a2107e36` — Mnemosyne Health Monitor |
| Schedule | `0 */12 * * *` (every 12 h) |
| Started | 2026-10-30 00:01:08.343024+08:00 |
| Finished | 2026-10-30 01:04:22.510091+08:00 (1 h 03 m 14 s) |
| Execution status | **`unknown`** |
| Job `last_status` | **`error`**, `failure_streak: 1` |
| Job `enabled` | **`false`** — auto-paused 2026-10-30 01:04:22 |
| Incident | `ea79a2_d93c3acdc4f2`, `failure_type: unknown`, state **`detected`** |
| Delivery | **failed — never sent** |
Error text on the execution row:
> Cron execution unverified; reconcile
> `/home/hermes/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`:
> Gateway connection failed; no local fallback
## The two traps in this one fault
### 1. A failure notice that was itself never delivered
`hermes cron list` reports for this job:
> The result was not delivered: live adapter send failed: send_path_degraded;
> live adapter delivery to `telegram:-1004321904721:102` failed
Confirmed at the artefact level: `deliveries.db` contains **zero rows created on 2026-10-30**
(2026-10-29 has 10 rows, all `delivered`). So the failure produced **no Telegram output at all** —
not even a notice about the failure.
This is the same class as [[incidents/2026-08-08-gitea-config-backup-push-dead]]: a job whose green
row hides a broken destination. Here the destination was broken *and* the row went red, but the
**alert path** was the part that was dead. If Anthony had only watched `hermes cron list` for a
non-clean row, he would have seen the job had paused; if he had only watched Telegram, he would have
seen nothing at all.
### 2. `unknown` is not `failed`
The execution closed `unknown`, not `failed`. Per the wiki's own convention
([[systems/scheduled-tasks]]), a `status: unknown` row with **no matching `deliveries.db` row** is a
probable non-delivery that the dashboard hides behind the previous slot's error. The scheduler then
auto-paused the job — so the fleet lost a **monitoring** job, not a reporting job.
## Root cause
**Unverified.** The error names a gateway-connection failure with "no local fallback" during the
reconciliation of an admission record. The admission file referenced in the error is
`~/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`.
This is **not** the same failure as the 2026-10-26 planned-restart orphan class
([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]), though the error string is similar
("Cron execution unverified; reconcile …"). The 10-26 rows were `alerted` and auto-closed; this row
is still `detected` and the job stayed paused. **No gateway restart was performed on 2026-10-30**
around that window, so the trigger is not established.
## Current state (verified 2026-10-30 03:3x AWST)
- Live `jobs.json`: `ea79a2107e36` → `enabled: false`, `last_status: error`, `failure_streak: 1`,
`paused_reason` = the gateway-connection text above.
- Live `cron_incidents` (table lives inside `executions.db`): `ea79a2_d93c3acdc4f2` is the **only**
new row since 2026-10-29; `closed_at` is **NULL**, `acked_at` is **NULL**, `alerted_at` is **NULL**.
- The other unclosed row is `0e753e_e7f8194a7d06` (Facebook monthly, `failure_type: agent`, `detected`
since 2026-10-01, clears on its 2026-11-01 slot) — **2 unclosed rows in total**.
- Fleet is now **29 jobs — 25 enabled / 4 disabled**; this job is the 4th disabled one.
- The rest of the fleet was healthy in the same window: 2026-10-29 settled **177 `completed` / 0
`failed`**, and 2026-10-30 (to 05:4x) shows 37 `completed`, 1 `running` (this stocktake), 1
`unknown` (this job), 0 `failed`.
## Actions taken
1. Recorded the incident and its two distinct failure modes here.
2. Updated [[systems/scheduled-tasks]] — the job's row, the Disabled Jobs table, the fleet counts
(29/25/4), the non-clean `last_status` count (3 → 4), the "Failing (last run failed outright)"
bullet (0 → 1), and the "Failing (delivery only)" bullet (1 → 2).
3. Updated [[infrastructure/hosts]] for the cron count.
## Actions NOT taken — and why
- **The job was not re-enabled.** Re-enabling a job whose gateway connection failed an hour ago
risks an immediate re-pause, and it would close the incident's purpose. The recovery belongs to
whoever owns the gateway fix.
- **No gateway restart was performed.** This was a stocktake run, and restarting the gateway is a
disruptive action with its own orphan-risk class ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]).
The fault is recorded, not papered over.
## Next step
Re-enable `ea79a2107e36` **after** the gateway connection is confirmed healthy, then confirm the
run completes and — critically — that a delivery row appears in `deliveries.db`. The point of this
monitor is to report on exception to topic `102`; if the topic path is still degraded, re-enabling it
recreates a silent monitor.
## Related
- [[systems/scheduled-tasks]] — fleet ledger
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — the earlier "execution unverified" class
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — green row, dead destination
- [[systems/mnemosyne]] — the subsystem being monitored
+3 -2
View File
File diff suppressed because one or more lines are too long
+6 -6
View File
@@ -3,8 +3,8 @@ title: Known Hosts and Paths
type: infrastructure type: infrastructure
status: partial status: partial
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-29 updated: 2026-10-30
verified_on: 2026-10-29 verified_on: 2026-10-30
confidence: medium confidence: medium
tags: [infrastructure, hosts, containers, paths] tags: [infrastructure, hosts, containers, paths]
sources: [] sources: []
@@ -20,11 +20,11 @@ sources: []
| litellm | LiteLLM proxy | `http://litellm:4000/v1` | Yes (from config) | | litellm | LiteLLM proxy | `http://litellm:4000/v1` | Yes (from config) |
| `custom:Omniroute:20128` | OmniRoute proxy (CT600 node `omniroute`) | `http://omniroute:20128/v1` → `100.88.81.19:20128` | Yes (live probe 2026-10-26: 6,480 models) | | `custom:Omniroute:20128` | OmniRoute proxy (CT600 node `omniroute`) | `http://omniroute:20128/v1` → `100.88.81.19:20128` | Yes (live probe 2026-10-26: 6,480 models) |
| `custom:omni-101` | OmniRoute proxy (CT101 node `omniroute-1`) — **the live default provider** | `http://100.78.63.20:20128` | Yes (live probe 2026-10-26) | | `custom:omni-101` | OmniRoute proxy (CT101 node `omniroute-1`) — **the live default provider** | `http://100.78.63.20:20128` | Yes (live probe 2026-10-26) |
| CT201 | Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) | | CT201 | Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.77` (LAN) | Yes (live probe 2026-10-30 — LAN address corrected from `192.168.178.166`) |
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net`, `100.118.5.51` (Tailscale) | Yes | | CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net`, `100.118.5.51` (Tailscale) | Yes |
| PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-10-26) | | PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-10-26) |
| PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-10-26) | | PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-10-26) |
| CT450 (hermesagent) | Hermes Agent **v0.22.3** (2026.9.24), local `5b147ab036d6` — same build as CT460, hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes | | CT450 (hermesagent) | Hermes Agent **v0.22.5** (2026.9.24), local `bf0203a794a0` — **same build as CT460**; hermes-webui, hermes-ui (Vite). ⚠ a stale **root-level** `/usr/local/bin/hermes` on CT450 still reports **v0.19.0** — the live agent is the `hermes`-user install under `~/.hermes/hermes-agent` | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes (live probe 2026-10-30 — version corrected from `v0.22.3`, build from `5b147ab036d6` to `bf0203a794a0`) |
## Project Paths ## Project Paths
@@ -35,7 +35,7 @@ sources: []
| `/home/hermes/.local/bin/hermes` | Hermes binary | | | `/home/hermes/.local/bin/hermes` | Hermes binary | |
| `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` | | `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` |
| `/home/hermes/.hermes/skills/` | Skills dir | 302 skills across 69 categories (`.archive/` excluded) | | `/home/hermes/.hermes/skills/` | Skills dir | 302 skills across 69 categories (`.archive/` excluded) |
| `/home/hermes/.hermes/cron/` | Cron jobs | 28 jobs — 25 enabled / 3 disabled | | `/home/hermes/.hermes/cron/` | Cron jobs | 29 jobs — 25 enabled / 4 disabled (the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure) |
| `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ | | `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ |
| `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 | | `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 |
| `~/.camofox/` | Camofox profiles | managed persistence | | `~/.camofox/` | Camofox profiles | managed persistence |
@@ -62,7 +62,7 @@ sources: []
- **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-10-26 (17 LXCs) - **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-10-26 (17 LXCs)
- **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-10-26 (19 LXCs). Guest IDs collide between the two hosts - **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-10-26 (19 LXCs). Guest IDs collide between the two hosts
- **CT600 (`omniroute`):** OmniRoute server, `100.88.81.19` — **running and live**; `omniroute` is the hostname that resolves to it - **CT600 (`omniroute`):** OmniRoute server, `100.88.81.19` — **running and live**; `omniroute` is the hostname that resolves to it
- **CT601 (`omni`):** the old CT601 OmniRoute guest **no longer exists** on either hypervisor (`pct config 601` → no such config). The `omni` Tailscale peer (100.93.204.4) is its orphaned node, offline 69d - **CT601 (`omni`):** the old CT601 OmniRoute guest **no longer exists** on either hypervisor (`pct config 601` → no such config on both `pve` and `m93p`). The orphaned `omni` Tailscale peer (`100.93.204.4`) has since **expired out of the tailnet entirely** — as of the 2026-10-30 live probe the tailnet contains **no** peer named `omni` and no peer at `100.93.204.4`, so the old "offline 69d" description no longer applies. The only OmniRoute nodes left are CT600 (`omniroute`) and CT101 (`omniroute-1`).
- **CT101 (`omniroute`, m93p):** second OmniRoute node, `100.78.63.20` — running - **CT101 (`omniroute`, m93p):** second OmniRoute node, `100.78.63.20` — running
## Unverified ## Unverified
+23 -26
View File
@@ -3,8 +3,8 @@ title: Tailscale
type: infrastructure type: infrastructure
status: active status: active
created: 2026-07-26 created: 2026-07-26
updated: 2026-10-29 updated: 2026-10-30
verified_on: 2026-10-29 verified_on: 2026-10-30
confidence: high confidence: high
tags: [tailscale, vpn, networking, infrastructure] tags: [tailscale, vpn, networking, infrastructure]
--- ---
@@ -19,12 +19,13 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`) - **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`) - **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
## Tailscale Node Inventory (verified 2026-10-25) ## Tailscale Node Inventory (verified 2026-10-30)
Full node list from `tailscale status` — **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age. Full node list from `tailscale status` — **50 non-funnel nodes** (49 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong). Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`. Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`.
Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**. Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**.
Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — **`octop` (100.92.163.122), `springroll` (100.100.153.125), `openmausbot` and `openmausbot-vm` were missing from the table body** (only mentioned in prose), `omarchymac` had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from `tailscale status --json`, keyed on DNS name rather than `HostName` (the JSON `HostName` for `freshrss` is `ct260`, and both `omniroute` nodes share the hostname `omniroute`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running. Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — **`octop` (100.92.163.122), `springroll` (100.100.153.125), `openmausbot` and `openmausbot-vm` were missing from the table body** (only mentioned in prose), `omarchymac` had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from `tailscale status --json`, keyed on DNS name rather than `HostName` (the JSON `HostName` for `freshrss` is `ct260`, and both `omniroute` nodes share the hostname `omniroute`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running.
Re-verified 2026-10-30 03:3x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 IP changes** (anthonys-mac-mini: 100.74.17.15 -> 100.76.60.37; omarchymac: 100.84.180.69 -> 100.126.173.115) and **4 nodes dropped out of the tailnet entirely** (anthonys-imac (100.112.123.124), cachyos (100.124.208.15), omarchymac-1 (100.126.173.115), omni (100.93.204.4)). `omarchymac` is now `100.126.173.115` (the peer formerly listed as `omarchymac-1`; the old `omarchymac` at `100.84.180.69` no longer matches any peer) and `anthonys-mac-mini` moved to `100.76.60.37` — both same-named nodes, new addresses. The four dropped peers are Tailscale expired/offline records (`anthonys-imac`, `cachyos`, `omni`, and the old `omarchymac` address), not newly-installed hardware. **`m93p` reads `Online: false` in the peer record for the third consecutive audit, yet the host is plainly up — `tailscale ping 100.104.51.98` → `pong from m93p via 192.168.178.118:41641 in 0s` and SSH returned `m93p / up 12 days, 11:02` — so the peer record is not authoritative for hypervisor reachability and the row records it as online. Node count corrected to **50 non-funnel nodes** (49 live IPv4 peers + `hermes` itself); `tailscale status --json` reports **72 peers total — 49 with a `100.` IPv4, 23 IPv6-only** (the `funnel-ingress-node` set). **Live IPv4 peers absent from the table: none.** Container identity re-confirmed from inside guests via `pct exec <vmid> -- tailscale ip -4` on `pve` (100.96.100.82) and `m93p` (100.104.51.98); CT600 is confirmed **running** and CT601 confirmed **absent from both hypervisors**. Serve tables re-probed per host: CT460's `:443` route set matches the table below exactly (12 paths), and CT201 (`pct exec 201 -- tailscale serve status`) serves only `/` → `127.0.0.1:8085` and `/mgr` → `127.0.0.1:8084` on **both** `:443` and `:8443`, with no `/steel` or `/steel-api` routes remaining.
Re-verified 2026-10-29 03:0x AWST (stocktake): State column rebuilt from `tailscale status --json` — **11 rows changed**. Two were **state transitions**, not age drift: **`sftpgo` (CT299, pve) came back online** (the 2026-10-28 audit had recorded it as `offline <1d`; live `tailscale ping 100.88.223.70` → `pong from sftpgo via 192.168.178.92:41641 in 0s`, `pct exec 299 -- systemctl is-active sftpgo` → `active`, `pct exec 299 -- tailscale ip -4` → `100.88.223.70`), and **`m93p` is online despite a stale peer record** — `tailscale status --json` still reports `Online: false` with `LastSeen 2026-10-28T18:40:00Z`, but `tailscale ping 100.104.51.98` → `pong from m93p via 192.168.178.118:41641 in 1ms` and SSH over the tailnet returned `m93p / up 11 days`. This is the **second** occurrence of the m93p stale-peer-record pattern (first seen 2026-10-26) — the peer record is not authoritative for hypervisor reachability. The remaining **9 rows were offline-age drift of exactly +1 day** (anthonys-imac 34d → 35d; cachyos 54d → 55d; cloudreve 95d → 96d — CT209 stopped; elementsynapse 98d → 99d — CT555 stopped; freshrss-new 71d → 72d; mattermost 38d → 39d — CT255 stopped; omarchybook 6d → 7d; omarchymac 10d → 11d; omni 71d → 72d — CT601 removed from both hypervisors; openmausbot 6d → 7d). Node count unchanged at **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself); `tailscale status --json` reports **76 peers total — 53 with a `100.` IPv4, 23 IPv6-only** (the `funnel-ingress-node` set). **Nodes in the table that are no longer in the tailnet: none. Live IPv4 peers absent from the table: none. IP mismatches: none.** Container identity re-confirmed from **inside** every LXC via `pct exec <vmid> -- tailscale ip -4` across **both** hypervisors (`pve` 100.96.100.82, `m93p` 100.104.51.98) — 34 guests enumerated; every peer IP mapped to its CT as listed, and the three stopped guests (`CT209` cloudreve, `CT255` mattermost, `CT555` elementsynapse) and the three running-but-unconnected guests (`CT105` conduit, `CT1264` openmausbot, `CT100` obsidian-livesync) confirmed. CT601 remains absent from both hypervisors. Re-verified 2026-10-29 03:0x AWST (stocktake): State column rebuilt from `tailscale status --json` — **11 rows changed**. Two were **state transitions**, not age drift: **`sftpgo` (CT299, pve) came back online** (the 2026-10-28 audit had recorded it as `offline <1d`; live `tailscale ping 100.88.223.70` → `pong from sftpgo via 192.168.178.92:41641 in 0s`, `pct exec 299 -- systemctl is-active sftpgo` → `active`, `pct exec 299 -- tailscale ip -4` → `100.88.223.70`), and **`m93p` is online despite a stale peer record** — `tailscale status --json` still reports `Online: false` with `LastSeen 2026-10-28T18:40:00Z`, but `tailscale ping 100.104.51.98` → `pong from m93p via 192.168.178.118:41641 in 1ms` and SSH over the tailnet returned `m93p / up 11 days`. This is the **second** occurrence of the m93p stale-peer-record pattern (first seen 2026-10-26) — the peer record is not authoritative for hypervisor reachability. The remaining **9 rows were offline-age drift of exactly +1 day** (anthonys-imac 34d → 35d; cachyos 54d → 55d; cloudreve 95d → 96d — CT209 stopped; elementsynapse 98d → 99d — CT555 stopped; freshrss-new 71d → 72d; mattermost 38d → 39d — CT255 stopped; omarchybook 6d → 7d; omarchymac 10d → 11d; omni 71d → 72d — CT601 removed from both hypervisors; openmausbot 6d → 7d). Node count unchanged at **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself); `tailscale status --json` reports **76 peers total — 53 with a `100.` IPv4, 23 IPv6-only** (the `funnel-ingress-node` set). **Nodes in the table that are no longer in the tailnet: none. Live IPv4 peers absent from the table: none. IP mismatches: none.** Container identity re-confirmed from **inside** every LXC via `pct exec <vmid> -- tailscale ip -4` across **both** hypervisors (`pve` 100.96.100.82, `m93p` 100.104.51.98) — 34 guests enumerated; every peer IP mapped to its CT as listed, and the three stopped guests (`CT209` cloudreve, `CT255` mattermost, `CT555` elementsynapse) and the three running-but-unconnected guests (`CT105` conduit, `CT1264` openmausbot, `CT100` obsidian-livesync) confirmed. CT601 remains absent from both hypervisors.
Re-verified 2026-10-28 00:2x (nightly audit): State column rebuilt from `tailscale status --json` — **14 rows changed** (anthonys-imac: idle/offline 32d -> offline 34d; cachyos: idle/offline 52d -> offline 54d; cloudreve: idle/offline 93d — CT209 stopped -> offline 95d — CT209 stopped; conduit: idle/offline 49d -> offline 52d; elementsynapse: idle/offline 96d — CT555 stopped -> offline 98d — CT555 stopped; freshrss-new: idle/offline 69d -> offline 71d; m93p: online -> offline <1d; mattermost: idle/offline 36d — CT255 stopped -> offline 38d — CT255 stopped; omarchybook: idle/offline 4d -> offline 6d; omarchymac: idle/offline 8d -> offline 10d; omni: idle/offline 69d — CT601 removed from both hypervisors -> offline 71d — CT601 removed from both hypervisors; openmausbot: idle/offline 4d -> offline 6d; sftpgo: online — CT299 (pve) -> offline <1d — CT299 (pve); ubuntu-server-xfce: idle/offline 66d -> offline 68d). Header count corrected to **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Nodes in the table that are no longer in the tailnet: **none**. Live IPv4 peers absent from the table: **none**. **State transitions inside the audit window (not just age drift): `m93p` online → offline <1d, `sftpgo` (CT299) online → offline <1d.** Re-verified 2026-10-28 00:2x (nightly audit): State column rebuilt from `tailscale status --json` — **14 rows changed** (anthonys-imac: idle/offline 32d -> offline 34d; cachyos: idle/offline 52d -> offline 54d; cloudreve: idle/offline 93d — CT209 stopped -> offline 95d — CT209 stopped; conduit: idle/offline 49d -> offline 52d; elementsynapse: idle/offline 96d — CT555 stopped -> offline 98d — CT555 stopped; freshrss-new: idle/offline 69d -> offline 71d; m93p: online -> offline <1d; mattermost: idle/offline 36d — CT255 stopped -> offline 38d — CT255 stopped; omarchybook: idle/offline 4d -> offline 6d; omarchymac: idle/offline 8d -> offline 10d; omni: idle/offline 69d — CT601 removed from both hypervisors -> offline 71d — CT601 removed from both hypervisors; openmausbot: idle/offline 4d -> offline 6d; sftpgo: online — CT299 (pve) -> offline <1d — CT299 (pve); ubuntu-server-xfce: idle/offline 66d -> offline 68d). Header count corrected to **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Nodes in the table that are no longer in the tailnet: **none**. Live IPv4 peers absent from the table: **none**. **State transitions inside the audit window (not just age drift): `m93p` online → offline <1d, `sftpgo` (CT299) online → offline <1d.**
@@ -34,29 +35,27 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|---|---|---| |---|---|---|
| hermes | 100.118.5.51 | **self** — CT460 (pve) | | hermes | 100.118.5.51 | **self** — CT460 (pve) |
| agentzero | 100.106.54.94 | online — CT475 (pve) | | agentzero | 100.106.54.94 | online — CT475 (pve) |
| anthonys-imac | 100.112.123.124 | offline 35d |
| anthonys-ipad | 100.111.198.120 | online | | anthonys-ipad | 100.111.198.120 | online |
| anthonys-iphone | 100.100.116.67 | online | | anthonys-iphone | 100.100.116.67 | online |
| anthonys-mac-mini | 100.74.17.15 | online | | anthonys-mac-mini | 100.76.60.37 | online |
| cachyos | 100.124.208.15 | offline 55d |
| clawtest | 100.122.189.81 | online — CT333 (pve) | | clawtest | 100.122.189.81 | online — CT333 (pve) |
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) | | cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
| cloudreve | 100.111.187.7 | offline 96d — CT209 stopped | | cloudreve | 100.111.187.7 | offline 97d — CT209 stopped |
| conduit | 100.81.79.87 | offline 52d | | conduit | 100.81.79.87 | offline 53d |
| docker | 100.96.150.69 | online — CT200 (pve) | | docker | 100.96.150.69 | online — CT200 (pve) |
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) | | docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
| elementsynapse | 100.75.250.37 | offline 99d — CT555 stopped | | elementsynapse | 100.75.250.37 | offline 100d — CT555 stopped |
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) | | freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
| freshrss | 100.83.60.93 | online — CT1260 (m93p) | | freshrss | 100.83.60.93 | online — CT1260 (m93p) |
| freshrss-new | 100.110.180.115 | offline 72d | | freshrss-new | 100.110.180.115 | offline 73d |
| gitea | 100.82.100.128 | online — CT221 (m93p) | | gitea | 100.82.100.128 | online — CT221 (m93p) |
| guest-vps | 100.117.14.84 | online | | guest-vps | 100.117.14.84 | online |
| hermesagent | 100.98.205.112 | online — CT450 (pve) | | hermesagent | 100.98.205.112 | online — CT450 (pve) |
| homeassistant | 100.68.139.5 | online | | homeassistant | 100.68.139.5 | online |
| litellm | 100.79.29.64 | online — CT500 (m93p) | | litellm | 100.79.29.64 | online — CT500 (m93p) |
| m93p | 100.104.51.98 | online — hypervisor (peer record stale: `tailscale ping` + SSH verified 2026-10-29) | | m93p | 100.104.51.98 | online — hypervisor (peer record stale: `tailscale ping` + SSH verified 2026-10-30) |
| mailflow | 100.115.60.92 | online — CT1262 (m93p) | | mailflow | 100.115.60.92 | online — CT1262 (m93p) |
| mattermost | 100.98.116.14 | offline 39d — CT255 stopped | | mattermost | 100.98.116.14 | offline 40d — CT255 stopped |
| n8n | 100.84.124.30 | online | | n8n | 100.84.124.30 | online |
| nextcloud | 100.90.102.19 | online — CT270 (pve) | | nextcloud | 100.90.102.19 | online — CT270 (pve) |
| ntfy | 100.67.153.116 | online — CT1241 (m93p) | | ntfy | 100.67.153.116 | online — CT1241 (m93p) |
@@ -64,13 +63,11 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| octop | 100.92.163.122 | online | | octop | 100.92.163.122 | online |
| ollama | 100.108.44.119 | online — CT205 (pve) | | ollama | 100.108.44.119 | online — CT205 (pve) |
| omarchy | 100.73.227.103 | online | | omarchy | 100.73.227.103 | online |
| omarchybook | 100.107.9.100 | offline 7d | | omarchybook | 100.107.9.100 | offline 8d |
| omarchymac | 100.84.180.69 | offline 11d | | omarchymac | 100.126.173.115 | online |
| omarchymac-1 | 100.126.173.115 | online |
| omni | 100.93.204.4 | offline 72d — CT601 removed from both hypervisors |
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute | | omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) | | omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
| openmausbot | 100.91.214.91 | offline 7d | | openmausbot | 100.91.214.91 | offline 8d |
| openmausbot-vm | 100.77.107.30 | online | | openmausbot-vm | 100.77.107.30 | online |
| openweb | 100.69.230.13 | online — CT245 (m93p) | | openweb | 100.69.230.13 | online — CT245 (m93p) |
| paseo | 100.72.188.93 | online — CT1261 (m93p) | | paseo | 100.72.188.93 | online — CT1261 (m93p) |
@@ -83,7 +80,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) | | social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
| springroll | 100.100.153.125 | online | | springroll | 100.100.153.125 | online |
| springroll-browser | 100.109.221.91 | online | | springroll-browser | 100.109.221.91 | online |
| ubuntu-server-xfce | 100.101.26.104 | offline 68d | | ubuntu-server-xfce | 100.101.26.104 | offline 70d |
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) | | vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
| webserver | 100.82.40.23 | online — CT207 (pve) | | webserver | 100.82.40.23 | online — CT207 (pve) |
@@ -118,14 +115,14 @@ Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 rou
Re-verified 2026-10-26 (stocktake): the node table was rebuilt again from `tailscale status --json` — **`octoagent` (100.117.10.68) and `omarchymac-1` (100.126.173.115) were both missing entirely** and have been added; `m93p` (100.104.51.98) now reads **online** (it was "idle/offline 13m ago" — a stale Tailscale peer record, and the host was plainly reachable throughout); every offline age was refreshed (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**, `ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**, `openmausbot` 3d → **4d**). The `omni` row now reads **"CT601 removed from both hypervisors"** rather than "CT601 stopped" — live `pct list` on both `pve` and `m93p` shows **no CT601 at all**, so the old guest is gone, not merely halted. Container identity re-confirmed from **inside** each LXC via `pct exec <vmid> -- tailscale ip -4` on both hypervisors; every peer whose LXC has no Tailscale IP was checked against `pct list` (`CT105`/`CT1264` run but do not connect — `conduit` and `openmausbot` offline; `CT209`/`CT255`/`CT555` are `stopped`). Serve tables re-probed per host with `pct exec 201 -- tailscale serve status`. Re-verified 2026-10-26 (stocktake): the node table was rebuilt again from `tailscale status --json` — **`octoagent` (100.117.10.68) and `omarchymac-1` (100.126.173.115) were both missing entirely** and have been added; `m93p` (100.104.51.98) now reads **online** (it was "idle/offline 13m ago" — a stale Tailscale peer record, and the host was plainly reachable throughout); every offline age was refreshed (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**, `ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**, `openmausbot` 3d → **4d**). The `omni` row now reads **"CT601 removed from both hypervisors"** rather than "CT601 stopped" — live `pct list` on both `pve` and `m93p` shows **no CT601 at all**, so the old guest is gone, not merely halted. Container identity re-confirmed from **inside** each LXC via `pct exec <vmid> -- tailscale ip -4` on both hypervisors; every peer whose LXC has no Tailscale IP was checked against `pct list` (`CT105`/`CT1264` run but do not connect — `conduit` and `openmausbot` offline; `CT209`/`CT255`/`CT555` are `stopped`). Serve tables re-probed per host with `pct exec 201 -- tailscale serve status`.
## CT201 Serve Routes ## CT201 Serve Routes
Verified live on 2026-10-26 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected. Re-verified live on 2026-10-30 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected.
| Host / Port | Path | Target | Notes | | Host / Port | Path | Target | Notes |
|---|---|---|---| |---|---|---|---|
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **502** (re-probed 2026-10-26; upstream 8082 has no listener) | | `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **502** (re-probed 2026-10-30; upstream 8082 has no listener — `curl http://127.0.0.1:8082/` → exit code 7 / `http=000`) |
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` container — **200** | | `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` container — **200** (re-probed 2026-10-30) |
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **502** (re-probed 2026-10-26) | | `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **502** (re-probed 2026-10-30) |
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` — **200** | | `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` — **200** (re-probed 2026-10-30) |
**Steel routes removed 2026-10-29.** This table previously carried two more rows on `:443` — `/steel` → **Steel routes removed 2026-10-29.** This table previously carried two more rows on `:443` — `/steel` →
`http://127.0.0.1:5173` and `/steel-api` → `http://127.0.0.1:3000`. Both were deleted with `http://127.0.0.1:5173` and `/steel-api` → `http://127.0.0.1:3000`. Both were deleted with
@@ -134,7 +131,7 @@ are `/` → `127.0.0.1:8085` and `/mgr` → `http://127.0.0.1:8084`, on **both**
→ [[comparisons/steel-vs-camoufox]] → [[comparisons/steel-vs-camoufox]]
Two corrections to the pre-2026-10-25 table: Two corrections to the pre-2026-10-25 table:
1. **`:8443` is served by `tailscaled` on CT201, but so is `:443`.** `ss -ltnp` on CT201 shows `tailscaled` bound to `100.96.244.39:443`, and the serve table above lists routes on **both** ports. The old note claiming "Port 443 occupied by Traefik on CT201" is **false as of 2026-10-25** — no Traefik container or process exists on CT201 (`docker ps -a` and `ps aux` both came back empty for `traefik`), and Tailscale owns 443 there. 1. **`:8443` is served by `tailscaled` on CT201, but so is `:443`.** `ss -ltnp` on CT201 shows `tailscaled` bound to `100.96.244.39:443`, and the serve table above lists routes on **both** ports. The old note claiming "Port 443 occupied by Traefik on CT201" is **false as of 2026-10-25** — no Traefik **process** runs on CT201 (`ps aux` returns nothing for `traefik`), and Tailscale owns 443 there. *(Evidence corrected 2026-10-30: the original wording said `docker ps -a` "came back empty for `traefik`" — it does not. Live `docker ps -a` lists `runtipi-reverse-proxy | traefik:v3.2 | Exited (255) 2 months ago`. The container is **stopped**, not absent; the conclusion that Traefik does not occupy 443 stands.)*
2. The old table routed `:8443` straight to `http://localhost:8082`. That port has **no listener**; the live target is `8085` (the `asset_proxy.py` front door), which itself proxies to the dead 8082 and returns 502. → [[systems/homepage-dashboard]] 2. The old table routed `:8443` straight to `http://localhost:8082`. That port has **no listener**; the live target is `8085` (the `asset_proxy.py` front door), which itself proxies to the dead 8082 and returns 502. → [[systems/homepage-dashboard]]
## Atomic Restore Script ## Atomic Restore Script
@@ -159,7 +156,7 @@ tailscale serve --set-path /jobs off
``` ```
## Notes ## Notes
- **CT201 now serves Tailscale on BOTH `:443` and `:8443`** (verified 2026-10-25). The old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" claim was **wrong**: no Traefik container or process exists on CT201, and `tailscaled` owns 443 there (`ss -ltnp` shows it bound to `100.96.244.39:443`). The CT201 Serve Routes table above now records both ports. - **CT201 now serves Tailscale on BOTH `:443` and `:8443`** (verified 2026-10-30). The old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" claim was **wrong**: no Traefik **process** runs on CT201, and `tailscaled` owns 443 there (`ss -ltnp` shows it bound to `100.96.244.39:443`). The CT201 Serve Routes table above now records both ports. *(Evidence corrected 2026-10-30: the original wording said "no Traefik container or process exists on CT201" — the container does exist, `runtipi-reverse-proxy | traefik:v3.2`, but it is **stopped** — `Exited (255) 2 months ago`. The conclusion that Tailscale owns 443 is unchanged.)*
- CT460 has no conflict — Tailscale uses 443 directly, with the public webhook Funnel on `:8443`. - CT460 has no conflict — Tailscale uses 443 directly, with the public webhook Funnel on `:8443`.
- Routes must be re-added in order (restore script handles this). - Routes must be re-added in order (restore script handles this).
- After every rebuild, verify that `tailscale serve status` contains both the `:443` Serve routes and the `:8443/webhook` Funnel route. - After every rebuild, verify that `tailscale serve status` contains both the `:443` Serve routes and the `:8443/webhook` Funnel route.
+62
View File
@@ -1584,3 +1584,65 @@ not pushed.
`log.md`. The wiki-maintained pages under `infrastructure/` remain a separate, known gap. `log.md`. The wiki-maintained pages under `infrastructure/` remain a separate, known gap.
- **Removed/archived:** none. **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a - **Removed/archived:** none. **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a
key or token value. key or token value.
## 2026-10-30 — Stocktake (03:00 AWST): 0 stale by age, but 5 pages had drifted from live state
- **Ran `wiki-maintain.sh`.** Regenerated `systems/skills-index.md` (**302 skills / 69 categories** — unchanged; the
generator de-dups one nested duplicate, so 302 is the correct unique-skill figure), wrote
`raw/stocktake/freshness-20261030.txt` (**0 stale / 109 fresh / 3 no-date of 112 pages**), rebuilt Quartz
(654 files) and deployed to `public/`.
- **Freshness is not accuracy.** The age scan found nothing, but a live-vs-page diff against the running estate found
real drift on five pages. All corrections below were verified against live sources before being written.
- **[[infrastructure/tailscale]] — node table rebuilt.** `tailscale status --json` now reports **50 non-funnel nodes**
(49 live IPv4 peers + `hermes`), not the 54 the page claimed. **2 IP changes** (`anthonys-mac-mini` 100.74.17.15 →
100.76.60.37; `omarchymac` 100.84.180.69 → 100.126.173.115 — the peer formerly listed as `omarchymac-1`) and **4 nodes
dropped out of the tailnet entirely** (`anthonys-imac`, `cachyos`, `omarchymac-1`, `omni`). Every offline age refreshed.
`m93p` reads `Online: false` for the **third** consecutive audit yet is plainly up (`tailscale ping` → pong, SSH → `up 12
days`), so the row records it online. Total peers **72** (49 IPv4 + 23 IPv6-only funnel). CT460's `:443` route set
matched the page exactly; CT201 serves only `/` → 8085 and `/mgr` → 8084 on both `:443` and `:8443`.
- **[[systems/scheduled-tasks]] — fleet ledger corrected.** Live `jobs.json` shows **29 jobs (25 enabled / 4 disabled)**, not
the 28/25/3 on the page: `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused 2026-10-30 01:04:22** after its
00:01:08 slot closed **`unknown`** on a gateway-connection failure. Non-clean `last_status` is **4** (not 3), and the
standing "every `failure_streak` reads 0" claim was false (`ea79a2107e36` = 1). "Failing (delivery only)" is **2**, not 1
(`fad5a86979d4` joined `066bb5191280`). The retired DHL row was dropped from the job table (it lives in Removed Jobs).
Seven job rows carried **stale `error` claims** for jobs that have since cleared to `ok` — each was marked superseded
with a dated correction rather than rewritten (SCHEMA rule 9). The `executions.db` retention floor note was corrected to
the live **2026-10-24 09:22:36**. Window tallies: **2026-10-29 = 177 / 0** and **2026-10-28 = 169 / 2** (both +1 on the
page — the 23:59 Wiki Audit row had still been `running` at audit time).
- **New incident page: [[incidents/2026-10-30-mnemosyne-health-monitor-paused]].** The Mnemosyne Health Monitor's own
failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) —
`deliveries.db` holds **zero rows for 2026-10-30**, so the incident row is the only trace. The job was **not re-enabled**
and the gateway was **not restarted** by this run; root cause is **unverified** and recorded as such.
- **[[infrastructure/hosts]] — four corrections.** CT201's LAN address is `192.168.178.77`, not `192.168.178.166`; CT450's Hermes Agent is **v0.22.5** (2026.9.24),
local **`bf0203a794a0`** — the same build as CT460 — so the page's "v0.22.3 / 5b147ab036d6" was wrong on both
(a stale *root-level* `/usr/local/bin/hermes` on CT450 still reports v0.19.0 and is **not** the live agent —
the live one is the `hermes`-user install under `~/.hermes/hermes-agent`); the cron count is now 29/25/4; and
the orphaned `omni`
peer has **expired out of the tailnet entirely** (no peer named `omni`, no peer at 100.93.204.4), so the "offline 69d"
description no longer applied. The **skills count of 302 was left alone** — it is correct.
- **[[systems/homepage-dashboard]] — five corrections, and a second pass to fix my own first pass.**
"All Apps" is **114 → 123 live apps**, not 122 (the API dedupes by `(name, normalised href)` across groups;
the literal `All Apps` group in `services.yaml` holds 122 and is not what the tab bar counts). The grid is a
responsive `repeat(auto-fill, minmax(230px, 1fr))` (served `/dash.css` line 3640), not a fixed 5-column grid.
The tab list was wrong — but **my first correction was wrong too** and had to be redone: the live tab bar is
built from the service *groups* (`renderTabs()` iterates `state.groups`), **not** from the `layout:` block in
`settings.yaml`. That block and its `tab:`/`columns:` keys are **inert** — `grep -c "Overview" /dash.js` = 0 and
`grep -c "columns" /dash.js` = 0. There is **no "Overview" tab**; the live tabs are `☆ Pinned`, `All (123)` and
the **14 groups** (AI Quotas 3, Infrastructure 13, Everyday 8, Agents 20, Media 11, AI & LLM 17, Tools 20,
Storage 6, More 13, Resources 6, System Health 0, Subscriptions 1, Quick Launch 2, Uncategorised 3). The saved
theme is **`{"pair":"Boarding Pass","mode":"light"}`** — one of **59** theme pairs; its light-mode amber is
`#795000`, so the old "dark base with warm amber (#d4a04a)" was wrong on both counts. **`manager.py` runs inside
the `homepage-manager` container** (its container `Cmd` is the `pip install … && python3 /opt/homepage/manager.py`
line, confirmed via `docker inspect` and `docker top`); only **`asset_proxy.py`** is a host process
(`0.0.0.0:8085 users:(("python3",pid=4038))`). And the "no Traefik container exists" evidence was wrong — the
`runtipi-reverse-proxy` Traefik container exists but is **stopped** (`Exited (255) 2 months ago`); the *conclusion*
that Traefik does not occupy 443 stands. The same evidence error was corrected in [[infrastructure/tailscale]].
- **[[current-state]]** and **[[index]]** updated with the new incident and the corrected fleet numbers.
- **Verified, not assumed:** CT600 **running** and CT601 **absent from both hypervisors** (`pct config 601` → no such
config on `pve` or `m93p`); the CT201 `:443`/`:8443` Serve tables; CT201 listeners 5173/3000/8082 (**none** — Steel's
removal confirmed); the homepage service groups and theme; and every cron counter, tally, incident row and delivery row
(read from `jobs.json`, `executions.db` — whose `cron_incidents` table lives inside it, the standalone
`cron_incidents.db`/`incidents.db` files are empty stubs — and `deliveries.db`).
- **Not changed:** `systems/skills-index.md` count (302 is correct), CT460's Serve route table (matched live exactly), and
the steel-removal records across the wiki (confirmed still true).
- **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a key or token value.
+125
View File
@@ -0,0 +1,125 @@
# Wiki freshness audit — 2026-10-30
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (109)
- [ 10d] CONTRIBUTING.md
- [ 4d] agents/hermes-chaos.md
- [ 4d] agents/hermes-production.md
- [ 10d] agents/index.md
- [ 1d] agents/nanobot.md
- [ 10d] agents/open-claw.md
- [ 19d] agents/qwenpaw.md
- [ 10d] agents/zeroclaw.md
- [ 1d] comparisons/camoufox-vs-chrome.md
- [ 10d] comparisons/hermes-vs-nanobot.md
- [ 10d] comparisons/memory-backends.md
- [ 10d] comparisons/model-routers.md
- [ 1d] comparisons/steel-vs-camoufox.md
- [ 10d] concepts/agent-memory.md
- [ 1d] concepts/browser-automation.md
- [ 10d] concepts/model-routing.md
- [ 10d] concepts/persistent-shells.md
- [ 10d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 1d] decisions/index.md
- [ 10d] decisions/keep-openrouter-as-fallback.md
- [ 10d] decisions/persistent-shell-enabled.md
- [ 1d] decisions/rollback-steel-to-camofox.md
- [ 10d] experiments/index.md
- [ 6d] incidents/2026-07-29-mnemosyne-relay.md
- [ 10d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 6d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 8d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 7d] incidents/2026-08-15-skills-index-drift.md
- [ 18d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 5d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 10d] incidents/2026-08-29-gmail-read-state.md
- [ 9d] incidents/2026-09-02-sftpgo-outage.md
- [ 19d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 1d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 4d] incidents/2026-10-03-ct450-hermes-update.md
- [ 4d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 1d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 6d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 21d] incidents/2026-10-09-memory-recall-relevance.md
- [ 14d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 3d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 11d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 3d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 8d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 5d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 4d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 4d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 4d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 2d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 0d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 0d] incidents/2026-10-29-conduit-voice-silence.md
- [ 0d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 0d] index.md
- [ 8d] infrastructure/backups.md
- [ 10d] infrastructure/credentials-map.md
- [ 1d] infrastructure/docker-services.md
- [ 10d] infrastructure/domains-and-tunnels.md
- [ 0d] infrastructure/hosts.md
- [ 16d] infrastructure/network-map.md
- [ 16d] infrastructure/proxmox.md
- [ 0d] infrastructure/tailscale.md
- [ 18d] queries/retained-research-answers.md
- [ 5d] reports/communities-wa-social-media-audit-2026.md
- [ 9d] reports/uber-eats/2026-07.md
- [ 9d] reports/uber-eats/2026-08.md
- [ 9d] reports/uber-eats/2026-09.md
- [ 9d] reports/uber-eats/index.md
- [ 10d] runbooks/backup-wiki.md
- [ 10d] runbooks/diagnose-docker.md
- [ 10d] runbooks/gateway-resource-resilience.md
- [ 0d] runbooks/job-radar.md
- [ 10d] runbooks/multiplexer-setup.md
- [ 4d] runbooks/provider-health.md
- [ 1d] runbooks/recover-docker-service.md
- [ 10d] runbooks/restart-browser.md
- [ 10d] runbooks/restart-hermes.md
- [ 10d] runbooks/restore-openrouter.md
- [ 10d] runbooks/test-openrouter.md
- [ 10d] runbooks/troubleshoot-tool-loop.md
- [ 4d] runbooks/update-hermes-safely.md
- [ 1d] systems/browser-backend.md
- [ 10d] systems/cloudflare-tunnel.md
- [ 0d] systems/conduit.md
- [ 4d] systems/current-profile.md
- [ 10d] systems/guanaco.md
- [ 10d] systems/headless-server.md
- [ 0d] systems/hermes-agent.md
- [ 10d] systems/hermes-ui.md
- [ 18d] systems/home-assistant.md
- [ 0d] systems/homepage-dashboard.md
- [ 19d] systems/ignis.md
- [ 4d] systems/memory-backend.md
- [ 4d] systems/messaging-integrations.md
- [ 6d] systems/mnemosyne.md
- [ 4d] systems/model-providers.md
- [ 18d] systems/nanobot.md
- [ 10d] systems/nextcloud.md
- [ 10d] systems/obsidian.md
- [ 8d] systems/omniroute.md
- [ 10d] systems/opencode-go.md
- [ 10d] systems/openrouter.md
- [ 19d] systems/paseo.md
- [ 11d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 10d] systems/terminal-backend.md
- [ 10d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 109 fresh, 3 no-date of 112 pages
+70 -17
View File
@@ -3,8 +3,8 @@ title: Homepage Dashboard
type: systems type: systems
status: degraded status: degraded
created: 2026-07-26 created: 2026-07-26
updated: 2026-10-29 updated: 2026-10-30
verified_on: 2026-10-29 verified_on: 2026-10-30
confidence: high confidence: high
tags: [dashboard, homepage, homelab, proxmox, monitoring] tags: [dashboard, homepage, homelab, proxmox, monitoring]
--- ---
@@ -32,30 +32,70 @@ The dashboard content itself is now **custom Python tooling** under `/opt/homepa
- **Local:** `http://100.96.244.39:8082/` → **no listener** - **Local:** `http://100.96.244.39:8082/` → **no listener**
- **Manager UI:** `http://runtipi.kangaroo-eel.ts.net:8084/` → `homepage-manager`, up - **Manager UI:** `http://runtipi.kangaroo-eel.ts.net:8084/` → `homepage-manager`, up
## Architecture (verified 2026-10-20) ## Architecture (re-verified 2026-10-30)
``` ```
CT201 (runtipi) CT201 (runtipi)
├── homarr container (port 8102) — live dashboard, HTTP 200 ├── homarr container (port 8102) — live dashboard, HTTP 200
├── homepage-manager container (8084) — web UI for adding/removing apps, up ├── homepage-manager container (8084) — web UI for adding/removing apps, up
├── homebox container (7745) — up ├── homebox container (7745) — up
├── asset_proxy.py (python3, port 8085) — proxies 8085 → 8082, currently 502 ├── asset_proxy.py (python3, port 8085) — HOST process, proxies 8085 → 8082, currently 502
├── manager.py (python3) — dashboard management service, up ├── manager.py (python3) — runs INSIDE the homepage-manager container (see below)
└── Tailscale serve (ports 443 AND 8443) — / → 8085, /mgr → 8084 on both ports └── Tailscale serve (ports 443 AND 8443) — / → 8085, /mgr → 8084 on both ports
(the former /steel and /steel-api routes were (the former /steel and /steel-api routes were
removed 2026-10-29) removed 2026-10-29)
``` ```
No container is listening on **8082** — `docker ps` on CT201 lists no `homepage` container at all. No container is listening on **8082** — `docker ps` on CT201 lists no `homepage` container at all.
> **Corrected 2026-10-30:** the old diagram implied `manager.py` also ran directly on the CT201 host.
> Live `docker inspect homepage-manager` shows its command is
> `["sh","-c","pip install --quiet --no-cache-dir PyYAML && python3 /opt/homepage/manager.py"]`
> with `/opt/homepage` bind-mounted — so **`manager.py` runs inside the `homepage-manager` container**
> (`MANAGER_PORT=8084`). Only **`asset_proxy.py` is a host process**: live `ss -ltnp` on CT201 shows
> `0.0.0.0:8085 users:(("python3",pid=4038))` — a bare `/usr/bin/python3 /opt/homepage/asset_proxy.py`
> with no container owning it. The host-side `:8084` listener is `docker-proxy`, not python.
## Tabs ## Tabs
| Tab | Content | > **Re-verified 2026-10-30 against the live manager API** (`GET http://127.0.0.1:8084/api/groups` on CT201),
|---|---| > the served `/dash.js` and the served `/dash.css`. The old table below was wrong in several ways — the
| Overview | Infrastructure (2 cols) + Everyday (4 cols) + Resources | > live **tab bar is built from the service *groups*** (`renderTabs()` in `dash.js` iterates `state.groups`),
| Media | Jellyfin, arr stack, RSS | > not from the `layout:` block in `settings.yaml`. That `layout:` block — and its `tab:`/`columns:` keys —
| AI | OmniRoute, LiteLLM, Lobe Chat, SearXNG | > is **inert**: `grep -c "Overview" /dash.js` returns **0**, and `grep -c "columns" /dash.js` returns **0**.
| Agents | Hermes, OpenClaw, Nanobot, Camofox | > There is **no "Overview" tab** and the `columns: N` values are never read.
| Tools | IT tools, Cyber Chef, etc. + Storage |
| More | Home Assistant, N8n, Jupyter | The live tab bar is: **`☆ Pinned`**, **`All (123)`**, then the **14 service groups** in order — each with a live
| All Apps | 114 apps in 5-column grid (alphabetical) | count badge:
| Tab | Live count | Apps |
|---|---|---|
| ☆ Pinned | — | pinned apps only (19 pinned in `dash_prefs.json`) |
| All (123) | 123 | every app, deduped by `(name, normalised href)` — **corrected from 114** |
| AI Quotas | 3 | Claude, Codex, Ollama |
| Infrastructure | 13 | Gitea, Grafana, Guacamole, LAN watcher, PVE, Portainer docker, Portainer openweb, Runtipi, SFTPGO, Syncthing Runtipi, Theme Studio, cloudreve, filebrowser runtipi |
| Everyday | 8 | Homelab dashboard, Investbrain, Linkwarden, Mealie, NextCloud, cardholder, homebox, read later |
| Agents | 20 | AISBF Dashboard, Agent Dashboard, Agent zero, Camofox VNC chaos, Camofox VNC rhino, Chaos webUI, Herdr omarchy, Hermes Control, Hermes web UI (rhino), Hindsight chaos, Honcho memory dashboard, Mausbot ×2, Octoagent, PAN Hermes, Zeroclaw dash, herdr omarchybook, hermes desktop rhino, hermes workspace, nanobot dashboard |
| Media | 11 | Audio bookshelf, Flaresolverr, FreshRSS, Jellyfin, MeTube, NewsKu, Prowlarr, RSS-Bridge, Transmission, YouTube, qBitTorrent |
| AI & LLM | 17 | Agent estate wiki, Cliproxy api, FreeLLMAPI, Khoj ADMIN, Khoj SH, LiteLLM, Lobe chat (docker), Manifest router, Omniroute gateway, Opencode server, Openweb, Reasonix, Searxng, Simple chat ai, Vercel ai cloud, Voice Studio, World monitor |
| Tools | 20 | 8mb video compression, Cobalt, Convert to it, Cyber chef, Dockage, Element synapse, Excaldraw, IT tools, Matermost, Quiet thanks, Retroarch, Social dashboard (communities), Stirling PDF, Subscription tracker, T3 code frontend, Wordpress, code-server debian, compose craft, convert anything?, subtitle generator |
| Storage | 6 | Obsidian agent vault, filebrowser Debby, filebrowser DebianVM, filebrowser DebianVM NEW, filebrowser Docker, guacano |
| More | 13 | Cloudfare apps, GitHub, Kubuntu desktop, N8n, Next step, Omni tools, Reactive resume, Sunshine Ubuntu's-server, Yeah koko tts, hermes desktop, hermes desktop chaos, home assistant, jupyter server |
| Resources | 6 | Communities reporting - socials, Dashboard Manager, Helper Scripts, Homelab Docs, RADAR, Self-Hosted Directory |
| System Health | 0 | *empty* — the tab renders but holds no app cards (the health panel is separate, widget-driven) |
| Subscriptions | 1 | Renewals |
| Quick Launch | 2 | Paseo, Portainer |
| Uncategorised | 3 | Hermes desktop ui, Openchamber, Vaultwarden |
**App-count reconciliation (two different numbers, both correct in context):**
- **123** — the live count the UI shows on the `All (123)` tab. `get_all_apps_flat()` walks every group except
`All Apps` and `AI Quotas & Rate Limits`, then **dedupes by `(name, normalised href)`** — so an app defined in
two groups collapses to one card (e.g. `Mausbot` appears in both `Agents` and `Tools`).
- **122** — the literal `All Apps` group in `/opt/homepage/config/services.yaml`, a generated roll-up that is
**not** what the tab bar counts. The old page's "114" was simply wrong.
**Grid layout (corrected):** the served `/dash.css` (385 121 bytes, = `/opt/homepage/config/dashboard.css`)
sets `.app-grid` at **line 3640** to `grid-template-columns: repeat(auto-fill, minmax(230px, 1fr))` — a
**responsive auto-fill**, not a fixed 5-column grid. Narrower breakpoints override it to
`repeat(auto-fill, minmax(160px, 1fr))` (line 5001) and then `1fr 1fr` (line 5010). The `columns:` figures in
`settings.yaml` are dead config.
## Proxmox Integration ## Proxmox Integration
- **Node widget:** Shows live VM/LXC counts, CPU, memory for PVE node - **Node widget:** Shows live VM/LXC counts, CPU, memory for PVE node
@@ -112,15 +152,28 @@ Web form at port 8084 for adding/removing apps without SSH.
Type anywhere on the dashboard to search services by name/description. Google search disabled — only searches your apps. Type anywhere on the dashboard to search services by name/description. Google search disabled — only searches your apps.
## Theme ## Theme
- Dark base with warm amber (#d4a04a) accents - **Corrected 2026-10-30:** the live saved theme is **`{"pair":"Boarding Pass","mode":"light"}`** — read from
`/opt/homepage/config/dash_prefs.json` key `dash.theme`. The previous "Dark base with warm amber
(#d4a04a) accents" claim is **wrong on both counts**: the mode is **light**, not dark, and
`#d4a04a` is not this theme's amber at all. **Boarding Pass** (`chassis: boarding`, "Boarding pass desk —
perforated stubs, gate codes, seat and route labels") is one of **59 theme pairs** in
`/opt/homepage/config/themes.json` under `dashboard.pairs`; in its **light** mode the amber accent is
**`#795000`** (dark mode would be `#f5c96a`). The dashboard also ships 26 legacy `homepage:` themes
(`obsidian-midnight`, `nord-frost`, `cyberpunk-terminal`, …) and three saved `custom:` themes in
`dash.customThemes` (E2E Test Theme, Whiteboard Ops 1, Evidence Wall 1), each with its own chassis and palette.
- Bigger tiles (68px min-height, 40px icons) - Bigger tiles (68px min-height, 40px icons)
- Custom CSS in `custom.css` - Custom CSS in `custom.css`
## Tailscale Serve ## Tailscale Serve
**CT201 serves Tailscale on BOTH `:443` and `:8443`** (verified live 2026-10-25 via `pct exec 201 -- tailscale serve status`). **CT201 serves Tailscale on BOTH `:443` and `:8443`** (verified live 2026-10-25 via `pct exec 201 -- tailscale serve status`).
The earlier note on this page — "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead" — was **wrong**: The earlier note on this page — "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead" — was **wrong**:
no Traefik container or process exists on CT201 (`docker ps -a` and `ps aux` both return nothing for `traefik`), and no Traefik **process** runs on CT201 (`ps aux` returns nothing for `traefik`), and
`ss -ltnp` shows `tailscaled` bound to `100.96.244.39:443`. Corrected on 2026-10-25. `ss -ltnp` shows `tailscaled` bound to `100.96.244.39:443`. Corrected on 2026-10-25.
> **Evidence corrected 2026-10-30:** the old wording said `docker ps -a` "returns nothing for
> `traefik`". That was **not accurate** — live `docker ps -a` on CT201 does list
> `runtipi-reverse-proxy | traefik:v3.2 | Exited (255) 2 months ago`. The **conclusion is unchanged**
> (Traefik is not running and does not occupy 443; `tailscaled` is bound to `100.96.244.39:443`),
> but the container is *stopped*, not absent.
## Maintenance ## Maintenance
- **Restart Manager:** `docker restart homepage-manager` - **Restart Manager:** `docker restart homepage-manager`
@@ -139,7 +192,7 @@ no Traefik container or process exists on CT201 (`docker ps -a` and `ps aux` bot
CT201's remaining routes are only `/` → 8085 and `/mgr` → 8084 on both `:443` and `:8443`. CT201's remaining routes are only `/` → 8085 and `/mgr` → 8084 on both `:443` and `:8443`.
→ [[infrastructure/tailscale]], [[comparisons/steel-vs-camoufox]] → [[infrastructure/tailscale]], [[comparisons/steel-vs-camoufox]]
- **2026-10-25 (live probe, stocktake):** the "Port 443 occupied by Traefik on CT201" note was **false** — no Traefik - **2026-10-25 (live probe, stocktake):** the "Port 443 occupied by Traefik on CT201" note was **false** — no Traefik
container or process exists on CT201, and `tailscaled` owns **both** `:443` and `:8443` there. Serve tables corrected: **process** runs on CT201 (the `runtipi-reverse-proxy` Traefik container exists but is **stopped** — `Exited (255) 2 months ago` — re-verified 2026-10-30), and `tailscaled` owns **both** `:443` and `:8443` there. Serve tables corrected:
`/` → 8085 and `/mgr` → 8084 on both ports *(the `/steel` and `/steel-api` routes listed at the time were `/` → 8085 and `/mgr` → 8084 on both ports *(the `/steel` and `/steel-api` routes listed at the time were
deleted on 2026-10-29 — see above)*. Re-confirmed live: 8084 → **200**, 8085 → **502**, deleted on 2026-10-29 — see above)*. Re-confirmed live: 8084 → **200**, 8085 → **502**,
8082 → **no listener**, `homepage-manager` up. → [[infrastructure/tailscale]] 8082 → **no listener**, `homepage-manager` up. → [[infrastructure/tailscale]]
+16 -15
View File
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **Current as of 2026-10-30 00:0x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**, and every `failure_streak` reads **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit). - **Current as of 2026-10-30 03:3x AWST (live `jobs.json` + `executions.db`) — the 10-29 clean day held, but the fleet changed shape overnight: the Mnemosyne Health Monitor was auto-paused and its failure notice could not be delivered.** Fleet now **29 jobs (25 enabled / 4 disabled)** — `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused at 2026-10-30 01:04:22** after its 00:01:08 slot closed **`unknown`** on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, which opened incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`). Non-clean `last_status` is now **4** — the three long-standing survivors (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`) **plus `ea79a2107e36` (`error`, `failure_streak: 1`)** — so the standing "every `failure_streak` reads 0" claim is now false. Window **2026-10-29 settled 177 `completed` / 0 `failed`** (the page previously said 176 — the 23:59 Wiki Audit row `cbfe434a3b5e` had still been `running` at audit time and has since settled `completed`); **2026-10-28 stands at 169 / 2** (also +1, same cause). **Unclosed incident rows 1 → 2:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving the Facebook monthly's `0e753e_e7f8194a7d06` (`detected`, cannot clear before its 2026-11-01 slot) **plus the new `ea79a2_d93c3acdc4f2`**. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report); **zero delivery rows were created on 2026-10-30**, because the Mnemosyne Health Monitor's own failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) — a failure notice that was itself never delivered. The radar's two-stage split held its ground: `dca8482e4f76` (stage 1) and `5fc292ff0f3b` (stage 2) both read `ok` with streak 0, and every previously-red row (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) has since **cleared to `ok`** on its own scheduled slot — the page's older per-row notes still claimed `error` for several of them. **Pins: 18 of the 25 enabled jobs pin `custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); the other 7 enabled rows are `no_agent` jobs with no pin.
- **Prior window — as of 2026-10-30 00:0x AWST *(superseded by the 2026-10-30 03:3x entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**, and every `failure_streak` read **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit).
- **Prior window — as of 2026-10-29 00:3x AWST *(superseded by the 2026-10-30 entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.** - **Prior window — as of 2026-10-29 00:3x AWST *(superseded by the 2026-10-30 entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.**
- **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)* - **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)*
- **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`). - **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`).
@@ -28,15 +29,15 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]] - **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] - **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
- **Total jobs:** **28 — 25 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-19 00:20 AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. - **Total jobs:** **29 — 25 enabled, 4 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-30 03:3x AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. The **4th disabled job is `ea79a2107e36` (Mnemosyne Health Monitor)**, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher) and `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
- **Non-clean `last_status`:** **12 as of 2026-10-20 00:09** — 10 `error` + 2 `delivery_failed`. The burst re-dirtied four rows that were clean at the 10-19 audit (`dca8482e4f76`, `85edf6772bca`, `08eb86f02d37`, `c0a9fa314542`) and deepened two of the existing seven (`0e519874d280` and `ecc1fc13655a` both to `failure_streak: 2`); `fad5a86979d4` newly reads `delivery_failed` (bare-`telegram` target). `b63ed5817212` (dirtied twice) and `96eb95ce12f2` (four derivative reds) **cleared the same day** (18:01:56 / 12:46:11), and `ea79a2107e36` cleared on its 10-20 00:01 slot. Survivors, with next run: `ecc1fc13655a` (streak **2**, 10-20 12:00), `0e519874d280` (**2**, Thu 10-22 08:00), `1b23c042e17d` (**2**, Fri 10-23 08:00), `dca8482e4f76` (1, **10-20 07:00**), `85edf6772bca` (1, **10-20 07:00**), `08eb86f02d37` (1, 10-20 08:30), `c0a9fa314542` (1, **Mon 10-26 09:00**), `06a0a1c8704f` (1, Thu 10-22 10:00), `c8d52904e1d6` (1, Thu 10-22 18:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 0, 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → **12 now**. Three of these fire within eight hours of this audit, so the count will move again by morning. - **Non-clean `last_status`:** **4 as of 2026-10-30 03:3x AWST** — 2 `error` + 2 `delivery_failed`. Live `jobs.json` (authoritative for the per-job counters): `ea79a2107e36` (Mnemosyne Health Monitor — `error`, `failure_streak: 1`, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident `ea79a2_d93c3acdc4f2` open `detected`), `0e753e9720fd` (Facebook Birthday Monthly Scrape — `error`, streak 0, next slot 11-01 09:00), `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — `delivery_failed`, streak 0, most recent failure **2026-10-26 03:08:15**) and `066bb5191280` (Uber Eats Monthly Spend Report — `delivery_failed`, streak 0, next slot 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → **4 now**, the increase being the Mnemosyne Health Monitor's overnight gateway fault.
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]]. - **Failing (last run failed outright):** **1** — `ea79a2107e36` (Mnemosyne Health Monitor), `failure_streak: 1`. Its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, and the job was **auto-paused** at that moment — it is now the 4th disabled job. ⚠ **Its failure notice was never delivered**: `hermes cron list` reports *"The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"*, and `deliveries.db` holds **zero rows for 2026-10-30**. A failure notice that fails to send is the same green-row/dead-job trap as [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the incident row `ea79a2_d93c3acdc4f2` is the only trace. The other previously-red jobs (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) have all **cleared to `ok` with streak 0** on their own scheduled slots.
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04). - Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
- **Date-bucketing note (added 2026-10-25):** `executions.db` stores `started_at` with a `+08:00` offset, and SQLite's `date()` normalises to UTC — so `date(started_at)` moves every row that ran 00:00–07:59 AWST onto the **previous** day. Daily tallies on this page are bucketed with `substr(started_at,1,10)`, which preserves AWST calendar days. A recount via `date()` reads low by roughly a day's early-morning runs; this is the same class of drift that put 10-21 at 168 instead of 171. - **Date-bucketing note (added 2026-10-25):** `executions.db` stores `started_at` with a `+08:00` offset, and SQLite's `date()` normalises to UTC — so `date(started_at)` moves every row that ran 00:00–07:59 AWST onto the **previous** day. Daily tallies on this page are bucketed with `substr(started_at,1,10)`, which preserves AWST calendar days. A recount via `date()` reads low by roughly a day's early-morning runs; this is the same class of drift that put 10-21 at 168 instead of 171.
- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window. - **Failing (delivery only):** **2** — `066bb5191280` (Uber Eats Monthly Spend Report: the report itself generated; the send failed against the DM thread — bare `telegram` target, 2026-10-01) and `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — the same dead-thread cause, most recently **2026-10-26 03:08:15**). Both carry `failure_streak: 0` because the *run* succeeded; only the delivery failed.
- **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs. - **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs.
- **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07); and `1a8e90ea2271` completed automatically on **2026-10-06 03:00**. Four of the seven are therefore verified by live run. The remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are repinned but not yet re-run — they are due on their normal schedules. `1a8e90ea2271` has since run **twice more on schedule** (2026-10-06 03:00 → `33f4e95`; 2026-10-07 03:00 → `8c63ef2`), so the stocktake repin is verified across consecutive automatic runs. - **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07); and `1a8e90ea2271` completed automatically on **2026-10-06 03:00**. Four of the seven are therefore verified by live run. The remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are repinned but not yet re-run — they are due on their normal schedules. `1a8e90ea2271` has since run **twice more on schedule** (2026-10-06 03:00 → `33f4e95`; 2026-10-07 03:00 → `8c63ef2`), so the stocktake repin is verified across consecutive automatic runs.
- **Streak counting note:** the `N failures in a row` figure in `hermes cron list` comes from the `failure_streak` counter in `jobs.json`. `executions.db` retains only rows from **2026-09-30 onward**, so DB-based recounts undercount older streaks. Live counters as of 2026-10-06: **all 0**. - **Streak counting note:** the `N failures in a row` figure in `hermes cron list` comes from the `failure_streak` counter in `jobs.json`. `executions.db` retains only a rolling window of rows — the live floor is now **2026-10-24 09:22:36** (1,001-row cap), so DB-based recounts undercount older streaks and cannot reproduce tallies for days before that floor. Quote the `jobs.json` counter as authoritative and state the floor whenever a DB recount is cited. Live counters as of 2026-10-30 03:3x AWST: **`ea79a2107e36` = 1; every other job = 0.**
- **Re-verified 2026-10-06 AWST:** job count unchanged at 30 (27 enabled / 3 disabled). `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found`). - **Re-verified 2026-10-06 AWST:** job count unchanged at 30 (27 enabled / 3 disabled). `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found`).
- **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 `completed`, 0 `failed`, 1 `running` (the audit itself); 14 sessions, all `cron` — **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`). - **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 `completed`, 0 `failed`, 1 `running` (the audit itself); 14 sessions, all `cron` — **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`).
- **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice. - **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice.
@@ -52,27 +53,26 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) | | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 | | AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 |
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | | | Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | `local` (failures → topic 1030) | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00. ✅ **First full split run 2026-10-29:** 07:00:59 → 10:16:12 (3 h 15 m 13 s), `completed`, wrote `handoff/2026-10-29.json` (170 discovered / 33 duplicates avoided / 37 verified); both of its ledger rows closed `resolved` at 10:16:12. **Delivery is now `local`** — stage 2 owns the report | | Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | `local` (failures → topic 1030) | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00. ✅ **First full split run 2026-10-29:** 07:00:59 → 10:16:12 (3 h 15 m 13 s), `completed`, wrote `handoff/2026-10-29.json` (170 discovered / 33 duplicates avoided / 37 verified); both of its ledger rows closed `resolved` at 10:16:12. **Delivery is now `local`** — stage 2 owns the report ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, `failure_streak: 1`" note above is **superseded**; the job cleared on its own 10-20 07:00 slot and has run clean since, including the successful 2026-10-29 split run. |
| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. ✅ **First full split run 2026-10-29:** fired 10:13:46, consumed the handoff at 10:40:59, closed `completed` 10:44:48 and delivered the same second — **20 pages + 2 cover letters, `missing_cover_letter_count 0`**, Notion board 455 → 475; the 11:30 and 14:30 slots then no-oped correctly. See [[incidents/2026-10-27-job-radar-budget-exhausted]] | | Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. ✅ **First full split run 2026-10-29:** fired 10:13:46, consumed the handoff at 10:40:59, closed `completed` 10:44:48 and delivered the same second — **20 pages + 2 cover letters, `missing_cover_letter_count 0`**, Notion board 455 → 475; the 11:30 and 14:30 slots then no-oped correctly. See [[incidents/2026-10-27-job-radar-budget-exhausted]] |
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant | | Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "still carries `last_status: error` … now **streak 2**" note above is **superseded**; the job cleared on its Fri 2026-10-23 08:00 slot and has run clean since. |
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] |
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent | | Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic | | Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic |
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). ⚠ **Live status is no longer `ok`:** the 2026-10-19 03:04:47 run completed but recorded `delivery_failed` — `configured thread_id 34795 for telegram:1793951355:34795 was not found` (the CLI renders it as "ran, but the result was not delivered", then "delivered without thread_id"). The run is fine; only the send is. **No incident row was created for this.** ⚠ Bare `telegram` target should move to a group topic | | Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). ⚠ **Live status is no longer `ok`:** the 2026-10-19 03:04:47 run completed but recorded `delivery_failed` — `configured thread_id 34795 for telegram:1793951355:34795 was not found` (the CLI renders it as "ran, but the result was not delivered", then "delivered without thread_id"). The run is fine; only the send is. **No incident row was created for this.** ⚠ Bare `telegram` target should move to a group topic |
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed that incident. ⚠ **Then 2026-10-18 12:00 failed differently** — a model repetition loop aborted the run at 12:08:25 **after** it had delivered a canned "Your request was not processed…" line to the topic (12:08:11) → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. ⚠ **Then the 2026-10-19 12:01 slot failed again, on the `omni-101` 503** — `ecc1fc_c9976ea24553` (a **10-16** row, refreshed 10-19 12:02:58, `alerted`) plus the separate `ecc1fc_21a72aa11cc4` degenerate-stream row from 10-18, so **this one job holds two open incidents**. Live `last_status: error`, `failure_streak: 2`, next slot **2026-10-20 12:00** | | FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed that incident. ⚠ **Then 2026-10-18 12:00 failed differently** — a model repetition loop aborted the run at 12:08:25 **after** it had delivered a canned "Your request was not processed…" line to the topic (12:08:11) → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. ⚠ **Then the 2026-10-19 12:01 slot failed again, on the `omni-101` 503** — `ecc1fc_c9976ea24553` (a **10-16** row, refreshed 10-19 12:02:58, `alerted`) plus the separate `ecc1fc_21a72aa11cc4` degenerate-stream row from 10-18, so **this one job holds two open incidents**. Live `last_status: error`, `failure_streak: 2`, next slot **2026-10-20 12:00** ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, `failure_streak: 2`" note above is **superseded**; the job cleared on its own daily 12:00 slot after 2026-10-19. |
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** ⚠ **The status reading changed again on 2026-10-19:** its scheduled Monday slot (`0 9 * * 1`) at **09:01 failed on the `omni-101` 503** (new incident `c0a9fa_764558b5ae83`, `alerted`), so **this week's roundup did not run at all** and `last_status` now reads `error` (streak 1). The token question is therefore **untested for a second consecutive Monday** — the job has not reached `digest.py` on a Monday since 2026-10-12. Next real test: **Mon 2026-10-26 09:00**. | | GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** ⚠ **The status reading changed again on 2026-10-19:** its scheduled Monday slot (`0 9 * * 1`) at **09:01 failed on the `omni-101` 503** (new incident `c0a9fa_764558b5ae83`, `alerted`), so **this week's roundup did not run at all** and `last_status` now reads `error` (streak 1). The token question is therefore **untested for a second consecutive Monday** — the job has not reached `digest.py` on a Monday since 2026-10-12. Next real test: **Mon 2026-10-26 09:00**. ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "reads `error` (streak 1)" note above is **superseded**; the job ran clean on its Mon 2026-10-26 09:00 slot, so the 10-19 outage has cleared. The **revoked-bot-token** finding above stands as a separate, still-unverified delivery-path question — that Monday run reached the model without raising a 503, which does not by itself prove the token works. |
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); and its **own 10-17 08:31** run restored a clean `last_status`. ⚠ **But the incident never closed: `08eb86_472c7b609c80` was refreshed on 2026-10-19 08:33 by the same 503, and is still `alerted`.** Live `last_status: error`, streak 1 | | Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); and its **own 10-17 08:31** run restored a clean `last_status`. ⚠ **But the incident never closed: `08eb86_472c7b609c80` was refreshed on 2026-10-19 08:33 by the same 503, and is still `alerted`.** Live `last_status: error`, streak 1 ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, streak 1" note above is **superseded**; the job cleared on its own daily 08:30 slot after 2026-10-19. |
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) | | Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
| Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent | | Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
| Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent | | Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
| Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent | | Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run. ⚠ **Both 2026-10-19 slots failed the same way** (06:01:05 and 12:01:16), taking its streak to **2** and re-triggering the watchdog; its **18:01 slot succeeded**, clearing the streak and auto-closing `b63ed5_bbe44c89aac1` at 18:01:56. Live `last_status: ok`. With `0 */6 * * *` this job is the fleet's best 503 sentinel — four probes a day | | Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run. ⚠ **Both 2026-10-19 slots failed the same way** (06:01:05 and 12:01:16), taking its streak to **2** and re-triggering the watchdog; its **18:01 slot succeeded**, clearing the streak and auto-closing `b63ed5_bbe44c89aac1` at 18:01:56. Live `last_status: ok`. With `0 */6 * * *` this job is the fleet's best 503 sentinel — four probes a day |
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 | | Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 ⚠ **Live status re-verified 2026-10-30 03:3x — the job is now DISABLED and failing:** its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`), and the scheduler **auto-paused the job at that moment**. Live `last_status: error`, `failure_streak: 1`, `enabled: false`. The "Live `last_status: ok`, streak 0" note above is **superseded**. ⚠ **The failure notice was never delivered** — `hermes cron list` reports *"live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"* and `deliveries.db` has **zero rows for 2026-10-30**. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults. **The pattern repeated exactly in burst 2** — four more reds on 2026-10-19 (06:16, 06:31, 12:16, 12:31), all quoting the same coverage-monitor error, all clearing at **12:46:11** (`96eb95_565ec1deb1c7`, a 10-16 row refreshed through 10-19 12:31). **8 of the 26 failure rows across the two bursts were this one sentinel** | | Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults. **The pattern repeated exactly in burst 2** — four more reds on 2026-10-19 (06:16, 06:31, 12:16, 12:31), all quoting the same coverage-monitor error, all clearing at **12:46:11** (`96eb95_565ec1deb1c7`, a 10-16 row refreshed through 10-19 12:31). **8 of the 26 failure rows across the two bursts were this one sentinel** |
| Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` | | Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` |
| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | ⚠ Two node faults, no code fault: 2026-10-15 08:06 `RuntimeError: Connection error.` (`0e5198_be2291809e56`) and **2026-10-19 08:01 `HTTP 503 … resource pressure`** (new incident `0e5198_ec2fac98a91d`, `alerted`). Both are open; `failure_streak: 2`. Next slot Thu 2026-10-22 08:00 | | | Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | ⚠ Two node faults, no code fault: 2026-10-15 08:06 `RuntimeError: Connection error.` (`0e5198_be2291809e56`) and **2026-10-19 08:01 `HTTP 503 … resource pressure`** (new incident `0e5198_ec2fac98a91d`, `alerted`). Both are open; `failure_streak: 2`. Next slot Thu 2026-10-22 08:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "`failure_streak: 2`" note above is **superseded**; the job cleared on its Mon/Thu 08:00 slots after 2026-10-19. |
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | ⚠ **2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `85edf6_256c5e150bc4`, `alerted`) — its first failure since its 10-15 connection-refused red cleared on 10-16. Live `last_status: error`, streak 1, next slot 2026-10-20 07:00 | | | Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | ⚠ **2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `85edf6_256c5e150bc4`, `alerted`) — its first failure since its 10-15 connection-refused red cleared on 10-16. Live `last_status: error`, streak 1, next slot 2026-10-20 07:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, streak 1" note above is **superseded**; the job cleared on its own daily 07:00 slot after 2026-10-19. |
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic | | Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic |
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 | | Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 |
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 | | weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
@@ -95,6 +95,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. | | Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| Mnemosyne Health Monitor | `ea79a2107e36` | **2026-10-30 01:04:22** | **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
## Known Issues ## Known Issues
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-29 updated: 2026-10-30
verified_on: 2026-10-29 verified_on: 2026-10-30
confidence: high confidence: high
tags: [skills, hermes] tags: [skills, hermes]
sources: [] sources: []