audit 2026-10-18: new incident page (omniroute-1 503 resource pressure, 9 calls / 7 jobs), fleet re-verified 28 jobs / 25 enabled / 8 non-clean, Mosquito Tracker removed, radar bot token dead
This commit is contained in:
+16
-2
@@ -3,8 +3,8 @@ title: Mnemosyne
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-16
|
||||
verified_on: 2026-10-16
|
||||
updated: 2026-10-18
|
||||
verified_on: 2026-10-18
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, backend]
|
||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||||
@@ -141,3 +141,17 @@ Primary agent memory store for Hermes.
|
||||
mode `0755`, so the monitor's non-executable finding is resolved.
|
||||
- **Open, unverified:** 13 orphan rows in `memory_embeddings` (oldest 2026-08-22) reported by a
|
||||
2026-10-10 monitor run - neither confirmed nor cleaned by this audit.
|
||||
|
||||
## Re-verified 2026-10-18 (audit window; live probes 2026-10-18 00:30-00:36 AWST)
|
||||
|
||||
- **Both monitors rode the `omniroute-1` 503 window; both are green again.** `ea79a2107e36` (health, every
|
||||
12 h) failed **10-17 00:00** with `RuntimeError: HTTP 503 ... resource pressure` and delivered its alert;
|
||||
`b63ed5817212` (coverage, every 6 h) failed **10-16 12:03** the same way. Since **10-17 12:00** every slot
|
||||
of both has completed with `delivery_outcome: suppressed` (nothing to report) — including **10-18 00:00**.
|
||||
- **No fresh coverage figures exist in the ledger.** These jobs report to Telegram, not to the execution
|
||||
row, so the **last explicit read remains 2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056
|
||||
(99.4 %)**, both above the 95 % reindex threshold. Nothing here indicates a reindex is due.
|
||||
- **The watchdog triple-checks the same thing.** `96eb95ce12f2` (every 15 min) exited 1 four times —
|
||||
10-16 12:15 + 12:30, 10-17 00:15 + 00:30 — each row quoting the **coverage monitor's own 503 error**, i.e.
|
||||
one sentinel producing four failure rows. Read failures **per job**, not per row. It has been clean on
|
||||
every slot since (latest 10-18 00:30).
|
||||
|
||||
+17
-2
@@ -3,8 +3,8 @@ title: OmniRoute
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-16
|
||||
verified_on: 2026-10-16
|
||||
updated: 2026-10-18
|
||||
verified_on: 2026-10-18
|
||||
confidence: high
|
||||
tags: [routing, provider, model, inference]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -66,6 +66,21 @@ The wiki previously recorded CT600 as retired and CT601 as the live successor. *
|
||||
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
|
||||
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
|
||||
|
||||
## Recent Changes (2026-10-18) — `omniroute-1` 503 resource pressure (2026-10-16 -> 2026-10-17)
|
||||
|
||||
- **A second, different failure mode from the same node.** Three days after the 34-hour TCP blackout,
|
||||
`omniroute-1` answered model calls with **`HTTP 503: Service temporarily unavailable due to resource
|
||||
pressure. Retry shortly.`** — **9 model calls across 7 jobs**, between **10-16 08:00:37** and
|
||||
**10-17 03:00:28** (~19 h). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
- **The control plane looked healthy throughout.** `/api/health` returned `200 {"status":"ok"}` and
|
||||
`/v1/models` returned `401` (auth-required) before, during and after the window — re-probed 2026-10-18
|
||||
00:33. A green health endpoint does **not** mean the model path is serving: 503 here is load-shedding
|
||||
under resource pressure, not a dead process, so process checks miss it.
|
||||
- **It recovered on its own; nothing was changed on the node.** Last 503 **10-17 03:00:28**; `10-17`
|
||||
closed **162 completed / 5 failed** (all five before 03:01) and `10-18` is green so far. Every affected
|
||||
job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17 08:31,
|
||||
`ecc1fc13655a` and `ea79a2107e36` both 10-17 12:02, auto-closing their own incidents.
|
||||
|
||||
## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm
|
||||
|
||||
- **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14
|
||||
|
||||
+17
-13
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-16
|
||||
verified_on: 2026-10-16
|
||||
updated: 2026-10-18
|
||||
verified_on: 2026-10-18
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
@@ -14,9 +14,10 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **✅ Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
||||
- **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, re-verified 2026-10-13 — unchanged; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10)
|
||||
- **Non-clean `last_status`:** **3** (was 5 at the 2026-10-12 audit) — `0e753e9720fd` (`error`, streak 0), `1b23c042e17d` (`error`, streak 1), `066bb5191280` (`delivery_failed`, streak 0). `fad5a86979d4` and `dca8482e4f76` cleared on **2026-10-12** when both ran clean on schedule.
|
||||
- **Total jobs:** **28 — 25 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-18 00:31 AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs.
|
||||
- **Non-clean `last_status`:** **8 as of 2026-10-18 00:31** — `1b23c042e17d` (`error`, streak 2), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`, streak 0). History: **3** at the 2026-10-12 audit (when `fad5a86979d4` and `dca8482e4f76` cleared on schedule) → **12** at the 2026-10-16 storm peak → 8 now. The six jobs that fixed themselves did so on their next scheduled slot; the remaining weekly ones (`1b23c042e17d` Fri, `0e519874d280` Mon, `06a0a1c8704f` + `c8d52904e1d6` Thu) cannot clear before their next run.
|
||||
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]].
|
||||
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
|
||||
- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window.
|
||||
@@ -40,21 +41,20 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
|
||||
| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. Live `last_status: ok`, streak 0. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open) |
|
||||
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant |
|
||||
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1. **Next scheduled slot: Fri 2026-10-16 08:00**, which should clear it |
|
||||
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] |
|
||||
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
||||
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic |
|
||||
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). Live `last_status: ok`, streak 0. ⚠ Bare `telegram` target should move to a group topic |
|
||||
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | |
|
||||
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed the incident. Live `last_status: ok` |
|
||||
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. `last_status: ok` because the 401 is inside the script, not the cron run. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** |
|
||||
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | |
|
||||
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); cleared by the 10-17 08:31 run, which auto-closed the incident. Live `last_status: ok` |
|
||||
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
|
||||
| Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
|
||||
| Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
|
||||
| Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
|
||||
| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95% |
|
||||
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
|
||||
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30) |
|
||||
| Mosquito Tracker | `1b31187397a5` | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 |
|
||||
| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run |
|
||||
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots (10-16 12:00, 10-17 00:00); cleared by its own **10-17 12:02** run |
|
||||
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults |
|
||||
| Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` |
|
||||
| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | |
|
||||
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
|
||||
@@ -62,8 +62,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic |
|
||||
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 |
|
||||
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
|
||||
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
|
||||
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken |
|
||||
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text |
|
||||
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken. ⚠ **The 2026-10-17 03:00 slot failed** on the same `omni-101` 503 — the first miss since the **2026-10-15** storm slot (which died with `RuntimeError: Connection error.`; the 10-16 slot succeeded). `raw/stocktake/freshness-20261017.txt` was not written and `systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 slot succeeded, so `freshness-20261016.txt` exists); the **2026-10-18 03:00** slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident `1a8e90_3c9c1d573cde` `detected`. Note the ledger tags this row `failure_type: unknown` even though the execution carries an explicit 503 message |
|
||||
| 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
||||
|
||||
## Removed Jobs
|
||||
@@ -71,6 +71,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
|---|---|---|
|
||||
| Workday start reminder | 2026-08-26 | No longer present in live cron list |
|
||||
| CT 221 Miner Watchdog | 2026-08-26 | Job `d5516d0a5147` removed; CT221 status unknown/inspected |
|
||||
| DHL Tracking - Parcel 8052838382 | 2026-10-10 | `62619aa6bcb3` — parcel delivered, monitor redundant |
|
||||
| Mosquito Tracker | 2026-10-16 (after the 15:00 run) | `1b31187397a5` — absent from `jobs.json` at the 2026-10-18 audit; last execution 2026-10-16 15:00 was `completed` / `delivered` |
|
||||
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
|
||||
|
||||
## Disabled Jobs (not removed)
|
||||
@@ -116,6 +118,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
- **Two wiki trees, drifted in both directions (NEW 2026-10-02):** `/home/hermes/workspace/agent-estate-wiki` is the canonical source (Gitea remote `Anthony/agent-estate-wiki`, Quartz build, `wiki-maintain.sh` target). `/home/hermes/wiki/agent-estate-wiki` is the **Obsidian/Ignis vault copy** the iPad reads — pages nested under `wiki/`, and **no git remote at all**. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged *backwards* (`systems/ignis.md`, `systems/nextcloud.md`, three `incidents/` pages exist only in the vault; `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` only in canonical) — a naive one-way sync deletes pages. See [[systems/hermes-agent]].
|
||||
- **Work mailbox unreachable (non-cron; affects on-demand lookups):** the `work` Outlook account is configured with password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` fails with `login not supported`. No cron job polls it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
|
||||
|
||||
- **The `omni-101` 503 burst left no node-side evidence (NEW 2026-10-18):** 9 model calls across 7 jobs died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` between 2026-10-16 08:00 and 2026-10-17 03:00 — while the node answered `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` before, during and after (re-probed 2026-10-18 00:33). Nothing on the Hermes side records the node's CPU/RAM/queue depth at failure time, so the "resource pressure" cause is taken on the router's word; if it recurs, capture host metrics *during* the burst, not afterwards. The incident ledger also tags **seven of the nine** rows `failure_type: unknown` **despite** the explicit message (only the two script-exit watchdog incidents get a type), so `failure_type` cannot be used to triage 503s — filter on the error text. Root cause **unknown**; not a repeat of the 10-14 TCP-refusal storm (which was reachability, this was serving). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
|
||||
### Historical
|
||||
- **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]].
|
||||
- **Session storage failures (2026-08-10):** Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See [[current-state]].
|
||||
|
||||
Reference in New Issue
Block a user