audit 2026-10-18: new incident page (omniroute-1 503 resource pressure, 9 calls / 7 jobs), fleet re-verified 28 jobs / 25 enabled / 8 non-clean, Mosquito Tracker removed, radar bot token dead
This commit is contained in:
+38
-3
@@ -3,8 +3,8 @@ title: Current State
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-16
|
updated: 2026-10-18
|
||||||
verified_on: 2026-10-16
|
verified_on: 2026-10-18
|
||||||
review_after: 2026-10-25
|
review_after: 2026-10-25
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system]
|
tags: [system]
|
||||||
@@ -12,7 +12,42 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
|
|||||||
---
|
---
|
||||||
|
|
||||||
# Current State
|
# Current State
|
||||||
Updated: 2026-10-16
|
Updated: 2026-10-18
|
||||||
|
|
||||||
|
## Recent Changes (2026-10-18, nightly audit; live probes 2026-10-18 00:31-00:36 AWST)
|
||||||
|
|
||||||
|
- **The `omniroute-1` node is up — and this time it shed load instead of refusing connections.** Three days
|
||||||
|
after the 34-hour TCP blackout, the same node produced a second, different failure mode: **9 model calls
|
||||||
|
across 7 jobs died with `RuntimeError: HTTP 503: Service temporarily unavailable due to resource
|
||||||
|
pressure.`** between **10-16 08:00:37 and 10-17 03:00:28** (~19 h), while `/api/health` answered
|
||||||
|
`200 {"status":"ok"}` and `/v1/models` answered `401` before, during and after (re-probed 10-18 00:33).
|
||||||
|
→ [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||||
|
- **It cleared itself and the fleet is green again.** `10-16 = 165 completed / 8 failed`, `10-17 = 162 / 5`
|
||||||
|
(**all five before 03:01**), `10-18` so far `7 / 0` — **no failure row of any kind since 10-17 03:00**.
|
||||||
|
Every job that failed re-ran clean on its next slot: `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17
|
||||||
|
08:31, `ecc1fc13655a` and `ea79a2107e36` both 10-17 12:02, each of those auto-closing its own incident.
|
||||||
|
- **Fleet count fell 29 → 28: the Mosquito Tracker was removed.** `1b31187397a5` is absent from
|
||||||
|
`jobs.json` (file `updated_at 2026-10-18T00:36:17+08:00`); its last execution was **10-16 15:00**
|
||||||
|
(`completed`, delivered). The three disabled jobs are unchanged — **25 enabled / 3 disabled**.
|
||||||
|
- **Non-clean `last_status`: 12 → 8.** Two of the eight cannot self-heal before their next slot —
|
||||||
|
`1b23c042e17d` sits at `error` (streak 2) until **Fri 10-23 08:00**; `0e519874d280`, `06a0a1c8704f`,
|
||||||
|
`c8d52904e1d6`, `0e753e9720fd` and `066bb5191280` are unchanged. **Open incidents 12 → 8**
|
||||||
|
(5 `alerted` / 3 `detected`); **9 opened and 6 closed** inside this window. → [[systems/scheduled-tasks]]
|
||||||
|
- **The watchdog double-counts — 4 of the 13 failure rows are one sentinel.** `96eb95ce12f2` exited 1 on
|
||||||
|
10-16 12:15 + 12:30 and 10-17 00:15 + 00:30, each row quoting the coverage monitor's own 503 error.
|
||||||
|
Read failures per job, not per row. → [[systems/mnemosyne]]
|
||||||
|
- **Second consecutive stocktake hole.** The 10-17 03:00 slot failed (503), so
|
||||||
|
`raw/stocktake/freshness-20261017.txt` does not exist and `systems/skills-index.md` is stamped **10-16**
|
||||||
|
(10-15 is missing too, from the storm — `RuntimeError: Connection error.`). The **10-18 03:00** slot
|
||||||
|
closes it; nothing was regenerated by hand, because a manual run would race the scheduled one.
|
||||||
|
- **Fifth consecutive window with no interactive session** — **30 sessions** across the two days, all
|
||||||
|
`cron`, and **zero** non-cron sessions since 2026-10-15. Every finding here comes from live system
|
||||||
|
state, not conversation.
|
||||||
|
- **The GitHub Trending Radar is dead at the delivery end.** The Telegram bot token in
|
||||||
|
`~/workspace/github-trending-radar/config.json` returns **HTTP 401** on `getMe` (probed 10-18 00:35);
|
||||||
|
the job's GitHub PAT still answers **200**, so only the delivery credential is gone. The cron row stays
|
||||||
|
green because `last_status` was set by the **10-12 09:00** run (`last_run_at 2026-10-12T09:00:12`).
|
||||||
|
Next exercise: **Mon 2026-10-19 09:00**. **Not repaired** — a replacement bot token is Anthony's call.
|
||||||
|
|
||||||
## Recent Changes (2026-10-16, nightly audit; live probes 2026-10-16 00:0x–00:2x AWST)
|
## Recent Changes (2026-10-16, nightly audit; live probes 2026-10-16 00:0x–00:2x AWST)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,119 @@
|
|||||||
|
---
|
||||||
|
title: OmniRoute `omni-101` Resource-Pressure 503s — Cron Failure Burst
|
||||||
|
type: incident
|
||||||
|
status: monitoring
|
||||||
|
created: 2026-10-18
|
||||||
|
updated: 2026-10-18
|
||||||
|
verified_on: 2026-10-18
|
||||||
|
review_after: 2026-10-25
|
||||||
|
confidence: high
|
||||||
|
tags: [cron, router, provider, infrastructure]
|
||||||
|
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
|
||||||
|
---
|
||||||
|
|
||||||
|
# OmniRoute `omni-101` Resource-Pressure 503s — Cron Failure Burst (2026-10-16 → 2026-10-17)
|
||||||
|
|
||||||
|
**Window:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
|
||||||
|
**Status:** **Monitoring.** Every slot from the 2026-10-17 08:30 run onward completed green
|
||||||
|
(10-16 = **165 completed / 8 failed**, 10-17 = **162 / 5**, and the 10-17 failures all pre-date 03:01).
|
||||||
|
The node answers `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` (auth-required) — re-probed
|
||||||
|
2026-10-18 00:33 AWST.
|
||||||
|
**Impact:** **13 failed execution rows** — **9 genuine model-call failures across 7 jobs**, plus **4 derivative
|
||||||
|
Mnemosyne Watchdog rows**. Against the 10-14/10-15 storm tail (10-15 = 152 / 24) and the last clean day
|
||||||
|
(10-12 = 175 / 0). Unlike the 10-14 storm, **the node was up and serving throughout** — model calls failed with
|
||||||
|
`503` resource-pressure responses rather than TCP refusals.
|
||||||
|
|
||||||
|
## What happened
|
||||||
|
|
||||||
|
Every model call on `custom:omni-101` (`http://100.78.63.20:20128/v1`) between the 10-16 08:00 and
|
||||||
|
10-17 03:00 slots died with the identical error:
|
||||||
|
|
||||||
|
```
|
||||||
|
RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.
|
||||||
|
```
|
||||||
|
|
||||||
|
| Job | Slot(s) AWST | Rows | Outcome |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `1b23c042e17d` Enhanced Weekend Adventure Planner | 10-16 08:00 (Fri slot) | 1 | failed; streak **2**; next slot **Fri 10-23 08:00** |
|
||||||
|
| `08eb86f02d37` Gmail Action Digest | 10-16 08:30 | 1 | failed; cleared 10-17 08:31 (own next run) |
|
||||||
|
| `b63ed5817212` Mnemosyne Embedding Coverage Monitor | 10-16 12:00, 10-17 00:00 | 2 | failed; cleared **10-17 06:01** |
|
||||||
|
| `ecc1fc13655a` FreshRSS Midday News Briefing | 10-16 12:00 | 1 | failed; cleared 10-17 12:02 |
|
||||||
|
| `ea79a2107e36` Mnemosyne Health Monitor | 10-16 12:00, 10-17 00:00 | 2 | failed; cleared 10-17 12:02 |
|
||||||
|
| `cbfe434a3b5e` Wiki Audit | 10-16 23:59 | 1 | failed; incident **open** (`detected`) — the 10-17 23:59 run is the one that produced this audit |
|
||||||
|
| `1a8e90ea2271` Wiki Stocktake | 10-17 03:00 | 1 | failed; incident **open** (`detected`) |
|
||||||
|
| `96eb95ce12f2` Mnemosyne Watchdog | 10-16 12:15 + 12:30, 10-17 00:15 + 00:30 | 4 | **derivative** — `Script exited with code 1`, its stdout quoting the watchdog's own log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…` |
|
||||||
|
|
||||||
|
**8 rows on 10-16, 5 on 10-17.** Reading per job rather than per row: **7 jobs actually failed**, and four of
|
||||||
|
the 13 rows are one sentinel watching another job's 503 — the same double-counting trap as the 10-14 storm
|
||||||
|
(see [[incidents/2026-10-14-omniroute-node-connection-refused-storm]], lesson 4).
|
||||||
|
|
||||||
|
Every 503 row sits on **`custom:omni-101`** — one alias, one node, again the single pin.
|
||||||
|
No other job failed in this window: `85edf6772bca` (Tech & AI) and `dca8482e4f76` (Job Radar) both ran
|
||||||
|
clean through it.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- `~/.hermes/cron/executions.db` — the 9 model-call rows each carry the literal
|
||||||
|
`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.`
|
||||||
|
The 4 watchdog rows carry `Script exited with code 1` plus the quoting stdout above.
|
||||||
|
- `~/.hermes/cron/executions.db:cron_incidents` — **9 incidents opened in the window, 6 closed, 3 still open**
|
||||||
|
(`1b23c0_103cbe04a79b` `alerted`; `cbfe43_9b89947a046a` and `1a8e90_3c9c1d573cde` `detected`).
|
||||||
|
**The ledger tags 7 of the 9 `failure_type: unknown`** even though the message was explicit (the two
|
||||||
|
watchdog rows are classed `script`, correctly — they *are* script exits); for the seven real 503s the
|
||||||
|
classifier had the message in hand and still filed no type. Open incidents overall: **12 → 8**
|
||||||
|
(5 `alerted` / 3 `detected`).
|
||||||
|
- **The node itself was never down.** `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` on
|
||||||
|
`100.78.63.20:20128` at 2026-10-18 00:33 AWST. A 503 is a *serving* failure, so reachability probes
|
||||||
|
(which the nightly audit and the operator both run) could not have caught this — only job-level error text did.
|
||||||
|
- **No node-side telemetry was retained.** Nothing on the Hermes side records host CPU/RAM/queue depth at
|
||||||
|
10-16 08:00, so the burst can be described but not explained. Treat the root cause as **unknown**.
|
||||||
|
|
||||||
|
## Why this is not a repeat of the 10-14 storm
|
||||||
|
|
||||||
|
| | 2026-10-14 → 10-15 | 2026-10-16 → 10-17 |
|
||||||
|
|---|---|---|
|
||||||
|
| Failure class | `ConnectError: [Errno 111] Connection refused` | HTTP `503` resource pressure |
|
||||||
|
| Node state | TCP-unreachable for ~34 h | up and answering probes throughout |
|
||||||
|
| Blast radius | 12 jobs, 24 model-call failures, every call for 34 h | 7 jobs, 9 model-call failures, bursty, self-cleared |
|
||||||
|
| Recovery | manual node recovery | none needed — cleared on its own |
|
||||||
|
|
||||||
|
Both are the **same single-node topology problem**: all enabled model-calling cron jobs ride
|
||||||
|
`custom:omni-101` with no failover, three days apart from the previous outage.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- **One stocktake hole:** `raw/stocktake/freshness-20261017.txt` was not written and
|
||||||
|
`systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 03:00 slot succeeded, so
|
||||||
|
`freshness-20261016.txt` exists). The **2026-10-18 03:00 slot** closes it — nothing was regenerated by hand,
|
||||||
|
because a manual run would race the scheduled one. A failure there would be the second consecutive miss.
|
||||||
|
- **The Wiki Audit lost its 10-16 slot**, so no wiki pass covered the 10-14→10-17 span until this run.
|
||||||
|
- **Stale flags clear only at the next success, and some slots are a week apart.** `1b23c042e17d` is a Friday
|
||||||
|
job: it will sit at `last_status: error`, streak 2, with an open incident until **Fri 2026-10-23 08:00**.
|
||||||
|
On a weekly cadence a one-slot failure reads as a week-long outage.
|
||||||
|
|
||||||
|
## Lessons
|
||||||
|
|
||||||
|
1. **Same topology, second failure mode in three days.** A 503 from a *live* node cannot be detected by
|
||||||
|
reachability probes; only per-job error text distinguishes "node down" from "node overloaded". Both
|
||||||
|
classes gate the whole scheduled estate.
|
||||||
|
2. **Watchdog rows inflate the count — read per job.** 4 of the 13 rows are one log sentinel reacting to
|
||||||
|
another job's failure, not independent faults.
|
||||||
|
3. **The incident classifier drops the failure type.** All **seven** genuine 503 incidents are filed
|
||||||
|
`failure_type: unknown` despite an explicit message in the execution row (only the two script-exit
|
||||||
|
watchdog incidents get a type), so `failure_type` cannot be used to triage — filter on the error text.
|
||||||
|
4. **A 503 burst leaves no evidence behind.** No host telemetry is captured at failure time, so the "resource
|
||||||
|
pressure" cause is taken on the router's word. If this recurs, capture the node's host metrics *during* the
|
||||||
|
burst, not afterwards.
|
||||||
|
|
||||||
|
## Open item (needs Anthony's call, unchanged from 10-14)
|
||||||
|
|
||||||
|
The durable fix is the same one flagged in [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]:
|
||||||
|
a second provider pin on a subset of jobs, or router-side failover between `omniroute-1` and CT600
|
||||||
|
`omniroute`. **Not applied** — that is a routing decision, not an audit one.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node.
|
||||||
|
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — the first multi-day storm from this topology.
|
||||||
|
- [[systems/omniroute]] — node layout, providers and pins.
|
||||||
|
- [[systems/scheduled-tasks]] — per-window fleet tally and incident ledger.
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# Anthony's Systems Wiki
|
# Anthony's Systems Wiki
|
||||||
> Operational knowledge for Anthony's agents, infrastructure,
|
> Operational knowledge for Anthony's agents, infrastructure,
|
||||||
> model routing and automation. Last updated: 2026-10-16
|
> model routing and automation. Last updated: 2026-10-18
|
||||||
>
|
>
|
||||||
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
||||||
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
|
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
|
||||||
@@ -33,10 +33,11 @@
|
|||||||
|
|
||||||
### Integrations
|
### Integrations
|
||||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
||||||
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-16 00:1x — the **`omniroute-1` connection-refused storm (2026-10-14 → 10-15)** cost **24 model-call failures across 12 jobs** and blanked ~34 h of digests; **all 18 model-calling jobs sit on `custom:omni-101` with no failover**. See [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]. Prior audit: **4 carried a non-clean `last_status`** as of 2026-10-14 00:15. **The clean streak broke on 2026-10-13: `167 completed` / 1 `failed`**, after 10-12 finalised `175 / 0` (10-11 `172 / 0`, 10-10 `175 / 0`) — `1a8e90ea2271` (wiki-stocktake-maintenance, 03:00) hit `HTTP 404: No active credentials for provider: opencode`, its **third failure in six days**, so **open incidents rose 2 → 3**. Two rows cleared: `fad5a86979d4` (10-12 03:05), `dca8482e4f76` (10-13 07:02). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. ⚠ A green cron row is not proof of delivery: GitHub Trending Radar's Telegram token is revoked (`getMe` → 401) while the row stays green — it next runs Mon 09:00. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
|
- [[systems/scheduled-tasks]] — **28 cron jobs (25 enabled, 3 disabled)**; **8 carry a non-clean `last_status`** as of 2026-10-18 00:3x — down from 12. The **`omniroute-1` 503 resource-pressure window (2026-10-16 08:00 → 10-17 03:00)** cost **9 model-call failures across 7 jobs** and was followed by a clean 10-17 06:00 → 10-18 window; the Mosquito Tracker was removed (29 → 28 jobs). See [[incidents/2026-10-16-omniroute-503-resource-pressure]]. Prior incident: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs** and ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
|
||||||
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
|
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. ⚠ **2026-10-17** stocktake slot failed (`HTTP 503`), so `freshness-20261017.txt` is missing and the index is stamped **10-16**; the **10-18 03:00** slot closes it. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
|
||||||
|
|
||||||
## Incidents
|
## Incidents
|
||||||
|
- [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **OmniRoute's second failure mode, 2026-10-16 → 10-17:** `omniroute-1` load-shed model calls with **`HTTP 503 … resource pressure`** — **9 model calls across 7 jobs** over ~19 h, while `/api/health` stayed `200 ok` and `/v1/models` stayed `401`. **Self-cleared** (last 503 10-17 03:00:28; 10-17 closed 162 / 5, 10-18 so far 7 / 0). 9 incidents opened, 6 closed; **8 open** overall, and the ledger tags **7 of the 9 `failure_type: unknown`**. **Single-node topology still open.**
|
||||||
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved)
|
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved)
|
||||||
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)
|
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)
|
||||||
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)
|
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)
|
||||||
|
|||||||
@@ -780,3 +780,68 @@ a second provider pin on a subset of jobs, or router-side failover. **Not applie
|
|||||||
decision, not an audit one.
|
decision, not an audit one.
|
||||||
|
|
||||||
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
|
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
|
||||||
|
|
||||||
|
## [2026-10-18] audit | nightly review of 2026-10-16 -> 2026-10-17 — OmniRoute's second failure mode (503 resource pressure), Mosquito Tracker removed, radar bot token dead
|
||||||
|
|
||||||
|
**Window:** cron executions 2026-10-16 00:00 → 2026-10-17 24:00 (+ 10-18 00:00 → 00:36 live probes), ran
|
||||||
|
2026-10-18 00:31 → 00:5x AWST. **Sessions reviewed:** 30, **all `cron`, zero interactive** — fifth
|
||||||
|
consecutive window with no interactive session, so every finding below is from live system state, not
|
||||||
|
conversation.
|
||||||
|
|
||||||
|
### Summary
|
||||||
|
- **`omniroute-1` (`custom:omni-101`) has a second failure mode: 503 load-shedding.** **9 model calls across
|
||||||
|
7 jobs** died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry
|
||||||
|
shortly.` between **10-16 08:00:37** and **10-17 03:00:28** (~19 h). The control plane stayed up the whole
|
||||||
|
time (`/api/health` `200 {"status":"ok"}`, `/v1/models` `401` — re-probed 10-18 00:33), so a green health
|
||||||
|
endpoint is not evidence that the model path is serving. This is **not** a repeat of the 10-14 storm:
|
||||||
|
that was reachability (`Connection error.`, Errno 111), this was the router serving and refusing under
|
||||||
|
load. → `incidents/2026-10-16-omniroute-503-resource-pressure` (new).
|
||||||
|
- **The burst ended on its own and the fleet is green again.** `10-16 = 165 completed / 8 failed`,
|
||||||
|
`10-17 = 162 / 5` (all five before 03:01), `10-18` so far `7 / 0`; **no failure row of any kind since
|
||||||
|
10-17 03:00**. Every affected job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01,
|
||||||
|
`08eb86f02d37` 10-17 08:31, `ecc1fc13655a` + `ea79a2107e36` both 10-17 12:02, each auto-closing its own
|
||||||
|
incident. **No change was made on the node**; nothing was restarted or repinned.
|
||||||
|
- **Mosquito Tracker removed — fleet 29 → 28 (`1b31187397a5`).** Absent from `jobs.json`
|
||||||
|
(`updated_at 2026-10-18T00:36:17+08:00`), last execution 10-16 15:00 (`completed`, delivered). The three
|
||||||
|
disabled jobs are unchanged, so **25 enabled / 3 disabled**. Non-clean `last_status` **12 → 8**; only
|
||||||
|
`1b23c042e17d` (`error`, streak 2, next slot Fri 10-23 08:00) is structurally stuck.
|
||||||
|
- **The watchdog double-counts.** `96eb95ce12f2` exited 1 four times (10-16 12:15 + 12:30, 10-17 00:15 +
|
||||||
|
00:30), each row quoting the coverage monitor's own 503 — 4 of the 13 failure rows in the window are one
|
||||||
|
sentinel. Read failures per job, not per row.
|
||||||
|
- **Second consecutive stocktake hole, again left alone deliberately.** The 10-17 03:00 slot failed, so
|
||||||
|
`raw/stocktake/freshness-20261017.txt` is missing and `systems/skills-index.md` is stamped 10-16;
|
||||||
|
10-15 is missing too (storm). The **10-18 03:00** slot closes it — no manual regeneration, because a
|
||||||
|
hand-run would race the scheduled one.
|
||||||
|
- **GitHub Trending Radar is dead at the delivery end.** The Telegram bot token in its `config.json`
|
||||||
|
returns **HTTP 401** on `getMe` (probe 10-18 00:35); the job's GitHub PAT still returns **200**, so only
|
||||||
|
the delivery credential is gone. The cron row stays green (`last_status` from the 10-12 09:00 run,
|
||||||
|
`last_run_at 2026-10-12T09:00:12`). Next exercise **Mon 2026-10-19 09:00**. **Not repaired** — a new bot
|
||||||
|
token is Anthony's call, and the secret is not written here.
|
||||||
|
- **Incidents:** **9 opened, 6 closed** in-window; open **12 → 8** (5 `alerted` / 3 `detected`). The ledger
|
||||||
|
files **7 of the 9** 503 rows `failure_type: unknown` despite the explicit message (the two script-exit
|
||||||
|
rows are typed `script`), so `failure_type` still cannot be used to triage.
|
||||||
|
- **Mnemosyne:** both monitors rode the burst (`ea79a2107e36` 10-17 00:00, `b63ed5817212` 10-16 12:03),
|
||||||
|
both clean on every slot since — including 10-18 00:00 (`delivery_outcome: suppressed`). **No fresh
|
||||||
|
coverage figures exist** (they report to Telegram, not the ledger); **last explicit read stays
|
||||||
|
2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %)**, above the 95 % reindex
|
||||||
|
threshold, so no reindex is due.
|
||||||
|
|
||||||
|
**Pages touched:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (**new**);
|
||||||
|
`systems/scheduled-tasks.md` (frontmatter + Count bullet + stocktake row + Known Issues entry);
|
||||||
|
`systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter +
|
||||||
|
new Re-verified section); `current-state.md` (frontmatter + `Updated:` line + new top section);
|
||||||
|
`index.md` (`Last updated`, new incident entry, scheduled-tasks bullet, skills-index note);
|
||||||
|
`log.md` (this entry). **Nothing removed or archived** — no page became obsolete in the window.
|
||||||
|
|
||||||
|
**Bridge housekeeping:** the shared pages are byte-identical between the authoring tree
|
||||||
|
(`/home/hermes/workspace/agent-estate-wiki`) and the vault copy (`/home/hermes/wiki/agent-estate-wiki`,
|
||||||
|
pages nested under `wiki/`), so this run mirrored the touched pages into the vault *after* writing the
|
||||||
|
authoring tree, and left the vault-only pages (`systems/ignis.md`, `systems/nextcloud.md`, three
|
||||||
|
`incidents/` pages, `raw/stocktake/`, `agents/`) untouched. No deletions, no one-way sync.
|
||||||
|
|
||||||
|
**Durable fix still open (needs Anthony's call):** the single-node topology — all 18 model-calling jobs on
|
||||||
|
`custom:omni-101` with no failover, now seen failing two different ways in five days. Options remain a
|
||||||
|
second provider pin on a subset of jobs, or router-side failover. **Not applied** — a routing decision, not
|
||||||
|
an audit one.
|
||||||
|
|
||||||
|
No secrets written; host names, IPs, VMIDs, job/incident IDs and probe status codes only.
|
||||||
|
|||||||
+16
-2
@@ -3,8 +3,8 @@ title: Mnemosyne
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-16
|
updated: 2026-10-18
|
||||||
verified_on: 2026-10-16
|
verified_on: 2026-10-18
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [memory, mnemosyne, backend]
|
tags: [memory, mnemosyne, backend]
|
||||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||||||
@@ -141,3 +141,17 @@ Primary agent memory store for Hermes.
|
|||||||
mode `0755`, so the monitor's non-executable finding is resolved.
|
mode `0755`, so the monitor's non-executable finding is resolved.
|
||||||
- **Open, unverified:** 13 orphan rows in `memory_embeddings` (oldest 2026-08-22) reported by a
|
- **Open, unverified:** 13 orphan rows in `memory_embeddings` (oldest 2026-08-22) reported by a
|
||||||
2026-10-10 monitor run - neither confirmed nor cleaned by this audit.
|
2026-10-10 monitor run - neither confirmed nor cleaned by this audit.
|
||||||
|
|
||||||
|
## Re-verified 2026-10-18 (audit window; live probes 2026-10-18 00:30-00:36 AWST)
|
||||||
|
|
||||||
|
- **Both monitors rode the `omniroute-1` 503 window; both are green again.** `ea79a2107e36` (health, every
|
||||||
|
12 h) failed **10-17 00:00** with `RuntimeError: HTTP 503 ... resource pressure` and delivered its alert;
|
||||||
|
`b63ed5817212` (coverage, every 6 h) failed **10-16 12:03** the same way. Since **10-17 12:00** every slot
|
||||||
|
of both has completed with `delivery_outcome: suppressed` (nothing to report) — including **10-18 00:00**.
|
||||||
|
- **No fresh coverage figures exist in the ledger.** These jobs report to Telegram, not to the execution
|
||||||
|
row, so the **last explicit read remains 2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056
|
||||||
|
(99.4 %)**, both above the 95 % reindex threshold. Nothing here indicates a reindex is due.
|
||||||
|
- **The watchdog triple-checks the same thing.** `96eb95ce12f2` (every 15 min) exited 1 four times —
|
||||||
|
10-16 12:15 + 12:30, 10-17 00:15 + 00:30 — each row quoting the **coverage monitor's own 503 error**, i.e.
|
||||||
|
one sentinel producing four failure rows. Read failures **per job**, not per row. It has been clean on
|
||||||
|
every slot since (latest 10-18 00:30).
|
||||||
|
|||||||
+17
-2
@@ -3,8 +3,8 @@ title: OmniRoute
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-16
|
updated: 2026-10-18
|
||||||
verified_on: 2026-10-16
|
verified_on: 2026-10-18
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [routing, provider, model, inference]
|
tags: [routing, provider, model, inference]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -66,6 +66,21 @@ The wiki previously recorded CT600 as retired and CT601 as the live successor. *
|
|||||||
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
|
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
|
||||||
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
|
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
|
||||||
|
|
||||||
|
## Recent Changes (2026-10-18) — `omniroute-1` 503 resource pressure (2026-10-16 -> 2026-10-17)
|
||||||
|
|
||||||
|
- **A second, different failure mode from the same node.** Three days after the 34-hour TCP blackout,
|
||||||
|
`omniroute-1` answered model calls with **`HTTP 503: Service temporarily unavailable due to resource
|
||||||
|
pressure. Retry shortly.`** — **9 model calls across 7 jobs**, between **10-16 08:00:37** and
|
||||||
|
**10-17 03:00:28** (~19 h). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||||
|
- **The control plane looked healthy throughout.** `/api/health` returned `200 {"status":"ok"}` and
|
||||||
|
`/v1/models` returned `401` (auth-required) before, during and after the window — re-probed 2026-10-18
|
||||||
|
00:33. A green health endpoint does **not** mean the model path is serving: 503 here is load-shedding
|
||||||
|
under resource pressure, not a dead process, so process checks miss it.
|
||||||
|
- **It recovered on its own; nothing was changed on the node.** Last 503 **10-17 03:00:28**; `10-17`
|
||||||
|
closed **162 completed / 5 failed** (all five before 03:01) and `10-18` is green so far. Every affected
|
||||||
|
job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17 08:31,
|
||||||
|
`ecc1fc13655a` and `ea79a2107e36` both 10-17 12:02, auto-closing their own incidents.
|
||||||
|
|
||||||
## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm
|
## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm
|
||||||
|
|
||||||
- **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14
|
- **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14
|
||||||
|
|||||||
+17
-13
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-16
|
updated: 2026-10-18
|
||||||
verified_on: 2026-10-16
|
verified_on: 2026-10-18
|
||||||
review_after: 2026-10-25
|
review_after: 2026-10-25
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, scheduled, automation]
|
tags: [cron, scheduled, automation]
|
||||||
@@ -14,9 +14,10 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
|||||||
# Scheduled Tasks
|
# Scheduled Tasks
|
||||||
|
|
||||||
## Count
|
## Count
|
||||||
|
- **✅ Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||||
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
||||||
- **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, re-verified 2026-10-13 — unchanged; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10)
|
- **Total jobs:** **28 — 25 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-18 00:31 AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs.
|
||||||
- **Non-clean `last_status`:** **3** (was 5 at the 2026-10-12 audit) — `0e753e9720fd` (`error`, streak 0), `1b23c042e17d` (`error`, streak 1), `066bb5191280` (`delivery_failed`, streak 0). `fad5a86979d4` and `dca8482e4f76` cleared on **2026-10-12** when both ran clean on schedule.
|
- **Non-clean `last_status`:** **8 as of 2026-10-18 00:31** — `1b23c042e17d` (`error`, streak 2), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`, streak 0). History: **3** at the 2026-10-12 audit (when `fad5a86979d4` and `dca8482e4f76` cleared on schedule) → **12** at the 2026-10-16 storm peak → 8 now. The six jobs that fixed themselves did so on their next scheduled slot; the remaining weekly ones (`1b23c042e17d` Fri, `0e519874d280` Mon, `06a0a1c8704f` + `c8d52904e1d6` Thu) cannot clear before their next run.
|
||||||
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]].
|
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]].
|
||||||
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
|
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
|
||||||
- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window.
|
- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window.
|
||||||
@@ -40,21 +41,20 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
|||||||
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
|
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
|
||||||
| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. Live `last_status: ok`, streak 0. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open) |
|
| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. Live `last_status: ok`, streak 0. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open) |
|
||||||
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant |
|
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant |
|
||||||
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1. **Next scheduled slot: Fri 2026-10-16 08:00**, which should clear it |
|
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] |
|
||||||
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
||||||
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic |
|
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic |
|
||||||
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). Live `last_status: ok`, streak 0. ⚠ Bare `telegram` target should move to a group topic |
|
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). Live `last_status: ok`, streak 0. ⚠ Bare `telegram` target should move to a group topic |
|
||||||
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | |
|
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed the incident. Live `last_status: ok` |
|
||||||
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. `last_status: ok` because the 401 is inside the script, not the cron run. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** |
|
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. `last_status: ok` because the 401 is inside the script, not the cron run. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** |
|
||||||
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | |
|
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); cleared by the 10-17 08:31 run, which auto-closed the incident. Live `last_status: ok` |
|
||||||
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
|
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
|
||||||
| Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
|
| Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
|
||||||
| Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
|
| Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
|
||||||
| Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
|
| Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
|
||||||
| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95% |
|
| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run |
|
||||||
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
|
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots (10-16 12:00, 10-17 00:00); cleared by its own **10-17 12:02** run |
|
||||||
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30) |
|
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults |
|
||||||
| Mosquito Tracker | `1b31187397a5` | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 |
|
|
||||||
| Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` |
|
| Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` |
|
||||||
| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | |
|
| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | |
|
||||||
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
|
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
|
||||||
@@ -62,8 +62,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
|||||||
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic |
|
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic |
|
||||||
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 |
|
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 |
|
||||||
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
|
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
|
||||||
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
|
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text |
|
||||||
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken |
|
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken. ⚠ **The 2026-10-17 03:00 slot failed** on the same `omni-101` 503 — the first miss since the **2026-10-15** storm slot (which died with `RuntimeError: Connection error.`; the 10-16 slot succeeded). `raw/stocktake/freshness-20261017.txt` was not written and `systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 slot succeeded, so `freshness-20261016.txt` exists); the **2026-10-18 03:00** slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident `1a8e90_3c9c1d573cde` `detected`. Note the ledger tags this row `failure_type: unknown` even though the execution carries an explicit 503 message |
|
||||||
| 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
| 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
||||||
|
|
||||||
## Removed Jobs
|
## Removed Jobs
|
||||||
@@ -71,6 +71,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Workday start reminder | 2026-08-26 | No longer present in live cron list |
|
| Workday start reminder | 2026-08-26 | No longer present in live cron list |
|
||||||
| CT 221 Miner Watchdog | 2026-08-26 | Job `d5516d0a5147` removed; CT221 status unknown/inspected |
|
| CT 221 Miner Watchdog | 2026-08-26 | Job `d5516d0a5147` removed; CT221 status unknown/inspected |
|
||||||
|
| DHL Tracking - Parcel 8052838382 | 2026-10-10 | `62619aa6bcb3` — parcel delivered, monitor redundant |
|
||||||
|
| Mosquito Tracker | 2026-10-16 (after the 15:00 run) | `1b31187397a5` — absent from `jobs.json` at the 2026-10-18 audit; last execution 2026-10-16 15:00 was `completed` / `delivered` |
|
||||||
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
|
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
|
||||||
|
|
||||||
## Disabled Jobs (not removed)
|
## Disabled Jobs (not removed)
|
||||||
@@ -116,6 +118,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
|||||||
- **Two wiki trees, drifted in both directions (NEW 2026-10-02):** `/home/hermes/workspace/agent-estate-wiki` is the canonical source (Gitea remote `Anthony/agent-estate-wiki`, Quartz build, `wiki-maintain.sh` target). `/home/hermes/wiki/agent-estate-wiki` is the **Obsidian/Ignis vault copy** the iPad reads — pages nested under `wiki/`, and **no git remote at all**. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged *backwards* (`systems/ignis.md`, `systems/nextcloud.md`, three `incidents/` pages exist only in the vault; `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` only in canonical) — a naive one-way sync deletes pages. See [[systems/hermes-agent]].
|
- **Two wiki trees, drifted in both directions (NEW 2026-10-02):** `/home/hermes/workspace/agent-estate-wiki` is the canonical source (Gitea remote `Anthony/agent-estate-wiki`, Quartz build, `wiki-maintain.sh` target). `/home/hermes/wiki/agent-estate-wiki` is the **Obsidian/Ignis vault copy** the iPad reads — pages nested under `wiki/`, and **no git remote at all**. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged *backwards* (`systems/ignis.md`, `systems/nextcloud.md`, three `incidents/` pages exist only in the vault; `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` only in canonical) — a naive one-way sync deletes pages. See [[systems/hermes-agent]].
|
||||||
- **Work mailbox unreachable (non-cron; affects on-demand lookups):** the `work` Outlook account is configured with password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` fails with `login not supported`. No cron job polls it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
|
- **Work mailbox unreachable (non-cron; affects on-demand lookups):** the `work` Outlook account is configured with password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` fails with `login not supported`. No cron job polls it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
|
||||||
|
|
||||||
|
- **The `omni-101` 503 burst left no node-side evidence (NEW 2026-10-18):** 9 model calls across 7 jobs died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` between 2026-10-16 08:00 and 2026-10-17 03:00 — while the node answered `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` before, during and after (re-probed 2026-10-18 00:33). Nothing on the Hermes side records the node's CPU/RAM/queue depth at failure time, so the "resource pressure" cause is taken on the router's word; if it recurs, capture host metrics *during* the burst, not afterwards. The incident ledger also tags **seven of the nine** rows `failure_type: unknown` **despite** the explicit message (only the two script-exit watchdog incidents get a type), so `failure_type` cannot be used to triage 503s — filter on the error text. Root cause **unknown**; not a repeat of the 10-14 TCP-refusal storm (which was reachability, this was serving). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||||
|
|
||||||
### Historical
|
### Historical
|
||||||
- **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]].
|
- **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]].
|
||||||
- **Session storage failures (2026-08-10):** Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See [[current-state]].
|
- **Session storage failures (2026-08-10):** Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See [[current-state]].
|
||||||
|
|||||||
Reference in New Issue
Block a user