🦉 wiki audit 2026-10-22: omni-101 pre-dispatch filter skip (3rd failure mode, new page), Hermes 0.22.3 update with snapshot skipped, Gitea backup backlog 76
- new: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md - systems/hermes-agent.md: version -> 0.22.3 (5b147ab036), 22:47-22:57 update, snapshot stage skipped, advisories unactioned - systems/scheduled-tasks.md: 2026-10-21 = 171 completed / 1 failed; non-clean stays 9 with 08eb86f02d37 out, dca8482e4f76 in - systems/omniroute.md: one pin, three signatures in eight days - incidents/2026-10-16-omniroute-503-resource-pressure.md: window update, stays monitoring - incidents/2026-08-08-gitea-config-backup-push-dead.md: 2026-10-21 run, backlog 75 -> 76, message_id 8449 - infrastructure/backups.md: 76 commits; pre_update_backup:false now exercised - current-state.md, index.md, log.md updated
This commit is contained in:
+34
-4
@@ -3,16 +3,46 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-21
|
||||
verified_on: 2026-10-21
|
||||
review_after: 2026-10-28
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
review_after: 2026-10-29
|
||||
confidence: high
|
||||
tags: [system]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-21
|
||||
Updated: 2026-10-22
|
||||
|
||||
## Recent Changes (2026-10-22, nightly audit; live probes 2026-10-21 23:59 → 2026-10-22 00:20 AWST)
|
||||
|
||||
- **Hermes Agent updated to `0.22.3`.** `hermes update` ran **2026-10-21 22:47:25 → 22:57:38 AWST**
|
||||
(action `9eed3a99e799347c691ea52a72e122a1`), from pre-update tree `0.22.2` / `d923a5eb487a17c0…` to commit
|
||||
`5b147ab036d6…` (detached, install stamp `displayVersion 0.22.3`, built 2026-10-21T14:57:38Z). Stages
|
||||
`plan` / `apply` (git) / `deps` / `build` all succeeded, **but `snapshot` = `skipped`**
|
||||
(`disabled by updates.pre_update_backup (mode: off)`). Gateway restarted via **systemd** — pid `868087`,
|
||||
`phase: running`, `api_server: connected`, heartbeat 2026-10-22 00:13 AWST — and the dashboard was respawned
|
||||
by argv on `:9119` (pid `869852`). Post-run advisories, **neither yet acted on**: rebuild the search index
|
||||
(`hermes sessions optimize-storage`) and *Preparing pinned cua-driver (Computer Use)…* (no completion line in
|
||||
the log). → [[systems/hermes-agent]]
|
||||
- **The only cron failure of 2026-10-21 is a new failure mode.** Window: **171 `completed` / 1 `failed`** (+ this
|
||||
audit). `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`) failed
|
||||
at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by
|
||||
pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string (zero such rows on 10-21;
|
||||
last 2026-10-19 09:02) — while the sibling 07:00 job `85edf6772bca` completed normally. Incident
|
||||
`dca848_014affeb6954` **`alerted`, unclosed**; the run produced no output (`cron_incomplete_no_output`,
|
||||
82 msgs / 47 tool calls). → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
|
||||
- **Incidents: 1 closed / 1 opened; unclosed stays 9** (8 `alerted` / 1 `detected`). `08eb86_472c7b609c80`
|
||||
**resolved 2026-10-21 08:37:21** after the job's 08:30 slot ran clean, which also cleared `08eb86f02d37` from
|
||||
the non-clean table. Non-clean `last_status` **stays 9** (7 `error` / 2 `delivery_failed`), one swap in/out.
|
||||
- **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the orphaned `08eb86f02d37` run. The previous
|
||||
audit's "168 / 0" was taken before the 23:59 slot landed.
|
||||
→ [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]
|
||||
- **Config backup: 76 commits now unpushed.** The 2026-10-21 11:00 run created commit `a8486a20` and the push
|
||||
failed for the 75th consecutive night; Telegram notice `message_id 8449` was delivered, AgentMail failed.
|
||||
→ [[incidents/2026-08-08-gitea-config-backup-push-dead]]
|
||||
- **12 sessions in the window, all `cron` — no interactive sessions.** Fleet unchanged at **28 jobs
|
||||
(25 enabled / 3 disabled)**; **17 pin `custom:omni-101`**, **7 carry no pin**, 1 pins `custom:omni`.
|
||||
|
||||
## Recent Changes (2026-10-21, 03:00 stocktake; live probes 03:00-03:12 AWST)
|
||||
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 Whi
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-21
|
||||
updated: 2026-10-21
|
||||
verified_on: 2026-10-21
|
||||
review_after: 2026-10-28
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
review_after: 2026-10-29
|
||||
confidence: high
|
||||
tags: [backup, gitea, cron, git, green-vs-actual]
|
||||
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki]
|
||||
@@ -83,6 +83,21 @@ tail -20 ~/.hermes/config-backup/backup.log
|
||||
tail -3 ~/.hermes/config-backup/.backup_history
|
||||
```
|
||||
|
||||
## Update 2026-10-21 (nightly run — the backlog grew again)
|
||||
|
||||
| Probe | Result |
|
||||
|---|---|
|
||||
| 2026-10-21 11:00 run | commit **`a8486a20`** created (`🦝 nightly backup: 2026-10-21 11:00:14 AWST`) — **push failed again**, `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address` |
|
||||
| `git rev-list --count origin/main..main` | **76** (was 75) — the backlog grows by exactly one commit per night |
|
||||
| Telegram notice | delivered: `🚨 BACKUP FAILED — 2026-10-21 11:00:14 AWST`, **`message_id 8449`**; the AgentMail step failed again |
|
||||
| GitHub fallback | `fatal: 'github' does not appear to be a git repository`; its "pushing clean branch" recovery (`github-clean-1792551629`) also failed and was deleted |
|
||||
| `.backup_history` | `2026-10-19 … FAIL`, `2026-10-20 … FAIL`, **`2026-10-21 … FAIL`** |
|
||||
| cron row `e27da6c34bbb` | still `last_status: ok` — the green-vs-actual gap is now **74 days** wide |
|
||||
|
||||
**Compounding factor found the same night:** the `hermes update` that ran at **2026-10-21 22:47 AWST** recorded
|
||||
its own `snapshot` stage as **`skipped`** (`disabled by updates.pre_update_backup (mode: off)`), so for that
|
||||
change as well the *only* off-box copy path was this dead push → [[infrastructure/backups]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[infrastructure/backups]]
|
||||
|
||||
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
|
||||
type: incident
|
||||
status: monitoring
|
||||
created: 2026-10-18
|
||||
updated: 2026-10-21
|
||||
verified_on: 2026-10-21
|
||||
review_after: 2026-10-28
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
review_after: 2026-10-29
|
||||
confidence: high
|
||||
tags: [cron, router, provider, infrastructure]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
|
||||
@@ -220,6 +220,25 @@ clean running, not a fix).
|
||||
`nous` (`https://inference-api.nousresearch.com/v1`), `model.default: smart-route` →
|
||||
`meituan/longcat-2.5-preview:free`. Pinned jobs are unaffected; unpinned jobs are. → [[systems/current-profile]]
|
||||
|
||||
## Update 2026-10-21 (window 2026-10-20 → 10-21)
|
||||
|
||||
- **A third failure mode from the same pin — and it is not 503.** The only failed cron row on **2026-10-21**
|
||||
was `dca8482e4f76` (Daily Marketing Job Radar, `custom:omni-101` / `smart-route`), which died after
|
||||
**40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch
|
||||
filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]].
|
||||
- **The resource-pressure string did not appear at all on 2026-10-21** (zero ledger rows; last occurrence
|
||||
remains 2026-10-19 09:02), so the second burst stays closed and this page stays `status: monitoring`.
|
||||
- **One row closed, one opened.** `08eb86_472c7b609c80` (the Gmail Action Digest 503s, first seen 10-16 08:33)
|
||||
**resolved 2026-10-21 08:37:21** — the job's own 08:30 slot ran clean that morning (its 10-20 run was the
|
||||
orphan; no incident row was ever raised for it), which also cleared `08eb86f02d37` from the non-clean table.
|
||||
`dca848_014affeb6954` opened 07:40:42 and is **still `alerted`**. Unclosed rows **stay at 9**
|
||||
(8 `alerted` / 1 `detected`).
|
||||
- **Windows:** 2026-10-20 settled at **169 `completed` / 1 `unknown`** (the gateway-orphaned run; the 10-21
|
||||
audit's 168 / 0 was a mid-flight snapshot taken before the 23:59 slot landed). **2026-10-21: 171 `completed`
|
||||
/ 1 `failed`** (+ this audit).
|
||||
- **Still no failover.** Every job that has failed on this node since 2026-10-14 still reads
|
||||
`provider: custom:omni-101` with `model: smart-route` or `free-stack`, and none has a second pin.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node.
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
title: Daily Marketing Job Radar — "All Targets Skipped by Pre-Dispatch Filters" (2026-10-21)
|
||||
type: incident
|
||||
status: unresolved
|
||||
created: 2026-10-22
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
review_after: 2026-10-29
|
||||
confidence: high
|
||||
tags: [cron, router, provider, job-search, failure-mode]
|
||||
sources: [live-jobs-json, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-cron-output-file]
|
||||
---
|
||||
|
||||
# Daily Marketing Job Radar — Router Pre-Dispatch Filter Skip (2026-10-21)
|
||||
|
||||
## Headline
|
||||
|
||||
The **Daily Marketing Job Radar** (`dca8482e4f76`) — the weekday **07:00** slot that scrapes
|
||||
SEEK / LinkedIn / Indeed / Jora, verifies against direct adverts, dedupes into the Notion job DB and
|
||||
delivers a Telegram summary — **failed on 2026-10-21 after 40 m 36 s** with a signature the wiki has not
|
||||
recorded before:
|
||||
|
||||
```
|
||||
RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s
|
||||
```
|
||||
|
||||
This is the **third distinct failure mode from the same provider pin** (`provider: custom:omni-101`,
|
||||
`model: smart-route`) after the connection-refused storm (2026-10-14/15) and the two
|
||||
`HTTP 503 … resource pressure` bursts (2026-10-16/17, 2026-10-19) documented in
|
||||
[[incidents/2026-10-16-omniroute-503-resource-pressure]]. **The 503 string does not appear in any
|
||||
2026-10-21 row.**
|
||||
|
||||
## Evidence (live, 2026-10-22 00:1x AWST)
|
||||
|
||||
| Probe | Result |
|
||||
|---|---|
|
||||
| `jobs.json` `dca8482e4f76` | `last_status: error`, `failure_streak: 1`, `schedule: 0 7 * * 1-5`, `provider: custom:omni-101`, `model: smart-route`, `deliver: telegram:-1004321904721:1030` |
|
||||
| executions row `cd1d1f6a64ed427b9de13f31086ffb78` | `status: failed` — claimed **07:00:00.99**, started **07:00:12.40**, last progress **07:40:01.33**, finished **07:40:48.54** (**40 m 36 s**), `delivery_outcome: delivered` |
|
||||
| `cron_incidents` row `dca848_014affeb6954` | `state: alerted`, `failure_type: unknown`, `first_seen_at: 2026-10-21T07:40:42.61`, `alerted_at: 2026-10-21T07:40:48.50`, **`closed_at: null`**, `error` = the string above |
|
||||
| session `cron_dca8482e4f76_20261021_070022` | **82 messages / 47 tool calls**, ended 07:40:42, `end_reason: cron_incomplete_no_output` |
|
||||
| cron output file | `/home/hermes/.hermes/cron/output/dca8482e4f76/2026-10-21_07-40-42.md` — `# Cron Job: Daily Marketing Job Radar (FAILED)` |
|
||||
| the same hour's sibling | `85edf6772bca` ran 07:00:16 → 07:03:44 `cron_complete` (4 msgs) — the node was serving other jobs at that hour |
|
||||
| whole 2026-10-21 fleet | **171 `completed` / 1 `failed`** — this row is the only failure of the day |
|
||||
| 503 resource-pressure rows, 2026-10-21 | **zero** (last occurrence remains 2026-10-19 09:02) |
|
||||
|
||||
## What the error says, and what it does not
|
||||
|
||||
- **Not a 503**, **not a connection error**, **not a repetition-loop cut** — those three strings are the
|
||||
entire recorded failure vocabulary of this fleet, and none of them matches.
|
||||
- The phrasing is the harness/router's own. The wiki has **no page documenting the pre-dispatch filter
|
||||
chain** that emits it, so the mechanism is unverified. What the string does assert: every dispatch target
|
||||
was filtered out **before** a model call was made, and the guard reset **19 m 20 s** into the run before the
|
||||
job was failed. The run itself spent **40 m 36 s** alive (07:00:12 → 07:40:48).
|
||||
- **Not caused by the 2026-10-21 `hermes update`.** That update ran 22:47 → 22:57 AWST — 15 h *after* this
|
||||
07:00 failure. → [[systems/hermes-agent]]
|
||||
- **Not a delivery problem.** `delivery_outcome: delivered`: the failure notice reached topic `:1030`, so the
|
||||
slot was visibly red in Telegram rather than silently blank.
|
||||
|
||||
## Impact
|
||||
|
||||
One weekday radar slot produced nothing: no new roles discovered, verified, de-duped or written to the Notion
|
||||
job DB, and no candidate summary delivered to topic `:1030`. The job recovers on its next weekday slot
|
||||
(2026-10-22 07:00) if the guard does not repeat.
|
||||
|
||||
## Open questions / next actions (proposed, not applied — needs Anthony's go-ahead)
|
||||
|
||||
1. **Find which component emits this string.** If it is a router-side guard with a fixed reset window, a
|
||||
job-level retry after the reset would have recovered the slot; if it is a local filter chain, one filter is
|
||||
probably mis-configured for this job's toolset.
|
||||
2. **Capture router-side telemetry at failure time** on the `omni-101` node — three distinct failure modes
|
||||
from one pin in eight days with no host telemetry captured for any of them.
|
||||
3. **Give the radar a second provider pin.** It has no failover: one node's guard blanks the slot.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
- [[systems/omniroute]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/hermes-agent]]
|
||||
@@ -1,6 +1,6 @@
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. Last updated: 2026-10-21
|
||||
> model routing and automation. Last updated: 2026-10-22
|
||||
>
|
||||
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
||||
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
|
||||
@@ -34,10 +34,11 @@
|
||||
### Integrations
|
||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP).
|
||||
- [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip.
|
||||
- [[systems/scheduled-tasks]] — **As of 2026-10-21 00:0x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12.** The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
|
||||
- [[systems/scheduled-tasks]] — **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
|
||||
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. The **2026-10-17** slot failed (`HTTP 503`), but the **10-18 03:00** slot closed the hole: `freshness-20261018.txt` written, index regenerated and committed (`dd53e5f`) — freshness **47 stale / 41 fresh / 7 no-date**. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
|
||||
|
||||
## Incidents
|
||||
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` **`alerted`, unclosed**; the only cron failure of the day (171 `completed` / 1 `failed`). **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
|
||||
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery.
|
||||
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]]
|
||||
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Backups
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
confidence: medium
|
||||
tags: [infrastructure, backups]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -49,13 +49,17 @@ How Hermes's own configuration/state gets backed up, and what's missing.
|
||||
> tarball step in [[runbooks/backup-wiki]] has never been executed (the runbook's `mkdir -p` would
|
||||
> create it on first use).
|
||||
- `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net.
|
||||
**Exercised on 2026-10-21:** the 22:47 `hermes update` (0.22.2 → 0.22.3) recorded its `snapshot` stage as
|
||||
`skipped` with reason `disabled by updates.pre_update_backup (mode: off)`, so that update shipped with **no
|
||||
local snapshot and no off-box copy** — the Gitea mirror below has been dead since 2026-08-08.
|
||||
→ [[systems/hermes-agent]]
|
||||
|
||||
## ⚠ Scheduled config backup — the Gitea mirror has been dead since 2026-08-08
|
||||
|
||||
`e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits
|
||||
locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its
|
||||
execution row reads `completed`.** Last commit that reached Gitea: **2026-08-07 11:00:47**; the local repo is
|
||||
**75 commits ahead of `origin/main`**. Root cause is a credential: `origin` is
|
||||
**76 commits ahead of `origin/main`**. Root cause is a credential: `origin` is
|
||||
`https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — HTTPS, no userinfo, no credential
|
||||
helper — so the non-interactive push dies on
|
||||
`fatal: could not read Username … No such device or address` → `✗ Push to Gitea FAILED` →
|
||||
@@ -64,7 +68,7 @@ helper — so the non-interactive push dies on
|
||||
|
||||
**Consequence for this page: everything above describes a topology that has not carried `~/.hermes` off the box
|
||||
since 2026-08-07.** The 2026-07-22 full zip and the wiki/authoring repos remain the only recent off-box copies.
|
||||
Telegram failure notices *have* been firing daily (last: 2026-10-20 11:00, `message_id 8447`); the AgentMail
|
||||
Telegram failure notices *have* been firing daily (last: 2026-10-21 11:00:14, `message_id 8449`); the AgentMail
|
||||
notification step failed.
|
||||
|
||||
## Relevant runbooks
|
||||
|
||||
@@ -1023,3 +1023,39 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st
|
||||
- **Fleet unchanged** — 28 jobs / 25 enabled. This job (`1a8e90ea2271`) is `active`, `last_status: ok`,
|
||||
`model: free-stack` / `provider: custom:omni-101`, next run 2026-10-22 03:00.
|
||||
- **No secrets written.** Only host names, IPs, container IDs, job IDs, port numbers and HTTP status codes.
|
||||
|
||||
## [2026-10-21] audit | Nightly wiki audit — window 2026-10-21 00:00 → 23:59 AWST: Hermes updated to 0.22.3 with its snapshot skipped, and a third failure signature from the `omni-101` pin
|
||||
|
||||
- **New page: [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]].** The **Daily Marketing Job Radar**
|
||||
(`dca8482e4f76`, 07:00 weekday, `custom:omni-101` / `smart-route`) failed **2026-10-21 07:40:48 after
|
||||
40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch
|
||||
filters reset after 19m 20s` — a string this fleet has never produced before, and **not** the 503
|
||||
resource-pressure error (zero such rows exist for 10-21; last 2026-10-19 09:02). Session
|
||||
`cron_dca8482e4f76_20261021_070022` wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls);
|
||||
incident `dca848_014affeb6954` is **`alerted` and unclosed**; it was the day's only failure
|
||||
(171 `completed` / 1 `failed`).
|
||||
- **Infrastructure: Hermes Agent updated to `0.22.3`** — `hermes update` ran **22:47:25 → 22:57:38 AWST**
|
||||
(action `9eed3a99e799347c691ea52a72e122a1`), pre `0.22.2` / `d923a5eb…` → commit `5b147ab036…` (detached),
|
||||
stamp built 2026-10-21T14:57:38Z. **Stage `snapshot` = `skipped`** (`disabled by
|
||||
updates.pre_update_backup (mode: off)`), so the update had no snapshot and the off-box copy path is still
|
||||
dead. Gateway restarted via systemd (pid `868087`, `phase: running`, heartbeat 00:13 AWST), dashboard
|
||||
respawned by argv (`:9119`, pid `869852`). Advisories printed, neither actioned: rebuild the search index
|
||||
(`hermes sessions optimize-storage`) and *Preparing pinned cua-driver…* (no completion line in the log).
|
||||
The intermediate `d923a5eb` rebuild was never recorded on the page, so the previously-noted
|
||||
`v0.21.5+6169.gab86a20` (2026-10-04) was already stale — corrected.
|
||||
- **Pages updated:** `systems/hermes-agent.md` (version/upstream bullets rewritten, frontmatter dated),
|
||||
`systems/scheduled-tasks.md` (new current-count bullet; 10-21 bullet marked superseded),
|
||||
`systems/omniroute.md` (new 2026-10-21 section: three signatures, one pin),
|
||||
`incidents/2026-10-16-omniroute-503-resource-pressure.md` (window-close update; stays `monitoring`),
|
||||
`incidents/2026-08-08-gitea-config-backup-push-dead.md` (2026-10-21 run: backlog 75 → **76**,
|
||||
`message_id 8449`), `infrastructure/backups.md` (76 commits; `pre_update_backup: false` now has a
|
||||
concrete exercised example), `current-state.md` (new dated section), `index.md` (new incident entry,
|
||||
scheduled-tasks summary, date).
|
||||
- **Ledger facts as recorded this window:** fleet **28 jobs / 25 enabled / 3 disabled**; non-clean
|
||||
`last_status` **9** (7 `error` / 2 `delivery_failed`) with one swap — `08eb86f02d37` out (its 08:30 slot ran
|
||||
clean and `08eb86_472c7b609c80` **resolved 08:37:21**), `dca8482e4f76` in; **1 incident opened / 1 closed,
|
||||
unclosed stays 9** (8 `alerted` / 1 `detected`); **12 sessions, all `cron`, no interactive sessions**;
|
||||
2026-10-20 settled at **169 `completed` / 1 `unknown`** (the orphan) — the 10-21 audit's "168 / 0" was taken
|
||||
before the 23:59 slot landed, and that correction is recorded in both `systems/scheduled-tasks.md` and
|
||||
`current-state.md`.
|
||||
- **No secrets written.** Host names, IPs, job IDs, incident IDs, ports, commit SHAs and HTTP status codes only.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Hermes Agent
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
confidence: high
|
||||
tags: [system, hermes, agent, llm]
|
||||
sources: [live-hermes-version, live-config, live-skill-inventory]
|
||||
@@ -16,10 +16,10 @@ sources: [live-hermes-version, live-config, live-skill-inventory]
|
||||
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
|
||||
|
||||
## Current Status
|
||||
- **Version:** v0.21.5+6169.gab86a20 (2026.9.24), upstream `ab86a20a` — verified live 2026-10-04. Same semver token as the earlier `v0.21.5+3837.gbfda74c` (`27062c34`), but a newer build: the install tree was rebuilt 2026-10-03 10:11 AWST.
|
||||
- **Version:** `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`.
|
||||
- **Install method:** git
|
||||
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
|
||||
- **Upstream status:** tree rebuilt 2026-10-03 from upstream `ab86a20a`; update availability not re-checked since.
|
||||
- **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on.
|
||||
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
||||
- **Managed runtimes:** 3 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv` (was 10 — the 2026-10-03 rebuild pruned superseded environments and created two new ones at 09:59/10:00; the third survives from 2026-09-28). `hermes-runtime-deps-guard.sh` runs every 15 min via `mnemosyne-watchdog.py` and now asserts `dotenv` + `mnemosyne` in the installer python *and every managed venv*, **plus** `fastembed` + `FASTEMBED_CACHE_PATH` + the local model dir in the bootstrap-selected venv — see [[incidents/2026-09-28-hermes-managed-runtime-deps]] and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||||
- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
|
||||
|
||||
+16
-2
@@ -3,8 +3,8 @@ title: OmniRoute
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
confidence: high
|
||||
tags: [routing, provider, model, inference]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -12,6 +12,20 @@ sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
|
||||
# OmniRoute
|
||||
|
||||
## Recent Changes (2026-10-21)
|
||||
|
||||
- **Third failure mode on the same node, and it is not the 503 string.** `dca8482e4f76` (Daily Marketing Job
|
||||
Radar) failed on 2026-10-21 at 07:40:48 after 40 m 36 s with
|
||||
`RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after
|
||||
19m 20s`. **Zero `HTTP 503 … resource pressure` rows exist for that day** (last: 2026-10-19 09:02) and the
|
||||
sibling 07:00 job `85edf6772bca` completed normally, so the node was serving work at that hour.
|
||||
→ [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
|
||||
- **One pin, three signatures in eight days:** connection-refused storm (10-14/15), resource-pressure 503s
|
||||
(10-16/17 and 10-19), pre-dispatch filter skip (10-21) — all on `custom:omni-101`, all with
|
||||
`model: smart-route`. No job in the fleet has a second provider pin.
|
||||
- **Router-side telemetry is still missing for every one of them** — `omni-101` host logs have not been
|
||||
captured during a failure, which is why none of the three has a confirmed root cause.
|
||||
|
||||
## Recent Changes (2026-10-20)
|
||||
|
||||
- **Recurring `503 … resource pressure` load-shedding on `omniroute-1` — second burst in four days.** On
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-21
|
||||
verified_on: 2026-10-21
|
||||
review_after: 2026-10-28
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
review_after: 2026-10-29
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **⚠ Current as of 2026-10-21 00:0x AWST (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
- **Current as of 2026-10-22 00:1x AWST (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]].
|
||||
- **⚠ Current as of 2026-10-21 00:0x AWST *(superseded by the 2026-10-22 entry above)* (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
- **Current as of 2026-10-20 00:09 AWST *(superseded by the 2026-10-21 entry above)* — the `omni-101` 503 fault is BACK, and it is now the fleet's defining risk.** Window 2026-10-19: **159 `completed` / 13 `failed`**, against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all carry the identical `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on `custom:omni-101`, in two clusters — **06:01** (one job) and **07:01 → 12:01** (seven more runs across seven jobs). The other **4 rows are derivative Mnemosyne Watchdog exits** (`96eb95ce12f2` at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the **second burst in four days** on the same node with the same blast radius, while `/api/health` read `200 {"status":"ok"}` throughout — so [[incidents/2026-10-16-omniroute-503-resource-pressure]] is back to **`status: active`** (it had been downgraded to `monitoring` 44 h earlier). **Fleet unchanged at 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` 7 → 12** (10 `error` + 2 `delivery_failed`); **unclosed incident rows 13** (12 `alerted` / 1 `detected`) — 4 opened and 2 closed on 10-19. Every `failure_streak` still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. **15 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
- **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
||||
- **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
|
||||
Reference in New Issue
Block a user