audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output

Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed).

NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
 - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed
 - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6)
   reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded
 - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row
 - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2

RESOLVED with output (dated Recovery sections added):
 - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0
 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B)
 - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved

UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md
(opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x),
incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
This commit is contained in:
Hermes Wiki Audit
2026-11-12 00:16:39 +08:00
parent c108eb1012
commit 729bb597cf
11 changed files with 463 additions and 20 deletions
+62 -4
View File
@@ -3,16 +3,74 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-11-10
verified_on: 2026-11-10
review_after: 2026-11-10
updated: 2026-11-11
verified_on: 2026-11-11
review_after: 2026-11-12
confidence: high
tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
---
# Current State
Updated: 2026-11-10
Updated: 2026-11-11
## Recent Changes (2026-11-11, nightly audit; live probes 2026-11-11 23:59 – 2026-11-12 00:1x AWST)
**Recovery night — plus one failure, and it was the auditor's own.**
→ [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]
- **The 2026-11-10 audit died on its own 429.** `cbfe434a3b5e` ran 2026-11-10T23:59:54.486572 →
**2026-11-11T00:12:29.646987 = 12 m 35 s**, `failed`, delivery `suppressed`:
`RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch
filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` — **6 candidates, `attempted 0`**, tenant
`tn-5ud77abtl1` quota exceeded. Row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 but
**`alerted_at: NULL`, `closed_at: NULL` — it is the estate's only open incident row**. Pool reset
≈ **06:36:57**; the 07:0x cohort ran clean.
⇒ **The 2026-11-10 window was never audited.** This audit covers two windows.
- **State reads:** fleet **29 (26 enabled / 3 disabled)**; **non-clean `last_status` 2** (was 1) —
`066bb5191280` Uber Eats (`delivery_failed`, streak 0) and `cbfe434a3b5e` (`error`, streak 1).
Ledger — **2026-11-09: 174 completed / 0 failed**; **2026-11-10: 157 rows, 155 completed / 2 failed**;
**2026-11-11: 172 rows, 171 completed / 1 running / 0 failed** (15 distinct jobs) — first zero-failure
day since 11-09. **Deliveries 11-11: 7 rows, 7 / 7 `delivered`, 0 `for_failure`** (last at 18:45:58).
- **Stocktake recovered with output.** `1a8e90ea2271` 03:00:56.081504 → **04:11:38.724852 = 1 h 10 m
42 s**, `completed`, **36 stale pages re-verified → 0 stale**, **commits `341dda9` + `c108eb1`**,
freshness report `raw/stocktake/freshness-20261111.txt` (6,235 B). `341dda9` also rewrote the live
fallback chain in [[runbooks/test-openrouter]] (primary back to `smart-route` / `custom:omni-101`),
corrected the CT333 `clawtest` naming claim, and replaced the nous refresh-token reading with the
404-credits signature. Closes [[incidents/2026-11-09-stocktake-completes-with-zero-edits]].
⚠ The job is **still pinned to `free-stack`**, the combo recorded broken on 2026-08-14, and succeeded.
- **Job Radar recovered with output.** Stage 1 `dca8482e4f76` 07:01:08.358004 → **07:37:34.076224 =
36 m 26 s**, `completed`, writing **`radar-ref/handoff/2026-11-11.json` (22,682 B, mtime 07:59)** — the
first handoff since **2026-11-06**. Stage 2 `5fc292ff0f3b`: recovery fire 07:35:56.632032 →
08:03:47.816811 (`delivered`), scheduled 11:30:56 → 11:35:18, 14:30:55 → 14:34:56 (`delivered`). Row
`5fc292_773822455856` auto-closed **`resolved` 2026-11-11T08:03:47.644226**, closing
[[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]].
- **Longest runs.** 11-11: stocktake **1 h 10 m 42 s**, Radar S1 **36 m 26 s**, Radar S2 **27 m 51 s**
(recovery fire), Newsletter Digest `ecc1fc13655a` 12:01:02.571781 → 12:19:38.965387 (**18 m 36 s**),
`85edf6772bca` 07:01:08.287627 → 07:11:42.858722 (**10 m 34 s**). **Nothing over 71 minutes.**
11-10: Radar S2 scheduled slot 11:30:35.267155 → 22:18:49.814215 = **10 h 48 m 14 s**, `failed` /
`interrupted`, delivery `delivered` with `for_failure: 1`.
- **Provider state (unchanged config; worsening symptoms).** `config.yaml` mtime still **2026-11-09
22:58:07** — nothing repaired. Top model `custom:omni-101` / `smart-route`
(`http://100.78.63.20:20128`). `fallback_providers` (6): nous `meituan/longcat-2.5-preview:free` →
**opencode-zen `deepseek-v4.1-flash` (EXPIRED key)** → `custom:guacano` → `custom:litellm`
`kilo-reflection-beam-free` → kilocode `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`)
`or-inclusionai_ling-2_6-1t`. **The expired opencode-zen key is no longer latent** — **7 × HTTP 401
`api_key_expired`** on 11-11 (07:01:13 … 19:19:39), each immediately after nous returned **14 × 404
`insufficient_credits_for_paid_model`** (07:01:11 … last 19:19:38, all from session
`20261031_190813_95fbde`, source `desktop`, opened 2026-10-31, **still open**, pinned to that paid
model). **Two consecutive dead hops on one chain.** → [[systems/opencode-go]]
- **Auxiliary slots:** nine slots plus `free_only: true`. Eight chain
`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` on `custom:omni-101`.
**`pulse_memory` remains the sole exception with no `fallback_chain`**
(`{"provider": "nous", "model": "anthropic/claude-haiku-4.5"}`) — 19 aux log lines on 11-11 across
06:0x / 08:0x / 13:0x / 18:0x, including **11 ×** *"main provider `custom:omni-101` is unavailable and
no fallback_chain / fallback_providers is configured — refusing to guess another logged-in provider"*.
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- **Error log:** `~/.hermes/logs/errors.log` = **6,179 lines** through **2026-11-12 00:10:11**.
- **Still open, untouched this window:** Uber Eats Monthly Spend Report `delivery_failed` streak 0 (no
execution row on 11-11 — monthly cadence); disabled trio unchanged (Important-mail monitor, Rhino
AgentMail Inbox Watcher, AgentMail Rhino Inbox Poll); the three inherited items below.
## Recent Changes (2026-11-10, nightly audit; live probes 2026-11-09 23:5x – 2026-11-10 01:0x AWST)
@@ -66,7 +66,31 @@ sources: [gateway-log, live-config, live-probe]
> currently erroring), and/or add credits for paid nous models. The original re-authenticate item
> is **superseded** — auth works now.
> **Escalated 2026-11-11 (nightly audit).** The single-slot exposure described above became the
> busiest error path on the host, and it now has a **downstream casualty**.
>
> - `anthropic/claude-haiku-4.5` on `nous` returned **404 `insufficient_credits_for_paid_model`**
> **14 times** on 2026-11-11 (07:01:11 … last **19:19:38**) — up from 3 in the whole preceding
> window. All from session `20261031_190813_95fbde` (**source `desktop`, started 2026-10-31
> 19:08:13, still open**), whose pinned model is that same paid nous model.
> - The **404 is immediately followed by a 401.** The session's fallback chain moves the turn on to
> `opencode-zen` / `deepseek-v4.1-flash`, whose key expired on 2026-11-08 — **7 × HTTP 401
> `api_key_expired`** on 11-11 (07:01:13, 07:19:30, 08:03:40, 13:03:28, 13:19:33, 18:03:39,
> 19:19:39). So **both hops of this chain are dead**, and the pair repeats every time the session is
> poked. → [[systems/opencode-go]]
> - `auxiliary.pulse_memory` logged **19 lines** on 11-11 (06:0x, 08:0x, 13:0x, 18:0x), including
> **11 ×** *"main provider `custom:omni-101` is unavailable and no fallback_chain /
> fallback_providers is configured — refusing to guess another logged-in provider."* Live config
> re-confirmed at audit time: `pulse_memory` is the **only** slot carrying **no `fallback_chain`**
> (`{"provider": "nous", "model": "anthropic/claude-haiku-4.5"}`), while the other eight all chain
> `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` on `custom:omni-101`.
> - **Net effect:** pulse/voice memory retrieval failed on every attempt this window. `config.yaml`
> mtime is unchanged since **2026-11-09 22:58:07** — nothing was fixed.
>
> **Open item, unchanged and now evidenced:** move `auxiliary.pulse_memory` onto `custom:omni-101`
> like the other eight slots and give it a `fallback_chain`, and re-point or retire the stale desktop
> session pinned to a paid nous model. No credentials were minted or modified.
## Related
- [[systems/model-providers]]
- [[systems/current-profile]]
- [[systems/hermes-agent]]
@@ -1,10 +1,10 @@
---
title: "2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op"
type: incident
status: active
status: resolved
created: 2026-11-10
updated: 2026-11-10
verified_on: 2026-11-10
updated: 2026-11-11
verified_on: 2026-11-11
confidence: high
tags: [cron, job-radar, stall, silent-failure, handoff, read-loop]
sources: ["executions.db (dca8482e4f76: 2026-11-09T07:00:48+08:00 → 2026-11-09T16:10:12+08:00, status completed, delivery_outcome suppressed; 5fc292ff0f3b: 11:30:38 → 12:10:07 and 14:30:34 → 14:37:27, both completed/suppressed)", "~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md (ends INCOMPLETE: tool budget exhausted; last words \\u201cI stopped retrying because I kept running read_file 5 times without making progress.\\u201d)", "~/.hermes/radar-ref/handoff/ (newest handoff on disk: 2026-11-06; no 2026-11-07/08/09 files)", "cron_incidents (no row raised for this run)"]
@@ -57,6 +57,31 @@ Two compounding faults, both previously documented for this job:
- Add a **read-progress guard** to Stage 1: when the same file has been read N times without a state change, stop and emit a partial handoff instead of burning the budget to `INCOMPLETE`.
- Note the pattern across incidents: **completion-time-only alerting hides every failure in this family** (stocktake 11-09, radar 11-09, radar 11-02/03).
## Recovery — 2026-11-11: the handoff landed, and Stage 2 delivered
| Field | 11-09 (this incident) | **11-11** |
|---|---|---|
| Stage 1 | 07:00:48 → 16:10:12 (**9 h 09 m 24 s**), no handoff | **07:01:08 → 07:37:34 = 36 m 26 s**, `completed` |
| Handoff on disk | newest was **2026-11-06** | **`radar-ref/handoff/2026-11-11.json` — 22,682 B, mtime 07:59** |
| Stage 2 (11:30 slot) | fired early, declined silently | **11:30:56 → 11:35:18**, `completed` |
| Stage 2 (14:30 slot) | declined silently | **14:30:55 → 14:34:56**, `completed` / **`delivered`** |
| Incident row | **none raised** | `5fc292_773822455856` closed **`resolved` 2026-11-11T08:03:47.644226** |
- **The first handoff in five days** (`2026-11-06` → `2026-11-11`) and the fastest Stage-1 cycle since
the two-stage split: **36 m 26 s** against 3 h 15 m (10-29, the last full success), 9 h 01 m (11-02),
6 h 28 m (11-03) and 9 h 09 m (11-09).
- **Stage 2's recovery fire** (`source: direct`, 07:35:56 → 08:03:47) consumed the handoff and delivered
at **08:03:47**; its completion is what auto-closed the row opened by 11-10's `interrupted` slot.
- ⚠ **Carried forward from 11-10:** the *scheduled* Stage-2 slot that day ran
**11:30:35 → 22:18:49 = 10 h 48 m 14 s** and closed **`failed` / `interrupted`** (delivery
`delivered`, `for_failure: 1`) — the longest run of either window, and itself a new data point for the
"completion-time-only alerting" family on this page.
- **Job state:** both radar jobs read `last_status: ok`, `failure_streak` 0.
**Status: `resolved`.** Both prevention rules on this page still stand and remain unimplemented: Stage 2
should hard-fail on a missing `handoff/<date>.json` rather than suppressing, and Stage 1 needs a
read-progress guard. Neither was needed on 11-11 because the run finished inside budget.
## Related
- [[systems/scheduled-tasks]]
@@ -1,10 +1,10 @@
---
title: "2026-11-09 — Stocktake Completes With Zero Edits"
type: incident
status: active
status: resolved
created: 2026-11-10
updated: 2026-11-10
verified_on: 2026-11-10
updated: 2026-11-11
verified_on: 2026-11-11
confidence: high
tags: [cron, wiki, stocktake, tool-budget, silent-failure, omni-101, free-stack]
sources: ["executions.db (1a8e90ea2271: 2026-11-09T03:00:43+08:00 → 2026-11-09T12:10:47+08:00, status completed, delivery_outcome suppressed)", "~/.hermes/cron/output/1a8e90ea2271/2026-11-09_12-10-46.md (8,305 bytes; RUN INCOMPLETE, \\u201ctool-iteration budget exhausted during verification\\u201d)", "cron_incidents (1a8e90_628d914f1f97, 1a8e90_2f370ef764f5, 1a8e90_a9cad99cb150 — all closed 2026-11-09T12:10:47.125205+08:00)", "wiki-stocktake-maintenance job 1a8e90ea2271 (provider custom:omni-101, model free-stack)"]
@@ -57,6 +57,35 @@ Also recorded this window: the `free-stack` pin on `1a8e90ea2271` — the only d
- **Alerting must fire on write progress, not completion.** This run produced 2 of 3 failure days silently and the third day produced a run that reported success while editing nothing. A check on `commit_count` / `edits_applied` for this job would have caught all three.
- Don't read `status`/`failure_streak` as the stocktake's health. Read whether a commit landed.
## Recovery — 2026-11-11: the pass ran in 1 h 10 m and landed 36 pages
The next scheduled slot **worked**, and worked quickly:
| Field | 11-09 (this incident) | **11-11** |
|---|---|---|
| Window | 03:00:43 → 12:10:47 | **03:00:56.081504 → 04:11:38.724852** |
| Duration | **9 h 10 m** | **1 h 10 m 42 s** |
| Terminal state | `completed` / `suppressed` | `completed` / `suppressed` |
| Edits landed | **0** | **36 stale pages re-verified → 0 stale** |
| Commit | **none** | **`341dda9`** + **`c108eb1`** |
| Freshness report | — | `raw/stocktake/freshness-20261111.txt` (**6,235 B**) |
- `341dda9` re-verified 30+ further pages live and de-flagged them, and applied real content
corrections — the live fallback chain rewritten in [[runbooks/test-openrouter]] (primary back to
`smart-route` / `custom:omni-101`), the CT333 `clawtest` naming claim corrected, and the nous
**404 `insufficient_credits_for_paid_model`** signature replacing the old refresh-token reading in
[[incidents/2026-10-22-nous-portal-refresh-token-invalid]].
- `c108eb1` regenerated the freshness report after the log/index addendum, so the published Quartz build
matches HEAD.
- **Job state:** `last_status: ok`, `failure_streak` 0, `last_run` 2026-11-11T03:00:56.
⚠ The job is **still pinned to `custom:omni-101` / `free-stack`** — the combo recorded broken on
2026-08-14 — and this run succeeded on it anyway. That is an observation, not a repin.
**Status: `resolved`.** The 11-07/11-08/11-09 run of failures is closed by output, not by a cosmetic
`failure_streak` reset. The page's own prevention rule stands: *alert on `commit_count` / `edits_applied`,
never on `status`* — this recovery is exactly why it matters, because 11-09 and 11-11 both read
`completed` / `suppressed` and only one of them did anything.
## Related
- [[systems/scheduled-tasks]]
@@ -0,0 +1,96 @@
---
title: Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited
type: incident
status: open
created: 2026-11-11
updated: 2026-11-11
verified_on: 2026-11-11
review_after: 2026-12-11
confidence: high
tags: [cron, incident, omniroute, rate-limit, audit]
sources: [live-executions-db, live-cron-incidents, gateway-log, live-jobs-json]
---
# Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited (2026-11-10 23:59 → 2026-11-11 00:12)
**This job is the estate's own observer, and on 2026-11-10 it became the estate's only failure.** Its
23:59 slot never produced a report, so **no page, no log entry and no incident alert describes the day
of 2026-11-10** — an unaudited day is invisible by construction.
## What happened
| Field | Value |
|---|---|
| Job | `cbfe434a3b5e` — **Wiki Audit — Nightly**, `custom:omni-101` / `deepseek-paid-smart` |
| Slot | 2026-11-10 23:59:54.486572 +08:00 (cron `59 23 * * *`) |
| Ended | **2026-11-11 00:12:29.646987 +08:00** — **12 m 35 s** |
| Status | `failed` · `delivery_outcome: suppressed` |
| Error | `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` |
| Output | `~/.hermes/cron/output/cbfe434a3b5e/2026-11-11_00-12-28.md` |
The three log lines from the run's own session `local-1000c1a3…` show the shape of the failure:
ERROR agent.chat_completion_helpers: Streaming failed before delivery: [429]: service failure: tenant tn-5ud77abtl1 quota exceeded
ERROR agent.chat_completion_helpers: Streaming failed before delivery: Error code: 429 -
ERROR agent.conversation_loop: API call failed after 3 retries. HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters
ERROR cron.scheduler: Job 'Wiki Audit — Nightly' failed: RuntimeError: HTTP 429: Service temporarily unavailable
- **The pool skipped every target.** Six candidates were evaluated and **`attempted 0`** — the run never
reached a provider. The only live target's credit balance read **100 %** consumed on connection
`conn:3021bab6`, which is why the router refused to dispatch rather than serving a degraded model.
- **`reset after 6h 24m 31s`** puts the pool's recovery at **≈ 2026-11-11 06:36:57 AWST**. The 07:0x
cron cohort ran clean the next morning, so the window closed on its own.
- The **same tenant** (`tn-5ud77abtl1`) appears in the `quota exceeded` line, so this is an
account-level quota fault on the router, not a single-model lockout.
## Incident row — raised, never alerted, still open
`cbfe43_ed01845310d8` — `job_id cbfe434a3b5e`, `failure_type: rate_limit`, state **`detected`**,
`alerted_at: NULL`, `closed_at: NULL`, opened **2026-11-11T00:12:28.578435+08:00**.
- It is the **only open row** in `cron_incidents` (135 rows total) — the previous open item,
`5fc292_773822455856`, closed `resolved` at **2026-11-11T08:03:47.644226**.
- So the estate's single outstanding incident is **the audit's own failure**, and nothing was told
about it.
## Impact
- **The 2026-11-10 window was never reviewed.** The nightly audit is the only producer of the per-day
ledger/wiki record; its failure means 11-10's 157 ledger rows were never reconciled on the day.
- **Fleet non-clean `last_status` 1 → 2.** `cbfe434a3b5e` now reads `last_status: error`,
`failure_streak: 1`, joining `066bb5191280` (*Uber Eats Monthly Spend Report*, `delivery_failed`,
streak 0).
- **The review published on 2026-11-11 23:59 carries both windows** — it is the recovery, not a
duplicate.
## Same signature, earlier appearances
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — the *identical* pre-dispatch-filter
string killed the Daily Marketing Job Radar on 2026-10-21 (`attempted 0`, no 503 anywhere in the day).
- 2026-11-05 — `deepseek-paid-smart` against the **CT600** endpoint returned `HTTP 503
ALL_TARGETS_SKIPPED` (`poolSize 5`, `attempted 0`) while `omniroute-1` answered 200 — recorded on
[[systems/scheduled-tasks]].
- This is the **third distinct symptom** of the same router behaviour: `attempted 0` with a healthy node
behind it is the thing to recognise, not the specific HTTP code.
## Fix Applied
**None — this is a detection-and-verification record.** No configuration changed this window. The run's
own recovery was the pool reset at ≈ 06:36:57 on 2026-11-11.
## Prevention
- **The observer needs an observer.** A failed nightly audit raises no delivery (suppressed), no alert
(`alerted_at: NULL`) and no page — every layer that would normally surface this is *inside the thing
that failed*. A one-line watchdog on `Wiki Audit — Nightly`'s `last_status != ok` (or on "no audit
commit in >24 h") is the only check that can catch it.
- **Alert on the artefact.** Verifying that a dated `audit(YYYY-MM-DD)` commit exists in this repo would
have caught both this failure and the stocktake's zero-edit days — the same lesson recorded on
[[incidents/2026-11-09-stocktake-completes-with-zero-edits]].
## Related
- [[systems/scheduled-tasks]]
- [[systems/omniroute]]
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]
+2 -1
View File
@@ -69,8 +69,9 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
- [[systems/skills-index]] — Installed skills inventory (**299 skills** (**67 categories**) as of the **2026-11-08** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
## Incidents
- [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **OPEN — the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]).
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - **RESOLVED (same day) — the hook was the fault, not the schedule.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** → [[systems/conduit]], [[systems/hermes-agent]]
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] - **RESOLVED BY ITSELF.** The nightly stocktake closed `completed` after **9 h 10 m** having landed **zero edits and no commit** — `failure_streak` 3 → 0 while the work stayed undone, and the top-priority item on its brief had been the monotonically worsening failure streak. Third consecutive day its findings never landed.
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] (resolved 2026-11-11) - **RESOLVED BY ITSELF.** The nightly stocktake closed `completed` after **9 h 10 m** having landed **zero edits and no commit** — `failure_streak` 3 → 0 while the work stayed undone, and the top-priority item on its brief had been the monotonically worsening failure streak. Third consecutive day its findings never landed.
- [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] - **NO INCIDENT RAISED.** Job Radar Stage 1 overran to **9 h 09 m**, skipped its own verification, wrote **no handoff file**, and both Stage-2 slots ran before it finished and declined silently. No incident row was created — the completion-time-only alerting gap leaves the failure invisible.
- [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - **OPEN.** A restart requested at **~04:29:52** sat in *after-turn* mode for the full **1800 s**, then timed out and drained at **05:00:43/47** — the same `#77184` behaviour as the 10-31 reload, but this time it killed **two** in-flight cron executions: *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42.918 -> 05:02:22.870) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31.515 -> 05:02:22.174), both closed **`failed`** on `lost its durable fire claim ownership`; systemd restarted the unit at **05:01:00** (`restart counter is at 3`). A **third** failure that night was the agent-log ERROR watchdog correctly catching the event 19 s later. The request landed **inside the `conduit_push` self-update window** (04:27:55 -> 04:32:32, its own ledger recording `restarting_gateway` = *failed*) — the **working hypothesis for the requester, not proven**; the 0.22.5 -> 0.23.0 updater is excluded (receipt starts 05:05:13). ⚠ **Closed 0 -> 2 incident rows and neither alerted**: both `1a8e90_*` rows sit `detected` with `alerted_at: null`. The stocktake slot has now failed **two days running** (`failure_streak: 2`). Job Radar ran clean. → [[systems/scheduled-tasks]], [[systems/hermes-agent]], [[systems/conduit]], [[systems/mnemosyne]]
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **OPEN.** A `direct`-source fire of `5fc292ff0f3b` (08:27:09 -> 13:26:22, 4 h 59 m 13 s, delivered) settled the claim while the scheduler's **own 11:30:32 slot** was still running, so that slot was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - a **new string variant** and the first claim-loss failure **with no gateway restart involved**. It raised **no `cron_incidents` row**, so a `failed` execution with no alert was the day's only red row; the discarded run left no output file. The 14:30 slot recovered the day in a normal **12 m 27 s** cycle. Candidate fixes (slot suppression when a claim is live, consuming the next slot for out-of-band fires, raising an incident row on discard) are listed on the page and **none was applied**.
+10
View File
@@ -2467,3 +2467,13 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
- **Estate shrank again (probed, not inferred):** `qm list` on `pve` now holds only VMs **100, 101, 110, 282** (VM **310** gone; the wiki had listed it) and `pct list` holds 17 CTs with **CT 601 gone**. VM 403/VM 401 and the `open-claw` Tailscale peer remain absent on both hypervisors → [[agents/open-claw]], [[agents/index]], `CONTRIBUTING.md`.
- **Delivery verified at the destination, not assumed:** before the commit, `http://100.118.5.51:9120/` and the changed pages (incl. `runbooks/test-openrouter`, `concepts/model-routing`, `incidents/2026-08-29-gmail-read-state`) all returned **200**, as did the external route `https://hermes.kangaroo-eel.ts.net/wiki/`. `CONTRIBUTING.md`'s access table was re-verified the same way and its `agents/open-claw/` scope row struck through ([[CONTRIBUTING]]).
- **No page deleted, no decision reversed, no secrets written.** No credential value appears in any page (the Gitea remote's embedded token is deliberately not reproduced). One pre-existing broken wikilink left as-is: `systems/guanaco.md` → `[[systems/ollama]]` (present in HEAD, page already qualifies it "if exists").
## 2026-11-12 00:1x AWST - Nightly wiki audit (window 2026-11-10 23:59 -> 2026-11-11 23:59)
- **The audit's own failure is the headline.** The 2026-11-10 23:59 slot (`cbfe434a3b5e`) ran **12 m 35 s** (23:59:54.486572 -> 2026-11-11 00:12:29.646987) and died on the router's pre-dispatch filter: `HTTP 429 ... all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` — **6 candidates, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded, delivery `suppressed`. **Consequence: the 2026-11-10 window was never reviewed.** Recovery was implicit (pool reset ≈ **06:36:57**; the 07:0x cohort ran clean), and this audit deliberately covers **both** windows. New page: [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]; new index entry.
- **The raised row was never alerted.** `cbfe43_ed01845310d8` (`rate_limit`) sits at `detected` with **`alerted_at: NULL`, `closed_at: NULL`** — the **only open row** in `cron_incidents` (the previous one, `5fc292_773822455856`, closed `resolved` at 08:03:47). Fleet non-clean `last_status` **1 -> 2** (`066bb5191280` Uber Eats `delivery_failed` streak 0, plus `cbfe434a3b5e` `error` streak 1).
- **Both long-running incidents recovered, with output.** Stocktake `1a8e90ea2271` **03:00:56.081504 -> 04:11:38.724852 = 1 h 10 m 42 s**, `completed`, **36 stale pages re-verified -> 0 stale**, commits **`341dda9`** + **`c108eb1`** (freshness report `raw/stocktake/freshness-20261111.txt`, 6,235 B) — closing [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]. Job Radar Stage 1 `dca8482e4f76` **07:01:08.358004 -> 07:37:34.076224 = 36 m 26 s**, writing the **first handoff since 2026-11-06** (`radar-ref/handoff/2026-11-11.json`, 22,682 B, mtime 07:59); Stage 2 `5fc292ff0f3b` recovery fire 07:35:56.632032 -> 08:03:47.816811 `delivered`, then 11:30:56 -> 11:35:18 and 14:30:55 -> 14:34:56 `delivered`, auto-closing the 11-10 row `resolved` — closing [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]. Both incident pages set to `status: resolved` with a dated Recovery section.
- **Ledger, both days re-parsed (times stored as ISO TEXT, grouped by `substr(started_at,1,10)`).** 2026-11-09 **174 completed / 0 failed**; **2026-11-10: 157 rows — 155 `completed` / 2 `failed`** (Radar S2 scheduled 11:30:35.267155 -> 22:18:49.814215 = **10 h 48 m 14 s** `interrupted` with `for_failure: 1`; the audit's own 12 m 35 s 429); **2026-11-11: 172 rows — 171 `completed` / 1 `running` / 0 `failed`**, 15 distinct jobs, **first zero-failure day since 11-09**. Longest 11-11 run **1 h 10 m 42 s** — **nothing over 71 minutes**, against 9 h 09 m (11-09) and 10 h 48 m (11-10). Fleet **29 (26 enabled / 3 disabled)** for a sixth window.
- **Deliveries checked at the destination:** 2026-11-11 = **7 rows, 7 / 7 `delivered`, 0 `for_failure`** (07:11:42, 07:31:01, 08:03:47, 08:34:36, 12:19:38, 14:34:55, 18:45:58); 2026-11-10's late row (22:18:48) is the one `for_failure: 1`. 2026-11-12 so far: 5 rows, 4 `completed` / 1 `running`.
- **Providers unchanged; symptoms worse.** `config.yaml` mtime still **2026-11-09 22:58:07** — nothing repaired this window. The expired `opencode-zen` key at `fallback_providers[1]` produced **7 x HTTP 401 `api_key_expired`** on 11-11 (07:01:13 -> 19:19:39), each **immediately after** nous returned **14 x 404 `insufficient_credits_for_paid_model`** (07:01:11 -> last 19:19:38, all from the still-open desktop session `20261031_190813_95fbde`). **Two consecutive dead hops on one chain** — [[systems/opencode-go]] updated from "latent risk" to observed failure; [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] gained the 11-11 escalation block. `auxiliary.pulse_memory` remains the **only slot with no `fallback_chain`** and logged **19 lines / 11 refusals** on 11-11.
- **Pages written:** 1 new incident; 2 incidents -> `resolved`; 2 system pages + `current-state.md` + `index.md` updated; `systems/scheduled-tasks.md` gained the 2026-11-11 count/state normalisation. `~/.hermes/logs/errors.log` is now **6,179 lines** through **2026-11-12 00:10:11**. **No page deleted, no decision reversed, no secrets written** — no credential value appears in any page; the opencode key is recorded by *state*, never by value.
+146
View File
@@ -0,0 +1,146 @@
# Wiki freshness audit — 2026-11-12
Threshold: >21 days since `updated`
## STALE (1)
- [ 22d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-10-21)
## FRESH (129)
- [ 1d] CONTRIBUTING.md
- [ 4d] agents/hermes-chaos.md
- [ 4d] agents/hermes-production.md
- [ 1d] agents/index.md
- [ 14d] agents/nanobot.md
- [ 1d] agents/open-claw.md
- [ 8d] agents/qwenpaw.md
- [ 1d] agents/zeroclaw.md
- [ 14d] comparisons/camoufox-vs-chrome.md
- [ 1d] comparisons/hermes-vs-nanobot.md
- [ 1d] comparisons/memory-backends.md
- [ 1d] comparisons/model-routers.md
- [ 14d] comparisons/steel-vs-camoufox.md
- [ 1d] concepts/agent-memory.md
- [ 14d] concepts/browser-automation.md
- [ 1d] concepts/model-routing.md
- [ 1d] concepts/persistent-shells.md
- [ 1d] concepts/tool-calling.md
- [ 1d] current-state.md
- [ 14d] decisions/index.md
- [ 1d] decisions/keep-openrouter-as-fallback.md
- [ 1d] decisions/persistent-shell-enabled.md
- [ 14d] decisions/rollback-steel-to-camofox.md
- [ 1d] experiments/index.md
- [ 19d] incidents/2026-07-29-mnemosyne-relay.md
- [ 1d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 19d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 5d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 7d] incidents/2026-08-15-skills-index-drift.md
- [ 8d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 18d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 1d] incidents/2026-08-29-gmail-read-state.md
- [ 8d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 14d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 17d] incidents/2026-10-03-ct450-hermes-update.md
- [ 17d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 14d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 19d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 11d] incidents/2026-10-09-memory-recall-relevance.md
- [ 6d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 16d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 2d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 16d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 21d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 18d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 1d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 17d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 17d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 15d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 13d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 13d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 13d] incidents/2026-10-29-conduit-voice-silence.md
- [ 12d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 13d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 12d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 11d] incidents/2026-10-31-asana-access-failures.md
- [ 11d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 11d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 10d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 9d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 8d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 6d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 6d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 7d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 3d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 7d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 5d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
- [ 3d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
- [ 3d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
- [ 1d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
- [ 1d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
- [ 1d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
- [ 1d] index.md
- [ 5d] infrastructure/backups.md
- [ 1d] infrastructure/credentials-map.md
- [ 5d] infrastructure/docker-services.md
- [ 1d] infrastructure/domains-and-tunnels.md
- [ 5d] infrastructure/hosts.md
- [ 5d] infrastructure/network-map.md
- [ 5d] infrastructure/proxmox.md
- [ 2d] infrastructure/tailscale.md
- [ 8d] queries/retained-research-answers.md
- [ 18d] reports/communities-wa-social-media-audit-2026.md
- [ 11d] reports/uber-eats/2026-07.md
- [ 11d] reports/uber-eats/2026-08.md
- [ 11d] reports/uber-eats/2026-09.md
- [ 11d] reports/uber-eats/2026-10.md
- [ 11d] reports/uber-eats/index.md
- [ 1d] runbooks/backup-wiki.md
- [ 1d] runbooks/diagnose-docker.md
- [ 1d] runbooks/gateway-resource-resilience.md
- [ 5d] runbooks/job-radar.md
- [ 1d] runbooks/multiplexer-setup.md
- [ 17d] runbooks/provider-health.md
- [ 14d] runbooks/recover-docker-service.md
- [ 1d] runbooks/restart-browser.md
- [ 1d] runbooks/restart-hermes.md
- [ 1d] runbooks/restore-openrouter.md
- [ 1d] runbooks/test-openrouter.md
- [ 1d] runbooks/troubleshoot-tool-loop.md
- [ 17d] runbooks/update-hermes-safely.md
- [ 4d] systems/browser-backend.md
- [ 1d] systems/cloudflare-tunnel.md
- [ 3d] systems/conduit.md
- [ 17d] systems/current-profile.md
- [ 1d] systems/guanaco.md
- [ 1d] systems/headless-server.md
- [ 3d] systems/hermes-agent.md
- [ 4d] systems/hermes-hq-push.md
- [ 1d] systems/hermes-ui.md
- [ 8d] systems/home-assistant.md
- [ 13d] systems/homepage-dashboard.md
- [ 8d] systems/ignis.md
- [ 17d] systems/memory-backend.md
- [ 10d] systems/messaging-integrations.md
- [ 3d] systems/mnemosyne.md
- [ 17d] systems/model-providers.md
- [ 8d] systems/nanobot.md
- [ 1d] systems/nextcloud.md
- [ 1d] systems/obsidian.md
- [ 2d] systems/omniroute.md
- [ 1d] systems/opencode-go.md
- [ 1d] systems/openrouter.md
- [ 8d] systems/paseo.md
- [ 2d] systems/push-notifications.md
- [ 1d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 1d] systems/terminal-backend.md
- [ 1d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 1 stale, 129 fresh, 3 no-date of 133 pages
+35 -3
View File
@@ -3,8 +3,8 @@ title: OpenCode Go
type: system
status: active
created: 2026-07-29
updated: 2026-11-08
verified_on: 2026-11-08
updated: 2026-11-11
verified_on: 2026-11-11
confidence: high
tags: [opencode, provider, models, subscription, limits]
sources: [https://opencode.ai/docs/go/]
@@ -160,12 +160,44 @@ both returned **HTTP 401** on the Go endpoint:
use.
- **To restore:** mint a new key in the OpenCode console, then replace it in **both** stores — the Hermes
provider entry and `~/.local/share/opencode/auth.json` — and re-probe `…/zen/go/v1/models` for a 200.
(Keys live in the vault; this page records only their state — [[references/hermes-api-vault]].)
(Keys live in the vault; this page records only their state — the `hermes-api-vault` skill (vault-backed; no value is stored in this wiki).)
**Config format:** `opencode-go/<model-id>` (e.g. `opencode-go/kimi-k3`)
**Model list API:** `https://opencode.ai/zen/go/v1/models`
## 2026-11-11 — the expired credential is no longer latent: it is in the live fallback path
The 2026-11-08 record above closed with *"**No execution failed on an auth error** — nothing in the fleet
was actually consuming this provider in-window. The risk is latent: anything pinned to it fails on first
use."* **That latent risk has now fired, repeatedly.**
- `fallback_providers[1]` is `opencode-zen` / `deepseek-v4.1-flash` (`https://opencode.ai/zen/v1`,
`api_mode: chat_completions`) — i.e. the **second hop of every session fallback chain**.
- On **2026-11-11** it produced **7 × HTTP 401 `api_key_expired`**, each one *immediately* after the
hop above it (nous `anthropic/claude-haiku-4.5`) returned **404 `insufficient_credits_for_paid_model`**:
| # | nous 404 | opencode-zen 401 |
|---|---|---|
| 1 | 07:01:11 | 07:01:13 |
| 2 | 07:19:28 | 07:19:30 |
| 3 | 08:03:38 | 08:03:40 |
| 4 | 13:03:27 | 13:03:28 |
| 5 | 13:19:31 | 13:19:33 |
| 6 | 18:03:38 | 18:03:39 |
| 7 | 19:19:37 | 19:19:39 |
- **Two consecutive dead hops** on one chain: a paid model with no credit, then an expired key. Every
affected turn has to fall through to `custom:guacano` / `custom:litellm` / `kilocode` to succeed.
- ⚠ Nothing in the fleet is *pinned* to `opencode-zen` — no cron job names it — so this never shows up as
a job failure. It shows up only as `agent.chat_completion_helpers: Streaming failed before delivery:
Error code: 401` in `~/.hermes/logs/errors.log`. → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
**To restore:** mint a new key in the OpenCode console and replace it in both stores (the Hermes provider
entry and `~/.local/share/opencode/auth.json`), then re-probe `…/zen/go/v1/models` for a 200 — or
**remove `opencode-zen` from `fallback_providers`**, since a dead second hop costs a round-trip on every
retry. Keys live in the vault; this page records only their state — the `hermes-api-vault` skill (vault-backed; no value is stored in this wiki).
## Privacy
Zero-retention policy from providers — your data is not used for model training.
+25 -3
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-11-10
verified_on: 2026-11-10
review_after: 2026-11-10
updated: 2026-11-11
verified_on: 2026-11-11
review_after: 2026-11-12
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
@@ -14,6 +14,28 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
# Scheduled Tasks
## Count
- **Current as of 2026-11-11 (window 2026-11-10 23:59 → 2026-11-11 23:59; nightly audit).**
**Fleet 29 (26 enabled / 3 disabled)** — unchanged for a sixth window. Three states moved, and the
observer itself is the story. **`cbfe434a3b5e` (Wiki Audit — Nightly) failed on its own 429** —
2026-11-10 23:59:54.486572 → **2026-11-11 00:12:29.646987 = 12 m 35 s**, `RuntimeError: HTTP 429 … all
targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s`,
delivery `suppressed`, incident row `cbfe43_ed01845310d8` (`rate_limit`, `alerted_at: NULL`) **still
open and the fleet's only open row** → [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]. That
failure means **the 2026-11-10 window was never audited**; this page covers both days.
**Non-clean `last_status` 1 → 2** (`066bb5191280` Uber Eats `delivery_failed` + `cbfe434a3b5e` `error`,
streak 1). Both long-running incidents **recovered with output**: the stocktake (03:00:56 → 04:11:38 =
**1 h 10 m 42 s**, **36 stale pages re-verified → 0**, commits `341dda9` + `c108eb1`) and Job Radar
Stage 1 (07:01:08 → 07:37:34 = **36 m 26 s**, writing the first handoff since **2026-11-06** —
`radar-ref/handoff/2026-11-11.json`, 22,682 B), whose Stage 2 then auto-closed its incident row
`resolved` at 08:03:47. **Ledger: 2026-11-10 = 157 rows / 155 completed / 2 failed; 2026-11-11 = 172
rows / 171 completed / 1 running / 0 failed** — a zero-failure day (first since 11-09). Longest 11-11
run **1 h 10 m 42 s**; **no run over 71 minutes**, against 9 h 09 m (11-09) and 10 h 48 m (11-10).
**Deliveries 11-11: 7 rows, 7 / 7 `delivered`, 0 `for_failure`.** Open rows: **1** (was 0 on 11-09).
⚠ **Provider/aux-path headwinds:** `opencode-zen`/`deepseek-v4.1-flash` sits at `fallback_providers[1]`
and threw **7 × HTTP 401 `api_key_expired`** on 11-11, each immediately after nous returned **14 ×
404 `insufficient_credits_for_paid_model`**; `auxiliary.pulse_memory` — the only slot with **no
`fallback_chain`** — logged 19 lines and 11 "refusing to guess another logged-in provider" refusals.
`config.yaml` unchanged since 2026-11-09 22:58:07.
- **Current as of 2026-11-10 (window 2026-11-09; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-10 00:0x–01:0x AWST):** **29 jobs (26 enabled / 3 disabled** — `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`), fifth consecutive audit at the same count. **2026-11-09 settled: 174 rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit) — the **first `failed`-free day since 2026-11-05** (11-06: 1 failed, 11-07: 3, 11-08: 1). Deliveries **8/8 `delivered`**, 0 `for_failure` (same as 11-08). **Unclosed incident rows 3 → 0** (134 rows total): the whole stocktake cluster — `1a8e90_628d914f1f97` (11-07T04:29:54, *"Restart-safe cron worker exited with status 1 after adopting…"*), `1a8e90_2f370ef764f5` (11-07T05:02:21, *"…lenovo… lost its…"*), `1a8e90_a9cad99cb150` (11-08T04:35:38, *"No reply: the turn stopped while a tool…"*) — **closed 2026-11-09T12:10:47.125205, all three at once, the instant the overrunning run finished**; none ever alerted. The most recent incident activity is now the `85edf6772bca` row `85edf6_0877ca1e0eac` (*"Cron execution unverified"*, alerted 2026-11-08T07:23:07, closed 08:06:54). **The stocktake's `failure_streak` 3 → 0 and `last_status ok` are cosmetic:** `1a8e90ea2271` ran **03:00:43 → 12:10:47 = 9 h 10 m** (its 11-07 and 11-08 runs died at 2 h 02 m and 1 h 35 m), closed `completed` with delivery `suppressed`, and landed **zero edits and no commit** — the run's own output file reads `RUN INCOMPLETE`, tool-iteration budget exhausted inside verification. **Third consecutive day the stocktake's work did not land** (11-07, 11-08, 11-09). → [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]. **The Job Radar landed nothing on 11-09 either:** Stage 1 (`dca8482e4f76`) ran **07:00:48 → 16:10:12 = 9 h 09 m** and never wrote `radar-ref/handoff/2026-11-09.json` (newest handoff on disk: **2026-11-06**); both Stage-2 slots (`5fc292ff0f3b`, 11:30:38→12:10:07 and 14:30:34→14:37:27) fired **before** Stage 1 finished and declined silently (`suppressed`). → [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]. **No other 11-09 run exceeded 74 min** (stocktake 550 min, Stage 1 549 min, then 74 / 40 / 11 min), and the 11-08 `pre_tool_call` hook cascade did not recur — see the follow-up on [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. The stocktake remains pinned to `custom:omni-101` / **`free-stack`**, a combo recorded broken on **2026-08-14**; this audit itself runs `custom:omni-101` / `deepseek-paid-smart`.
- **Current as of 2026-11-08 (window 2026-11-07; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-09 00:0x-00:1x AWST):** **29 jobs (26 enabled / 3 disabled** — `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; the count and split are unchanged across **six** consecutive audits, but fleet `updated_at` **moved** to **2026-11-09T00:06:50+08:00** — treat that as a run-state stamp, **not** a definition change, because no job was added, retired, disabled or repinned: re-verified every field on the nine jobs looked at and the enabled/disabled split is identical). **11-08 settled: 169 rows across 16 jobs — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33). The **only** failure is **`1a8e90ea2271` (wiki-stocktake-maintenance), 03:01:11 → 04:36:08 (1 h 34 m 57 s)**, and it is a **new terminal signature for that job**: `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` — the previous two red days both read *"lost its durable fire claim ownership"*. **`failure_streak` is now 3** — **three consecutive failed days**. *Working hypothesis, not proven:* the `conduit_push` `pre_tool_call` hook that was **timing out at 30 s 58 times between 2026-11-07 14:14:03 and 2026-11-08 08:21:03** holds a tool result open in exactly the way this error describes, but the string does not name the hook → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. The hole in the day was **the hook, not the schedule**: *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)**, *FreshRSS Midday Briefing* `ecc1fc13655a` **12:00:50 → 14:10:43 (2 h 09 m 53 s)**, both delivered. Deliveries **8 rows / 8 `delivered` / 0 `for_failure`** (11-07 was 7 / 7). **Unclosed incident rows 2 → 3** — a **third** silent `detected` row appeared (`1a8e90_a9cad99cb150`, `alerted_at: null`), and the two 11-07 stocktake rows are still `detected`; **the alerting path is not the fault** (the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window — its `database is locked` admission, closed when the run completed). Nothing added, retired, disabled or repinned.
- **Current as of 2026-11-07 (window 2026-11-06; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-07 00:3x-00:5x AWST):** **29 jobs (26 enabled / 3 disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; count and split unchanged across five consecutive audits, fleet `updated_at` **2026-11-07T00:39:46+08:00**). **11-06 settled: 172 rows, 16 distinct jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:31). The single `failed` row is a **new, non-restart failure signature**: the Job Radar's **11:30 Stage-2 slot** (`95347fa5…`) was **discarded as stale** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job, started 08:27:09, was still holding the claim** - that fire completed **13:26:22** and delivered, and the ledger records **no gateway restart** in the window -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; the radar still landed the day in full at 14:30 (`MISSING_COVER_LETTER_COUNT 0`, 4 same-day strong rows, 4 cover letters, 20 non-empty artifacts, *Run status: COMPLETE*). Fleet non-clean `last_status` unchanged at **1** (`066bb5191280`, `delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0** - no ledger row has `last_seen_at` on or after 2026-11-06 (newest activity remains the 2026-11-05T12:36:17 closures), so **the discarded run raised no incident row and no alert**; [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] remains the one open page. The stocktake's overrun streak **fully unwound**: `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Nothing added, retired, disabled or repinned.
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system
status: active
created: 2026-07-22
updated: 2026-11-11
verified_on: 2026-11-11
updated: 2026-11-12
verified_on: 2026-11-12
confidence: high
tags: [skills, hermes]
sources: []