diff --git a/current-state.md b/current-state.md index 6ed398c..d062b83 100644 --- a/current-state.md +++ b/current-state.md @@ -3,16 +3,74 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-11-10 -verified_on: 2026-11-10 -review_after: 2026-11-10 +updated: 2026-11-11 +verified_on: 2026-11-11 +review_after: 2026-11-12 confidence: high tags: [system] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db] --- # Current State -Updated: 2026-11-10 +Updated: 2026-11-11 + +## Recent Changes (2026-11-11, nightly audit; live probes 2026-11-11 23:59 – 2026-11-12 00:1x AWST) + +**Recovery night — plus one failure, and it was the auditor's own.** +→ [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] + +- **The 2026-11-10 audit died on its own 429.** `cbfe434a3b5e` ran 2026-11-10T23:59:54.486572 → + **2026-11-11T00:12:29.646987 = 12 m 35 s**, `failed`, delivery `suppressed`: + `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch + filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` — **6 candidates, `attempted 0`**, tenant + `tn-5ud77abtl1` quota exceeded. Row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 but + **`alerted_at: NULL`, `closed_at: NULL` — it is the estate's only open incident row**. Pool reset + ≈ **06:36:57**; the 07:0x cohort ran clean. + ⇒ **The 2026-11-10 window was never audited.** This audit covers two windows. +- **State reads:** fleet **29 (26 enabled / 3 disabled)**; **non-clean `last_status` 2** (was 1) — + `066bb5191280` Uber Eats (`delivery_failed`, streak 0) and `cbfe434a3b5e` (`error`, streak 1). + Ledger — **2026-11-09: 174 completed / 0 failed**; **2026-11-10: 157 rows, 155 completed / 2 failed**; + **2026-11-11: 172 rows, 171 completed / 1 running / 0 failed** (15 distinct jobs) — first zero-failure + day since 11-09. **Deliveries 11-11: 7 rows, 7 / 7 `delivered`, 0 `for_failure`** (last at 18:45:58). +- **Stocktake recovered with output.** `1a8e90ea2271` 03:00:56.081504 → **04:11:38.724852 = 1 h 10 m + 42 s**, `completed`, **36 stale pages re-verified → 0 stale**, **commits `341dda9` + `c108eb1`**, + freshness report `raw/stocktake/freshness-20261111.txt` (6,235 B). `341dda9` also rewrote the live + fallback chain in [[runbooks/test-openrouter]] (primary back to `smart-route` / `custom:omni-101`), + corrected the CT333 `clawtest` naming claim, and replaced the nous refresh-token reading with the + 404-credits signature. Closes [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]. + ⚠ The job is **still pinned to `free-stack`**, the combo recorded broken on 2026-08-14, and succeeded. +- **Job Radar recovered with output.** Stage 1 `dca8482e4f76` 07:01:08.358004 → **07:37:34.076224 = + 36 m 26 s**, `completed`, writing **`radar-ref/handoff/2026-11-11.json` (22,682 B, mtime 07:59)** — the + first handoff since **2026-11-06**. Stage 2 `5fc292ff0f3b`: recovery fire 07:35:56.632032 → + 08:03:47.816811 (`delivered`), scheduled 11:30:56 → 11:35:18, 14:30:55 → 14:34:56 (`delivered`). Row + `5fc292_773822455856` auto-closed **`resolved` 2026-11-11T08:03:47.644226**, closing + [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]. +- **Longest runs.** 11-11: stocktake **1 h 10 m 42 s**, Radar S1 **36 m 26 s**, Radar S2 **27 m 51 s** + (recovery fire), Newsletter Digest `ecc1fc13655a` 12:01:02.571781 → 12:19:38.965387 (**18 m 36 s**), + `85edf6772bca` 07:01:08.287627 → 07:11:42.858722 (**10 m 34 s**). **Nothing over 71 minutes.** + 11-10: Radar S2 scheduled slot 11:30:35.267155 → 22:18:49.814215 = **10 h 48 m 14 s**, `failed` / + `interrupted`, delivery `delivered` with `for_failure: 1`. +- **Provider state (unchanged config; worsening symptoms).** `config.yaml` mtime still **2026-11-09 + 22:58:07** — nothing repaired. Top model `custom:omni-101` / `smart-route` + (`http://100.78.63.20:20128`). `fallback_providers` (6): nous `meituan/longcat-2.5-preview:free` → + **opencode-zen `deepseek-v4.1-flash` (EXPIRED key)** → `custom:guacano` → `custom:litellm` + `kilo-reflection-beam-free` → kilocode `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) + `or-inclusionai_ling-2_6-1t`. **The expired opencode-zen key is no longer latent** — **7 × HTTP 401 + `api_key_expired`** on 11-11 (07:01:13 … 19:19:39), each immediately after nous returned **14 × 404 + `insufficient_credits_for_paid_model`** (07:01:11 … last 19:19:38, all from session + `20261031_190813_95fbde`, source `desktop`, opened 2026-10-31, **still open**, pinned to that paid + model). **Two consecutive dead hops on one chain.** → [[systems/opencode-go]] +- **Auxiliary slots:** nine slots plus `free_only: true`. Eight chain + `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` on `custom:omni-101`. + **`pulse_memory` remains the sole exception with no `fallback_chain`** + (`{"provider": "nous", "model": "anthropic/claude-haiku-4.5"}`) — 19 aux log lines on 11-11 across + 06:0x / 08:0x / 13:0x / 18:0x, including **11 ×** *"main provider `custom:omni-101` is unavailable and + no fallback_chain / fallback_providers is configured — refusing to guess another logged-in provider"*. + → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] +- **Error log:** `~/.hermes/logs/errors.log` = **6,179 lines** through **2026-11-12 00:10:11**. +- **Still open, untouched this window:** Uber Eats Monthly Spend Report `delivery_failed` streak 0 (no + execution row on 11-11 — monthly cadence); disabled trio unchanged (Important-mail monitor, Rhino + AgentMail Inbox Watcher, AgentMail Rhino Inbox Poll); the three inherited items below. ## Recent Changes (2026-11-10, nightly audit; live probes 2026-11-09 23:5x – 2026-11-10 01:0x AWST) diff --git a/incidents/2026-10-22-nous-portal-refresh-token-invalid.md b/incidents/2026-10-22-nous-portal-refresh-token-invalid.md index cacd5d2..8e56d56 100644 --- a/incidents/2026-10-22-nous-portal-refresh-token-invalid.md +++ b/incidents/2026-10-22-nous-portal-refresh-token-invalid.md @@ -66,7 +66,31 @@ sources: [gateway-log, live-config, live-probe] > currently erroring), and/or add credits for paid nous models. The original re-authenticate item > is **superseded** — auth works now. +> **Escalated 2026-11-11 (nightly audit).** The single-slot exposure described above became the +> busiest error path on the host, and it now has a **downstream casualty**. +> +> - `anthropic/claude-haiku-4.5` on `nous` returned **404 `insufficient_credits_for_paid_model`** +> **14 times** on 2026-11-11 (07:01:11 … last **19:19:38**) — up from 3 in the whole preceding +> window. All from session `20261031_190813_95fbde` (**source `desktop`, started 2026-10-31 +> 19:08:13, still open**), whose pinned model is that same paid nous model. +> - The **404 is immediately followed by a 401.** The session's fallback chain moves the turn on to +> `opencode-zen` / `deepseek-v4.1-flash`, whose key expired on 2026-11-08 — **7 × HTTP 401 +> `api_key_expired`** on 11-11 (07:01:13, 07:19:30, 08:03:40, 13:03:28, 13:19:33, 18:03:39, +> 19:19:39). So **both hops of this chain are dead**, and the pair repeats every time the session is +> poked. → [[systems/opencode-go]] +> - `auxiliary.pulse_memory` logged **19 lines** on 11-11 (06:0x, 08:0x, 13:0x, 18:0x), including +> **11 ×** *"main provider `custom:omni-101` is unavailable and no fallback_chain / +> fallback_providers is configured — refusing to guess another logged-in provider."* Live config +> re-confirmed at audit time: `pulse_memory` is the **only** slot carrying **no `fallback_chain`** +> (`{"provider": "nous", "model": "anthropic/claude-haiku-4.5"}`), while the other eight all chain +> `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` on `custom:omni-101`. +> - **Net effect:** pulse/voice memory retrieval failed on every attempt this window. `config.yaml` +> mtime is unchanged since **2026-11-09 22:58:07** — nothing was fixed. +> +> **Open item, unchanged and now evidenced:** move `auxiliary.pulse_memory` onto `custom:omni-101` +> like the other eight slots and give it a `fallback_chain`, and re-point or retire the stale desktop +> session pinned to a paid nous model. No credentials were minted or modified. + ## Related -- [[systems/model-providers]] - [[systems/current-profile]] - [[systems/hermes-agent]] diff --git a/incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md b/incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md index 773bca1..cd0b82e 100644 --- a/incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md +++ b/incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md @@ -1,10 +1,10 @@ --- title: "2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op" type: incident -status: active +status: resolved created: 2026-11-10 -updated: 2026-11-10 -verified_on: 2026-11-10 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [cron, job-radar, stall, silent-failure, handoff, read-loop] sources: ["executions.db (dca8482e4f76: 2026-11-09T07:00:48+08:00 → 2026-11-09T16:10:12+08:00, status completed, delivery_outcome suppressed; 5fc292ff0f3b: 11:30:38 → 12:10:07 and 14:30:34 → 14:37:27, both completed/suppressed)", "~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md (ends INCOMPLETE: tool budget exhausted; last words \\u201cI stopped retrying because I kept running read_file 5 times without making progress.\\u201d)", "~/.hermes/radar-ref/handoff/ (newest handoff on disk: 2026-11-06; no 2026-11-07/08/09 files)", "cron_incidents (no row raised for this run)"] @@ -57,6 +57,31 @@ Two compounding faults, both previously documented for this job: - Add a **read-progress guard** to Stage 1: when the same file has been read N times without a state change, stop and emit a partial handoff instead of burning the budget to `INCOMPLETE`. - Note the pattern across incidents: **completion-time-only alerting hides every failure in this family** (stocktake 11-09, radar 11-09, radar 11-02/03). +## Recovery — 2026-11-11: the handoff landed, and Stage 2 delivered + +| Field | 11-09 (this incident) | **11-11** | +|---|---|---| +| Stage 1 | 07:00:48 → 16:10:12 (**9 h 09 m 24 s**), no handoff | **07:01:08 → 07:37:34 = 36 m 26 s**, `completed` | +| Handoff on disk | newest was **2026-11-06** | **`radar-ref/handoff/2026-11-11.json` — 22,682 B, mtime 07:59** | +| Stage 2 (11:30 slot) | fired early, declined silently | **11:30:56 → 11:35:18**, `completed` | +| Stage 2 (14:30 slot) | declined silently | **14:30:55 → 14:34:56**, `completed` / **`delivered`** | +| Incident row | **none raised** | `5fc292_773822455856` closed **`resolved` 2026-11-11T08:03:47.644226** | + +- **The first handoff in five days** (`2026-11-06` → `2026-11-11`) and the fastest Stage-1 cycle since + the two-stage split: **36 m 26 s** against 3 h 15 m (10-29, the last full success), 9 h 01 m (11-02), + 6 h 28 m (11-03) and 9 h 09 m (11-09). +- **Stage 2's recovery fire** (`source: direct`, 07:35:56 → 08:03:47) consumed the handoff and delivered + at **08:03:47**; its completion is what auto-closed the row opened by 11-10's `interrupted` slot. +- ⚠ **Carried forward from 11-10:** the *scheduled* Stage-2 slot that day ran + **11:30:35 → 22:18:49 = 10 h 48 m 14 s** and closed **`failed` / `interrupted`** (delivery + `delivered`, `for_failure: 1`) — the longest run of either window, and itself a new data point for the + "completion-time-only alerting" family on this page. +- **Job state:** both radar jobs read `last_status: ok`, `failure_streak` 0. + +**Status: `resolved`.** Both prevention rules on this page still stand and remain unimplemented: Stage 2 +should hard-fail on a missing `handoff/.json` rather than suppressing, and Stage 1 needs a +read-progress guard. Neither was needed on 11-11 because the run finished inside budget. + ## Related - [[systems/scheduled-tasks]] diff --git a/incidents/2026-11-09-stocktake-completes-with-zero-edits.md b/incidents/2026-11-09-stocktake-completes-with-zero-edits.md index 7e05459..105824f 100644 --- a/incidents/2026-11-09-stocktake-completes-with-zero-edits.md +++ b/incidents/2026-11-09-stocktake-completes-with-zero-edits.md @@ -1,10 +1,10 @@ --- title: "2026-11-09 — Stocktake Completes With Zero Edits" type: incident -status: active +status: resolved created: 2026-11-10 -updated: 2026-11-10 -verified_on: 2026-11-10 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [cron, wiki, stocktake, tool-budget, silent-failure, omni-101, free-stack] sources: ["executions.db (1a8e90ea2271: 2026-11-09T03:00:43+08:00 → 2026-11-09T12:10:47+08:00, status completed, delivery_outcome suppressed)", "~/.hermes/cron/output/1a8e90ea2271/2026-11-09_12-10-46.md (8,305 bytes; RUN INCOMPLETE, \\u201ctool-iteration budget exhausted during verification\\u201d)", "cron_incidents (1a8e90_628d914f1f97, 1a8e90_2f370ef764f5, 1a8e90_a9cad99cb150 — all closed 2026-11-09T12:10:47.125205+08:00)", "wiki-stocktake-maintenance job 1a8e90ea2271 (provider custom:omni-101, model free-stack)"] @@ -57,6 +57,35 @@ Also recorded this window: the `free-stack` pin on `1a8e90ea2271` — the only d - **Alerting must fire on write progress, not completion.** This run produced 2 of 3 failure days silently and the third day produced a run that reported success while editing nothing. A check on `commit_count` / `edits_applied` for this job would have caught all three. - Don't read `status`/`failure_streak` as the stocktake's health. Read whether a commit landed. +## Recovery — 2026-11-11: the pass ran in 1 h 10 m and landed 36 pages + +The next scheduled slot **worked**, and worked quickly: + +| Field | 11-09 (this incident) | **11-11** | +|---|---|---| +| Window | 03:00:43 → 12:10:47 | **03:00:56.081504 → 04:11:38.724852** | +| Duration | **9 h 10 m** | **1 h 10 m 42 s** | +| Terminal state | `completed` / `suppressed` | `completed` / `suppressed` | +| Edits landed | **0** | **36 stale pages re-verified → 0 stale** | +| Commit | **none** | **`341dda9`** + **`c108eb1`** | +| Freshness report | — | `raw/stocktake/freshness-20261111.txt` (**6,235 B**) | + +- `341dda9` re-verified 30+ further pages live and de-flagged them, and applied real content + corrections — the live fallback chain rewritten in [[runbooks/test-openrouter]] (primary back to + `smart-route` / `custom:omni-101`), the CT333 `clawtest` naming claim corrected, and the nous + **404 `insufficient_credits_for_paid_model`** signature replacing the old refresh-token reading in + [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. +- `c108eb1` regenerated the freshness report after the log/index addendum, so the published Quartz build + matches HEAD. +- **Job state:** `last_status: ok`, `failure_streak` 0, `last_run` 2026-11-11T03:00:56. + ⚠ The job is **still pinned to `custom:omni-101` / `free-stack`** — the combo recorded broken on + 2026-08-14 — and this run succeeded on it anyway. That is an observation, not a repin. + +**Status: `resolved`.** The 11-07/11-08/11-09 run of failures is closed by output, not by a cosmetic +`failure_streak` reset. The page's own prevention rule stands: *alert on `commit_count` / `edits_applied`, +never on `status`* — this recovery is exactly why it matters, because 11-09 and 11-11 both read +`completed` / `suppressed` and only one of them did anything. + ## Related - [[systems/scheduled-tasks]] diff --git a/incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md b/incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md new file mode 100644 index 0000000..5118226 --- /dev/null +++ b/incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md @@ -0,0 +1,96 @@ +--- +title: Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited +type: incident +status: open +created: 2026-11-11 +updated: 2026-11-11 +verified_on: 2026-11-11 +review_after: 2026-12-11 +confidence: high +tags: [cron, incident, omniroute, rate-limit, audit] +sources: [live-executions-db, live-cron-incidents, gateway-log, live-jobs-json] +--- + +# Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited (2026-11-10 23:59 → 2026-11-11 00:12) + +**This job is the estate's own observer, and on 2026-11-10 it became the estate's only failure.** Its +23:59 slot never produced a report, so **no page, no log entry and no incident alert describes the day +of 2026-11-10** — an unaudited day is invisible by construction. + +## What happened + +| Field | Value | +|---|---| +| Job | `cbfe434a3b5e` — **Wiki Audit — Nightly**, `custom:omni-101` / `deepseek-paid-smart` | +| Slot | 2026-11-10 23:59:54.486572 +08:00 (cron `59 23 * * *`) | +| Ended | **2026-11-11 00:12:29.646987 +08:00** — **12 m 35 s** | +| Status | `failed` · `delivery_outcome: suppressed` | +| Error | `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` | +| Output | `~/.hermes/cron/output/cbfe434a3b5e/2026-11-11_00-12-28.md` | + +The three log lines from the run's own session `local-1000c1a3…` show the shape of the failure: + + ERROR agent.chat_completion_helpers: Streaming failed before delivery: [429]: service failure: tenant tn-5ud77abtl1 quota exceeded + ERROR agent.chat_completion_helpers: Streaming failed before delivery: Error code: 429 - + ERROR agent.conversation_loop: API call failed after 3 retries. HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters + ERROR cron.scheduler: Job 'Wiki Audit — Nightly' failed: RuntimeError: HTTP 429: Service temporarily unavailable + +- **The pool skipped every target.** Six candidates were evaluated and **`attempted 0`** — the run never + reached a provider. The only live target's credit balance read **100 %** consumed on connection + `conn:3021bab6`, which is why the router refused to dispatch rather than serving a degraded model. +- **`reset after 6h 24m 31s`** puts the pool's recovery at **≈ 2026-11-11 06:36:57 AWST**. The 07:0x + cron cohort ran clean the next morning, so the window closed on its own. +- The **same tenant** (`tn-5ud77abtl1`) appears in the `quota exceeded` line, so this is an + account-level quota fault on the router, not a single-model lockout. + +## Incident row — raised, never alerted, still open + +`cbfe43_ed01845310d8` — `job_id cbfe434a3b5e`, `failure_type: rate_limit`, state **`detected`**, +`alerted_at: NULL`, `closed_at: NULL`, opened **2026-11-11T00:12:28.578435+08:00**. + +- It is the **only open row** in `cron_incidents` (135 rows total) — the previous open item, + `5fc292_773822455856`, closed `resolved` at **2026-11-11T08:03:47.644226**. +- So the estate's single outstanding incident is **the audit's own failure**, and nothing was told + about it. + +## Impact + +- **The 2026-11-10 window was never reviewed.** The nightly audit is the only producer of the per-day + ledger/wiki record; its failure means 11-10's 157 ledger rows were never reconciled on the day. +- **Fleet non-clean `last_status` 1 → 2.** `cbfe434a3b5e` now reads `last_status: error`, + `failure_streak: 1`, joining `066bb5191280` (*Uber Eats Monthly Spend Report*, `delivery_failed`, + streak 0). +- **The review published on 2026-11-11 23:59 carries both windows** — it is the recovery, not a + duplicate. + +## Same signature, earlier appearances + +- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — the *identical* pre-dispatch-filter + string killed the Daily Marketing Job Radar on 2026-10-21 (`attempted 0`, no 503 anywhere in the day). +- 2026-11-05 — `deepseek-paid-smart` against the **CT600** endpoint returned `HTTP 503 + ALL_TARGETS_SKIPPED` (`poolSize 5`, `attempted 0`) while `omniroute-1` answered 200 — recorded on + [[systems/scheduled-tasks]]. +- This is the **third distinct symptom** of the same router behaviour: `attempted 0` with a healthy node + behind it is the thing to recognise, not the specific HTTP code. + +## Fix Applied + +**None — this is a detection-and-verification record.** No configuration changed this window. The run's +own recovery was the pool reset at ≈ 06:36:57 on 2026-11-11. + +## Prevention + +- **The observer needs an observer.** A failed nightly audit raises no delivery (suppressed), no alert + (`alerted_at: NULL`) and no page — every layer that would normally surface this is *inside the thing + that failed*. A one-line watchdog on `Wiki Audit — Nightly`'s `last_status != ok` (or on "no audit + commit in >24 h") is the only check that can catch it. +- **Alert on the artefact.** Verifying that a dated `audit(YYYY-MM-DD)` commit exists in this repo would + have caught both this failure and the stocktake's zero-edit days — the same lesson recorded on + [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]. + +## Related + +- [[systems/scheduled-tasks]] +- [[systems/omniroute]] +- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] +- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] diff --git a/index.md b/index.md index ab97ae8..2a921ba 100644 --- a/index.md +++ b/index.md @@ -69,8 +69,9 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi - [[systems/skills-index]] — Installed skills inventory (**299 skills** (**67 categories**) as of the **2026-11-08** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents +- [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **OPEN — the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]). - [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - **RESOLVED (same day) — the hook was the fault, not the schedule.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** → [[systems/conduit]], [[systems/hermes-agent]] -- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] - **RESOLVED BY ITSELF.** The nightly stocktake closed `completed` after **9 h 10 m** having landed **zero edits and no commit** — `failure_streak` 3 → 0 while the work stayed undone, and the top-priority item on its brief had been the monotonically worsening failure streak. Third consecutive day its findings never landed. +- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] (resolved 2026-11-11) - **RESOLVED BY ITSELF.** The nightly stocktake closed `completed` after **9 h 10 m** having landed **zero edits and no commit** — `failure_streak` 3 → 0 while the work stayed undone, and the top-priority item on its brief had been the monotonically worsening failure streak. Third consecutive day its findings never landed. - [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] - **NO INCIDENT RAISED.** Job Radar Stage 1 overran to **9 h 09 m**, skipped its own verification, wrote **no handoff file**, and both Stage-2 slots ran before it finished and declined silently. No incident row was created — the completion-time-only alerting gap leaves the failure invisible. - [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - **OPEN.** A restart requested at **~04:29:52** sat in *after-turn* mode for the full **1800 s**, then timed out and drained at **05:00:43/47** — the same `#77184` behaviour as the 10-31 reload, but this time it killed **two** in-flight cron executions: *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42.918 -> 05:02:22.870) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31.515 -> 05:02:22.174), both closed **`failed`** on `lost its durable fire claim ownership`; systemd restarted the unit at **05:01:00** (`restart counter is at 3`). A **third** failure that night was the agent-log ERROR watchdog correctly catching the event 19 s later. The request landed **inside the `conduit_push` self-update window** (04:27:55 -> 04:32:32, its own ledger recording `restarting_gateway` = *failed*) — the **working hypothesis for the requester, not proven**; the 0.22.5 -> 0.23.0 updater is excluded (receipt starts 05:05:13). ⚠ **Closed 0 -> 2 incident rows and neither alerted**: both `1a8e90_*` rows sit `detected` with `alerted_at: null`. The stocktake slot has now failed **two days running** (`failure_streak: 2`). Job Radar ran clean. → [[systems/scheduled-tasks]], [[systems/hermes-agent]], [[systems/conduit]], [[systems/mnemosyne]] - [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **OPEN.** A `direct`-source fire of `5fc292ff0f3b` (08:27:09 -> 13:26:22, 4 h 59 m 13 s, delivered) settled the claim while the scheduler's **own 11:30:32 slot** was still running, so that slot was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - a **new string variant** and the first claim-loss failure **with no gateway restart involved**. It raised **no `cron_incidents` row**, so a `failed` execution with no alert was the day's only red row; the discarded run left no output file. The 14:30 slot recovered the day in a normal **12 m 27 s** cycle. Candidate fixes (slot suppression when a claim is live, consuming the next slot for out-of-band fires, raising an incident row on discard) are listed on the page and **none was applied**. diff --git a/log.md b/log.md index cc611b6..fb15769 100644 --- a/log.md +++ b/log.md @@ -2467,3 +2467,13 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted* - **Estate shrank again (probed, not inferred):** `qm list` on `pve` now holds only VMs **100, 101, 110, 282** (VM **310** gone; the wiki had listed it) and `pct list` holds 17 CTs with **CT 601 gone**. VM 403/VM 401 and the `open-claw` Tailscale peer remain absent on both hypervisors → [[agents/open-claw]], [[agents/index]], `CONTRIBUTING.md`. - **Delivery verified at the destination, not assumed:** before the commit, `http://100.118.5.51:9120/` and the changed pages (incl. `runbooks/test-openrouter`, `concepts/model-routing`, `incidents/2026-08-29-gmail-read-state`) all returned **200**, as did the external route `https://hermes.kangaroo-eel.ts.net/wiki/`. `CONTRIBUTING.md`'s access table was re-verified the same way and its `agents/open-claw/` scope row struck through ([[CONTRIBUTING]]). - **No page deleted, no decision reversed, no secrets written.** No credential value appears in any page (the Gitea remote's embedded token is deliberately not reproduced). One pre-existing broken wikilink left as-is: `systems/guanaco.md` → `[[systems/ollama]]` (present in HEAD, page already qualifies it "if exists"). + +## 2026-11-12 00:1x AWST - Nightly wiki audit (window 2026-11-10 23:59 -> 2026-11-11 23:59) + +- **The audit's own failure is the headline.** The 2026-11-10 23:59 slot (`cbfe434a3b5e`) ran **12 m 35 s** (23:59:54.486572 -> 2026-11-11 00:12:29.646987) and died on the router's pre-dispatch filter: `HTTP 429 ... all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` — **6 candidates, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded, delivery `suppressed`. **Consequence: the 2026-11-10 window was never reviewed.** Recovery was implicit (pool reset ≈ **06:36:57**; the 07:0x cohort ran clean), and this audit deliberately covers **both** windows. New page: [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]; new index entry. +- **The raised row was never alerted.** `cbfe43_ed01845310d8` (`rate_limit`) sits at `detected` with **`alerted_at: NULL`, `closed_at: NULL`** — the **only open row** in `cron_incidents` (the previous one, `5fc292_773822455856`, closed `resolved` at 08:03:47). Fleet non-clean `last_status` **1 -> 2** (`066bb5191280` Uber Eats `delivery_failed` streak 0, plus `cbfe434a3b5e` `error` streak 1). +- **Both long-running incidents recovered, with output.** Stocktake `1a8e90ea2271` **03:00:56.081504 -> 04:11:38.724852 = 1 h 10 m 42 s**, `completed`, **36 stale pages re-verified -> 0 stale**, commits **`341dda9`** + **`c108eb1`** (freshness report `raw/stocktake/freshness-20261111.txt`, 6,235 B) — closing [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]. Job Radar Stage 1 `dca8482e4f76` **07:01:08.358004 -> 07:37:34.076224 = 36 m 26 s**, writing the **first handoff since 2026-11-06** (`radar-ref/handoff/2026-11-11.json`, 22,682 B, mtime 07:59); Stage 2 `5fc292ff0f3b` recovery fire 07:35:56.632032 -> 08:03:47.816811 `delivered`, then 11:30:56 -> 11:35:18 and 14:30:55 -> 14:34:56 `delivered`, auto-closing the 11-10 row `resolved` — closing [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]. Both incident pages set to `status: resolved` with a dated Recovery section. +- **Ledger, both days re-parsed (times stored as ISO TEXT, grouped by `substr(started_at,1,10)`).** 2026-11-09 **174 completed / 0 failed**; **2026-11-10: 157 rows — 155 `completed` / 2 `failed`** (Radar S2 scheduled 11:30:35.267155 -> 22:18:49.814215 = **10 h 48 m 14 s** `interrupted` with `for_failure: 1`; the audit's own 12 m 35 s 429); **2026-11-11: 172 rows — 171 `completed` / 1 `running` / 0 `failed`**, 15 distinct jobs, **first zero-failure day since 11-09**. Longest 11-11 run **1 h 10 m 42 s** — **nothing over 71 minutes**, against 9 h 09 m (11-09) and 10 h 48 m (11-10). Fleet **29 (26 enabled / 3 disabled)** for a sixth window. +- **Deliveries checked at the destination:** 2026-11-11 = **7 rows, 7 / 7 `delivered`, 0 `for_failure`** (07:11:42, 07:31:01, 08:03:47, 08:34:36, 12:19:38, 14:34:55, 18:45:58); 2026-11-10's late row (22:18:48) is the one `for_failure: 1`. 2026-11-12 so far: 5 rows, 4 `completed` / 1 `running`. +- **Providers unchanged; symptoms worse.** `config.yaml` mtime still **2026-11-09 22:58:07** — nothing repaired this window. The expired `opencode-zen` key at `fallback_providers[1]` produced **7 x HTTP 401 `api_key_expired`** on 11-11 (07:01:13 -> 19:19:39), each **immediately after** nous returned **14 x 404 `insufficient_credits_for_paid_model`** (07:01:11 -> last 19:19:38, all from the still-open desktop session `20261031_190813_95fbde`). **Two consecutive dead hops on one chain** — [[systems/opencode-go]] updated from "latent risk" to observed failure; [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] gained the 11-11 escalation block. `auxiliary.pulse_memory` remains the **only slot with no `fallback_chain`** and logged **19 lines / 11 refusals** on 11-11. +- **Pages written:** 1 new incident; 2 incidents -> `resolved`; 2 system pages + `current-state.md` + `index.md` updated; `systems/scheduled-tasks.md` gained the 2026-11-11 count/state normalisation. `~/.hermes/logs/errors.log` is now **6,179 lines** through **2026-11-12 00:10:11**. **No page deleted, no decision reversed, no secrets written** — no credential value appears in any page; the opencode key is recorded by *state*, never by value. diff --git a/raw/stocktake/freshness-20261112.txt b/raw/stocktake/freshness-20261112.txt new file mode 100644 index 0000000..8476a5c --- /dev/null +++ b/raw/stocktake/freshness-20261112.txt @@ -0,0 +1,146 @@ +# Wiki freshness audit — 2026-11-12 +Threshold: >21 days since `updated` + +## STALE (1) + +- [ 22d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-10-21) + +## FRESH (129) + +- [ 1d] CONTRIBUTING.md +- [ 4d] agents/hermes-chaos.md +- [ 4d] agents/hermes-production.md +- [ 1d] agents/index.md +- [ 14d] agents/nanobot.md +- [ 1d] agents/open-claw.md +- [ 8d] agents/qwenpaw.md +- [ 1d] agents/zeroclaw.md +- [ 14d] comparisons/camoufox-vs-chrome.md +- [ 1d] comparisons/hermes-vs-nanobot.md +- [ 1d] comparisons/memory-backends.md +- [ 1d] comparisons/model-routers.md +- [ 14d] comparisons/steel-vs-camoufox.md +- [ 1d] concepts/agent-memory.md +- [ 14d] concepts/browser-automation.md +- [ 1d] concepts/model-routing.md +- [ 1d] concepts/persistent-shells.md +- [ 1d] concepts/tool-calling.md +- [ 1d] current-state.md +- [ 14d] decisions/index.md +- [ 1d] decisions/keep-openrouter-as-fallback.md +- [ 1d] decisions/persistent-shell-enabled.md +- [ 14d] decisions/rollback-steel-to-camofox.md +- [ 1d] experiments/index.md +- [ 19d] incidents/2026-07-29-mnemosyne-relay.md +- [ 1d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 19d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 5d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 7d] incidents/2026-08-15-skills-index-drift.md +- [ 8d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 18d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 1d] incidents/2026-08-29-gmail-read-state.md +- [ 8d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 14d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 17d] incidents/2026-10-03-ct450-hermes-update.md +- [ 17d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 14d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 19d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 11d] incidents/2026-10-09-memory-recall-relevance.md +- [ 6d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 16d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 2d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 16d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 21d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 18d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 1d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 17d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 17d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 15d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 13d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 13d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md +- [ 13d] incidents/2026-10-29-conduit-voice-silence.md +- [ 12d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +- [ 13d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md +- [ 12d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 11d] incidents/2026-10-31-asana-access-failures.md +- [ 11d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md +- [ 11d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md +- [ 10d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md +- [ 9d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md +- [ 8d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md +- [ 6d] incidents/2026-11-03-job-radar-stream-active-timeout.md +- [ 6d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md +- [ 7d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md +- [ 3d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md +- [ 7d] incidents/2026-11-05-skills-index-generator-divergence.md +- [ 5d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md +- [ 3d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md +- [ 3d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md +- [ 1d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md +- [ 1d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md +- [ 1d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md +- [ 1d] index.md +- [ 5d] infrastructure/backups.md +- [ 1d] infrastructure/credentials-map.md +- [ 5d] infrastructure/docker-services.md +- [ 1d] infrastructure/domains-and-tunnels.md +- [ 5d] infrastructure/hosts.md +- [ 5d] infrastructure/network-map.md +- [ 5d] infrastructure/proxmox.md +- [ 2d] infrastructure/tailscale.md +- [ 8d] queries/retained-research-answers.md +- [ 18d] reports/communities-wa-social-media-audit-2026.md +- [ 11d] reports/uber-eats/2026-07.md +- [ 11d] reports/uber-eats/2026-08.md +- [ 11d] reports/uber-eats/2026-09.md +- [ 11d] reports/uber-eats/2026-10.md +- [ 11d] reports/uber-eats/index.md +- [ 1d] runbooks/backup-wiki.md +- [ 1d] runbooks/diagnose-docker.md +- [ 1d] runbooks/gateway-resource-resilience.md +- [ 5d] runbooks/job-radar.md +- [ 1d] runbooks/multiplexer-setup.md +- [ 17d] runbooks/provider-health.md +- [ 14d] runbooks/recover-docker-service.md +- [ 1d] runbooks/restart-browser.md +- [ 1d] runbooks/restart-hermes.md +- [ 1d] runbooks/restore-openrouter.md +- [ 1d] runbooks/test-openrouter.md +- [ 1d] runbooks/troubleshoot-tool-loop.md +- [ 17d] runbooks/update-hermes-safely.md +- [ 4d] systems/browser-backend.md +- [ 1d] systems/cloudflare-tunnel.md +- [ 3d] systems/conduit.md +- [ 17d] systems/current-profile.md +- [ 1d] systems/guanaco.md +- [ 1d] systems/headless-server.md +- [ 3d] systems/hermes-agent.md +- [ 4d] systems/hermes-hq-push.md +- [ 1d] systems/hermes-ui.md +- [ 8d] systems/home-assistant.md +- [ 13d] systems/homepage-dashboard.md +- [ 8d] systems/ignis.md +- [ 17d] systems/memory-backend.md +- [ 10d] systems/messaging-integrations.md +- [ 3d] systems/mnemosyne.md +- [ 17d] systems/model-providers.md +- [ 8d] systems/nanobot.md +- [ 1d] systems/nextcloud.md +- [ 1d] systems/obsidian.md +- [ 2d] systems/omniroute.md +- [ 1d] systems/opencode-go.md +- [ 1d] systems/openrouter.md +- [ 8d] systems/paseo.md +- [ 2d] systems/push-notifications.md +- [ 1d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 1d] systems/terminal-backend.md +- [ 1d] systems/tool-search.md + +## NO updated DATE (3) + +- SCHEMA.md +- experiments/active/example.md +- log.md + +TOTAL: 1 stale, 129 fresh, 3 no-date of 133 pages \ No newline at end of file diff --git a/systems/opencode-go.md b/systems/opencode-go.md index 94bbaeb..31d1a75 100644 --- a/systems/opencode-go.md +++ b/systems/opencode-go.md @@ -3,8 +3,8 @@ title: OpenCode Go type: system status: active created: 2026-07-29 -updated: 2026-11-08 -verified_on: 2026-11-08 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [opencode, provider, models, subscription, limits] sources: [https://opencode.ai/docs/go/] @@ -160,12 +160,44 @@ both returned **HTTP 401** on the Go endpoint: use. - **To restore:** mint a new key in the OpenCode console, then replace it in **both** stores — the Hermes provider entry and `~/.local/share/opencode/auth.json` — and re-probe `…/zen/go/v1/models` for a 200. - (Keys live in the vault; this page records only their state — [[references/hermes-api-vault]].) + (Keys live in the vault; this page records only their state — the `hermes-api-vault` skill (vault-backed; no value is stored in this wiki).) **Config format:** `opencode-go/` (e.g. `opencode-go/kimi-k3`) **Model list API:** `https://opencode.ai/zen/go/v1/models` +## 2026-11-11 — the expired credential is no longer latent: it is in the live fallback path + +The 2026-11-08 record above closed with *"**No execution failed on an auth error** — nothing in the fleet +was actually consuming this provider in-window. The risk is latent: anything pinned to it fails on first +use."* **That latent risk has now fired, repeatedly.** + +- `fallback_providers[1]` is `opencode-zen` / `deepseek-v4.1-flash` (`https://opencode.ai/zen/v1`, + `api_mode: chat_completions`) — i.e. the **second hop of every session fallback chain**. +- On **2026-11-11** it produced **7 × HTTP 401 `api_key_expired`**, each one *immediately* after the + hop above it (nous `anthropic/claude-haiku-4.5`) returned **404 `insufficient_credits_for_paid_model`**: + + | # | nous 404 | opencode-zen 401 | + |---|---|---| + | 1 | 07:01:11 | 07:01:13 | + | 2 | 07:19:28 | 07:19:30 | + | 3 | 08:03:38 | 08:03:40 | + | 4 | 13:03:27 | 13:03:28 | + | 5 | 13:19:31 | 13:19:33 | + | 6 | 18:03:38 | 18:03:39 | + | 7 | 19:19:37 | 19:19:39 | + +- **Two consecutive dead hops** on one chain: a paid model with no credit, then an expired key. Every + affected turn has to fall through to `custom:guacano` / `custom:litellm` / `kilocode` to succeed. +- ⚠ Nothing in the fleet is *pinned* to `opencode-zen` — no cron job names it — so this never shows up as + a job failure. It shows up only as `agent.chat_completion_helpers: Streaming failed before delivery: + Error code: 401` in `~/.hermes/logs/errors.log`. → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] + +**To restore:** mint a new key in the OpenCode console and replace it in both stores (the Hermes provider +entry and `~/.local/share/opencode/auth.json`), then re-probe `…/zen/go/v1/models` for a 200 — or +**remove `opencode-zen` from `fallback_providers`**, since a dead second hop costs a round-trip on every +retry. Keys live in the vault; this page records only their state — the `hermes-api-vault` skill (vault-backed; no value is stored in this wiki). + ## Privacy Zero-retention policy from providers — your data is not used for model training. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index b9c7c87..017a8e8 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-11-10 -verified_on: 2026-11-10 -review_after: 2026-11-10 +updated: 2026-11-11 +verified_on: 2026-11-11 +review_after: 2026-11-12 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db] @@ -14,6 +14,28 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db # Scheduled Tasks ## Count +- **Current as of 2026-11-11 (window 2026-11-10 23:59 → 2026-11-11 23:59; nightly audit).** + **Fleet 29 (26 enabled / 3 disabled)** — unchanged for a sixth window. Three states moved, and the + observer itself is the story. **`cbfe434a3b5e` (Wiki Audit — Nightly) failed on its own 429** — + 2026-11-10 23:59:54.486572 → **2026-11-11 00:12:29.646987 = 12 m 35 s**, `RuntimeError: HTTP 429 … all + targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s`, + delivery `suppressed`, incident row `cbfe43_ed01845310d8` (`rate_limit`, `alerted_at: NULL`) **still + open and the fleet's only open row** → [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]. That + failure means **the 2026-11-10 window was never audited**; this page covers both days. + **Non-clean `last_status` 1 → 2** (`066bb5191280` Uber Eats `delivery_failed` + `cbfe434a3b5e` `error`, + streak 1). Both long-running incidents **recovered with output**: the stocktake (03:00:56 → 04:11:38 = + **1 h 10 m 42 s**, **36 stale pages re-verified → 0**, commits `341dda9` + `c108eb1`) and Job Radar + Stage 1 (07:01:08 → 07:37:34 = **36 m 26 s**, writing the first handoff since **2026-11-06** — + `radar-ref/handoff/2026-11-11.json`, 22,682 B), whose Stage 2 then auto-closed its incident row + `resolved` at 08:03:47. **Ledger: 2026-11-10 = 157 rows / 155 completed / 2 failed; 2026-11-11 = 172 + rows / 171 completed / 1 running / 0 failed** — a zero-failure day (first since 11-09). Longest 11-11 + run **1 h 10 m 42 s**; **no run over 71 minutes**, against 9 h 09 m (11-09) and 10 h 48 m (11-10). + **Deliveries 11-11: 7 rows, 7 / 7 `delivered`, 0 `for_failure`.** Open rows: **1** (was 0 on 11-09). + ⚠ **Provider/aux-path headwinds:** `opencode-zen`/`deepseek-v4.1-flash` sits at `fallback_providers[1]` + and threw **7 × HTTP 401 `api_key_expired`** on 11-11, each immediately after nous returned **14 × + 404 `insufficient_credits_for_paid_model`**; `auxiliary.pulse_memory` — the only slot with **no + `fallback_chain`** — logged 19 lines and 11 "refusing to guess another logged-in provider" refusals. + `config.yaml` unchanged since 2026-11-09 22:58:07. - **Current as of 2026-11-10 (window 2026-11-09; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-10 00:0x–01:0x AWST):** **29 jobs (26 enabled / 3 disabled** — `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`), fifth consecutive audit at the same count. **2026-11-09 settled: 174 rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit) — the **first `failed`-free day since 2026-11-05** (11-06: 1 failed, 11-07: 3, 11-08: 1). Deliveries **8/8 `delivered`**, 0 `for_failure` (same as 11-08). **Unclosed incident rows 3 → 0** (134 rows total): the whole stocktake cluster — `1a8e90_628d914f1f97` (11-07T04:29:54, *"Restart-safe cron worker exited with status 1 after adopting…"*), `1a8e90_2f370ef764f5` (11-07T05:02:21, *"…lenovo… lost its…"*), `1a8e90_a9cad99cb150` (11-08T04:35:38, *"No reply: the turn stopped while a tool…"*) — **closed 2026-11-09T12:10:47.125205, all three at once, the instant the overrunning run finished**; none ever alerted. The most recent incident activity is now the `85edf6772bca` row `85edf6_0877ca1e0eac` (*"Cron execution unverified"*, alerted 2026-11-08T07:23:07, closed 08:06:54). **The stocktake's `failure_streak` 3 → 0 and `last_status ok` are cosmetic:** `1a8e90ea2271` ran **03:00:43 → 12:10:47 = 9 h 10 m** (its 11-07 and 11-08 runs died at 2 h 02 m and 1 h 35 m), closed `completed` with delivery `suppressed`, and landed **zero edits and no commit** — the run's own output file reads `RUN INCOMPLETE`, tool-iteration budget exhausted inside verification. **Third consecutive day the stocktake's work did not land** (11-07, 11-08, 11-09). → [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]. **The Job Radar landed nothing on 11-09 either:** Stage 1 (`dca8482e4f76`) ran **07:00:48 → 16:10:12 = 9 h 09 m** and never wrote `radar-ref/handoff/2026-11-09.json` (newest handoff on disk: **2026-11-06**); both Stage-2 slots (`5fc292ff0f3b`, 11:30:38→12:10:07 and 14:30:34→14:37:27) fired **before** Stage 1 finished and declined silently (`suppressed`). → [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]. **No other 11-09 run exceeded 74 min** (stocktake 550 min, Stage 1 549 min, then 74 / 40 / 11 min), and the 11-08 `pre_tool_call` hook cascade did not recur — see the follow-up on [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. The stocktake remains pinned to `custom:omni-101` / **`free-stack`**, a combo recorded broken on **2026-08-14**; this audit itself runs `custom:omni-101` / `deepseek-paid-smart`. - **Current as of 2026-11-08 (window 2026-11-07; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-09 00:0x-00:1x AWST):** **29 jobs (26 enabled / 3 disabled** — `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; the count and split are unchanged across **six** consecutive audits, but fleet `updated_at` **moved** to **2026-11-09T00:06:50+08:00** — treat that as a run-state stamp, **not** a definition change, because no job was added, retired, disabled or repinned: re-verified every field on the nine jobs looked at and the enabled/disabled split is identical). **11-08 settled: 169 rows across 16 jobs — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33). The **only** failure is **`1a8e90ea2271` (wiki-stocktake-maintenance), 03:01:11 → 04:36:08 (1 h 34 m 57 s)**, and it is a **new terminal signature for that job**: `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` — the previous two red days both read *"lost its durable fire claim ownership"*. **`failure_streak` is now 3** — **three consecutive failed days**. *Working hypothesis, not proven:* the `conduit_push` `pre_tool_call` hook that was **timing out at 30 s 58 times between 2026-11-07 14:14:03 and 2026-11-08 08:21:03** holds a tool result open in exactly the way this error describes, but the string does not name the hook → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. The hole in the day was **the hook, not the schedule**: *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)**, *FreshRSS Midday Briefing* `ecc1fc13655a` **12:00:50 → 14:10:43 (2 h 09 m 53 s)**, both delivered. Deliveries **8 rows / 8 `delivered` / 0 `for_failure`** (11-07 was 7 / 7). **Unclosed incident rows 2 → 3** — a **third** silent `detected` row appeared (`1a8e90_a9cad99cb150`, `alerted_at: null`), and the two 11-07 stocktake rows are still `detected`; **the alerting path is not the fault** (the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window — its `database is locked` admission, closed when the run completed). Nothing added, retired, disabled or repinned. - **Current as of 2026-11-07 (window 2026-11-06; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-07 00:3x-00:5x AWST):** **29 jobs (26 enabled / 3 disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; count and split unchanged across five consecutive audits, fleet `updated_at` **2026-11-07T00:39:46+08:00**). **11-06 settled: 172 rows, 16 distinct jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:31). The single `failed` row is a **new, non-restart failure signature**: the Job Radar's **11:30 Stage-2 slot** (`95347fa5…`) was **discarded as stale** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job, started 08:27:09, was still holding the claim** - that fire completed **13:26:22** and delivered, and the ledger records **no gateway restart** in the window -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; the radar still landed the day in full at 14:30 (`MISSING_COVER_LETTER_COUNT 0`, 4 same-day strong rows, 4 cover letters, 20 non-empty artifacts, *Run status: COMPLETE*). Fleet non-clean `last_status` unchanged at **1** (`066bb5191280`, `delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0** - no ledger row has `last_seen_at` on or after 2026-11-06 (newest activity remains the 2026-11-05T12:36:17 closures), so **the discarded run raised no incident row and no alert**; [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] remains the one open page. The stocktake's overrun streak **fully unwound**: `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Nothing added, retired, disabled or repinned. diff --git a/systems/skills-index.md b/systems/skills-index.md index 2167ec4..34469f2 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-11-11 -verified_on: 2026-11-11 +updated: 2026-11-12 +verified_on: 2026-11-12 confidence: high tags: [skills, hermes] sources: []