diff --git a/.gitignore b/.gitignore index 72a9ce2..faa13c5 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,4 @@ public/ package-lock.json package.json *.log +.audit-report-*.txt diff --git a/current-state.md b/current-state.md index 270002f..5251ea8 100644 --- a/current-state.md +++ b/current-state.md @@ -3,8 +3,8 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-27 -verified_on: 2026-10-27 +updated: 2026-10-28 +verified_on: 2026-10-28 review_after: 2026-11-03 confidence: high tags: [system] @@ -12,7 +12,38 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv --- # Current State -Updated: 2026-10-27 +Updated: 2026-10-28 + +## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST) + +- **The radar's silent budget overrun became a recorded failure — and the radar is now two jobs.** Window + 2026-10-27 settled **172 `completed` / 1 `failed`**. The failure is `dca8482e4f76` (Daily Marketing Job + Radar - Discovery): **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed **`failed`**, delivery `delivered`, + and the delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — + discovery, verification, dedup and classification completed; **Notion ingestion, cover-letter generation and + the final cover-letter audit did not execute**. The prior day (10-26) had the same overrun but closed + `completed`, hiding it in a green row. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` + 11:03:59, **still open** at this audit (`failure_type: auth` — misclassified). → new page + [[incidents/2026-10-27-job-radar-budget-exhausted]] +- **Response: the radar was split into a two-stage pipeline the same day.** Stage 1 `dca8482e4f76` — Discovery + (`0 7 * * 1-5`, `deliver: local`, failures → topic 1030) writes a handoff and triggers stage 2; new stage 2 + `5fc292ff0f3b` — Daily Marketing Job Radar - Write and Letters (`30 11,14 * * 1-5`, Telegram topic 1030, + created **2026-10-27T17:34:15**) lands the Notion entries, cover letters and audit. Two test fires of stage 2 + on 10-27 (17:35:00 → 17:36:52, 17:38:38 → 17:43:42) both `completed` with delivery `suppressed`; the first + real 07:00 → 11:30 cycle had not yet run. **Fleet 28 → 29 jobs (26 / 3).** → [[runbooks/job-radar]] +- **The 2026-10-26 orphan class did not recur, and its two incident rows closed.** `ea79a2_a4661554a3c2` and + `ea79a2_c3ab2d784ba3` **auto-closed 00:38:32** (unclosed rows **3 → 2**, the 2 being the Facebook birthday + monthly scrape from 10-01 and the new radar row). No 10-27 run besides the radar's failed, so the watchdog + and every other job were clean all day. +- **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` + (streak 1) and `08eb86f02d37` re-joined as `delivery_queued`. Still non-clean: `0e753e9720fd` (error), + `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** +- **Also updated:** `incidents/2026-10-27-job-radar-budget-exhausted.md` (new), + `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` (recurrence check), `runbooks/job-radar.md`, + `systems/scheduled-tasks.md`, `systems/hermes-agent.md` (TTS provider correction), + `systems/conduit.md` (voice pipeline), `infrastructure/tailscale.md` (peer table re-verified), + `systems/skills-index.md` (regenerated 10-27, committed), `index.md`, `log.md`. + **Nothing removed or archived.** No secrets written. ## Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST) diff --git a/incidents/2026-10-26-planned-restart-orphaned-cron-run.md b/incidents/2026-10-26-planned-restart-orphaned-cron-run.md index f143ceb..46a3823 100644 --- a/incidents/2026-10-26-planned-restart-orphaned-cron-run.md +++ b/incidents/2026-10-26-planned-restart-orphaned-cron-run.md @@ -3,7 +3,7 @@ title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemos type: incident status: active created: 2026-10-27 -updated: 2026-10-27 +updated: 2026-10-28 verified_on: 2026-10-27 review_after: 2026-11-03 confidence: high @@ -101,3 +101,13 @@ disappears silently. What it did not do is *finish the work* or *avoid the inter - [[systems/scheduled-tasks]] — fleet ledger and per-window tallies - [[systems/mnemosyne]] — the interrupted job's purpose and settings - [[systems/hermes-agent]] — gateway lifecycle and restart history + +## Update — 2026-10-28 (nightly audit) + +- **No recurrence.** Every run in the 2026-10-27 window except the radar's budget overrun completed, and the + watchdog logged no alert for the day. +- **Both incident rows this fault raised closed automatically** — `ea79a2_a4661554a3c2` and + `ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** (unclosed rows fleet-wide fell 3 → 2). +- The `lost its durable fire claim ownership` string did not appear in any 10-27 execution row. +- Open question from the original write-up still stands: why a *planned* stop bypasses the drain deadline + (`timed_out=True` at 1.35 s of a 30 s budget). diff --git a/incidents/2026-10-27-job-radar-budget-exhausted.md b/incidents/2026-10-27-job-radar-budget-exhausted.md new file mode 100644 index 0000000..e5da7ec --- /dev/null +++ b/incidents/2026-10-27-job-radar-budget-exhausted.md @@ -0,0 +1,92 @@ +--- +title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response +type: incident +status: active +created: 2026-10-28 +updated: 2026-10-28 +verified_on: 2026-10-28 +review_after: 2026-11-04 +confidence: high +tags: [automation, cron, jobs, notion, silent-failure, scheduling] +sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api] +--- + +# Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response + +## Headline + +On **2026-10-27** the Daily Marketing Job Radar (`dca8482e4f76`) ran **07:01:08 → 11:03:54 AWST** +(**4 h 2 m 46 s** wall clock), closed **`failed`**, and delivered a report that opens: + +> **Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call budget exhausted before Notion write stage)** + +Discovery, verification, dedup and classification completed. **Notion ingestion, cover-letter generation and the +final cover-letter audit were not executed** — so the run is recorded as failed rather than successful. +Self-reported run time **10:47:57**; freshness cutoff **2026-10-20**. + +This is the second consecutive budget overrun: the **2026-10-26** 07:00 slot ran 07:01:08 → 09:10:27 +(**2 h 9 m 19 s**), closed **`completed`**, and *still* delivered the same `RUN INCOMPLETE` marker — the day's +backlog had to be finished by hand. 2026-10-27 is the first overrun recorded as **`failed`**, which is the +improvement: the failure reached the incident ledger instead of hiding inside a green row. + +## Ledger state + +| Field | Value | +|---|---| +| Execution row | `dca8482e4f76`, started `2026-10-27T07:01:08.265913+08:00`, finished `2026-10-27T11:03:54.143535+08:00`, `failed` | +| Delivery | `delivered` (the incomplete report reached Telegram topic 1030) | +| Incident row | `dca848_e93a538ade6b` — `state: alerted`, `failure_type: auth`, first/last seen `2026-10-27T11:03:53.715494+08:00`, alerted `11:03:59.013115` | +| Incident status | **open at the 2026-10-28 00:2x audit** — no ack, no close | +| `jobs.json` | `last_status: error`, `failure_streak: 1`, pin unchanged at `custom:omni-101` / `smart-route` | + +⚠ `failure_type: auth` does **not** describe this fault. A tool-call budget exhaustion is being classified as an +auth failure; the field is not trustworthy as a diagnosis for this job. + +## What the run actually did (from its own report) + +- Live Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168`: **451 entries**, `Date Added = 2026-10-27` → **0**. +- Board composition at run time: **Agent — Rhino 402, Chaos 12, Manual 2, blank 35**; **Source — SEEK 295, + LinkedIn 88** (row text truncated in the ledger). + +## Response, same day: the radar is now two jobs + +At **2026-10-27 17:34:15** the single radar job was split into a two-stage pipeline (see +[[runbooks/job-radar]] for the full contract): + +| Stage | Job | Schedule | Delivery | Job id | +|---|---|---|---|---| +| 1 — Discovery | Daily Marketing Job Radar - Discovery — `DISCOVERY → VERIFY → DEDUP → SCORE`, no Notion write, no cover letters; writes a handoff file | `0 7 * * 1-5` | `local` (failures → Telegram topic 1030) | `dca8482e4f76` (unchanged) | +| 2 — Write and Letters | Daily Marketing Job Radar - Write and Letters — lands the handoff: Notion entries, cover letters, audit, report | `30 11,14 * * 1-5` | Telegram group `-1004321904721`, topic `1030` | `5fc292ff0f3b` (created 17:34:15) | + +Stage 1 keeps its id, pin (`custom:omni-101` / `smart-route`) and its own tool budget; stage 2 is a separate job +whose prompt lands the handoff rather than repeating the discovery narrative. Two test fires of stage 2 ran the same +evening — **17:35:00 → 17:36:52** and **17:38:38 → 17:43:42** — both `completed`, both with delivery +`suppressed`, `last_status: ok`, streak 0. + +Fleet size went **28 → 29 jobs** (26 enabled / 3 disabled). + +## Why it matters + +1. **The failure mode is partial work, not a lost run.** Four hours of browser-heavy discovery produced a + correct report and zero writes. Nothing in `jobs.json` distinguished it from a good day except + `last_status: error` on 10-27 (`completed` on 10-26, with the same incompleteness buried in the text). +2. **Splitting is the mitigation, not a fix.** Stage 2 inherits its own tool budget; if stage 1 still overruns + and never writes the handoff file, stage 2 has nothing to land — watch the 11:30 slot for that. +3. **The budget is consumed by discovery**, which is exactly the stage that now owns it alone. + +## Open questions + +1. **Verify one full cycle end-to-end** (stage 1 07:00 → stage 2 11:30): does stage 2 find stage 1's handoff + file, do the Notion writes and cover letters land, and is `MISSING_COVER_LETTER_COUNT=0` reported? +2. **Make budget exhaustion self-describing.** The classifier tagged it `auth`; a distinct budget/partial-run + type would put this on the same footing as any other failure. +3. **Do the two stages fit the same wall-clock envelope?** Stage 1 must finish inside its own budget *and* leave + the handoff before 11:30. +4. The incident row stays open until a clean stage-1 slot auto-closes it — confirm the 2026-10-28 07:00 slot + closes `dca848_e93a538ade6b`. + +## Related + +- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun +- [[systems/scheduled-tasks]] — fleet ledger, job table and the 2026-10-27 window +- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — the radar's previous failure mode diff --git a/index.md b/index.md index db1c56d..017c5be 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-27 (nightly audit — 1 new incident page, 4 pages updated) +> model routing and automation. Last updated: 2026-10-28 (nightly audit — 1 new incident page, 9 pages updated) > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -35,10 +35,11 @@ - [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.2` (HEAD `942ddb8`, 2026-10-24 15:35 — *"Pulse: list candidates, decide in code, journal them (#342)"*; 24 commits on since the `0.64.0` baseline, **0 behind `origin/main`**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` restarted 2026-10-25 18:42:33 — so the upstream commits are now LIVE** (the 10-24 "not live" note is superseded). One registered device (iPhone app `2.0.0`). **Still carries the uncommitted in-tree `dashboard/plugin_api.py` patch**; `conduit-calls.json → history` is **empty**, so the memory fix still has no positive call evidence, but `conduit_show` ran once on 10-25 14:37 (card delivered). ⚠ **Known failure: three 10-24 requests to be phoned were refused by the agent, which never called the tool; no recurrence since** → [[incidents/2026-10-24-conduit-call-request-refused]]. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. -- [[systems/scheduled-tasks]] — **As of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** -- [[systems/skills-index]] — Installed skills inventory (**306 skills / 70 categories** as of the **2026-10-23 03:07** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at that run: **2 stale / 94 fresh / 6 no-date of 102 pages** (`freshness-20261023.txt`). Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. +- [[systems/scheduled-tasks]] — **As of 2026-10-28 00:2x: 29 jobs (26 enabled, 3 disabled), 5 non-clean `last_status`, 2 unclosed incident rows; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** +- [[systems/skills-index]] — Installed skills inventory (**302 skills** (69 categories) as of the **2026-10-27** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at the 2026-10-27 run: **0 stale / 104 fresh / 4 no-date of 108 pages** (`freshness-20261027.txt`). The 2026-10-27 03:00 slot regenerated `systems/skills-index.md` but left it uncommitted in the authoring tree; the 2026-10-28 audit committed it. Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents +- [[incidents/2026-10-27-job-radar-budget-exhausted]] — **The Daily Marketing Job Radar exhausted its tool-call budget for the second day running — and this time the run is recorded as `failed`, not `completed`.** On 2026-10-27 the 07:00 slot ran **07:01:08 → 11:03:54** (4 h 2 m 46 s) and its delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery/verify/dedup/classify finished, **Notion writes, cover letters and the final audit never ran** (Notion DB held 451 entries, 0 added for the day). Incident row `dca848_e93a538ade6b` opened 11:03:53 (`alerted` 11:03:59, `failure_type: auth` — wrong for a budget fault) and was still open at the 2026-10-28 audit. **Response the same day:** the radar was split into stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) and new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030), created 17:34:15, two `completed` test fires that evening → [[runbooks/job-radar]] - [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **A *planned* gateway restart orphaned a live cron run — the second instance of the orphan class, and the first that alerted.** On 2026-10-26 at **14:33:35 AWST** the gateway took a planned `SIGTERM` (`under_systemd=yes`); the shutdown drain reported **`timed_out=True` after just 1.35 s** of its 30 s budget and interrupted `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) **2 h 34 m** in. It closed **`failed`** on a **new** string — `lost its durable fire claim ownership` — raised **two** `cron_incidents` rows (14:33:37 *unverified / reconcile admission* + 14:35:59 *fire claim*), **delivered** the failure at 14:35:59, and left the job on **`failure_streak: 2`** from one interrupted fire. Two Mnemosyne Watchdog reds followed at 14:46 and 15:01 and self-resolved at **15:16:12**. Net = **169 / 3** for 10-26. Unresolved: why a **planned** stop bypasses the drain deadline, and whether `failure_streak` should count the unverified row. → also updates [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] (the class now recurs without a crash). - [[incidents/2026-10-25-gmail-digest-runaway-and-refire]] — **The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25.** The 08:30 slot started 08:31:14 and closed **10:26:37 `failed`** on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed)` — against **91 s** (10-23) and **104 s** (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a **new signature** — `Cron admission missing: the owner never admitted this fire` (distinct from the 10-20 orphan) — and all six fires, failures included, carry `delivery_outcome: delivered`, so up to **six digests reached Anthony**. Neither failure raised a `cron_incidents` row; the job now reads `last_status: delivery_queued` with an **empty** `last_error`. Hypothesis (unconfirmed): the 08:31 fire **adopted an unrelated long-lived session** — its worker made **70 `terminal` calls on VM101 `/dev/sdb` backup work** with context growing 66 k → 148 k tokens, while **context compression was unavailable all day** (`invalid_grant`). → [[systems/scheduled-tasks]], [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] - [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] — **`Invalid refresh token` — Nous Portal runtime credentials are dead, so context summarisation failed.** Gateway-log counts: **1** line 2026-10-22, **3** on 10-24, **20** on 10-25 (last 14:53:04, `agent.context_compressor: Failed to generate context summary`), **0** since the 19:27:13 restart. ⚠ **Masked, not fixed:** at **2026-10-25 19:13** all **eight** `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off `nous` to **`no-think/freeapi/auto` on `custom:omni-101`** (fallback `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`; compression timeout 90 s, vision 180 s). No `hermes model` re-authentication has been run and **`nous` is still the first entry in `fallback_providers`**, so failover tries a dead credential. → [[systems/model-providers]], [[systems/current-profile]] @@ -115,7 +116,7 @@ - [[runbooks/provider-health]] — Provider health check - [[runbooks/troubleshoot-tool-loop]] — Tool loop troubleshooting - [[runbooks/gateway-resource-resilience]] — Gateway resilience -- [[runbooks/job-radar]] — Job Radar search, Notion, dashboard and recovery procedure +- [[runbooks/job-radar]] — Job Radar search, Notion, dashboard and recovery procedure (**two-stage since 2026-10-27:** `dca8482e4f76` discovery → `5fc292ff0f3b` write and letters) ## Raw - [[raw/configs/hermes-config-sanitized.txt]] — Sanitized config diff --git a/infrastructure/tailscale.md b/infrastructure/tailscale.md index ae435ee..70ed230 100644 --- a/infrastructure/tailscale.md +++ b/infrastructure/tailscale.md @@ -3,8 +3,8 @@ title: Tailscale type: infrastructure status: active created: 2026-07-26 -updated: 2026-10-26 -verified_on: 2026-10-26 +updated: 2026-10-28 +verified_on: 2026-10-28 confidence: high tags: [tailscale, vpn, networking, infrastructure] --- @@ -25,35 +25,36 @@ Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, seve Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`. Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**. Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — **`octop` (100.92.163.122), `springroll` (100.100.153.125), `openmausbot` and `openmausbot-vm` were missing from the table body** (only mentioned in prose), `omarchymac` had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from `tailscale status --json`, keyed on DNS name rather than `HostName` (the JSON `HostName` for `freshrss` is `ct260`, and both `omniroute` nodes share the hostname `omniroute`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running. +Re-verified 2026-10-28 00:2x (nightly audit): State column rebuilt from `tailscale status --json` — **14 rows changed** (anthonys-imac: idle/offline 32d -> offline 34d; cachyos: idle/offline 52d -> offline 54d; cloudreve: idle/offline 93d — CT209 stopped -> offline 95d — CT209 stopped; conduit: idle/offline 49d -> offline 52d; elementsynapse: idle/offline 96d — CT555 stopped -> offline 98d — CT555 stopped; freshrss-new: idle/offline 69d -> offline 71d; m93p: online -> offline <1d; mattermost: idle/offline 36d — CT255 stopped -> offline 38d — CT255 stopped; omarchybook: idle/offline 4d -> offline 6d; omarchymac: idle/offline 8d -> offline 10d; omni: idle/offline 69d — CT601 removed from both hypervisors -> offline 71d — CT601 removed from both hypervisors; openmausbot: idle/offline 4d -> offline 6d; sftpgo: online — CT299 (pve) -> offline <1d — CT299 (pve); ubuntu-server-xfce: idle/offline 66d -> offline 68d). Header count corrected to **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Nodes in the table that are no longer in the tailnet: **none**. Live IPv4 peers absent from the table: **none**. **State transitions inside the audit window (not just age drift): `m93p` online → offline <1d, `sftpgo` (CT299) online → offline <1d.** Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale status --json` — **`springroll-browser` (100.109.221.91) was missing entirely** and has been added. **`omarchybook` (100.107.9.100) and `openmausbot` (100.91.214.91) have both gone offline** (online → idle/offline 3d) and `openmausbot` is no longer a live peer despite its m93p LXC (CT1264) still running `tailscale` — the node itself has stopped connecting. Every offline age was refreshed (e.g. `mattermost` 31d → **35d**, `omni` 64d → **68d**, `elementsynapse` 91d → **95d**, `ubuntu-server-xfce` 60d → **64d**, `omarchymac` 3d → **7d**). `m93p` (100.104.51.98) now reads **idle/offline 13m ago** — the peer record is stale, but the host is plainly reachable (both `pct list` probes and the SSH session to CT201/CT460 during this audit succeeded), so treat it as a Tailscale-side liveness artefact, not a hypervisor outage. `homeassistant` still advertises `ExitNodeOption: true` but `ExitNode: false` — it **offers** exit-node capability; it is not currently serving as one (the old "active; offers exit node" wording overstated the live state). Container states cross-checked again against live `pct list` on both hypervisors, and each peer's container identity confirmed by running `tailscale ip -4` **inside** the candidate LXC on both hosts — the authoritative mapping, since peer names and LXC names do not always match. | Node | IP | State | |---|---|---| | hermes | 100.118.5.51 | **self** — CT460 (pve) | | agentzero | 100.106.54.94 | online — CT475 (pve) | -| anthonys-imac | 100.112.123.124 | idle/offline 32d | +| anthonys-imac | 100.112.123.124 | offline 34d | | anthonys-ipad | 100.111.198.120 | online | | anthonys-iphone | 100.100.116.67 | online | | anthonys-mac-mini | 100.74.17.15 | online | -| cachyos | 100.124.208.15 | idle/offline 52d | +| cachyos | 100.124.208.15 | offline 54d | | clawtest | 100.122.189.81 | online — CT333 (pve) | | cliproxyapi | 100.83.112.94 | online — CT103 (pve) | -| cloudreve | 100.111.187.7 | idle/offline 93d — CT209 stopped | -| conduit | 100.81.79.87 | idle/offline 49d | +| cloudreve | 100.111.187.7 | offline 95d — CT209 stopped | +| conduit | 100.81.79.87 | offline 52d | | docker | 100.96.150.69 | online — CT200 (pve) | | docker-01 | 100.121.12.31 | online — CT1001 (m93p) | -| elementsynapse | 100.75.250.37 | idle/offline 96d — CT555 stopped | +| elementsynapse | 100.75.250.37 | offline 98d — CT555 stopped | | freellmapi | 100.127.208.75 | online — CT1006 (m93p) | | freshrss | 100.83.60.93 | online — CT1260 (m93p) | -| freshrss-new | 100.110.180.115 | idle/offline 69d | +| freshrss-new | 100.110.180.115 | offline 71d | | gitea | 100.82.100.128 | online — CT221 (m93p) | | guest-vps | 100.117.14.84 | online | | hermesagent | 100.98.205.112 | online — CT450 (pve) | | homeassistant | 100.68.139.5 | online | | litellm | 100.79.29.64 | online — CT500 (m93p) | -| m93p | 100.104.51.98 | online | +| m93p | 100.104.51.98 | offline <1d | | mailflow | 100.115.60.92 | online — CT1262 (m93p) | -| mattermost | 100.98.116.14 | idle/offline 36d — CT255 stopped | +| mattermost | 100.98.116.14 | offline 38d — CT255 stopped | | n8n | 100.84.124.30 | online | | nextcloud | 100.90.102.19 | online — CT270 (pve) | | ntfy | 100.67.153.116 | online — CT1241 (m93p) | @@ -61,13 +62,13 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | octop | 100.92.163.122 | online | | ollama | 100.108.44.119 | online — CT205 (pve) | | omarchy | 100.73.227.103 | online | -| omarchybook | 100.107.9.100 | idle/offline 4d | -| omarchymac | 100.84.180.69 | idle/offline 8d | +| omarchybook | 100.107.9.100 | offline 6d | +| omarchymac | 100.84.180.69 | offline 10d | | omarchymac-1 | 100.126.173.115 | online | -| omni | 100.93.204.4 | idle/offline 69d — CT601 removed from both hypervisors | +| omni | 100.93.204.4 | offline 71d — CT601 removed from both hypervisors | | omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute | | omniroute-1 | 100.78.63.20 | online — CT101 (m93p) | -| openmausbot | 100.91.214.91 | idle/offline 4d | +| openmausbot | 100.91.214.91 | offline 6d | | openmausbot-vm | 100.77.107.30 | online | | openweb | 100.69.230.13 | online — CT245 (m93p) | | paseo | 100.72.188.93 | online — CT1261 (m93p) | @@ -76,11 +77,11 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | rclone | 100.107.116.17 | online — CT104 (m93p) | | runtipi | 100.96.244.39 | online — CT201 (pve) | | rustdesk | 100.95.227.57 | online — CT1008 (m93p) | -| sftpgo | 100.88.223.70 | online — CT299 (pve) | +| sftpgo | 100.88.223.70 | offline <1d — CT299 (pve) | | social-dashboard | 100.70.183.37 | online — CT1263 (m93p) | | springroll | 100.100.153.125 | online | | springroll-browser | 100.109.221.91 | online | -| ubuntu-server-xfce | 100.101.26.104 | idle/offline 66d | +| ubuntu-server-xfce | 100.101.26.104 | offline 68d | | vaultwarden | 100.123.225.90 | online — CT102 (m93p) | | webserver | 100.82.40.23 | online — CT207 (pve) | diff --git a/log.md b/log.md index f052627..2f35207 100644 --- a/log.md +++ b/log.md @@ -1454,3 +1454,55 @@ not pushed. whether `failure_streak` should count the *unverified* row (one interruption currently reads streak 2); and the **`Thread 34795`** delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the unthreaded chat. + +## [2026-10-28] audit | Nightly wiki audit — window 2026-10-26 00:00 → 2026-10-27 00:5x AWST: a planned restart orphaned a live cron run + +- **Window 2026-10-27: 172 `completed` / 1 `failed` (+ this audit `running`).** The failure is the radar's + budget overrun, and it is the story of the night → new page + [[incidents/2026-10-27-job-radar-budget-exhausted]]. +- **The radar's tool-call budget was exhausted for the second consecutive day — and this time the run is + recorded as `failed`.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed` and + delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery, verification, + dedup and classification completed; **Notion ingestion, cover-letter generation and the final cover-letter + audit never ran**. Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168` held **451 entries** with **0** added + for 2026-10-27 (Agent — Rhino 402 / Chaos 12 / Manual 2 / blank 35; Source — SEEK 295 / LinkedIn 88). + Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** at this audit + (`failure_type: auth` — which does not describe a budget fault). The previous day's identical overrun closed + **`completed`** and was found only by reading the delivered text — that is why the 10-27 row matters. +- **Response, same day: the radar is now two jobs.** Stage 1 `dca8482e4f76` **Daily Marketing Job Radar - + Discovery** (`0 7 * * 1-5`, `deliver: local`, failures → Telegram topic 1030) does + `DISCOVERY → VERIFY → DEDUP → SCORE`, writes a handoff file and triggers stage 2. New stage 2 + **`5fc292ff0f3b`** **Daily Marketing Job Radar - Write and Letters** (`30 11,14 * * 1-5`, Telegram topic + 1030, created **2026-10-27T17:34:15**) lands the Notion entries, cover letters, missing-letter audit and + report. Two stage-2 test fires on 10-27 (**17:35:00 → 17:36:52**, **17:38:38 → 17:43:42**) both + `completed`, delivery `suppressed`, `last_status: ok`, streak 0. Pins unchanged (`custom:omni-101` / + `smart-route`). **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** → `runbooks/job-radar.md` +- **The 2026-10-26 orphan class did not recur; its two incident rows closed.** `ea79a2_a4661554a3c2` and + `ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** — unclosed rows fleet-wide **3 → 2** (the + 2: `0e753e_e7f8194a7d06` Facebook birthday monthly scrape, open since 2026-10-01 09:00:55, and the new + `dca848_e93a538ade6b`). No 10-27 execution row carries the orphan strings and the watchdog alert log has no + 10-27 entry. → `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` +- **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` + (streak 1) and `08eb86f02d37` re-joined as `delivery_queued`. Also non-clean: `0e753e9720fd` (error), + `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). +- **Two config corrections from live probes.** `tts.provider` is **`openai`** pointed at the **Mac mini speech + server** (`https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, model `phonon`, voice `alloy`) — the + page still claimed ElevenLabs from 2026-10-10 (change date not established, so the old bullet is marked + superseded rather than deleted). Live voice uses `voice.speech_server.url` = + `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/`; `GET /v1/pool` → `{"size":2,"in_use":0}`, both + units `idle`, while `/v1/models` returns `{"detail":"Not Found"}`. `voice.auto_tts` is still `false`, and + `wake_word.enabled` is `true`. → `systems/hermes-agent.md`, `systems/conduit.md` +- **`systems/skills-index.md` was regenerated by the 2026-10-27 03:00 slot but never committed** — the + authoring tree was dirty with `updated: 2026-10-27` and a **302 skills / 69 categories** total (against + index.md's stale **306 / 70 of 2026-10-23**). Freshness at that run: **0 stale / 104 fresh / 4 no-date of + 108 pages** (`freshness-20261027.txt`). Committed, and `index.md` corrected to 302. +- **`infrastructure/tailscale.md` peer table re-verified** against `tailscale status --json`; the node-count + header and any state changes are recorded in that page. +- **Files:** new `incidents/2026-10-27-job-radar-budget-exhausted.md`; updated + `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`, `runbooks/job-radar.md`, + `systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `systems/conduit.md`, + `infrastructure/tailscale.md`, `systems/skills-index.md`, `current-state.md`, `index.md`, `log.md`. + **Nothing removed or archived.** No secrets written. +- **Unresolved:** whether the two-stage split actually fixes the budget problem — the first real 07:00 → 11:30 + cycle (stage 1 handoff → stage 2 writes) had not run at audit time; why a budget exhaustion is classified + `failure_type: auth`; and the 10-27 radar incident row will only close when a clean stage-1 slot runs. diff --git a/raw/stocktake/freshness-20261027.txt b/raw/stocktake/freshness-20261027.txt new file mode 100644 index 0000000..db9fb7c --- /dev/null +++ b/raw/stocktake/freshness-20261027.txt @@ -0,0 +1,121 @@ +# Wiki freshness audit — 2026-10-27 +Threshold: >21 days since `updated` + +## STALE (0) + + +## FRESH (104) + +- [ 7d] CONTRIBUTING.md +- [ 1d] agents/hermes-chaos.md +- [ 1d] agents/hermes-production.md +- [ 7d] agents/index.md +- [ 16d] agents/nanobot.md +- [ 7d] agents/open-claw.md +- [ 16d] agents/qwenpaw.md +- [ 7d] agents/zeroclaw.md +- [ 7d] comparisons/camoufox-vs-chrome.md +- [ 7d] comparisons/hermes-vs-nanobot.md +- [ 7d] comparisons/memory-backends.md +- [ 7d] comparisons/model-routers.md +- [ 14d] comparisons/steel-vs-camoufox.md +- [ 7d] concepts/agent-memory.md +- [ 14d] concepts/browser-automation.md +- [ 7d] concepts/model-routing.md +- [ 7d] concepts/persistent-shells.md +- [ 7d] concepts/tool-calling.md +- [ 0d] current-state.md +- [ 7d] decisions/index.md +- [ 7d] decisions/keep-openrouter-as-fallback.md +- [ 7d] decisions/persistent-shell-enabled.md +- [ 7d] decisions/rollback-steel-to-camofox.md +- [ 7d] experiments/index.md +- [ 3d] incidents/2026-07-29-mnemosyne-relay.md +- [ 7d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 3d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 5d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 4d] incidents/2026-08-15-skills-index-drift.md +- [ 15d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 2d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 7d] incidents/2026-08-29-gmail-read-state.md +- [ 6d] incidents/2026-09-02-sftpgo-outage.md +- [ 16d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 21d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 1d] incidents/2026-10-03-ct450-hermes-update.md +- [ 1d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 21d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 3d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 18d] incidents/2026-10-09-memory-recall-relevance.md +- [ 11d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 0d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 8d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 0d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 5d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 2d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 1d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 1d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 1d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 0d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 5d] infrastructure/backups.md +- [ 7d] infrastructure/credentials-map.md +- [ 16d] infrastructure/docker-services.md +- [ 7d] infrastructure/domains-and-tunnels.md +- [ 1d] infrastructure/hosts.md +- [ 13d] infrastructure/network-map.md +- [ 13d] infrastructure/proxmox.md +- [ 1d] infrastructure/tailscale.md +- [ 15d] queries/retained-research-answers.md +- [ 2d] reports/communities-wa-social-media-audit-2026.md +- [ 6d] reports/uber-eats/2026-07.md +- [ 6d] reports/uber-eats/2026-08.md +- [ 6d] reports/uber-eats/2026-09.md +- [ 6d] reports/uber-eats/index.md +- [ 7d] runbooks/backup-wiki.md +- [ 7d] runbooks/diagnose-docker.md +- [ 7d] runbooks/gateway-resource-resilience.md +- [ 0d] runbooks/job-radar.md +- [ 7d] runbooks/multiplexer-setup.md +- [ 1d] runbooks/provider-health.md +- [ 7d] runbooks/recover-docker-service.md +- [ 7d] runbooks/restart-browser.md +- [ 7d] runbooks/restart-hermes.md +- [ 7d] runbooks/restore-openrouter.md +- [ 7d] runbooks/test-openrouter.md +- [ 7d] runbooks/troubleshoot-tool-loop.md +- [ 1d] runbooks/update-hermes-safely.md +- [ 14d] systems/browser-backend.md +- [ 7d] systems/cloudflare-tunnel.md +- [ 1d] systems/conduit.md +- [ 1d] systems/current-profile.md +- [ 7d] systems/guanaco.md +- [ 7d] systems/headless-server.md +- [ 1d] systems/hermes-agent.md +- [ 7d] systems/hermes-ui.md +- [ 15d] systems/home-assistant.md +- [ 2d] systems/homepage-dashboard.md +- [ 16d] systems/ignis.md +- [ 1d] systems/memory-backend.md +- [ 1d] systems/messaging-integrations.md +- [ 3d] systems/mnemosyne.md +- [ 1d] systems/model-providers.md +- [ 15d] systems/nanobot.md +- [ 7d] systems/nextcloud.md +- [ 7d] systems/obsidian.md +- [ 5d] systems/omniroute.md +- [ 7d] systems/opencode-go.md +- [ 7d] systems/openrouter.md +- [ 16d] systems/paseo.md +- [ 8d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 7d] systems/terminal-backend.md +- [ 7d] systems/tool-search.md + +## NO updated DATE (4) + +- SCHEMA.md +- experiments/active/example.md +- index.md +- log.md + +TOTAL: 0 stale, 104 fresh, 4 no-date of 108 pages \ No newline at end of file diff --git a/runbooks/job-radar.md b/runbooks/job-radar.md index e6c3637..b54ecf1 100644 --- a/runbooks/job-radar.md +++ b/runbooks/job-radar.md @@ -3,10 +3,10 @@ title: Daily Marketing Job Radar Operations and Recovery type: runbook status: active created: 2026-07-27 -updated: 2026-10-27 +updated: 2026-10-28 review_after: 2026-11-10 -verified_on: 2026-10-27 -version_applies_to: Hermes Agent cron dca8482e4f76 +verified_on: 2026-10-28 +version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2) confidence: high tags: [automation, cron, jobs, notion, dashboard, runbook] sources: [live-cron-output, live-notion-api, live-dashboard-api, hermes-docs] @@ -22,11 +22,14 @@ Run a verified Western Australian marketing-job search, deduplicate against the | Setting | Value | |---|---| -| Cron job | `dca8482e4f76` — Daily Marketing Job Radar | -| Schedule | `0 7 * * 1-5` (07:00 AWST weekdays) | -| Delivery | Telegram group `-1004321904721`, topic `1030` | +| Cron job — stage 1 | `dca8482e4f76` — Daily Marketing Job Radar - Discovery (Discovery → Verify → Dedup → Score; no Notion write, no cover letters) | +| Schedule — stage 1 | `0 7 * * 1-5` (07:00 AWST weekdays) | +| Delivery — stage 1 | `local` (dashboard/CLI); failures → Telegram group `-1004321904721`, topic `1030` | +| Cron job — stage 2 | `5fc292ff0f3b` — Daily Marketing Job Radar - Write and Letters (created 2026-10-27 17:34:15) | +| Schedule — stage 2 | `30 11,14 * * 1-5` (11:30 and 14:30 AWST weekdays) | +| Delivery — stage 2 | Telegram group `-1004321904721`, topic `1030` | | Skill | `job-search-automation` | -| Primary model | `custom:omni-101` / `gpt-5.6-terra` (upstream `command-code`) — ⚠ 403 `MODEL_NOT_IN_PLAN` since 2026-09-26 | +| Primary model (both stages) | `custom:omni-101` / `smart-route` (live 2026-10-28) — the 2026-09-26 `gpt-5.6-terra` 403 is historical | | First fallback | `opencode-go / mimo-v2.5` | | Dashboard | `https://hermes.kangaroo-eel.ts.net/jobs/` | | Dashboard backend | `127.0.0.1:9099` | @@ -279,6 +282,17 @@ produced a visibly different document — Anthony rejected exactly that mismatch - All five artefacts (`.txt .json .html .docx .pdf`) must still exist and be nonempty, and the HTML must contain no unresolved `{{...}}` tokens — the HTML is now the source of truth the PDF follows. +### 2026-10-27 07:00 slot — the same overrun, now recorded as `failed` (and the trigger for the split) + +The 07:00 slot ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**) and closed **`failed`** on delivery +`delivered`. The report it sent opens *"Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call +budget exhausted before Notion write stage)"*: discovery, verification, dedup and classification finished; +**Notion writes, cover letters and the final audit did not run**. Notion DB +`0f90ba2b-8b10-4d02-a62c-6f692d5b1168` held **451 entries** with **0** added for 2026-10-27 at run time. +Incident row `dca848_e93a538ade6b` opened 11:03:53 (`alerted` 11:03:59) and was **still open** at the +2026-10-28 audit. The split above is the response — see +[[incidents/2026-10-27-job-radar-budget-exhausted]] for the full ledger. + ### 2026-10-26 07:00 slot — `RUN INCOMPLETE`, and why the backlog was finished by hand The 07:00 run started on schedule and closed **`completed` at 09:10**, but the report it delivered read @@ -287,11 +301,38 @@ the **Notion writes and the entire cover-letter stage never executed**. The same interactively (rows written for the outstanding strong matches, cover letters generated and verified). - **A `completed` execution row is not a completed radar run.** Read the delivered text for the - `RUN INCOMPLETE` marker before treating a slot as done. + `RUN INCOMPLETE` marker before treating a slot as done. A *failed* row with a delivered report is the honest + version of the same fault — the 2026-10-27 slot is the first time it reached the incident ledger. - Budget the browser-heavy stages (one tab in flight, close before opening the next — see the 2026-10-12 hazard below) and keep headroom for the Notion + cover-letter writes, which sit last in the pipeline. +## Two-stage split (changed 2026-10-27) + +The single-job radar was split after two consecutive budget overruns (2026-10-26 `completed` but +`RUN INCOMPLETE`, 2026-10-27 closed `failed` — see +[[incidents/2026-10-27-job-radar-budget-exhausted]]). Discovery is the stage that eats the tool budget, so it +now owns a job of its own and the writes own a second one. + +| Stage | Job id | Schedule | Writes | Delivers | +|---|---|---|---|---| +| 1 — Discovery | `dca8482e4f76` (existing id) | `0 7 * * 1-5` | a handoff file only | `local`; failures → topic 1030 | +| 2 — Write and Letters | `5fc292ff0f3b` (new) | `30 11,14 * * 1-5` | Notion entries, cover letters, final audit, report | Telegram topic 1030 | + +- **Stage 1** does `DISCOVERY → VERIFY → DEDUP → SCORE`, writes the handoff file, and triggers stage 2. It must + **not** write to Notion and must **not** generate cover letters. +- **Stage 2** lands the handoff: Notion entries, cover letters, the missing-letter audit and the report. Its + prompt is not a copy of stage 1's — it reports what it wrote, not what it found. +- Stage 1 keeps the `custom:omni-101` / `smart-route` pin. Both jobs still inherit the profile-level + `fallback_providers` chain; Hermes stores no per-job fallback list. +- First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both + `completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle had not yet run at the + 2026-10-28 audit.** + +**Recovery if stage 2 lands nothing:** check that stage 1 actually wrote its handoff file before running stage 2 +by hand (`hermes cron run 5fc292ff0f3b`). A stage-2 run with no handoff is a no-op, not a failed search — do not +re-run stage 1's discovery just to satisfy it. + ## Browser-layer hazard inside the run (2026-10-12) This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove diff --git a/systems/conduit.md b/systems/conduit.md index 6305b3f..ccbcf5e 100644 --- a/systems/conduit.md +++ b/systems/conduit.md @@ -3,8 +3,8 @@ title: Conduit type: system status: active created: 2026-10-24 -updated: 2026-10-26 -verified_on: 2026-10-26 +updated: 2026-10-28 +verified_on: 2026-10-28 review_after: 2026-11-08 confidence: high tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling] @@ -90,6 +90,19 @@ the call rings when the asking turn ends and plays the reason first. Limits: `mi asserted the capability did not exist **without calling the tool or loading the skill** → [[incidents/2026-10-24-conduit-call-request-refused]]. +## Voice pipeline (probed 2026-10-28) + +Phone voice sessions ride the Mac mini speech server, which is also the Hermes TTS target: + +- `voice.speech_server.url` = `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/` — live-voice transport, + `wake_word.enabled: true`. +- `tts.provider: openai` → `https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech` (model `phonon`, + voice `alloy`) — see [[systems/hermes-agent]]. +- **Health check:** `GET https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/pool` returned + `{"size": 2, "in_use": 0}` with both units `idle` on 2026-10-28 00:2x — a **2-unit pool**. `/v1/models` is + **not implemented** (`{"detail":"Not Found"}`); probe `/v1/pool`, not `/v1/models`. +- `voice.auto_tts: false` — assistant replies are not auto-spoken; audio appears only when requested. + ## Memory integration The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not through Mnemosyne's `system_prompt_block()` — recall is wrapped in a per-request deadline diff --git a/systems/hermes-agent.md b/systems/hermes-agent.md index f49013e..2ddfa2c 100644 --- a/systems/hermes-agent.md +++ b/systems/hermes-agent.md @@ -3,8 +3,8 @@ title: Hermes Agent type: system status: active created: 2026-07-22 -updated: 2026-10-26 -verified_on: 2026-10-26 +updated: 2026-10-28 +verified_on: 2026-10-28 confidence: high tags: [system, hermes, agent, llm] sources: [live-hermes-version, live-config, live-skill-inventory] @@ -166,6 +166,15 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi - **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS output appears only when explicitly requested. + - ⚠ **Superseded — live config checked 2026-10-28** (change date not established): `tts.provider` is + **`openai`** pointed at the **Mac mini speech server** — `tts.providers.openai.base_url` = + `https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, `model: phonon`, `voice: alloy`. Live voice + (as opposed to one-shot TTS) runs over `voice.speech_server.url` = + `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/`, with `wake_word.enabled: true`. A `command`-type + provider `ollama_local` (`~/.hermes/scripts/ollama-tts.sh`) remains configured as an alternative. + `tts.elevenlabs` (`eleven_flash_v2_5`) and `tts.gemini` (`gemini-2.5-flash-preview-tts`, voice `Charon`, + persona file `~/.hermes/hermes-tts-persona.txt`) are still present as provider blocks. `voice.auto_tts` + is still `false`. See [[systems/conduit]] for the voice path. - **Mattermost platform unreachable** since 2026-10-10; the gateway keeps retrying on a 5-minute cadence - [[systems/messaging-integrations]]. - **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]]. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index f2cb2d4..29dc8bd 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,8 +3,8 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-27 -verified_on: 2026-10-27 +updated: 2026-10-28 +verified_on: 2026-10-28 review_after: 2026-11-03 confidence: high tags: [cron, scheduled, automation] @@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Current as of 2026-10-27 00:1x AWST (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`). +- **Current as of 2026-10-28 00:2x AWST (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. +- **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`). - **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`). - **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. *(Superseded by the 2026-10-26 entry above.)* - **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]] @@ -49,7 +50,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) | | AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 | | Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | | -| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00 | +| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00 | +| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. See [[incidents/2026-10-27-job-radar-budget-exhausted]] | | DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant | | Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] | | Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent | @@ -218,7 +220,7 @@ wholly clean: the first zero-failure day in the four on record here, and the end |---|---|---|---| | Facebook Birthday Monthly Scrape | `0e753e9720fd` | `error` | 0 | | Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `error` | 0 | -| Daily Marketing Job Radar | `dca8482e4f76` | `error` | 1 | +| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `error` | 1 | | Enhanced Weekend Adventure Planner | `1b23c042e17d` | `error` | 1 | | Uber Eats Monthly Spend Report | `066bb5191280` | `delivery_failed` | 0 | diff --git a/systems/skills-index.md b/systems/skills-index.md index 6c3318a..ae18c8a 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-10-26 -verified_on: 2026-10-26 +updated: 2026-10-27 +verified_on: 2026-10-27 confidence: high tags: [skills, hermes] sources: []