From e4086ec18ee718a14e89a714ee7c50626aacb5e3 Mon Sep 17 00:00:00 2001 From: hermes Date: Thu, 5 Nov 2026 01:04:31 +0800 Subject: [PATCH] =?UTF-8?q?wiki:=20nightly=20audit=202026-11-04=20window?= =?UTF-8?q?=20=E2=80=94=20new=20incident=20(Job=20Radar=20+=20Newsletter?= =?UTF-8?q?=20Digest=20both=20killed=20by=20upstream=20stream=20failure;?= =?UTF-8?q?=20Stage=202=20rebuilt=20handoff=20and=20delivered=205=20roles?= =?UTF-8?q?=20+=201=20letter);=20CT299=20second=20freeze=20documented;=20C?= =?UTF-8?q?L=5FMODEL=20402=20caveat=20in=20job-radar=20runbook;=20schedule?= =?UTF-8?q?d-tasks/current-state/index/log=20updated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- current-state.md | 38 +++++- ...-04-upstream-stream-failure-0700-cohort.md | 120 +++++++++++++++++ ...11-04-vzdump-freeze-ct299-sftpgo-outage.md | 121 ++++++++++++++++++ index.md | 11 +- infrastructure/tailscale.md | 41 +++--- log.md | 37 ++++++ runbooks/job-radar.md | 19 ++- systems/scheduled-tasks.md | 9 +- 8 files changed, 361 insertions(+), 35 deletions(-) create mode 100644 incidents/2026-11-04-upstream-stream-failure-0700-cohort.md create mode 100644 incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md diff --git a/current-state.md b/current-state.md index e3e56c3..29cd0e8 100644 --- a/current-state.md +++ b/current-state.md @@ -3,16 +3,46 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-31 -verified_on: 2026-10-31 +updated: 2026-11-05 +verified_on: 2026-11-05 review_after: 2026-11-07 confidence: high tags: [system] -sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] +sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents] --- # Current State -Updated: 2026-10-30 +Updated: 2026-11-05 + +## Recent Changes (2026-11-05, nightly audit; live probes 2026-11-05 00:0x-00:5x AWST) + +- **A dead route took out two 07:00 jobs at once - and the safety-net slot saved the day.** `dca8482e4f76` + (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) + both died on `RuntimeError: Upstream stream failed before completion.` - **36 seconds apart**, at **09:41:44** + and **09:42:20**, each after **2 h 41 m** on `custom:omni-101` / `smart-route`. Neither produced output; both + raised and alerted a `cron_incidents` row. The 11:30 Stage 2 slot then **reconstructed the missing handoff + from Stage 1's candidate files**, re-verified every candidate live in Camofox, and wrote **5 roles (board + 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at + 16:08:59 - see [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. +- **Fleet: 158 jobs (144 enabled / 14 disabled).** In-window ledger: **170 rows - 167 `completed` / 2 `failed` / + 1 `item_suppressed`**. **Non-clean `last_status` 2 -> 3**; **unclosed incident rows 1 -> 3** (all + `state = alerted`, incl. the 11-03 `timeout` row `dca848_dfd4bea84f2a`). Deliveries on 11-04: **8 rows, 8 + `delivered`** - the failure notices were delivered, not silent. +- **The cover-letter route is only alive by override.** `regenerate_cover_letter.py` line 82 reads `CL_MODEL`, + defaulting to **`deepseek-paid-smart`**, which returned **HTTP 402** during the 11-04 run; that process was + pinned to **`gemini/gemini-3.5-flash`**. **No `CL_MODEL` exists in `~/.hermes/.env`**, so the override did not + persist. Endpoint check 2026-11-05: `omniroute:20128` -> **401** and `100.78.63.20:20128` -> **401** (live, + key-gated), and the same model still serves other jobs - so re-test before calling the route dead. +- **CT299 froze a second time.** A manual `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and + wedged; the guest was frozen until **~00:30 on 11-05** (~16 h). Live probe **2026-11-05 00:54:21 AWST**: + `LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump processes; + SFTPGo reachable from CT460 at `100.88.223.70:8080`. Abandoned temp dir + `/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299` still on disk. -> + [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]] +- **wiki-stocktake-maintenance overran for the third day running** - `1a8e90ea2271` ran **03:00:31 -> 08:42:17 + (5 h 41 m 46 s)**, hit the turn ceiling, reported incomplete, delivery `suppressed`. Pin: + `custom:omni-101` / **`free-stack`**. + ## Recent Changes (2026-10-30, nightly audit; live probes 2026-10-30 00:0x AWST) diff --git a/incidents/2026-11-04-upstream-stream-failure-0700-cohort.md b/incidents/2026-11-04-upstream-stream-failure-0700-cohort.md new file mode 100644 index 0000000..aac767a --- /dev/null +++ b/incidents/2026-11-04-upstream-stream-failure-0700-cohort.md @@ -0,0 +1,120 @@ +--- +title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)" +type: incident +status: open +created: 2026-11-05 +updated: 2026-11-05 +verified_on: 2026-11-05 +confidence: high +tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident] +sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json] +--- + +# Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04) + +**Window:** 2026-11-04 07:00:45 to 09:42:20 AWST (2 h 41 m 35 s) - **Status:** both incidents still open +(`state = alerted`); no data lost - the Job Radar's day was recovered by its Stage 2 safety-net slot. + +## Summary + +Two independent 07:00 jobs died on the **same upstream fault, 36 seconds apart** on 2026-11-04: + +- `dca8482e4f76` **Daily Marketing Job Radar - Discovery** (`0 7 * * 1-5`) +- `85edf6772bca` **Tech & AI Newsletter Digest (Analytical)** (`0 7 * * *`) + +Both ran **2 h 41 m** and ended with the identical error, `RuntimeError: Upstream stream failed before +completion.` Both are pinned to `custom:omni-101` / `smart-route`. Neither produced output. Two +`cron_incidents` rows were raised and alerted, and both notices were delivered. + +Unlike the 2026-11-02 and 2026-11-03 incidents - where an aborted Stage 1 cost the day's candidate set - +**the Job Radar's day survived**: the 11:30 Stage 2 slot reconstructed the missing handoff, re-verified every +candidate live, and still wrote 5 roles and 1 cover letter. See *The recovery* below. + +## Evidence + +Live `executions.db`, `cron_incidents`, `deliveries.db` and `jobs.json`, read 2026-11-05 00:5x AWST: + +| Job | Start | End | Duration | Status | Incident row | +|---|---|---|---|---|---| +| `dca8482e4f76` Daily Marketing Job Radar - Discovery | 07:00:45 | 09:42:20 | 2 h 41 m 35 s | `error` | `dca848_3b1bb1e9b003` (`unknown`, open) | +| `85edf6772bca` Tech & AI Newsletter Digest (Analytical) | 07:00:45 | 09:41:44 | 2 h 40 m 59 s | `error` | `85edf6_c52a11a62044` (`unknown`, open) | + +Both error strings: `RuntimeError: Upstream stream failed before completion.` + +All three open incident rows carry `state = alerted` with `closed_at` NULL - the 2026-11-03 `timeout` row +`dca848_dfd4bea84f2a` (first seen `2026-11-03T13:28:51`) plus these two (`09:41:43` and `09:42:19`). +**Unclosed incident rows 1 -> 3.** + +Deliveries on 11-04: **8 rows, 8 `delivered`** - including both failure notices (09:41:43, 09:42:20). The +alerts were not silent. + +## Why this is not simply the 2026-11-03 incident again + +| | 2026-11-03 | 2026-11-04 | +|---|---|---| +| Error | `RuntimeError: stream active timeout` | `RuntimeError: Upstream stream failed before completion.` | +| Duration | 6 h 28 m 18 s | 2 h 41 m (both jobs) | +| Jobs hit | 1 (`dca8482e4f76`) | 2, in the same minute | +| Row | `dca848_dfd4bea84f2a` | `85edf6_c52a11a62044`, `dca848_3b1bb1e9b003` | + +A timeout is a stall; "stream failed before completion" is an upstream drop. Both land in the stream layer +of the `smart-route` combo, on `custom:omni-101`. Two unrelated prompts - job discovery and a newsletter +digest - failing within 36 seconds of each other points at the route/provider, not at either job. + +## The recovery + +Stage 2 (`5fc292ff0f3b`) ran in its 11:30 safety-net slot and finished **15:58-16:09 AWST: Result PASS, +0 missing cover letters**: + +- Stage 1 wrote **no handoff**. Stage 2 **reconstructed it from Stage 1's own candidate files** and + re-verified every surviving candidate **live in Camofox** before writing anything. +- **5 roles written to Notion** (board **479 -> 484**): Colab Resources (Group Marketing Manager, strong + match), Foundation for the WA Museum, Mindful Meditation Australia, HPH Solutions, RecruitShop. +- Discovery totals carried forward: **34 discovered / 10 duplicates avoided / 20 rejected**. +- **1 strong match -> 1 canonical cover letter** (Colab Resources) rendered through the canonical teal + template: `.txt`/`.json`/`.html`/`.docx`/`.pdf` all present and non-empty, no `{{}}` tokens, HTTP 200 on + `:9099/jobs/drafts/...`. `SAME_DAY_STRONG_COUNT 1` / `VERIFIED_COVER_LETTER_COUNT 1` / + `MISSING_COVER_LETTER_COUNT 0`. +- Handoff marked `consumed: true` at **16:08:59** (atomic write), so the 14:30 slot stayed silent; delivery + at 16:12:22. +- The run also caught **its own** drafting error: the first Colab letter misstated three resume facts (an + invented "40% sales increase", Horizon Power programmes attributed to Pacific Energy, a stray en-dash). + It was corrected against `current_resume_from_website.txt` and re-rendered - a live demonstration of the + runbook's "verify against the canonical resume" rule. + +**So the 11-04 failure cost a Stage 1 run, not the day** - the first time the split pipeline's safety-net +slot has had to carry the day alone. + +## The cover-letter route is only alive by override + +`regenerate_cover_letter.py` line 82 reads `os.environ.get("CL_MODEL", "deepseek-paid-smart")`. During the +11-04 run that default returned **HTTP 402 (billing)**, so generation was pinned to +**`gemini/gemini-3.5-flash`** for that process. `grep` finds **no `CL_MODEL` in `~/.hermes/.env`** - the +override never persisted, so the next run inherits the default again. The same model name still serves other +cron jobs (`cbfe434a3b5e`, this nightly audit, `last_status: ok`), so treat a 402 as **route/time-specific +until re-tested** rather than as proof the route is dead. See [[runbooks/job-radar]]. + +## Other in-window cron faults + +- `1a8e90ea2271` **wiki-stocktake-maintenance** ran **03:00:31 -> 08:42:17 (5 h 41 m 46 s)**, hit the turn + ceiling, reported incomplete, delivery `suppressed` - the **third consecutive overrun** of this job. Its + pin is `custom:omni-101` / **`free-stack`**. It overlapped the 07:00 window Stage 1 was running in. +- `066bb5191280` **Uber Eats Monthly Spend Report** carries `last_status: delivery_failed` (streak 0) - part + of the non-clean count, not a new failure. + +## Open questions / next actions + +1. Both 11-04 failures share a signature and a route with the 11-03 timeout. **No output file exists for + either** - the fault lands upstream of the agent's first turn. +2. `failure_type` is stored as **`unknown`** for a stream fault, so neither new row groups with the 11-03 + `timeout` row. The incident table undercounts what looks like one route-level pattern (same miscoding + seen on the 2026-10-27 row, stored as `auth` for a 400). +3. All three rows are `alerted` and unclosed. Decide whether the two 11-04 rows are two incidents or one + route incident - and whether the 11-03 row should be closed. +4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]]. + +## Related + +- [[incidents/2026-11-03-job-radar-stream-active-timeout]] +- [[systems/scheduled-tasks]] +- [[runbooks/job-radar]] diff --git a/incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md b/incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md new file mode 100644 index 0000000..60c7b91 --- /dev/null +++ b/incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md @@ -0,0 +1,121 @@ +--- +title: vzdump Freeze Took CT299 / SFTPGo Offline for ~24h +type: incident +status: resolved +created: 2026-11-04 +updated: 2026-11-05 +verified_on: 2026-11-05 +confidence: high +tags: [incident, proxmox, vzdump, backup, nfs, ct299, sftpgo, tailscale, manual-backup, ssd0] +sources: [live-pve-probe, live-tailscale-probe, live-ssh, live-pve-tasks, live-probe-2026-11-05] +--- + +# vzdump Froze CT299 / SFTPGo Twice - 11-03 (NFS target) and 11-04 (manual ssd0 run) + +> **Updated 2026-11-05 (nightly audit) - CT299 was frozen twice, not once.** The page below documents the +> 2026-11-03 freeze. A **second, separate freeze** ran from **2026-11-04 08:30:04** to **2026-11-05 ~00:30** +> (~16 h), caused by a **manual** `vzdump 299 ... --storage ssd0` run. Live probe at **00:54:21 AWST**: +> `LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active` - **service up**. +> See *Second freeze* below. + +**Window:** ~2026-11-03 03:00 AWST → 2026-11-04 04:4x AWST (≈25 h) +**Status:** Resolved — wedged `vzdump` killed, guest unfrozen, SFTPGo verified healthy +**Impact:** `sftpgo` (LXC CT299 on `pve`) was **frozen solid** — the container's filesystem was suspended by the +backup job, so `sftpgo`, `tailscaled` and everything else inside it stopped. The tailnet peer went offline and the +service was unreachable for the better part of a day. **Nothing alerted.** + +## Symptoms + +- Tailscale peer `sftpgo` (`100.88.223.70`) offline; `tailscale ping` → timeout. +- `ssh` into `100.88.223.70` hung until timeout; `pct exec 299 -- ...` also hung (exit 124). +- The guest was not crashed — it was **frozen**: the backup job had taken an LVM snapshot and suspended the guest + filesystem, then stalled before it could release it. + +## Root cause + +A scheduled Proxmox backup job (`vzdump: backup-tue-g1-299`, `schedule tue 03:00`, `mode snapshot`, +`storage fred-qnap`) started on **Tue 2026-11-03 at 03:00 AWST** and **never completed**. The task wedged at +`create storage snapshot` on the **`fred-qnap` NFS storage** (`192.168.178.31:/Proxmox` mounted at `/mnt/pve/fred-qnap`): + +- The NFS write path was unresponsive — a `dd` write probe to `/mnt/pve/fred-qnap` timed out while the task was stuck. +- The `vzdump` child process was in **`D` state** (uninterruptible I/O) on a FUSE `request_wait_answer` — it could not + be killed by a normal signal while blocked in the kernel. +- Its temp directory `/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/` was left behind, and the guest's filesystem + remained suspended for the whole ~24 h. + +**The failure mode is the important part: a backup task that cannot finish also cannot release the guest.** Proxmox's +snapshot mode suspends the guest filesystem while the snapshot is created and the data is read out. A stalled task +therefore does not merely "fail" — it **takes the guest down with it**, silently, until someone notices. + +## Remediation (2026-11-04 04:4x AWST) + +1. Identified the stuck task on `pve`: `vzdump` parent PID **274590**, child PID **274760** (state `D`). +2. `kill -TERM` on both PIDs — the parent died, the child remained blocked in `D` state on the FUSE wait. +3. `kill -KILL` on the remaining child — the kernel eventually released it once the NFS request timed out. +4. **The guest unfroze immediately.** `pct exec 299 -- /bin/echo ALIVE` → `ALIVE` (previously exit 124 / timeout). +5. Verified end to end: + - `pct exec 299 -- systemctl is-active sftpgo` → **`active`** + - `pct exec 299 -- ss -ltn` → `LISTEN *:8080` (WebAdmin) and `:2022` (SFTP) + - `tailscale status` → peer `sftpgo` **Online: true** again + - `tailscale ping 100.88.223.70` → **`pong from sftpgo via 192.168.178.92:41641 in 0s`** (direct, not DERP) + - `curl http://100.88.223.70:8080/` → **HTTP 200** + +## Post-remediation storage check + +`fred-qnap` was healthy again by the time the task was cleared: + +- `pvesm status` → `fred-qnap nfs active 14,380,169,216 KiB total / 12,084,987,904 used / 84.04 %` +- `dd` write probe of 5 MiB with `oflag=direct` → **28.6 MB/s**, exit 0 — no stall. +- The other backup task running concurrently (`vzdump: backup-sun-g1-100`, started 2026-11-03 10:36 AWST) **finished + successfully** at 2026-11-04 04:56:59 (`TASK OK`, 00:16:25) against the same storage. + +So the NFS stall was **transient, not a dead mount** — it hit the one task that happened to hold a guest suspended. + +## Second freeze - 2026-11-04 08:30:04 to 2026-11-05 ~00:30 (~16 h) + +The 11-03 freeze documented above was **not the only one**. A live probe of pve (`100.96.100.82`) on +**2026-11-05 00:54:21 AWST**, plus the node's task history, shows a **second, separate wedge** the next morning: + +- `pvesh get /nodes/pve/tasks` lists **two CT299 `vzdump` tasks on 2026-11-04** - and **neither is the 03:00 + fleet job**: + - **`2026-11-04 07:36:44` - `vzdump` / status `job errors`** - a clean failure after ~11 minutes. Its log + survives on the dump target: `/mnt/ssd0/dump/vzdump-lxc-299-2026_11_04-07_36_44.log` (597 B, mtime Nov 4 07:47). + - **`2026-11-04 08:30:04` - `vzdump` / status `unexpected status`** - the wedge. `unexpected status` is what a + killed task leaves behind; the 11-03 `03:00:04` row reads the same. +- Three `push_file` tasks from client **`100.118.5.51` (CT460)** ran at **07:17:27 / 07:27:18 / 07:33:07**, + immediately before the first attempt - so the sequence reads as operator/agent-initiated, not scheduled. +- The abandoned temp dir **`/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299`** is still on disk (the dump dir holds + 8.8 G total; the only other artefact is `openweb-245-preupgrade-20261101-211545.tar`). +- **Both attempts targeted the SSD that is bind-mounted into the guest** (`--storage ssd0`, `/mnt/ssd0/dump`) - + the same hazard class as the 11-03 NFS target: the guest is snapshotted onto storage it is actively using. + +**Recovery state, verified live 2026-11-05 00:54:21 AWST:** `LOCK: 0` in `/etc/pve/lxc/299.conf`; +`/sys/fs/cgroup/lxc/299/cgroup.freeze = 0` **and** `ns/cgroup.freeze = 0`; `pct exec 299 -- /bin/echo ALIVE` +returns `ALIVE`; `systemctl is-active sftpgo` -> `active`; **0** `vzdump`/`lxc-freeze` processes; SFTPGo +reachable end-to-end from CT460 at `http://100.88.223.70:8080`. **CT299 is not frozen now.** + +**Open:** why a 299 dump was attempted twice on 11-04 outside any schedule, and whether the `ssd0` target +should be used for this guest at all. See [[infrastructure/proxmox]] and the open items below. The abandoned +`vzdumptmp3750030_299` directory should be reclaimed once nobody needs it. + +## Prevention / open items + +- **Nothing monitored the backup tasks.** A `vzdump` that hangs for 24 h on a guest-holding storage is an outage + generator, and no page, cron job or dashboard covers it. A watchdog that alerts on a `vzdump` task older than + ~2 h — or that checks for `vzdump` tasks holding a suspended guest — is the missing control. +- **NFS `hard` mount semantics amplify this.** `fred-qnap` is mounted `hard,fatal_neterrors=none`, so a stalled NFS + server parks I/O in uninterruptible sleep (`D` state) indefinitely rather than failing fast. A `soft`/`intr` mount + or a `timeout=` tuned for the backup path would let the task die instead of freezing the guest. +- **The temp dir must be reclaimed manually.** `/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/` was left on disk by the + killed task; the existing cleanup cron skips a directory whose task is still registered. +- **Do not assume "guest offline" means "guest crashed."** This guest was healthy and simply frozen. The first two + probes (ping, ssh) both time out in exactly the same way as a dead guest — the distinguishing signal is a + `vzdump` task in `D` state with an open snapshot on that guest. + +## Related +- [[infrastructure/proxmox]] — host, guest and storage inventory +- [[infrastructure/backups]] — Hermes-side config backups (a separate mechanism from PVE `vzdump`) +- [[incidents/2026-09-02-sftpgo-outage]] — the previous CT299 outage (swap exhaustion + cgroup pinning) — a + **different** root cause with the same symptom +- [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] — prior NFS-layer failure in this estate +- [[systems/scheduled-tasks]] — cron fleet (no backup-task watchdog exists today) diff --git a/index.md b/index.md index 8454911..b2fbb4a 100644 --- a/index.md +++ b/index.md @@ -3,16 +3,18 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-04 -verified_on: 2026-11-04 +updated: 2026-11-05 +verified_on: 2026-11-05 confidence: high -tags: [index, infrastructure, systems] +tags: [index, infrastructure, systems, stream-failure, ct299] --- # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-11-03 (nightly audit — the 11-02 fleet settled **177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`**, the **first fully clean ledger day in five**, and **both 11-01 `unknown` rows resolved in-window**: the Sunday 18:00 **Weekly Review reconciled at 01:27:22** after **7 h 26 m 43 s** and **delivered**, so *"the lost review" was never lost* — incident `79f9bb_9c443b602eec` auto-closed **01:27:21.606691** and its page is now **`resolved`** — while `ecc1fc13655a`'s 11-01 ghost `error` cleared on its 12:00:33 slot; **unclosed incident rows 1 → 0** (the first zero in the period), non-clean `last_status` **2 → 1**, deliveries **9 rows / 9 delivered**; ⚠ **but two ~9-hour runs failed invisibly.** The **Job Radar Stage 1** (`dca8482e4f76`, 07:00:33 → 16:01:52, **9 h 01 m 19 s**) hit the **agent iteration cap before writing its handoff**, so **both Stage 2 slots exited `[SILENT]` and nothing reached Notion** — and it **raised no `cron_incidents` row at all**, where the same class raised incidents on 10-27 and 10-28, because its terminal state flipped to `completed` with `error: NULL`; the **03:00 stocktake** (`1a8e90ea2271`) likewise ran **9 h 43 m 29 s**, **delivered no report** and left its output uncommitted in the working tree (committed by this audit).) +> model routing and automation. Last updated: 2026-11-05 (nightly audit - the 11-04 fleet settled 170 rows: 167 `completed` / 2 `failed` / 1 `item_suppressed`; non-clean `last_status` 2 -> 3; unclosed incident rows 1 -> 3; deliveries 8/8 `delivered`. Both 07:00 failures were one upstream stream fault - and the Job Radar's Stage 2 slot reconstructed the missing handoff and saved the day. CT299 froze a second time.) > +> **2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze:** the **11-04 fleet settled 170 rows - 167 `completed` / 2 `failed` / 1 `item_suppressed`**, **non-clean `last_status` 2 -> 3**, **unclosed incident rows 1 -> 3**, **deliveries 8 rows / 8 `delivered`**. Both failures are the *same* upstream fault: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) died **36 seconds apart at 09:41-09:42**, each after **2 h 41 m**, on `RuntimeError: Upstream stream failed before completion.` - both pinned `custom:omni-101` / `smart-route`. **The day was not lost:** the 11:30 Stage 2 slot **reconstructed the missing handoff from Stage 1's own candidate files**, re-verified every candidate live, and still wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. The cover-letter route survived only on an override: `regenerate_cover_letter.py` defaults to `deepseek-paid-smart`, which threw **HTTP 402** mid-run, so the letter was pinned to `gemini/gemini-3.5-flash` via `CL_MODEL` - and that override **was never written to `.env`**, so it did not persist. And **CT299 was frozen twice, not once**: a second, **manual** `vzdump 299 ... --storage ssd0` wedged from **11-04 08:30:04** until **11-05 ~00:30** (~16 h); the live probe at 00:54:21 AWST shows `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. + > **2026-11-04 (nightly audit) — the ledger raised its own red row:** the **11-03 fleet settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) — **unclosed incident rows 0 → 1**, non-clean `last_status` **1 → 2**, deliveries **7 rows / 7 `delivered`**. The single failure is the **Job Radar Stage 1** (`dca8482e4f76`), which ran **07:00:33 → 13:28:52 (6 h 28 m 18 s)** and died on **`RuntimeError: stream active timeout`** — a **new terminal signature**, not the iteration cap — after going **silent for 65 m 21 s**; its incident `dca848_dfd4bea84f2a` (`timeout`) is **`alerted` and open** and the notice **was delivered** (13:28:51.741724) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. That answers the 11-02 page's open question: **output is suppressed when the run ends `completed`, delivered when it ends `failed`** — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so **2026-11-02's and 2026-11-03's candidate sets are both unrecovered**. ⚠ **The 03:00 stocktake overran for a second day** (`1a8e90ea2271`, 03:00:30 → 10:41:13 = **7 h 40 m 43 s**, `suppressed`, nothing delivered, artefacts uncommitted again). **New page:** [[systems/hermes-hq-push]] (v0.9.27, enabled; manifest says it **replaces `dispatch-push`**, and no trace of the old plugin survives). > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered @@ -53,6 +55,7 @@ tags: [index, infrastructure, systems] - [[systems/skills-index]] — Installed skills inventory (**302 skills** (69 categories) as of the **2026-10-30** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at the 2026-10-30 run: **0 stale / 109 fresh / 3 no-date of 112 pages** (`freshness-20261030.txt`). Note: `SKILL.md` files on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so **302 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents +- [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter). - [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] — **OPEN — the app's "local" voice mode never applies, and the connection test lies.** The test checks **socket reachability only**, while every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects `session.update` as an unknown event**. `voice.mode` is a **phone-side** selector — **no `mode` key exists** in `config.yaml → voice` or `conduit-settings.json` — so a "mode did not apply" cannot be diagnosed on host config. Found via session `20261031_193709_40b9c2` (19:37:09, 30 msgs / 17 tool calls); `voice.local_live.front_model` was repointed to `mistral/voxtral-small-latest` during it and **nothing was changed on the Mac**. → [[systems/conduit]] - [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] — **A gateway reload waited its full 1,800 s drain cap, then force-drained and orphaned a live cron run.** At **18:06:41** the gateway took the reload; the drain reported `timed_out=True` at the cap and force-drained at **18:37:38** (new gateway **18:38:43**). `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:01:06 slot, **6 h 38 m 09 s** in) closed **`failed`** at **18:39:15** on *"lost its durable fire claim ownership"*. Same class as [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **second instance, and the first on a plain reload**. A **second, clean** restart followed at **19:23:39 → 19:24:40**; 2026-10-31 closed **160 / 4**. → [[systems/scheduled-tasks]], [[systems/hermes-agent]] - [[incidents/2026-10-31-asana-access-failures]] — **Both Asana paths on CT460 are non-functional.** The MCP server `asana` (`https://mcp.asana.com/v2/mcp`, `auth: oauth`, `enabled: true`) has **never authenticated** — `~/.hermes/mcp-tokens/` holds only `asana.client.json` (**319 B**, mtime 2026-10-26 01:07) and the logs carry *"MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run `hermes mcp login asana` interactively first"*; the hosted connector also rejects the call shape. The fix needs an **interactive** `hermes mcp login asana`, which a cron context cannot perform. → [[systems/hermes-agent]] diff --git a/infrastructure/tailscale.md b/infrastructure/tailscale.md index b8e8655..cc089cb 100644 --- a/infrastructure/tailscale.md +++ b/infrastructure/tailscale.md @@ -3,8 +3,8 @@ title: Tailscale type: infrastructure status: active created: 2026-07-26 -updated: 2026-11-01 -verified_on: 2026-11-01 +updated: 2026-11-05 +verified_on: 2026-11-04 confidence: high tags: [tailscale, vpn, networking, infrastructure] --- @@ -20,7 +20,8 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access - **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`) ## Tailscale Node Inventory (verified 2026-10-31) -Full node list from `tailscale status` — **50 non-funnel nodes** (49 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age. +Full node list from `tailscale status` — **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age. +Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route. Re-verified 2026-10-31 05:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **one state transition and seven offline-age drifts, no IP changes and no missing peers.** **`mattermost` came back online**: the peer record reads `Online: true` for the first time since the outage, and the guest behind it is genuinely healthy — live `pct exec 255 -- systemctl is-active mattermost` → **`active`**, `ss -ltnp` shows `mattermost` listening on `*:8065`, `pct exec 255 -- tailscale ip -4` → `100.98.116.14`, and the gateway's own `gateway.log` records **`✓ mattermost connected` at 2026-10-30 16:36:17** (after an earlier successful reconnect at 07:19:37 the same day). CT255 is therefore **running** on `pve` — the 40-day outage is over. The **7 offline ages** were refreshed (cloudreve 97d → **98d**; conduit 53d → **55d**; elementsynapse 100d → **102d**; freshrss-new 73d → **74d**; omarchybook 8d → **9d**; openmausbot 8d → **9d**; ubuntu-server-xfce 70d → **71d**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Two container annotations were added for peers that had none: `openmausbot` (**CT1264, m93p — stopped**) and `openmausbot-vm` (**VM106, m93p**), both confirmed from live `pct list` / `qm list` on `m93p`. The `m93p` row's "peer record stale" note was updated — its peer record read `Online: false` on 10-29 and 10-30 and reads **`Online: true`** today, the third consecutive day the record disagreed with the (reachable) host. CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route. Re-verified 2026-11-01 03:0x–04:0x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition and 5 offline-age drifts, no IP changes and no missing peers.** **`omarchymac` went offline**: the peer record reads `Online: false` with `LastSeen 2026-10-31T13:20:00Z` and `tailscale ping 100.126.173.115` times out, so this is a **genuine** transition (the JSON `OS` field reads `linux`, not a Mac). **`m93p`'s peer record reads `Online: false` again** — the **4th consecutive** stale record — while `tailscale ping 100.104.51.98` → `pong from m93p via DERP(syd) in 98ms` and SSH over the tailnet returned `up 1 day, 12:02`, so the row stays **online** and the peer record remains non-authoritative for hypervisor reachability. The **5 offline ages** were refreshed (cloudreve 98d → **99d**; freshrss-new 74d → **75d**; omarchybook 9d → **10d**; openmausbot 9d → **10d**; ubuntu-server-xfce 71d → **72d**). **`conduit` (55d) and `elementsynapse` (102d) were already correct** and are unchanged. **Every peer IP in the table still matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Container identity re-confirmed from inside the guests: `pct exec 201 -- tailscale ip -4` → `100.96.244.39` (CT201, runtipi), `pct exec 1001 -- docker ps` → `sshwifty` (CT1001 on `m93p`), CT209 `cloudreve` and CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT255 `mattermost` **running** with `systemctl is-active mattermost` → `active` and `LISTEN *:8065`. Both OmniRoute nodes re-probed: `100.88.81.19` (CT600) and `100.78.63.20` (CT101) both return **401** on `/v1/models` (alive, auth required). Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong). @@ -42,34 +43,36 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | anthonys-mac-mini | 100.76.60.37 | online | | clawtest | 100.122.189.81 | online — CT333 (pve) | | cliproxyapi | 100.83.112.94 | online — CT103 (pve) | -| cloudreve | 100.111.187.7 | offline 99d — CT209 stopped | -| conduit | 100.81.79.87 | offline 55d | +| cloudreve | 100.111.187.7 | offline 102d — CT209 stopped | +| conduit | 100.81.79.87 | offline 58d | | docker | 100.96.150.69 | online — CT200 (pve) | | docker-01 | 100.121.12.31 | online — CT1001 (m93p) | -| elementsynapse | 100.75.250.37 | offline 102d — CT555 stopped | +| elementsynapse | 100.75.250.37 | offline 105d — CT555 stopped | | freellmapi | 100.127.208.75 | online — CT1006 (m93p) | | freshrss | 100.83.60.93 | online — CT1260 (m93p) | -| freshrss-new | 100.110.180.115 | offline 75d | +| freshrss-new | 100.110.180.115 | offline 78d | | gitea | 100.82.100.128 | online — CT221 (m93p) | | guest-vps | 100.117.14.84 | online | | hermesagent | 100.98.205.112 | online — CT450 (pve) | -| homeassistant | 100.68.139.5 | online | +| homeassistant | 100.68.139.5 | online — VM100 (pve) | | litellm | 100.79.29.64 | online — CT500 (m93p) | -| m93p | 100.104.51.98 | online — hypervisor (peer record reads `Online: false` again on 2026-11-01, 4th consecutive stale record; `tailscale ping` → pong) | +| m93p | 100.104.51.98 | online — hypervisor (peer record reads `Online: true` again on 2026-11-04 — the stale-record pattern of 10-29 → 11-01 has cleared; `tailscale ping` → pong) | | mailflow | 100.115.60.92 | online — CT1262 (m93p) | | mattermost | 100.98.116.14 | online — CT255 (pve) | +| mem0 | 100.73.94.53 | online — CT107 (m93p) | | n8n | 100.84.124.30 | online | | nextcloud | 100.90.102.19 | online — CT270 (pve) | | ntfy | 100.67.153.116 | online — CT1241 (m93p) | -| octoagent | 100.117.10.68 | online | -| octop | 100.92.163.122 | online | +| obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) | +| octoagent | 100.117.10.68 | offline 2d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | +| octop | 100.92.163.122 | offline <1d — last seen 2026-11-03 09:03Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | | ollama | 100.108.44.119 | online — CT205 (pve) | | omarchy | 100.73.227.103 | online | -| omarchybook | 100.107.9.100 | offline 10d | -| omarchymac | 100.126.173.115 | offline <1d — last seen 2026-10-31 13:20Z (state transition 2026-11-01; `tailscale ping` times out) | +| omarchybook | 100.107.9.100 | offline 13d | +| omarchymac | 100.126.173.115 | offline 2d — last seen 2026-11-01 11:13Z | | omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute | | omniroute-1 | 100.78.63.20 | online — CT101 (m93p) | -| openmausbot | 100.91.214.91 | offline 10d — CT1264 (m93p) stopped | +| openmausbot | 100.91.214.91 | offline 13d — CT1264 (m93p) stopped | | openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) | | openweb | 100.69.230.13 | online — CT245 (m93p) | | paseo | 100.72.188.93 | online — CT1261 (m93p) | @@ -78,11 +81,11 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | rclone | 100.107.116.17 | online — CT104 (m93p) | | runtipi | 100.96.244.39 | online — CT201 (pve) | | rustdesk | 100.95.227.57 | online — CT1008 (m93p) | -| sftpgo | 100.88.223.70 | online — CT299 (pve) | +| sftpgo | 100.88.223.70 | online — CT299 (pve) — **recovered 2026-11-04 (freeze 1) and again 2026-11-05 (freeze 2, manual `ssd0` run)**: a wedged `vzdump` had frozen the guest for ~24 h; killing the task unfroze it (see [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]) | | social-dashboard | 100.70.183.37 | online — CT1263 (m93p) | | springroll | 100.100.153.125 | online | | springroll-browser | 100.109.221.91 | online | -| ubuntu-server-xfce | 100.101.26.104 | offline 72d | +| ubuntu-server-xfce | 100.101.26.104 | offline 75d | | vaultwarden | 100.123.225.90 | online — CT102 (m93p) | | webserver | 100.82.40.23 | online — CT207 (pve) | @@ -94,15 +97,15 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | Path | Target | Notes | |---|---|---| | `/` | `http://127.0.0.1:9119` | Hermes WebUI | -| `/ui` | `http://127.0.0.1:5175/` | Hermes UI (Vite) | +| `/ui` | `http://127.0.0.1:5175` | Hermes UI (Vite) | | `/jobs` | `http://127.0.0.1:9099` | Jobs Dashboard | | `/wiki` | `http://100.118.5.51:9120` | Agent Estate Wiki (Quartz) | | `/ignis` | `http://127.0.0.1:8080` | Ignis (browser-native Obsidian) | -| `/webui` | `http://127.0.0.1:8787/` | Hermes WebUI (Standalone) | +| `/webui` | `http://127.0.0.1:8787` | Hermes WebUI (Standalone) | | `/gitea` | `http://100.82.100.128:3000` | Gitea | | `/desktop` | `http://127.0.0.1:6081` | CUA noVNC | | `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC | -| `/synergy-prep` | `http://127.0.0.1:9123/` | Synergy prep | +| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep | | `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) | | `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid | diff --git a/log.md b/log.md index cc3fbb0..8522ad6 100644 --- a/log.md +++ b/log.md @@ -1960,3 +1960,40 @@ audit committed them. **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no decision reversed. **No secrets, passwords, tokens or keys written or quoted** — a Gitea remote URL with an embedded credential was seen during verification and is not reproduced here ([REDACTED]). + +## [2026-11-05] audit - window 2026-11-04 + +- **New page:** `incidents/2026-11-04-upstream-stream-failure-0700-cohort.md` - two 07:00 jobs + (`dca8482e4f76` **Daily Marketing Job Radar - Discovery**, `85edf6772bca` **Tech & AI Newsletter Digest + (Analytical)**) died **36 s apart** (09:41:44 / 09:42:20) on `RuntimeError: Upstream stream failed before + completion.` after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`; both incidents still open + (`state = alerted`). Records the recovery: Stage 2 reconstructed the missing handoff from Stage 1's own + candidate files, re-verified every candidate live, and wrote **5 roles (board 479 -> 484) + 1 canonical + cover letter** (`MISSING_COVER_LETTER_COUNT 0`), consuming the handoff at 16:08:59. +- **Updated `incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md`** - CT299 was frozen **twice**, not + once. Second freeze: a **manual** `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and wedged + (`unexpected status`) until **11-05 ~00:30**. Also records the earlier clean failure at **07:36:44** + (log `vzdump-lxc-299-2026_11_04-07_36_44.log` on the dump target) and three `push_file` tasks from CT460 + at 07:17-07:33, plus the abandoned `/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299`. Live probe 2026-11-05 + 00:54:21 AWST: `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump procs. +- **Updated [[infrastructure/tailscale]]** - the sftpgo row now records recovery from *two* freezes. +- **Updated [[systems/scheduled-tasks]]** - new ledger bullet: 11-04 settled **170 rows - 167 `completed` / + 2 `failed` / 1 `item_suppressed`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3**; + deliveries **8/8 `delivered`**; the two failures are one upstream signature; stocktake overran for the third + day (5 h 41 m 46 s, `free-stack` pin). +- **Updated [[runbooks/job-radar]]** - cover-letter model route: `regenerate_cover_letter.py` defaults to + `CL_MODEL=deepseek-paid-smart` (line 82); the 11-04 run pinned `gemini/gemini-3.5-flash` after **HTTP 402**. + `grep` shows **no `CL_MODEL` in `~/.hermes/.env`** - the override was process-local and did not persist, so + the next run inherits the default again. Endpoint probe 2026-11-05: `omniroute:20128/v1/models` -> **401** + and `100.78.63.20:20128/v1/models` -> **401** (live, key-gated) - so the route is reachable; the same model + serves other cron jobs successfully. Treat a 402 as route/time-specific until re-tested. +- **Updated [[current-state]]** - new "Recent Changes (2026-11-05)" block; the page had been sitting at the + 2026-10-30 block across five audits. +- **Updated [[index]]** - new dated ledger paragraph, incidents list gains the new page, "Last updated" moved + 2026-11-03 -> 2026-11-05. +- **Live verification this audit:** pve (100.96.100.82) SSH probe at 2026-11-05 00:54:21 AWST (lock, cgroup + freeze flags, guest ALIVE, sftpgo active, proc count, task list, dump dir); SFTPGo reached end-to-end from + CT460 on `100.88.223.70:8080`; `jobs.json`, `executions.db`, `cron_incidents`, `deliveries.db` read live. +- **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or re-pinned; no + decision reversed. **No secrets, passwords, tokens or keys written or quoted.** + diff --git a/runbooks/job-radar.md b/runbooks/job-radar.md index ff759e9..d15e439 100644 --- a/runbooks/job-radar.md +++ b/runbooks/job-radar.md @@ -3,13 +3,13 @@ title: Daily Marketing Job Radar Operations and Recovery type: runbook status: active created: 2026-07-27 -updated: 2026-10-30 +updated: 2026-11-05 review_after: 2026-11-13 -verified_on: 2026-10-30 +verified_on: 2026-11-05 version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2) confidence: high -tags: [automation, cron, jobs, notion, dashboard, runbook] -sources: [live-cron-output, live-notion-api, live-dashboard-api, hermes-docs] +tags: [automation, cron, jobs, notion, dashboard, runbook, cover-letter, model-route] +sources: [live-cron-output, live-notion-api, live-dashboard-api, hermes-docs, live-script-inspection, live-endpoint-probe] --- # Daily Marketing Job Radar Operations and Recovery @@ -39,6 +39,17 @@ Cron jobs inherit the profile-level `fallback_providers` chain; Hermes does not ## Cover letters are non-negotiable +> **Model route caveat - verified 2026-11-05.** `regenerate_cover_letter.py` (line 82) reads +> `os.environ.get("CL_MODEL", "deepseek-paid-smart")`. On the 2026-11-04 run that default returned **HTTP 402 +> (billing)**, and the run pinned **`gemini/gemini-3.5-flash`** via `CL_MODEL` to get the strong-match letter out. +> `grep` finds **no `CL_MODEL` in `~/.hermes/.env`** - the override existed only inside that process, so **the +> next run inherits the default again**. If a run logs 402/503 while generating letters, re-pin `CL_MODEL` for the +> process and confirm the letters rendered - do **not** ship a run without its cover letters. Note the same model +> name still serves other cron jobs successfully (e.g. the 23:59 nightly audit), and the endpoint itself is live: +> `omniroute:20128/v1/models` -> **401** and `100.78.63.20:20128/v1/models` -> **401** on 2026-11-05 (key-gated, +> not down). Treat a 402 as **route/time-specific until re-tested**, not as proof the route is dead. + + A successful radar run includes the cover-letter stage. - Audit **every Strong Match added today**, including pages created by an interrupted earlier run. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index fbb6968..3930367 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,17 +3,18 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-11-04 -verified_on: 2026-11-04 -review_after: 2026-11-11 +updated: 2026-11-05 +verified_on: 2026-11-05 +review_after: 2026-11-06 confidence: high tags: [cron, scheduled, automation] -sources: [live-cron-list, live-jobs-json, cron-executions-db] +sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db] --- # Scheduled Tasks ## Count +- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **158 jobs (144 enabled / 14 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `item_suppressed`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. - **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and