wiki: nightly audit 2026-11-04 window — new incident (Job Radar + Newsletter Digest both killed by upstream stream failure; Stage 2 rebuilt handoff and delivered 5 roles + 1 letter); CT299 second freeze documented; CL_MODEL 402 caveat in job-radar runbook; scheduled-tasks/current-state/index/log updated
This commit is contained in:
@@ -0,0 +1,120 @@
|
||||
---
|
||||
title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)"
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-11-05
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
confidence: high
|
||||
tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident]
|
||||
sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json]
|
||||
---
|
||||
|
||||
# Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)
|
||||
|
||||
**Window:** 2026-11-04 07:00:45 to 09:42:20 AWST (2 h 41 m 35 s) - **Status:** both incidents still open
|
||||
(`state = alerted`); no data lost - the Job Radar's day was recovered by its Stage 2 safety-net slot.
|
||||
|
||||
## Summary
|
||||
|
||||
Two independent 07:00 jobs died on the **same upstream fault, 36 seconds apart** on 2026-11-04:
|
||||
|
||||
- `dca8482e4f76` **Daily Marketing Job Radar - Discovery** (`0 7 * * 1-5`)
|
||||
- `85edf6772bca` **Tech & AI Newsletter Digest (Analytical)** (`0 7 * * *`)
|
||||
|
||||
Both ran **2 h 41 m** and ended with the identical error, `RuntimeError: Upstream stream failed before
|
||||
completion.` Both are pinned to `custom:omni-101` / `smart-route`. Neither produced output. Two
|
||||
`cron_incidents` rows were raised and alerted, and both notices were delivered.
|
||||
|
||||
Unlike the 2026-11-02 and 2026-11-03 incidents - where an aborted Stage 1 cost the day's candidate set -
|
||||
**the Job Radar's day survived**: the 11:30 Stage 2 slot reconstructed the missing handoff, re-verified every
|
||||
candidate live, and still wrote 5 roles and 1 cover letter. See *The recovery* below.
|
||||
|
||||
## Evidence
|
||||
|
||||
Live `executions.db`, `cron_incidents`, `deliveries.db` and `jobs.json`, read 2026-11-05 00:5x AWST:
|
||||
|
||||
| Job | Start | End | Duration | Status | Incident row |
|
||||
|---|---|---|---|---|---|
|
||||
| `dca8482e4f76` Daily Marketing Job Radar - Discovery | 07:00:45 | 09:42:20 | 2 h 41 m 35 s | `error` | `dca848_3b1bb1e9b003` (`unknown`, open) |
|
||||
| `85edf6772bca` Tech & AI Newsletter Digest (Analytical) | 07:00:45 | 09:41:44 | 2 h 40 m 59 s | `error` | `85edf6_c52a11a62044` (`unknown`, open) |
|
||||
|
||||
Both error strings: `RuntimeError: Upstream stream failed before completion.`
|
||||
|
||||
All three open incident rows carry `state = alerted` with `closed_at` NULL - the 2026-11-03 `timeout` row
|
||||
`dca848_dfd4bea84f2a` (first seen `2026-11-03T13:28:51`) plus these two (`09:41:43` and `09:42:19`).
|
||||
**Unclosed incident rows 1 -> 3.**
|
||||
|
||||
Deliveries on 11-04: **8 rows, 8 `delivered`** - including both failure notices (09:41:43, 09:42:20). The
|
||||
alerts were not silent.
|
||||
|
||||
## Why this is not simply the 2026-11-03 incident again
|
||||
|
||||
| | 2026-11-03 | 2026-11-04 |
|
||||
|---|---|---|
|
||||
| Error | `RuntimeError: stream active timeout` | `RuntimeError: Upstream stream failed before completion.` |
|
||||
| Duration | 6 h 28 m 18 s | 2 h 41 m (both jobs) |
|
||||
| Jobs hit | 1 (`dca8482e4f76`) | 2, in the same minute |
|
||||
| Row | `dca848_dfd4bea84f2a` | `85edf6_c52a11a62044`, `dca848_3b1bb1e9b003` |
|
||||
|
||||
A timeout is a stall; "stream failed before completion" is an upstream drop. Both land in the stream layer
|
||||
of the `smart-route` combo, on `custom:omni-101`. Two unrelated prompts - job discovery and a newsletter
|
||||
digest - failing within 36 seconds of each other points at the route/provider, not at either job.
|
||||
|
||||
## The recovery
|
||||
|
||||
Stage 2 (`5fc292ff0f3b`) ran in its 11:30 safety-net slot and finished **15:58-16:09 AWST: Result PASS,
|
||||
0 missing cover letters**:
|
||||
|
||||
- Stage 1 wrote **no handoff**. Stage 2 **reconstructed it from Stage 1's own candidate files** and
|
||||
re-verified every surviving candidate **live in Camofox** before writing anything.
|
||||
- **5 roles written to Notion** (board **479 -> 484**): Colab Resources (Group Marketing Manager, strong
|
||||
match), Foundation for the WA Museum, Mindful Meditation Australia, HPH Solutions, RecruitShop.
|
||||
- Discovery totals carried forward: **34 discovered / 10 duplicates avoided / 20 rejected**.
|
||||
- **1 strong match -> 1 canonical cover letter** (Colab Resources) rendered through the canonical teal
|
||||
template: `.txt`/`.json`/`.html`/`.docx`/`.pdf` all present and non-empty, no `{{}}` tokens, HTTP 200 on
|
||||
`:9099/jobs/drafts/...`. `SAME_DAY_STRONG_COUNT 1` / `VERIFIED_COVER_LETTER_COUNT 1` /
|
||||
`MISSING_COVER_LETTER_COUNT 0`.
|
||||
- Handoff marked `consumed: true` at **16:08:59** (atomic write), so the 14:30 slot stayed silent; delivery
|
||||
at 16:12:22.
|
||||
- The run also caught **its own** drafting error: the first Colab letter misstated three resume facts (an
|
||||
invented "40% sales increase", Horizon Power programmes attributed to Pacific Energy, a stray en-dash).
|
||||
It was corrected against `current_resume_from_website.txt` and re-rendered - a live demonstration of the
|
||||
runbook's "verify against the canonical resume" rule.
|
||||
|
||||
**So the 11-04 failure cost a Stage 1 run, not the day** - the first time the split pipeline's safety-net
|
||||
slot has had to carry the day alone.
|
||||
|
||||
## The cover-letter route is only alive by override
|
||||
|
||||
`regenerate_cover_letter.py` line 82 reads `os.environ.get("CL_MODEL", "deepseek-paid-smart")`. During the
|
||||
11-04 run that default returned **HTTP 402 (billing)**, so generation was pinned to
|
||||
**`gemini/gemini-3.5-flash`** for that process. `grep` finds **no `CL_MODEL` in `~/.hermes/.env`** - the
|
||||
override never persisted, so the next run inherits the default again. The same model name still serves other
|
||||
cron jobs (`cbfe434a3b5e`, this nightly audit, `last_status: ok`), so treat a 402 as **route/time-specific
|
||||
until re-tested** rather than as proof the route is dead. See [[runbooks/job-radar]].
|
||||
|
||||
## Other in-window cron faults
|
||||
|
||||
- `1a8e90ea2271` **wiki-stocktake-maintenance** ran **03:00:31 -> 08:42:17 (5 h 41 m 46 s)**, hit the turn
|
||||
ceiling, reported incomplete, delivery `suppressed` - the **third consecutive overrun** of this job. Its
|
||||
pin is `custom:omni-101` / **`free-stack`**. It overlapped the 07:00 window Stage 1 was running in.
|
||||
- `066bb5191280` **Uber Eats Monthly Spend Report** carries `last_status: delivery_failed` (streak 0) - part
|
||||
of the non-clean count, not a new failure.
|
||||
|
||||
## Open questions / next actions
|
||||
|
||||
1. Both 11-04 failures share a signature and a route with the 11-03 timeout. **No output file exists for
|
||||
either** - the fault lands upstream of the agent's first turn.
|
||||
2. `failure_type` is stored as **`unknown`** for a stream fault, so neither new row groups with the 11-03
|
||||
`timeout` row. The incident table undercounts what looks like one route-level pattern (same miscoding
|
||||
seen on the 2026-10-27 row, stored as `auth` for a 400).
|
||||
3. All three rows are `alerted` and unclosed. Decide whether the two 11-04 rows are two incidents or one
|
||||
route incident - and whether the 11-03 row should be closed.
|
||||
4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[runbooks/job-radar]]
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
title: vzdump Freeze Took CT299 / SFTPGo Offline for ~24h
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-11-04
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
confidence: high
|
||||
tags: [incident, proxmox, vzdump, backup, nfs, ct299, sftpgo, tailscale, manual-backup, ssd0]
|
||||
sources: [live-pve-probe, live-tailscale-probe, live-ssh, live-pve-tasks, live-probe-2026-11-05]
|
||||
---
|
||||
|
||||
# vzdump Froze CT299 / SFTPGo Twice - 11-03 (NFS target) and 11-04 (manual ssd0 run)
|
||||
|
||||
> **Updated 2026-11-05 (nightly audit) - CT299 was frozen twice, not once.** The page below documents the
|
||||
> 2026-11-03 freeze. A **second, separate freeze** ran from **2026-11-04 08:30:04** to **2026-11-05 ~00:30**
|
||||
> (~16 h), caused by a **manual** `vzdump 299 ... --storage ssd0` run. Live probe at **00:54:21 AWST**:
|
||||
> `LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active` - **service up**.
|
||||
> See *Second freeze* below.
|
||||
|
||||
**Window:** ~2026-11-03 03:00 AWST → 2026-11-04 04:4x AWST (≈25 h)
|
||||
**Status:** Resolved — wedged `vzdump` killed, guest unfrozen, SFTPGo verified healthy
|
||||
**Impact:** `sftpgo` (LXC CT299 on `pve`) was **frozen solid** — the container's filesystem was suspended by the
|
||||
backup job, so `sftpgo`, `tailscaled` and everything else inside it stopped. The tailnet peer went offline and the
|
||||
service was unreachable for the better part of a day. **Nothing alerted.**
|
||||
|
||||
## Symptoms
|
||||
|
||||
- Tailscale peer `sftpgo` (`100.88.223.70`) offline; `tailscale ping` → timeout.
|
||||
- `ssh` into `100.88.223.70` hung until timeout; `pct exec 299 -- ...` also hung (exit 124).
|
||||
- The guest was not crashed — it was **frozen**: the backup job had taken an LVM snapshot and suspended the guest
|
||||
filesystem, then stalled before it could release it.
|
||||
|
||||
## Root cause
|
||||
|
||||
A scheduled Proxmox backup job (`vzdump: backup-tue-g1-299`, `schedule tue 03:00`, `mode snapshot`,
|
||||
`storage fred-qnap`) started on **Tue 2026-11-03 at 03:00 AWST** and **never completed**. The task wedged at
|
||||
`create storage snapshot` on the **`fred-qnap` NFS storage** (`192.168.178.31:/Proxmox` mounted at `/mnt/pve/fred-qnap`):
|
||||
|
||||
- The NFS write path was unresponsive — a `dd` write probe to `/mnt/pve/fred-qnap` timed out while the task was stuck.
|
||||
- The `vzdump` child process was in **`D` state** (uninterruptible I/O) on a FUSE `request_wait_answer` — it could not
|
||||
be killed by a normal signal while blocked in the kernel.
|
||||
- Its temp directory `/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/` was left behind, and the guest's filesystem
|
||||
remained suspended for the whole ~24 h.
|
||||
|
||||
**The failure mode is the important part: a backup task that cannot finish also cannot release the guest.** Proxmox's
|
||||
snapshot mode suspends the guest filesystem while the snapshot is created and the data is read out. A stalled task
|
||||
therefore does not merely "fail" — it **takes the guest down with it**, silently, until someone notices.
|
||||
|
||||
## Remediation (2026-11-04 04:4x AWST)
|
||||
|
||||
1. Identified the stuck task on `pve`: `vzdump` parent PID **274590**, child PID **274760** (state `D`).
|
||||
2. `kill -TERM` on both PIDs — the parent died, the child remained blocked in `D` state on the FUSE wait.
|
||||
3. `kill -KILL` on the remaining child — the kernel eventually released it once the NFS request timed out.
|
||||
4. **The guest unfroze immediately.** `pct exec 299 -- /bin/echo ALIVE` → `ALIVE` (previously exit 124 / timeout).
|
||||
5. Verified end to end:
|
||||
- `pct exec 299 -- systemctl is-active sftpgo` → **`active`**
|
||||
- `pct exec 299 -- ss -ltn` → `LISTEN *:8080` (WebAdmin) and `:2022` (SFTP)
|
||||
- `tailscale status` → peer `sftpgo` **Online: true** again
|
||||
- `tailscale ping 100.88.223.70` → **`pong from sftpgo via 192.168.178.92:41641 in 0s`** (direct, not DERP)
|
||||
- `curl http://100.88.223.70:8080/` → **HTTP 200**
|
||||
|
||||
## Post-remediation storage check
|
||||
|
||||
`fred-qnap` was healthy again by the time the task was cleared:
|
||||
|
||||
- `pvesm status` → `fred-qnap nfs active 14,380,169,216 KiB total / 12,084,987,904 used / 84.04 %`
|
||||
- `dd` write probe of 5 MiB with `oflag=direct` → **28.6 MB/s**, exit 0 — no stall.
|
||||
- The other backup task running concurrently (`vzdump: backup-sun-g1-100`, started 2026-11-03 10:36 AWST) **finished
|
||||
successfully** at 2026-11-04 04:56:59 (`TASK OK`, 00:16:25) against the same storage.
|
||||
|
||||
So the NFS stall was **transient, not a dead mount** — it hit the one task that happened to hold a guest suspended.
|
||||
|
||||
## Second freeze - 2026-11-04 08:30:04 to 2026-11-05 ~00:30 (~16 h)
|
||||
|
||||
The 11-03 freeze documented above was **not the only one**. A live probe of pve (`100.96.100.82`) on
|
||||
**2026-11-05 00:54:21 AWST**, plus the node's task history, shows a **second, separate wedge** the next morning:
|
||||
|
||||
- `pvesh get /nodes/pve/tasks` lists **two CT299 `vzdump` tasks on 2026-11-04** - and **neither is the 03:00
|
||||
fleet job**:
|
||||
- **`2026-11-04 07:36:44` - `vzdump` / status `job errors`** - a clean failure after ~11 minutes. Its log
|
||||
survives on the dump target: `/mnt/ssd0/dump/vzdump-lxc-299-2026_11_04-07_36_44.log` (597 B, mtime Nov 4 07:47).
|
||||
- **`2026-11-04 08:30:04` - `vzdump` / status `unexpected status`** - the wedge. `unexpected status` is what a
|
||||
killed task leaves behind; the 11-03 `03:00:04` row reads the same.
|
||||
- Three `push_file` tasks from client **`100.118.5.51` (CT460)** ran at **07:17:27 / 07:27:18 / 07:33:07**,
|
||||
immediately before the first attempt - so the sequence reads as operator/agent-initiated, not scheduled.
|
||||
- The abandoned temp dir **`/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299`** is still on disk (the dump dir holds
|
||||
8.8 G total; the only other artefact is `openweb-245-preupgrade-20261101-211545.tar`).
|
||||
- **Both attempts targeted the SSD that is bind-mounted into the guest** (`--storage ssd0`, `/mnt/ssd0/dump`) -
|
||||
the same hazard class as the 11-03 NFS target: the guest is snapshotted onto storage it is actively using.
|
||||
|
||||
**Recovery state, verified live 2026-11-05 00:54:21 AWST:** `LOCK: 0` in `/etc/pve/lxc/299.conf`;
|
||||
`/sys/fs/cgroup/lxc/299/cgroup.freeze = 0` **and** `ns/cgroup.freeze = 0`; `pct exec 299 -- /bin/echo ALIVE`
|
||||
returns `ALIVE`; `systemctl is-active sftpgo` -> `active`; **0** `vzdump`/`lxc-freeze` processes; SFTPGo
|
||||
reachable end-to-end from CT460 at `http://100.88.223.70:8080`. **CT299 is not frozen now.**
|
||||
|
||||
**Open:** why a 299 dump was attempted twice on 11-04 outside any schedule, and whether the `ssd0` target
|
||||
should be used for this guest at all. See [[infrastructure/proxmox]] and the open items below. The abandoned
|
||||
`vzdumptmp3750030_299` directory should be reclaimed once nobody needs it.
|
||||
|
||||
## Prevention / open items
|
||||
|
||||
- **Nothing monitored the backup tasks.** A `vzdump` that hangs for 24 h on a guest-holding storage is an outage
|
||||
generator, and no page, cron job or dashboard covers it. A watchdog that alerts on a `vzdump` task older than
|
||||
~2 h — or that checks for `vzdump` tasks holding a suspended guest — is the missing control.
|
||||
- **NFS `hard` mount semantics amplify this.** `fred-qnap` is mounted `hard,fatal_neterrors=none`, so a stalled NFS
|
||||
server parks I/O in uninterruptible sleep (`D` state) indefinitely rather than failing fast. A `soft`/`intr` mount
|
||||
or a `timeout=` tuned for the backup path would let the task die instead of freezing the guest.
|
||||
- **The temp dir must be reclaimed manually.** `/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/` was left on disk by the
|
||||
killed task; the existing cleanup cron skips a directory whose task is still registered.
|
||||
- **Do not assume "guest offline" means "guest crashed."** This guest was healthy and simply frozen. The first two
|
||||
probes (ping, ssh) both time out in exactly the same way as a dead guest — the distinguishing signal is a
|
||||
`vzdump` task in `D` state with an open snapshot on that guest.
|
||||
|
||||
## Related
|
||||
- [[infrastructure/proxmox]] — host, guest and storage inventory
|
||||
- [[infrastructure/backups]] — Hermes-side config backups (a separate mechanism from PVE `vzdump`)
|
||||
- [[incidents/2026-09-02-sftpgo-outage]] — the previous CT299 outage (swap exhaustion + cgroup pinning) — a
|
||||
**different** root cause with the same symptom
|
||||
- [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] — prior NFS-layer failure in this estate
|
||||
- [[systems/scheduled-tasks]] — cron fleet (no backup-task watchdog exists today)
|
||||
Reference in New Issue
Block a user