wiki: nightly audit 2026-11-04 window — new incident (Job Radar + Newsletter Digest both killed by upstream stream failure; Stage 2 rebuilt handoff and delivered 5 roles + 1 letter); CT299 second freeze documented; CL_MODEL 402 caveat in job-radar runbook; scheduled-tasks/current-state/index/log updated
This commit is contained in:
+34
-4
@@ -3,16 +3,46 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-31
|
||||
verified_on: 2026-10-31
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [system]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-30
|
||||
Updated: 2026-11-05
|
||||
|
||||
## Recent Changes (2026-11-05, nightly audit; live probes 2026-11-05 00:0x-00:5x AWST)
|
||||
|
||||
- **A dead route took out two 07:00 jobs at once - and the safety-net slot saved the day.** `dca8482e4f76`
|
||||
(**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**)
|
||||
both died on `RuntimeError: Upstream stream failed before completion.` - **36 seconds apart**, at **09:41:44**
|
||||
and **09:42:20**, each after **2 h 41 m** on `custom:omni-101` / `smart-route`. Neither produced output; both
|
||||
raised and alerted a `cron_incidents` row. The 11:30 Stage 2 slot then **reconstructed the missing handoff
|
||||
from Stage 1's candidate files**, re-verified every candidate live in Camofox, and wrote **5 roles (board
|
||||
479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at
|
||||
16:08:59 - see [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]].
|
||||
- **Fleet: 158 jobs (144 enabled / 14 disabled).** In-window ledger: **170 rows - 167 `completed` / 2 `failed` /
|
||||
1 `item_suppressed`**. **Non-clean `last_status` 2 -> 3**; **unclosed incident rows 1 -> 3** (all
|
||||
`state = alerted`, incl. the 11-03 `timeout` row `dca848_dfd4bea84f2a`). Deliveries on 11-04: **8 rows, 8
|
||||
`delivered`** - the failure notices were delivered, not silent.
|
||||
- **The cover-letter route is only alive by override.** `regenerate_cover_letter.py` line 82 reads `CL_MODEL`,
|
||||
defaulting to **`deepseek-paid-smart`**, which returned **HTTP 402** during the 11-04 run; that process was
|
||||
pinned to **`gemini/gemini-3.5-flash`**. **No `CL_MODEL` exists in `~/.hermes/.env`**, so the override did not
|
||||
persist. Endpoint check 2026-11-05: `omniroute:20128` -> **401** and `100.78.63.20:20128` -> **401** (live,
|
||||
key-gated), and the same model still serves other jobs - so re-test before calling the route dead.
|
||||
- **CT299 froze a second time.** A manual `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and
|
||||
wedged; the guest was frozen until **~00:30 on 11-05** (~16 h). Live probe **2026-11-05 00:54:21 AWST**:
|
||||
`LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump processes;
|
||||
SFTPGo reachable from CT460 at `100.88.223.70:8080`. Abandoned temp dir
|
||||
`/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299` still on disk. ->
|
||||
[[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]
|
||||
- **wiki-stocktake-maintenance overran for the third day running** - `1a8e90ea2271` ran **03:00:31 -> 08:42:17
|
||||
(5 h 41 m 46 s)**, hit the turn ceiling, reported incomplete, delivery `suppressed`. Pin:
|
||||
`custom:omni-101` / **`free-stack`**.
|
||||
|
||||
|
||||
## Recent Changes (2026-10-30, nightly audit; live probes 2026-10-30 00:0x AWST)
|
||||
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
---
|
||||
title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)"
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-11-05
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
confidence: high
|
||||
tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident]
|
||||
sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json]
|
||||
---
|
||||
|
||||
# Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)
|
||||
|
||||
**Window:** 2026-11-04 07:00:45 to 09:42:20 AWST (2 h 41 m 35 s) - **Status:** both incidents still open
|
||||
(`state = alerted`); no data lost - the Job Radar's day was recovered by its Stage 2 safety-net slot.
|
||||
|
||||
## Summary
|
||||
|
||||
Two independent 07:00 jobs died on the **same upstream fault, 36 seconds apart** on 2026-11-04:
|
||||
|
||||
- `dca8482e4f76` **Daily Marketing Job Radar - Discovery** (`0 7 * * 1-5`)
|
||||
- `85edf6772bca` **Tech & AI Newsletter Digest (Analytical)** (`0 7 * * *`)
|
||||
|
||||
Both ran **2 h 41 m** and ended with the identical error, `RuntimeError: Upstream stream failed before
|
||||
completion.` Both are pinned to `custom:omni-101` / `smart-route`. Neither produced output. Two
|
||||
`cron_incidents` rows were raised and alerted, and both notices were delivered.
|
||||
|
||||
Unlike the 2026-11-02 and 2026-11-03 incidents - where an aborted Stage 1 cost the day's candidate set -
|
||||
**the Job Radar's day survived**: the 11:30 Stage 2 slot reconstructed the missing handoff, re-verified every
|
||||
candidate live, and still wrote 5 roles and 1 cover letter. See *The recovery* below.
|
||||
|
||||
## Evidence
|
||||
|
||||
Live `executions.db`, `cron_incidents`, `deliveries.db` and `jobs.json`, read 2026-11-05 00:5x AWST:
|
||||
|
||||
| Job | Start | End | Duration | Status | Incident row |
|
||||
|---|---|---|---|---|---|
|
||||
| `dca8482e4f76` Daily Marketing Job Radar - Discovery | 07:00:45 | 09:42:20 | 2 h 41 m 35 s | `error` | `dca848_3b1bb1e9b003` (`unknown`, open) |
|
||||
| `85edf6772bca` Tech & AI Newsletter Digest (Analytical) | 07:00:45 | 09:41:44 | 2 h 40 m 59 s | `error` | `85edf6_c52a11a62044` (`unknown`, open) |
|
||||
|
||||
Both error strings: `RuntimeError: Upstream stream failed before completion.`
|
||||
|
||||
All three open incident rows carry `state = alerted` with `closed_at` NULL - the 2026-11-03 `timeout` row
|
||||
`dca848_dfd4bea84f2a` (first seen `2026-11-03T13:28:51`) plus these two (`09:41:43` and `09:42:19`).
|
||||
**Unclosed incident rows 1 -> 3.**
|
||||
|
||||
Deliveries on 11-04: **8 rows, 8 `delivered`** - including both failure notices (09:41:43, 09:42:20). The
|
||||
alerts were not silent.
|
||||
|
||||
## Why this is not simply the 2026-11-03 incident again
|
||||
|
||||
| | 2026-11-03 | 2026-11-04 |
|
||||
|---|---|---|
|
||||
| Error | `RuntimeError: stream active timeout` | `RuntimeError: Upstream stream failed before completion.` |
|
||||
| Duration | 6 h 28 m 18 s | 2 h 41 m (both jobs) |
|
||||
| Jobs hit | 1 (`dca8482e4f76`) | 2, in the same minute |
|
||||
| Row | `dca848_dfd4bea84f2a` | `85edf6_c52a11a62044`, `dca848_3b1bb1e9b003` |
|
||||
|
||||
A timeout is a stall; "stream failed before completion" is an upstream drop. Both land in the stream layer
|
||||
of the `smart-route` combo, on `custom:omni-101`. Two unrelated prompts - job discovery and a newsletter
|
||||
digest - failing within 36 seconds of each other points at the route/provider, not at either job.
|
||||
|
||||
## The recovery
|
||||
|
||||
Stage 2 (`5fc292ff0f3b`) ran in its 11:30 safety-net slot and finished **15:58-16:09 AWST: Result PASS,
|
||||
0 missing cover letters**:
|
||||
|
||||
- Stage 1 wrote **no handoff**. Stage 2 **reconstructed it from Stage 1's own candidate files** and
|
||||
re-verified every surviving candidate **live in Camofox** before writing anything.
|
||||
- **5 roles written to Notion** (board **479 -> 484**): Colab Resources (Group Marketing Manager, strong
|
||||
match), Foundation for the WA Museum, Mindful Meditation Australia, HPH Solutions, RecruitShop.
|
||||
- Discovery totals carried forward: **34 discovered / 10 duplicates avoided / 20 rejected**.
|
||||
- **1 strong match -> 1 canonical cover letter** (Colab Resources) rendered through the canonical teal
|
||||
template: `.txt`/`.json`/`.html`/`.docx`/`.pdf` all present and non-empty, no `{{}}` tokens, HTTP 200 on
|
||||
`:9099/jobs/drafts/...`. `SAME_DAY_STRONG_COUNT 1` / `VERIFIED_COVER_LETTER_COUNT 1` /
|
||||
`MISSING_COVER_LETTER_COUNT 0`.
|
||||
- Handoff marked `consumed: true` at **16:08:59** (atomic write), so the 14:30 slot stayed silent; delivery
|
||||
at 16:12:22.
|
||||
- The run also caught **its own** drafting error: the first Colab letter misstated three resume facts (an
|
||||
invented "40% sales increase", Horizon Power programmes attributed to Pacific Energy, a stray en-dash).
|
||||
It was corrected against `current_resume_from_website.txt` and re-rendered - a live demonstration of the
|
||||
runbook's "verify against the canonical resume" rule.
|
||||
|
||||
**So the 11-04 failure cost a Stage 1 run, not the day** - the first time the split pipeline's safety-net
|
||||
slot has had to carry the day alone.
|
||||
|
||||
## The cover-letter route is only alive by override
|
||||
|
||||
`regenerate_cover_letter.py` line 82 reads `os.environ.get("CL_MODEL", "deepseek-paid-smart")`. During the
|
||||
11-04 run that default returned **HTTP 402 (billing)**, so generation was pinned to
|
||||
**`gemini/gemini-3.5-flash`** for that process. `grep` finds **no `CL_MODEL` in `~/.hermes/.env`** - the
|
||||
override never persisted, so the next run inherits the default again. The same model name still serves other
|
||||
cron jobs (`cbfe434a3b5e`, this nightly audit, `last_status: ok`), so treat a 402 as **route/time-specific
|
||||
until re-tested** rather than as proof the route is dead. See [[runbooks/job-radar]].
|
||||
|
||||
## Other in-window cron faults
|
||||
|
||||
- `1a8e90ea2271` **wiki-stocktake-maintenance** ran **03:00:31 -> 08:42:17 (5 h 41 m 46 s)**, hit the turn
|
||||
ceiling, reported incomplete, delivery `suppressed` - the **third consecutive overrun** of this job. Its
|
||||
pin is `custom:omni-101` / **`free-stack`**. It overlapped the 07:00 window Stage 1 was running in.
|
||||
- `066bb5191280` **Uber Eats Monthly Spend Report** carries `last_status: delivery_failed` (streak 0) - part
|
||||
of the non-clean count, not a new failure.
|
||||
|
||||
## Open questions / next actions
|
||||
|
||||
1. Both 11-04 failures share a signature and a route with the 11-03 timeout. **No output file exists for
|
||||
either** - the fault lands upstream of the agent's first turn.
|
||||
2. `failure_type` is stored as **`unknown`** for a stream fault, so neither new row groups with the 11-03
|
||||
`timeout` row. The incident table undercounts what looks like one route-level pattern (same miscoding
|
||||
seen on the 2026-10-27 row, stored as `auth` for a 400).
|
||||
3. All three rows are `alerted` and unclosed. Decide whether the two 11-04 rows are two incidents or one
|
||||
route incident - and whether the 11-03 row should be closed.
|
||||
4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[runbooks/job-radar]]
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
title: vzdump Freeze Took CT299 / SFTPGo Offline for ~24h
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-11-04
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
confidence: high
|
||||
tags: [incident, proxmox, vzdump, backup, nfs, ct299, sftpgo, tailscale, manual-backup, ssd0]
|
||||
sources: [live-pve-probe, live-tailscale-probe, live-ssh, live-pve-tasks, live-probe-2026-11-05]
|
||||
---
|
||||
|
||||
# vzdump Froze CT299 / SFTPGo Twice - 11-03 (NFS target) and 11-04 (manual ssd0 run)
|
||||
|
||||
> **Updated 2026-11-05 (nightly audit) - CT299 was frozen twice, not once.** The page below documents the
|
||||
> 2026-11-03 freeze. A **second, separate freeze** ran from **2026-11-04 08:30:04** to **2026-11-05 ~00:30**
|
||||
> (~16 h), caused by a **manual** `vzdump 299 ... --storage ssd0` run. Live probe at **00:54:21 AWST**:
|
||||
> `LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active` - **service up**.
|
||||
> See *Second freeze* below.
|
||||
|
||||
**Window:** ~2026-11-03 03:00 AWST → 2026-11-04 04:4x AWST (≈25 h)
|
||||
**Status:** Resolved — wedged `vzdump` killed, guest unfrozen, SFTPGo verified healthy
|
||||
**Impact:** `sftpgo` (LXC CT299 on `pve`) was **frozen solid** — the container's filesystem was suspended by the
|
||||
backup job, so `sftpgo`, `tailscaled` and everything else inside it stopped. The tailnet peer went offline and the
|
||||
service was unreachable for the better part of a day. **Nothing alerted.**
|
||||
|
||||
## Symptoms
|
||||
|
||||
- Tailscale peer `sftpgo` (`100.88.223.70`) offline; `tailscale ping` → timeout.
|
||||
- `ssh` into `100.88.223.70` hung until timeout; `pct exec 299 -- ...` also hung (exit 124).
|
||||
- The guest was not crashed — it was **frozen**: the backup job had taken an LVM snapshot and suspended the guest
|
||||
filesystem, then stalled before it could release it.
|
||||
|
||||
## Root cause
|
||||
|
||||
A scheduled Proxmox backup job (`vzdump: backup-tue-g1-299`, `schedule tue 03:00`, `mode snapshot`,
|
||||
`storage fred-qnap`) started on **Tue 2026-11-03 at 03:00 AWST** and **never completed**. The task wedged at
|
||||
`create storage snapshot` on the **`fred-qnap` NFS storage** (`192.168.178.31:/Proxmox` mounted at `/mnt/pve/fred-qnap`):
|
||||
|
||||
- The NFS write path was unresponsive — a `dd` write probe to `/mnt/pve/fred-qnap` timed out while the task was stuck.
|
||||
- The `vzdump` child process was in **`D` state** (uninterruptible I/O) on a FUSE `request_wait_answer` — it could not
|
||||
be killed by a normal signal while blocked in the kernel.
|
||||
- Its temp directory `/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/` was left behind, and the guest's filesystem
|
||||
remained suspended for the whole ~24 h.
|
||||
|
||||
**The failure mode is the important part: a backup task that cannot finish also cannot release the guest.** Proxmox's
|
||||
snapshot mode suspends the guest filesystem while the snapshot is created and the data is read out. A stalled task
|
||||
therefore does not merely "fail" — it **takes the guest down with it**, silently, until someone notices.
|
||||
|
||||
## Remediation (2026-11-04 04:4x AWST)
|
||||
|
||||
1. Identified the stuck task on `pve`: `vzdump` parent PID **274590**, child PID **274760** (state `D`).
|
||||
2. `kill -TERM` on both PIDs — the parent died, the child remained blocked in `D` state on the FUSE wait.
|
||||
3. `kill -KILL` on the remaining child — the kernel eventually released it once the NFS request timed out.
|
||||
4. **The guest unfroze immediately.** `pct exec 299 -- /bin/echo ALIVE` → `ALIVE` (previously exit 124 / timeout).
|
||||
5. Verified end to end:
|
||||
- `pct exec 299 -- systemctl is-active sftpgo` → **`active`**
|
||||
- `pct exec 299 -- ss -ltn` → `LISTEN *:8080` (WebAdmin) and `:2022` (SFTP)
|
||||
- `tailscale status` → peer `sftpgo` **Online: true** again
|
||||
- `tailscale ping 100.88.223.70` → **`pong from sftpgo via 192.168.178.92:41641 in 0s`** (direct, not DERP)
|
||||
- `curl http://100.88.223.70:8080/` → **HTTP 200**
|
||||
|
||||
## Post-remediation storage check
|
||||
|
||||
`fred-qnap` was healthy again by the time the task was cleared:
|
||||
|
||||
- `pvesm status` → `fred-qnap nfs active 14,380,169,216 KiB total / 12,084,987,904 used / 84.04 %`
|
||||
- `dd` write probe of 5 MiB with `oflag=direct` → **28.6 MB/s**, exit 0 — no stall.
|
||||
- The other backup task running concurrently (`vzdump: backup-sun-g1-100`, started 2026-11-03 10:36 AWST) **finished
|
||||
successfully** at 2026-11-04 04:56:59 (`TASK OK`, 00:16:25) against the same storage.
|
||||
|
||||
So the NFS stall was **transient, not a dead mount** — it hit the one task that happened to hold a guest suspended.
|
||||
|
||||
## Second freeze - 2026-11-04 08:30:04 to 2026-11-05 ~00:30 (~16 h)
|
||||
|
||||
The 11-03 freeze documented above was **not the only one**. A live probe of pve (`100.96.100.82`) on
|
||||
**2026-11-05 00:54:21 AWST**, plus the node's task history, shows a **second, separate wedge** the next morning:
|
||||
|
||||
- `pvesh get /nodes/pve/tasks` lists **two CT299 `vzdump` tasks on 2026-11-04** - and **neither is the 03:00
|
||||
fleet job**:
|
||||
- **`2026-11-04 07:36:44` - `vzdump` / status `job errors`** - a clean failure after ~11 minutes. Its log
|
||||
survives on the dump target: `/mnt/ssd0/dump/vzdump-lxc-299-2026_11_04-07_36_44.log` (597 B, mtime Nov 4 07:47).
|
||||
- **`2026-11-04 08:30:04` - `vzdump` / status `unexpected status`** - the wedge. `unexpected status` is what a
|
||||
killed task leaves behind; the 11-03 `03:00:04` row reads the same.
|
||||
- Three `push_file` tasks from client **`100.118.5.51` (CT460)** ran at **07:17:27 / 07:27:18 / 07:33:07**,
|
||||
immediately before the first attempt - so the sequence reads as operator/agent-initiated, not scheduled.
|
||||
- The abandoned temp dir **`/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299`** is still on disk (the dump dir holds
|
||||
8.8 G total; the only other artefact is `openweb-245-preupgrade-20261101-211545.tar`).
|
||||
- **Both attempts targeted the SSD that is bind-mounted into the guest** (`--storage ssd0`, `/mnt/ssd0/dump`) -
|
||||
the same hazard class as the 11-03 NFS target: the guest is snapshotted onto storage it is actively using.
|
||||
|
||||
**Recovery state, verified live 2026-11-05 00:54:21 AWST:** `LOCK: 0` in `/etc/pve/lxc/299.conf`;
|
||||
`/sys/fs/cgroup/lxc/299/cgroup.freeze = 0` **and** `ns/cgroup.freeze = 0`; `pct exec 299 -- /bin/echo ALIVE`
|
||||
returns `ALIVE`; `systemctl is-active sftpgo` -> `active`; **0** `vzdump`/`lxc-freeze` processes; SFTPGo
|
||||
reachable end-to-end from CT460 at `http://100.88.223.70:8080`. **CT299 is not frozen now.**
|
||||
|
||||
**Open:** why a 299 dump was attempted twice on 11-04 outside any schedule, and whether the `ssd0` target
|
||||
should be used for this guest at all. See [[infrastructure/proxmox]] and the open items below. The abandoned
|
||||
`vzdumptmp3750030_299` directory should be reclaimed once nobody needs it.
|
||||
|
||||
## Prevention / open items
|
||||
|
||||
- **Nothing monitored the backup tasks.** A `vzdump` that hangs for 24 h on a guest-holding storage is an outage
|
||||
generator, and no page, cron job or dashboard covers it. A watchdog that alerts on a `vzdump` task older than
|
||||
~2 h — or that checks for `vzdump` tasks holding a suspended guest — is the missing control.
|
||||
- **NFS `hard` mount semantics amplify this.** `fred-qnap` is mounted `hard,fatal_neterrors=none`, so a stalled NFS
|
||||
server parks I/O in uninterruptible sleep (`D` state) indefinitely rather than failing fast. A `soft`/`intr` mount
|
||||
or a `timeout=` tuned for the backup path would let the task die instead of freezing the guest.
|
||||
- **The temp dir must be reclaimed manually.** `/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/` was left on disk by the
|
||||
killed task; the existing cleanup cron skips a directory whose task is still registered.
|
||||
- **Do not assume "guest offline" means "guest crashed."** This guest was healthy and simply frozen. The first two
|
||||
probes (ping, ssh) both time out in exactly the same way as a dead guest — the distinguishing signal is a
|
||||
`vzdump` task in `D` state with an open snapshot on that guest.
|
||||
|
||||
## Related
|
||||
- [[infrastructure/proxmox]] — host, guest and storage inventory
|
||||
- [[infrastructure/backups]] — Hermes-side config backups (a separate mechanism from PVE `vzdump`)
|
||||
- [[incidents/2026-09-02-sftpgo-outage]] — the previous CT299 outage (swap exhaustion + cgroup pinning) — a
|
||||
**different** root cause with the same symptom
|
||||
- [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] — prior NFS-layer failure in this estate
|
||||
- [[systems/scheduled-tasks]] — cron fleet (no backup-task watchdog exists today)
|
||||
@@ -3,16 +3,18 @@ title: Anthony's Systems Wiki
|
||||
type: collection
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
confidence: high
|
||||
tags: [index, infrastructure, systems]
|
||||
tags: [index, infrastructure, systems, stream-failure, ct299]
|
||||
---
|
||||
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. Last updated: 2026-11-03 (nightly audit — the 11-02 fleet settled **177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`**, the **first fully clean ledger day in five**, and **both 11-01 `unknown` rows resolved in-window**: the Sunday 18:00 **Weekly Review reconciled at 01:27:22** after **7 h 26 m 43 s** and **delivered**, so *"the lost review" was never lost* — incident `79f9bb_9c443b602eec` auto-closed **01:27:21.606691** and its page is now **`resolved`** — while `ecc1fc13655a`'s 11-01 ghost `error` cleared on its 12:00:33 slot; **unclosed incident rows 1 → 0** (the first zero in the period), non-clean `last_status` **2 → 1**, deliveries **9 rows / 9 delivered**; ⚠ **but two ~9-hour runs failed invisibly.** The **Job Radar Stage 1** (`dca8482e4f76`, 07:00:33 → 16:01:52, **9 h 01 m 19 s**) hit the **agent iteration cap before writing its handoff**, so **both Stage 2 slots exited `[SILENT]` and nothing reached Notion** — and it **raised no `cron_incidents` row at all**, where the same class raised incidents on 10-27 and 10-28, because its terminal state flipped to `completed` with `error: NULL`; the **03:00 stocktake** (`1a8e90ea2271`) likewise ran **9 h 43 m 29 s**, **delivered no report** and left its output uncommitted in the working tree (committed by this audit).)
|
||||
> model routing and automation. Last updated: 2026-11-05 (nightly audit - the 11-04 fleet settled 170 rows: 167 `completed` / 2 `failed` / 1 `item_suppressed`; non-clean `last_status` 2 -> 3; unclosed incident rows 1 -> 3; deliveries 8/8 `delivered`. Both 07:00 failures were one upstream stream fault - and the Job Radar's Stage 2 slot reconstructed the missing handoff and saved the day. CT299 froze a second time.)
|
||||
>
|
||||
> **2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze:** the **11-04 fleet settled 170 rows - 167 `completed` / 2 `failed` / 1 `item_suppressed`**, **non-clean `last_status` 2 -> 3**, **unclosed incident rows 1 -> 3**, **deliveries 8 rows / 8 `delivered`**. Both failures are the *same* upstream fault: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) died **36 seconds apart at 09:41-09:42**, each after **2 h 41 m**, on `RuntimeError: Upstream stream failed before completion.` - both pinned `custom:omni-101` / `smart-route`. **The day was not lost:** the 11:30 Stage 2 slot **reconstructed the missing handoff from Stage 1's own candidate files**, re-verified every candidate live, and still wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. The cover-letter route survived only on an override: `regenerate_cover_letter.py` defaults to `deepseek-paid-smart`, which threw **HTTP 402** mid-run, so the letter was pinned to `gemini/gemini-3.5-flash` via `CL_MODEL` - and that override **was never written to `.env`**, so it did not persist. And **CT299 was frozen twice, not once**: a second, **manual** `vzdump 299 ... --storage ssd0` wedged from **11-04 08:30:04** until **11-05 ~00:30** (~16 h); the live probe at 00:54:21 AWST shows `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]].
|
||||
|
||||
> **2026-11-04 (nightly audit) — the ledger raised its own red row:** the **11-03 fleet settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) — **unclosed incident rows 0 → 1**, non-clean `last_status` **1 → 2**, deliveries **7 rows / 7 `delivered`**. The single failure is the **Job Radar Stage 1** (`dca8482e4f76`), which ran **07:00:33 → 13:28:52 (6 h 28 m 18 s)** and died on **`RuntimeError: stream active timeout`** — a **new terminal signature**, not the iteration cap — after going **silent for 65 m 21 s**; its incident `dca848_dfd4bea84f2a` (`timeout`) is **`alerted` and open** and the notice **was delivered** (13:28:51.741724) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. That answers the 11-02 page's open question: **output is suppressed when the run ends `completed`, delivered when it ends `failed`** — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so **2026-11-02's and 2026-11-03's candidate sets are both unrecovered**. ⚠ **The 03:00 stocktake overran for a second day** (`1a8e90ea2271`, 03:00:30 → 10:41:13 = **7 h 40 m 43 s**, `suppressed`, nothing delivered, artefacts uncommitted again). **New page:** [[systems/hermes-hq-push]] (v0.9.27, enabled; manifest says it **replaces `dispatch-push`**, and no trace of the old plugin survives).
|
||||
>
|
||||
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
||||
@@ -53,6 +55,7 @@ tags: [index, infrastructure, systems]
|
||||
- [[systems/skills-index]] — Installed skills inventory (**302 skills** (69 categories) as of the **2026-10-30** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at the 2026-10-30 run: **0 stale / 109 fresh / 3 no-date of 112 pages** (`freshness-20261030.txt`). Note: `SKILL.md` files on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so **302 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
|
||||
|
||||
## Incidents
|
||||
- [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter).
|
||||
- [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] — **OPEN — the app's "local" voice mode never applies, and the connection test lies.** The test checks **socket reachability only**, while every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects `session.update` as an unknown event**. `voice.mode` is a **phone-side** selector — **no `mode` key exists** in `config.yaml → voice` or `conduit-settings.json` — so a "mode did not apply" cannot be diagnosed on host config. Found via session `20261031_193709_40b9c2` (19:37:09, 30 msgs / 17 tool calls); `voice.local_live.front_model` was repointed to `mistral/voxtral-small-latest` during it and **nothing was changed on the Mac**. → [[systems/conduit]]
|
||||
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] — **A gateway reload waited its full 1,800 s drain cap, then force-drained and orphaned a live cron run.** At **18:06:41** the gateway took the reload; the drain reported `timed_out=True` at the cap and force-drained at **18:37:38** (new gateway **18:38:43**). `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:01:06 slot, **6 h 38 m 09 s** in) closed **`failed`** at **18:39:15** on *"lost its durable fire claim ownership"*. Same class as [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **second instance, and the first on a plain reload**. A **second, clean** restart followed at **19:23:39 → 19:24:40**; 2026-10-31 closed **160 / 4**. → [[systems/scheduled-tasks]], [[systems/hermes-agent]]
|
||||
- [[incidents/2026-10-31-asana-access-failures]] — **Both Asana paths on CT460 are non-functional.** The MCP server `asana` (`https://mcp.asana.com/v2/mcp`, `auth: oauth`, `enabled: true`) has **never authenticated** — `~/.hermes/mcp-tokens/` holds only `asana.client.json` (**319 B**, mtime 2026-10-26 01:07) and the logs carry *"MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run `hermes mcp login asana` interactively first"*; the hosted connector also rejects the call shape. The fix needs an **interactive** `hermes mcp login asana`, which a cron context cannot perform. → [[systems/hermes-agent]]
|
||||
|
||||
+22
-19
@@ -3,8 +3,8 @@ title: Tailscale
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-26
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-04
|
||||
confidence: high
|
||||
tags: [tailscale, vpn, networking, infrastructure]
|
||||
---
|
||||
@@ -20,7 +20,8 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
|
||||
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
|
||||
|
||||
## Tailscale Node Inventory (verified 2026-10-31)
|
||||
Full node list from `tailscale status` — **50 non-funnel nodes** (49 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
|
||||
Full node list from `tailscale status` — **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
|
||||
Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec <vmid> -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
|
||||
Re-verified 2026-10-31 05:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **one state transition and seven offline-age drifts, no IP changes and no missing peers.** **`mattermost` came back online**: the peer record reads `Online: true` for the first time since the outage, and the guest behind it is genuinely healthy — live `pct exec 255 -- systemctl is-active mattermost` → **`active`**, `ss -ltnp` shows `mattermost` listening on `*:8065`, `pct exec 255 -- tailscale ip -4` → `100.98.116.14`, and the gateway's own `gateway.log` records **`✓ mattermost connected` at 2026-10-30 16:36:17** (after an earlier successful reconnect at 07:19:37 the same day). CT255 is therefore **running** on `pve` — the 40-day outage is over. The **7 offline ages** were refreshed (cloudreve 97d → **98d**; conduit 53d → **55d**; elementsynapse 100d → **102d**; freshrss-new 73d → **74d**; omarchybook 8d → **9d**; openmausbot 8d → **9d**; ubuntu-server-xfce 70d → **71d**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Two container annotations were added for peers that had none: `openmausbot` (**CT1264, m93p — stopped**) and `openmausbot-vm` (**VM106, m93p**), both confirmed from live `pct list` / `qm list` on `m93p`. The `m93p` row's "peer record stale" note was updated — its peer record read `Online: false` on 10-29 and 10-30 and reads **`Online: true`** today, the third consecutive day the record disagreed with the (reachable) host. CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
|
||||
Re-verified 2026-11-01 03:0x–04:0x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition and 5 offline-age drifts, no IP changes and no missing peers.** **`omarchymac` went offline**: the peer record reads `Online: false` with `LastSeen 2026-10-31T13:20:00Z` and `tailscale ping 100.126.173.115` times out, so this is a **genuine** transition (the JSON `OS` field reads `linux`, not a Mac). **`m93p`'s peer record reads `Online: false` again** — the **4th consecutive** stale record — while `tailscale ping 100.104.51.98` → `pong from m93p via DERP(syd) in 98ms` and SSH over the tailnet returned `up 1 day, 12:02`, so the row stays **online** and the peer record remains non-authoritative for hypervisor reachability. The **5 offline ages** were refreshed (cloudreve 98d → **99d**; freshrss-new 74d → **75d**; omarchybook 9d → **10d**; openmausbot 9d → **10d**; ubuntu-server-xfce 71d → **72d**). **`conduit` (55d) and `elementsynapse` (102d) were already correct** and are unchanged. **Every peer IP in the table still matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Container identity re-confirmed from inside the guests: `pct exec 201 -- tailscale ip -4` → `100.96.244.39` (CT201, runtipi), `pct exec 1001 -- docker ps` → `sshwifty` (CT1001 on `m93p`), CT209 `cloudreve` and CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT255 `mattermost` **running** with `systemctl is-active mattermost` → `active` and `LISTEN *:8065`. Both OmniRoute nodes re-probed: `100.88.81.19` (CT600) and `100.78.63.20` (CT101) both return **401** on `/v1/models` (alive, auth required).
|
||||
Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
|
||||
@@ -42,34 +43,36 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
||||
| anthonys-mac-mini | 100.76.60.37 | online |
|
||||
| clawtest | 100.122.189.81 | online — CT333 (pve) |
|
||||
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
|
||||
| cloudreve | 100.111.187.7 | offline 99d — CT209 stopped |
|
||||
| conduit | 100.81.79.87 | offline 55d |
|
||||
| cloudreve | 100.111.187.7 | offline 102d — CT209 stopped |
|
||||
| conduit | 100.81.79.87 | offline 58d |
|
||||
| docker | 100.96.150.69 | online — CT200 (pve) |
|
||||
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
|
||||
| elementsynapse | 100.75.250.37 | offline 102d — CT555 stopped |
|
||||
| elementsynapse | 100.75.250.37 | offline 105d — CT555 stopped |
|
||||
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
|
||||
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
|
||||
| freshrss-new | 100.110.180.115 | offline 75d |
|
||||
| freshrss-new | 100.110.180.115 | offline 78d |
|
||||
| gitea | 100.82.100.128 | online — CT221 (m93p) |
|
||||
| guest-vps | 100.117.14.84 | online |
|
||||
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
|
||||
| homeassistant | 100.68.139.5 | online |
|
||||
| homeassistant | 100.68.139.5 | online — VM100 (pve) |
|
||||
| litellm | 100.79.29.64 | online — CT500 (m93p) |
|
||||
| m93p | 100.104.51.98 | online — hypervisor (peer record reads `Online: false` again on 2026-11-01, 4th consecutive stale record; `tailscale ping` → pong) |
|
||||
| m93p | 100.104.51.98 | online — hypervisor (peer record reads `Online: true` again on 2026-11-04 — the stale-record pattern of 10-29 → 11-01 has cleared; `tailscale ping` → pong) |
|
||||
| mailflow | 100.115.60.92 | online — CT1262 (m93p) |
|
||||
| mattermost | 100.98.116.14 | online — CT255 (pve) |
|
||||
| mem0 | 100.73.94.53 | online — CT107 (m93p) |
|
||||
| n8n | 100.84.124.30 | online |
|
||||
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
|
||||
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
|
||||
| octoagent | 100.117.10.68 | online |
|
||||
| octop | 100.92.163.122 | online |
|
||||
| obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) |
|
||||
| octoagent | 100.117.10.68 | offline 2d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
|
||||
| octop | 100.92.163.122 | offline <1d — last seen 2026-11-03 09:03Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
|
||||
| ollama | 100.108.44.119 | online — CT205 (pve) |
|
||||
| omarchy | 100.73.227.103 | online |
|
||||
| omarchybook | 100.107.9.100 | offline 10d |
|
||||
| omarchymac | 100.126.173.115 | offline <1d — last seen 2026-10-31 13:20Z (state transition 2026-11-01; `tailscale ping` times out) |
|
||||
| omarchybook | 100.107.9.100 | offline 13d |
|
||||
| omarchymac | 100.126.173.115 | offline 2d — last seen 2026-11-01 11:13Z |
|
||||
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
|
||||
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
|
||||
| openmausbot | 100.91.214.91 | offline 10d — CT1264 (m93p) stopped |
|
||||
| openmausbot | 100.91.214.91 | offline 13d — CT1264 (m93p) stopped |
|
||||
| openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) |
|
||||
| openweb | 100.69.230.13 | online — CT245 (m93p) |
|
||||
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
|
||||
@@ -78,11 +81,11 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
||||
| rclone | 100.107.116.17 | online — CT104 (m93p) |
|
||||
| runtipi | 100.96.244.39 | online — CT201 (pve) |
|
||||
| rustdesk | 100.95.227.57 | online — CT1008 (m93p) |
|
||||
| sftpgo | 100.88.223.70 | online — CT299 (pve) |
|
||||
| sftpgo | 100.88.223.70 | online — CT299 (pve) — **recovered 2026-11-04 (freeze 1) and again 2026-11-05 (freeze 2, manual `ssd0` run)**: a wedged `vzdump` had frozen the guest for ~24 h; killing the task unfroze it (see [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]) |
|
||||
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
|
||||
| springroll | 100.100.153.125 | online |
|
||||
| springroll-browser | 100.109.221.91 | online |
|
||||
| ubuntu-server-xfce | 100.101.26.104 | offline 72d |
|
||||
| ubuntu-server-xfce | 100.101.26.104 | offline 75d |
|
||||
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
|
||||
| webserver | 100.82.40.23 | online — CT207 (pve) |
|
||||
|
||||
@@ -94,15 +97,15 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
||||
| Path | Target | Notes |
|
||||
|---|---|---|
|
||||
| `/` | `http://127.0.0.1:9119` | Hermes WebUI |
|
||||
| `/ui` | `http://127.0.0.1:5175/` | Hermes UI (Vite) |
|
||||
| `/ui` | `http://127.0.0.1:5175` | Hermes UI (Vite) |
|
||||
| `/jobs` | `http://127.0.0.1:9099` | Jobs Dashboard |
|
||||
| `/wiki` | `http://100.118.5.51:9120` | Agent Estate Wiki (Quartz) |
|
||||
| `/ignis` | `http://127.0.0.1:8080` | Ignis (browser-native Obsidian) |
|
||||
| `/webui` | `http://127.0.0.1:8787/` | Hermes WebUI (Standalone) |
|
||||
| `/webui` | `http://127.0.0.1:8787` | Hermes WebUI (Standalone) |
|
||||
| `/gitea` | `http://100.82.100.128:3000` | Gitea |
|
||||
| `/desktop` | `http://127.0.0.1:6081` | CUA noVNC |
|
||||
| `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC |
|
||||
| `/synergy-prep` | `http://127.0.0.1:9123/` | Synergy prep |
|
||||
| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep |
|
||||
| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) |
|
||||
| `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid |
|
||||
|
||||
|
||||
@@ -1960,3 +1960,40 @@ audit committed them.
|
||||
**Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no
|
||||
decision reversed. **No secrets, passwords, tokens or keys written or quoted** — a Gitea remote URL with an
|
||||
embedded credential was seen during verification and is not reproduced here ([REDACTED]).
|
||||
|
||||
## [2026-11-05] audit - window 2026-11-04
|
||||
|
||||
- **New page:** `incidents/2026-11-04-upstream-stream-failure-0700-cohort.md` - two 07:00 jobs
|
||||
(`dca8482e4f76` **Daily Marketing Job Radar - Discovery**, `85edf6772bca` **Tech & AI Newsletter Digest
|
||||
(Analytical)**) died **36 s apart** (09:41:44 / 09:42:20) on `RuntimeError: Upstream stream failed before
|
||||
completion.` after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`; both incidents still open
|
||||
(`state = alerted`). Records the recovery: Stage 2 reconstructed the missing handoff from Stage 1's own
|
||||
candidate files, re-verified every candidate live, and wrote **5 roles (board 479 -> 484) + 1 canonical
|
||||
cover letter** (`MISSING_COVER_LETTER_COUNT 0`), consuming the handoff at 16:08:59.
|
||||
- **Updated `incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md`** - CT299 was frozen **twice**, not
|
||||
once. Second freeze: a **manual** `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and wedged
|
||||
(`unexpected status`) until **11-05 ~00:30**. Also records the earlier clean failure at **07:36:44**
|
||||
(log `vzdump-lxc-299-2026_11_04-07_36_44.log` on the dump target) and three `push_file` tasks from CT460
|
||||
at 07:17-07:33, plus the abandoned `/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299`. Live probe 2026-11-05
|
||||
00:54:21 AWST: `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump procs.
|
||||
- **Updated [[infrastructure/tailscale]]** - the sftpgo row now records recovery from *two* freezes.
|
||||
- **Updated [[systems/scheduled-tasks]]** - new ledger bullet: 11-04 settled **170 rows - 167 `completed` /
|
||||
2 `failed` / 1 `item_suppressed`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3**;
|
||||
deliveries **8/8 `delivered`**; the two failures are one upstream signature; stocktake overran for the third
|
||||
day (5 h 41 m 46 s, `free-stack` pin).
|
||||
- **Updated [[runbooks/job-radar]]** - cover-letter model route: `regenerate_cover_letter.py` defaults to
|
||||
`CL_MODEL=deepseek-paid-smart` (line 82); the 11-04 run pinned `gemini/gemini-3.5-flash` after **HTTP 402**.
|
||||
`grep` shows **no `CL_MODEL` in `~/.hermes/.env`** - the override was process-local and did not persist, so
|
||||
the next run inherits the default again. Endpoint probe 2026-11-05: `omniroute:20128/v1/models` -> **401**
|
||||
and `100.78.63.20:20128/v1/models` -> **401** (live, key-gated) - so the route is reachable; the same model
|
||||
serves other cron jobs successfully. Treat a 402 as route/time-specific until re-tested.
|
||||
- **Updated [[current-state]]** - new "Recent Changes (2026-11-05)" block; the page had been sitting at the
|
||||
2026-10-30 block across five audits.
|
||||
- **Updated [[index]]** - new dated ledger paragraph, incidents list gains the new page, "Last updated" moved
|
||||
2026-11-03 -> 2026-11-05.
|
||||
- **Live verification this audit:** pve (100.96.100.82) SSH probe at 2026-11-05 00:54:21 AWST (lock, cgroup
|
||||
freeze flags, guest ALIVE, sftpgo active, proc count, task list, dump dir); SFTPGo reached end-to-end from
|
||||
CT460 on `100.88.223.70:8080`; `jobs.json`, `executions.db`, `cron_incidents`, `deliveries.db` read live.
|
||||
- **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or re-pinned; no
|
||||
decision reversed. **No secrets, passwords, tokens or keys written or quoted.**
|
||||
|
||||
|
||||
+15
-4
@@ -3,13 +3,13 @@ title: Daily Marketing Job Radar Operations and Recovery
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-27
|
||||
updated: 2026-10-30
|
||||
updated: 2026-11-05
|
||||
review_after: 2026-11-13
|
||||
verified_on: 2026-10-30
|
||||
verified_on: 2026-11-05
|
||||
version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2)
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, dashboard, runbook]
|
||||
sources: [live-cron-output, live-notion-api, live-dashboard-api, hermes-docs]
|
||||
tags: [automation, cron, jobs, notion, dashboard, runbook, cover-letter, model-route]
|
||||
sources: [live-cron-output, live-notion-api, live-dashboard-api, hermes-docs, live-script-inspection, live-endpoint-probe]
|
||||
---
|
||||
|
||||
# Daily Marketing Job Radar Operations and Recovery
|
||||
@@ -39,6 +39,17 @@ Cron jobs inherit the profile-level `fallback_providers` chain; Hermes does not
|
||||
|
||||
## Cover letters are non-negotiable
|
||||
|
||||
> **Model route caveat - verified 2026-11-05.** `regenerate_cover_letter.py` (line 82) reads
|
||||
> `os.environ.get("CL_MODEL", "deepseek-paid-smart")`. On the 2026-11-04 run that default returned **HTTP 402
|
||||
> (billing)**, and the run pinned **`gemini/gemini-3.5-flash`** via `CL_MODEL` to get the strong-match letter out.
|
||||
> `grep` finds **no `CL_MODEL` in `~/.hermes/.env`** - the override existed only inside that process, so **the
|
||||
> next run inherits the default again**. If a run logs 402/503 while generating letters, re-pin `CL_MODEL` for the
|
||||
> process and confirm the letters rendered - do **not** ship a run without its cover letters. Note the same model
|
||||
> name still serves other cron jobs successfully (e.g. the 23:59 nightly audit), and the endpoint itself is live:
|
||||
> `omniroute:20128/v1/models` -> **401** and `100.78.63.20:20128/v1/models` -> **401** on 2026-11-05 (key-gated,
|
||||
> not down). Treat a 402 as **route/time-specific until re-tested**, not as proof the route is dead.
|
||||
|
||||
|
||||
A successful radar run includes the cover-letter stage.
|
||||
|
||||
- Audit **every Strong Match added today**, including pages created by an interrupted earlier run.
|
||||
|
||||
@@ -3,17 +3,18 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
review_after: 2026-11-11
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
review_after: 2026-11-06
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db]
|
||||
---
|
||||
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **158 jobs (144 enabled / 14 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `item_suppressed`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]].
|
||||
- **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
`deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` /
|
||||
1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and
|
||||
|
||||
Reference in New Issue
Block a user