resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly

- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo
- Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved)
- Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB
- Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification,
  decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree,
  honest AgentMail/Telegram delivery reporting
- Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed
- Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated
- Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with
  'lost its durable fire claim ownership' and left the tree dirty)
This commit is contained in:
Tony0410
2026-11-07 05:28:39 +08:00
parent b820993ecc
commit 0b4ff25fd2
14 changed files with 370 additions and 110 deletions
+4 -4
View File
@@ -3,8 +3,8 @@ title: Hermes Chaos
type: system
status: active
created: 2026-08-15
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-07
verified_on: 2026-11-07
confidence: high
tags: [hermes, chaos, ct450, experimental, agent]
sources: [live-probe, config-inspection]
@@ -28,10 +28,10 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
## Current State
- **Status:** Active (experimental)
- **Hermes version:** `v0.22.3` (2026.9.24), local `5b147ab036d6` — **the same build as CT460** (updated 2026-10-21). **0 commits behind `origin/main`** (verified 2026-10-26; the old "3 commits behind" claim is stale), though `hermes update --check` reports stable **v0.22.5** available. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]].
- **Hermes version:** `v0.22.5` (2026.9.24), local `bf0203a794a0` — **the same build as CT460** (updated 2026-10-21). ⚠ **Corrected 2026-11-07:** this line still read *"`v0.22.3` … local `5b147ab036d6`"* while asserting it was *"the same build as CT460"* — it was not. Live `hermes --version` on CT450 now returns **`v0.22.5` … local `bf0203a794a0`**, byte-identical to CT460's own `hermes --version`, so the *same build* claim is finally true. `hermes update --check` reports stable **v0.23.0** (`2bd378ed1b8d`) available. Python 3.14.9. See [[incidents/2026-10-03-ct450-hermes-update]].
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
- **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` 2026-10-26). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460)
- **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` **2026-11-07**; unchanged). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460) — both confirmed by a live set-diff of the two `jobs.json` files on 2026-11-07
- **Production counterparts:** CT460 has **28 jobs / 25 enabled** — the two fleets no longer share a job count, and the CT450 duplicates are paused
## Providers & Routing
+4 -4
View File
@@ -3,8 +3,8 @@ title: Hermes Production
type: system
status: active
created: 2026-08-15
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-07
verified_on: 2026-11-07
confidence: high
tags: [hermes, production, ct460, agent]
sources: [live-probe, config-inspection]
@@ -43,7 +43,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Providers & Routing
- **Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, CT101/m93p) — verified live 2026-10-26. The `custom:omni` provider key does not exist in `custom_providers`, but **2 cron jobs still pin it** (see [[systems/scheduled-tasks]])
- **Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, CT101/m93p) — verified live 2026-10-26, **re-verified 2026-11-07** (live `config.yaml` `model.default.provider: custom:omni-101`; a keyed `deepseek-paid-smart` chat probe against `100.78.63.20:20128` returns **HTTP 200**). The `custom:omni` provider key does not exist in `custom_providers`. ⚠ **Corrected 2026-11-07:** the old claim that *"2 cron jobs still pin `custom:omni`"* is **no longer true** — live `jobs.json` shows **no job pinned to the bare `custom:omni`**; **all 21 provider-pinned jobs use `custom:omni-101`** (see [[systems/scheduled-tasks]])
- **Default model:** `smart-route` (previously `freellmapi-auto`)
- **Fallback providers** (live chain, re-verified 2026-10-26): `nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6_1t`. ⚠ The **first** entry is the dead `nous` credential — a fallover tries it first
- **Auxiliary models:** all eight `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) are **`no-think/freeapi/auto` on `custom:omni-101`** with a `freeapi/deepseek-v4-flash` → `kg/kilo-auto/free` fallback chain (moved 2026-10-25 19:13)
@@ -74,7 +74,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
| Aspect | Production (CT460) | Chaos (CT450) |
|--------|-------------------|---------------|
| Purpose | Stable daily operations | Experimental/testing |
| Cron jobs | 28 (25 enabled / 3 disabled) | 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460 |
| Cron jobs | **29 (26 enabled / 3 disabled)** (re-verified 2026-11-07) | 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460 (re-verified 2026-11-07) |
| Config changes | Requires verification | Safe to experiment |
| Memory | Production Mnemosyne | Separate state |
+39
View File
@@ -14,6 +14,45 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
# Current State
Updated: 2026-11-07
## Recent Changes (2026-11-07, stocktake; live probes 2026-11-07 03:1x-03:2x AWST)
- **The tailnet lost a peer and the m93p guest list lost a guest — both corrections came from live probes, not the age scan.**
[[infrastructure/tailscale]] was rebuilt from `tailscale status --json`: **`octop` (`100.92.163.122`) is back
online** (peer record `Online: true`, `tailscale ping` → `pong from octop via DERP(syd) in 98ms`; it remains
**no LXC or VM on either hypervisor** — an orphaned tailnet peer), and the **9 remaining offline ages** were
refreshed (+1d each). All **51 live IPv4 peers** match the table's IPs exactly, with no missing peers and no
orphan rows (**52 rows** = 51 peers + `hermes` itself). The CT460 Serve route table still matches
`tailscale serve status` **exactly (12 paths on :443)**; both previously-flagged dead targets were
**re-confirmed dead** — still no listener on `8769` (`/agent-estate`) or `9123` (`/synergy-prep`).
- **⚠ `m93p` actually runs 20 LXCs, not 19 — CT107 (`mem0`) was missing from the Proxmox page.**
Live `pct list` on `m93p` returns **20 LXCs — 18 running, 2 stopped** (CT998 `nightbridge`, CT1264
`openmausbot`). The omitted guest is **CT107 `mem0`** (`mem0-mcp.service` active on `:8890`,
`pct exec 107 -- tailscale ip -4` → `100.73.94.53`). Added to [[infrastructure/proxmox]]; the
**2026-10-31 reading of 19 was already short for the same reason**. `pve` still returns exactly
**17 LXCs + 4 VMs**, and both hosts still run the identical `pve-manager 9.2.10` build.
- **⚠ `m93p`'s uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*.
Live `uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18** at probe time, load
**0.76 / 1.08 / 1.24** — normal). The 2026-10-30 reboot was real but belonged to **`pve`**
(boot **2026-10-30 16:24:00**), not `m93p`.
- **Storage percentages refreshed on both hosts** against live `pvesm status` — `pve`'s `ssd0` moved
**76.43 % → 78.29 %** (~152 GiB free), `fred-qnap` **82.77 % → 84.21 %**, `local` **79.66 % → 81.62 %**;
on `m93p` the iMac shares and `local` moved **60.25 % → 76.21 %** and `omb-internal`
**57.92 % → 64.66 %**.
- **Docker inventories diffed set-wise against live `docker ps` on all three hosts.** CT200 matches
**61 running / 68 total** exactly; CT201 matches **20 / 26** exactly; CT333 matches **8 / 9** — but its
**enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name is
**`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected in [[infrastructure/docker-services]].
- **Two small cross-page corrections.** [[infrastructure/network-map]]'s `/gitea` route was annotated
*"Gitea (ct207 tailnet IP)"* — it is **CT221 on `m93p`** (`100.82.100.128`; ct207's own IP is
`100.82.40.23`); the target answers **HTTP 200**. [[agents/hermes-chaos]]'s version line read
*"`v0.22.3` … local `5b147ab036d6` — the same build as CT460"*, which contradicted itself: live
`hermes --version` on CT450 is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to CT460's.
[[agents/hermes-production]]'s claim that *2 cron jobs still pin `custom:omni`* is **no longer true** —
live `jobs.json` shows **0** jobs on the bare key and **21** on `custom:omni-101`.
- **Cron fleet re-verified unchanged:** **29 jobs (26 enabled / 3 disabled)**, matching the nightly audit
hours earlier. CT450's fleet is **25 jobs (13 enabled / 12 disabled)** with **9 shared names** — confirmed
by a live set-diff of both `jobs.json` files. → [[systems/scheduled-tasks]]
## Recent Changes (2026-11-07, nightly audit; live probes 2026-11-07 00:3x-00:5x AWST)
- **A red row with no alert: the Job Radar's 11:30 slot was discarded as stale while a concurrent fire
@@ -1,18 +1,23 @@
---
title: Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 While the Job Reads `ok`
title: Hermes Config Backup to Gitea — Push Was Dead 2026-08-08 → 2026-11-07 While the Job Read `ok`
type: incident
status: active
status: resolved
created: 2026-10-21
updated: 2026-10-22
verified_on: 2026-10-22
review_after: 2026-10-29
updated: 2026-11-07
resolved_on: 2026-11-07
verified_on: 2026-11-07
review_after: 2027-02-07
confidence: high
tags: [backup, gitea, cron, git, green-vs-actual]
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki]
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki, gitea-api, github-api, sandbox-e2e-test]
---
# Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08
> **RESOLVED 2026-11-07.** The push is live again, the backlog is drained, and the script now exits
> non-zero on failure so the cron row cannot lie. See [Resolution 2026-11-07](#resolution-2026-11-07)
> at the foot of this page for the full record. The original investigation is preserved below.
## Headline
The daily **Hermes Config Backup to Gitea** job (`e27da6c34bbb`, `0 11 * * *`, no-agent, `hermes-config-backup.sh`)
@@ -103,3 +108,92 @@ change as well the *only* off-box copy path was this dead push → [[infrastruct
- [[infrastructure/backups]]
- [[systems/scheduled-tasks]]
- [[systems/hermes-agent]]
---
# Resolution 2026-11-07
Anthony asked to fix the failing push (and to check the Notion vault for the API key). Resolved in
one session: the push is live, the backlog is drained, and the script can no longer report a false green.
## Root causes (four, all independent)
1. **No Gitea credential.** `origin` was HTTPS with no userinfo, and the global `credential.helper=store`
file `~/.git-credentials` held only a `github.com` entry — so the cron worker could not authenticate
non-interactively. The token in the Notion vault was valid and identical to the one the wiki repo
carries in its remote URL; it simply was never wired into the backup repo.
2. **`backup.sh` had no `exit` statement at all.** It always returned 0, so the scheduler
(`success = (returncode == 0)` in `cron/scheduler_script.py`) recorded `last_status: ok` on every
failed night. This is why the outage went unnoticed for 92 days — it was a *false green*, not a
missing signal. (The incident's earlier note blamed the scheduler; the scheduler was correct. The
script was the liar.)
3. **A 1.76 GB `checkpoints/` backlog** sat in the unpushed range — 13 blobs over 100 MB, all junk
that `.gitignore` already excluded for new commits but that history still carried.
4. **Two broken secondary legs.** The `github` remote did not exist, and the fallback
`_github_clean_push` pushed a branch's full ancestry — which contains a 7.45 GB tarball blob, so
GitHub's 100 MB file limit could never accept it. The AgentMail alert called a
`python3 send_email.py` that no longer exists, via a `chaos-agent@agentmail.to` inbox that no
longer exists.
## Fix applied
- **Credential:** stored a host-scoped Gitea entry via `git credential approve` (not embedded in the
remote URL). `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`.
- **Backlog:** `git filter-repo --partial --refs origin/main..main --path checkpoints --invert-paths
--prune-empty never --force` stripped `checkpoints/` from the unpushed range only (so `origin/main`
was untouched and the push stayed a fast-forward), then `checkpoints/` was untracked at the tip.
Pre-rewrite history is preserved under the tag `backup/pre-rewrite-20261107-032658`.
**Push set: ~2 GB → 264.8 MB.** The 7.45 GB tarball was already on `origin/main` and was never resent.
- **Script hardening** (`~/.hermes/config-backup/backup.sh`, rewritten via OpenCode + a correction pass):
- exits **0 only on verified success**, `1` otherwise — the cron row can no longer lie;
- `.last_backup_ok` is stamped **only** after the remote is confirmed to hold the commit;
- the push decision compares **remote vs local**, not "were there new working-tree changes" — a
local-ahead-of-remote with a clean tree now still pushes (this is exactly the mechanism that
created the 92-commit backlog);
- every `git ls-remote` is guarded with `|| true`, so an unreachable remote reaches the alert path
instead of killing the script at exit 128 before `notify_failure` fires;
- the GitHub leg now pushes a **parentless orphan snapshot** built from `HEAD^{tree}` via
`git commit-tree` — no history, no >100 MB blob, HEAD/branch/index untouched — and is a
warning-only secondary target that can never fail the job;
- the AgentMail leg uses the real CLI (`agentmail inboxes:messages send --inbox-id <addr>`) with the
key read from `.env` at runtime and passed via the environment, never argv;
- the Telegram leg uses `curl --fail` and requires `"ok":true` in the body, so it reports `sent`
only on a real delivery.
## Verification (live, 2026-11-07 AWST)
| Check | Result |
|---|---|
| Real `backup.sh` run | exit **0**, 47 s |
| `git rev-parse HEAD` vs `git ls-remote origin refs/heads/main` | identical — `a50a5170c1c0e2af5235fa63a95a732e2d780703` |
| `git rev-list --count origin/main..main` | **0** (was 92) |
| Gitea API repo `Anthony/hermes-config-backup` | HTTP 200, default branch `main`, updated `2026-11-07T05:10:46+08:00`, 8.1 MB |
| Gitea API `branches/main` | `a50a5170c1c0` — `🦝 nightly backup: 2026-11-07 05:10:33 AWST` |
| `.last_backup_ok` | refreshed to the run timestamp (was stale from 2026-11-06 11:00) |
| GitHub `main` | `faed9fb5c677`, **0 parents** (orphan snapshot confirmed), 30,086 files, largest blob 23.5 MB |
| GitHub `master` branch | untouched at `c99c94e6b13b` |
| GitHub force-push data loss | none — GitHub's tree was a strict **subset** of local HEAD (only `BACKUP-INDEX.md` added) |
| Token leakage scan (HEAD tree, working tree, all unpushed blobs) | **clean** — token absent everywhere |
| AgentMail leg (self-test to the agent's own inbox) | delivered, `message_id` returned |
| Telegram leg (`getMe`, no message sent to Anthony) | `"ok":true` detected → would report `sent` |
| Sandbox E2E: unreachable remote | exit **1**, `notify_failure` fired, health file **not** stamped |
| Sandbox E2E: local ahead of remote, clean working tree | pushed + verified, exit **0**, health stamped |
| Sandbox E2E: already up to date | no push, exit **0**, health stamped |
## Verification command set (for the next audit)
```
git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
git -C ~/.hermes/config-backup rev-list --count origin/main..main
tail -20 ~/.hermes/config-backup/backup.log
tail -3 ~/.hermes/config-backup/.backup_history
```
## Still open (not part of this incident)
- Gitea host SSH login is denied for `anthonymau` — free disk headroom on the Gitea box is still
unverified (the Gitea API reports repo size, not host free space).
- `hermes-gateway-runtime-health.service` still fails every 5 min because the Mattermost host
`100.98.116.14` has been offline since ~Sept 2026. Separate incident.
- The `checkpoints/` directory is still excluded from backups by policy; the live snapshots remain
at `~/.hermes/checkpoints/` on-box only.
+3 -3
View File
@@ -28,7 +28,7 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
> **2026-11-04 (nightly audit) — the ledger raised its own red row:** the **11-03 fleet settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) — **unclosed incident rows 0 → 1**, non-clean `last_status` **1 → 2**, deliveries **7 rows / 7 `delivered`**. The single failure is the **Job Radar Stage 1** (`dca8482e4f76`), which ran **07:00:33 → 13:28:52 (6 h 28 m 18 s)** and died on **`RuntimeError: stream active timeout`** — a **new terminal signature**, not the iteration cap — after going **silent for 65 m 21 s**; its incident `dca848_dfd4bea84f2a` (`timeout`) is **`alerted` and open** and the notice **was delivered** (13:28:51.741724) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. That answers the 11-02 page's open question: **output is suppressed when the run ends `completed`, delivered when it ends `failed`** — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so **2026-11-02's and 2026-11-03's candidate sets are both unrecovered**. ⚠ **The 03:00 stocktake overran for a second day** (`1a8e90ea2271`, 03:00:30 → 10:41:13 = **7 h 40 m 43 s**, `suppressed`, nothing delivered, artefacts uncommitted again). **New page:** [[systems/hermes-hq-push]] (v0.9.27, enabled; manifest says it **replaces `dispatch-push`**, and no trace of the old plugin survives).
>
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.10), was discovered during the 2026-10-14 stocktake.
> Guest IDs collide between the two hosts.
## Start here
@@ -125,8 +125,8 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
- [[infrastructure/backups]] — hermes backup CLI, retention, and gaps
### Networking
- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 9.2.10 + `m93p` 9.2.10, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 76.43 % / ~169 GiB free, 2026-10-31)
- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **50-node** tailnet inventory (re-verified 2026-10-31: `mattermost` back **online** after ~40 days; 7 offline ages refreshed)
- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 9.2.10 + `m93p` 9.2.10, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 78.29 % / ~152 GiB free, 2026-11-07)
- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **52-node** tailnet inventory (re-verified 2026-11-07: `octop` back **online**; 9 offline ages refreshed; **CT107 `mem0`** added to the m93p guest list)
- [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly
## Concepts
+36 -26
View File
@@ -3,8 +3,8 @@ title: Backups
type: infrastructure
status: active
created: 2026-07-22
updated: 2026-11-04
verified_on: 2026-11-04
updated: 2026-11-07
verified_on: 2026-11-07
confidence: medium
tags: [infrastructure, backups]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -54,7 +54,7 @@ How Hermes's own configuration/state gets backed up, and what's missing.
local snapshot and no off-box copy** — the Gitea mirror below has been dead since 2026-08-08.
→ [[systems/hermes-agent]]
## ⚠ Scheduled config backup — the Gitea mirror has been dead since 2026-08-08
## ✅ Scheduled config backup — RESOLVED 2026-11-07 (was dead since 2026-08-08)
`e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits
locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its
@@ -66,31 +66,41 @@ helper — so the non-interactive push dies on
`--- Backup complete: FAIL ---`. Full evidence and the proposed fix:
[[incidents/2026-08-08-gitea-config-backup-push-dead]].
**Consequence for this page: everything above describes a topology that has not carried `~/.hermes` off the box
since 2026-08-07.** The 2026-07-22 full zip and the wiki/authoring repos remain the only recent off-box copies.
Telegram failure notices *have* been firing daily (last: 2026-10-21 11:00:14, `message_id 8449`); the AgentMail
notification step failed.
> **RESOLVED 2026-11-07.** The 92-commit backlog is drained and pushed to Gitea, and the script has been
> hardened so the cron row can no longer report a false green. Full record:
> [[incidents/2026-08-08-gitea-config-backup-push-dead#Resolution 2026-11-07]].
**Re-verified 2026-11-04 — still dead, and the drift is now 89 commits.** Live `git -C ~/.hermes/config-backup`:
`origin` = `https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` (**unchanged** — still HTTPS with
no userinfo and no credential helper, so the credential root cause has not been addressed); `rev-list --count
origin/main..HEAD` = **89**; `log origin/main -1` = **2026-08-07 11:00:47 +0800 1e1eb208** (`🦝 nightly backup:
2026-08-07 11:00:29 AWST`) — i.e. **nothing has reached Gitea in 89 days**. `backup.log` confirms the nightly
failure is current, not historical: the last two runs read
`[2026-11-02 11:00:32 AWST] ✗ Push to Gitea FAILED` / `--- Backup complete: FAIL ---` and
`[2026-11-03 11:00:31 AWST] ✗ Push to Gitea FAILED` / `--- Backup complete: FAIL ---`.
**A second, separate push failure is now also present:** the script attempts a `github` remote that **does not
exist** — `fatal: 'github' does not appear to be a git repository` → `✗ Push to GitHub FAILED`. So both off-box
targets are broken; the GitHub leg is a missing-remote error rather than a credential error, and would need
`git remote add github …` with credentials before it can work at all.
**The cron row still lies:** `e27da6c34bbb` reads `enabled: true, last_status: ok` in `jobs.json` while every
nightly execution has failed the push — the classic green-row/dead-artifact pattern.
**What was wrong (four independent causes):** (1) no Gitea credential — `origin` was HTTPS and
`~/.git-credentials` held only a `github.com` entry, so the non-interactive push could not authenticate;
(2) **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's
`success = (returncode == 0)` recorded `ok` on every failed night (the scheduler was correct; the script
was the liar); (3) a 1.76 GB `checkpoints/` backlog sat in the unpushed range, 13 blobs over 100 MB;
(4) both secondary legs were broken — the `github` remote did not exist, and its fallback pushed a
branch's full ancestry containing a 7.45 GB tarball blob that GitHub's 100 MB limit could never accept.
**Full-backup gap unchanged:** the one full `hermes backup` zip on disk is still
`/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, 22 Jul) — **now 105 days old**. No new full zip has
been taken. `crontab -l -u hermes` is still empty and there are still **0 user systemd timers**, so nothing
schedules the full zip; only `e5ed215fc91c` (**Mnemosyne DB Backup**, weekly `0 3 * * 0`) and `e27da6c34bbb`
(config backup, daily) exist in the cron fleet, and neither takes a full `hermes backup`.
**What was done:** host-scoped Gitea credential stored via `git credential approve`; `checkpoints/`
stripped from the unpushed range with `git filter-repo --partial` (push set ~2 GB → 264.8 MB, pre-rewrite
history preserved under tag `backup/pre-rewrite-20261107-032658`); the script rewritten to exit non-zero on
failure, stamp `.last_backup_ok` only after remote verification, decide the push by **remote-vs-local**
rather than by working-tree changes, guard every `ls-remote` with `|| true` so an unreachable remote still
reaches the alert path, push GitHub as a **parentless orphan snapshot**, and report Telegram/AgentMail
delivery honestly (`curl --fail` + `"ok":true`; real `agentmail` CLI).
**Re-verified 2026-11-07 — live and green.** Real run exit **0** in 47 s; `git rev-parse HEAD` and
`git ls-remote origin refs/heads/main` both `a50a5170c1c0e2af5235fa63a95a732e2d780703`;
`rev-list --count origin/main..main` = **0**; Gitea API confirms `main` at `a50a5170c1c0`, updated
`2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub mirror
now carries a parentless orphan snapshot (`faed9fb5c677`, 0 parents, 30,086 files, largest blob 23.5 MB) —
no data lost, since GitHub's tree was a strict subset of local HEAD. The AgentMail and Telegram alert
paths were each exercised directly and report honestly.
**Consequence for this page: the topology above carries `~/.hermes` off the box again.** The one full
`hermes backup` zip on disk is still `/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, 22 Jul) —
**now 108 days old**, and no new full zip has been taken. `crontab -l -u hermes` is still empty and there
are still **0 user systemd timers**, so nothing schedules the full zip; only `e5ed215fc91c`
(**Mnemosyne DB Backup**, weekly `0 3 * * 0`) and `e27da6c34bbb` (config backup, daily) exist in the cron
fleet, and neither takes a full `hermes backup`. The nightly config backup is a *config* mirror, not a
full-home backup — the gap on the full zip is unchanged.
## Relevant runbooks
- [[runbooks/backup-wiki]]
+6 -6
View File
@@ -3,8 +3,8 @@ title: Docker Services
type: infrastructure
status: mixed
created: 2026-07-22
updated: 2026-11-06
verified_on: 2026-11-06
updated: 2026-11-07
verified_on: 2026-11-07
confidence: high
tags: [docker, homelab, infrastructure]
sources: [raw/configs/hermes-config-sanitized.txt, live-probe]
@@ -20,13 +20,13 @@ Docker runs on **four hosts**, and CT460's socket is not readable by the agent u
| Host | Access | Containers running |
|---|---|---|
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** running / 68 total — main Docker host (**re-verified 2026-11-06**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) |
| CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** running / 26 total — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**re-verified 2026-11-05**) |
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total — `flamboyant_mahavira`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → `flamboyant_mahavira` (2026-11-04) → **`nostalgic_keldysh`** (2026-11-06) — all `ghcr.io/github/github-mcp-server`, the live one created 2026-11-05T22:15:57Z |
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** running / 68 total — main Docker host (**re-verified 2026-11-07**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) |
| CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** running / 26 total — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**re-verified 2026-11-07** — set-wise diff against live `docker ps`: no additions, no removals) |
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total (**re-verified 2026-11-07**) — `nostalgic_keldysh`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → `flamboyant_mahavira` (2026-11-04) → **`nostalgic_keldysh`** (2026-11-06) — all `ghcr.io/github/github-mcp-server`, the live one created 2026-11-05T22:15:57Z. ⚠ **Corrected 2026-11-07:** the enumerated list still opened with `flamboyant_mahavira`, which is **not running** — live `docker ps` on CT333 returns **`nostalgic_keldysh`** (created 2026-11-05 22:15:57, `Up 29 hours`), `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1` (8 running), plus `qdrant` **exited** (9 total) |
| CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** running / 9 total — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented |
| CT460 `hermes` | `pct exec 460 -- docker ps` **as root** from the PVE host — socket is `root:docker srw-rw----`, the `hermes` user is not in group `docker` (its own `docker ps` → `permission denied while trying to connect to the docker API`) | `ignis` — **running** (`Up 5 days`, `ignis-subpath:0.8.9`, `0.0.0.0:8080->8080/tcp`), re-verified 2026-11-05 |
## CT200 live inventory (re-verified 2026-11-06, 61 running / 68 total, excluding 0–2 transient `worldmonitor-seeder-*` containers)
## CT200 live inventory (re-verified 2026-11-07, 61 running / 68 total, excluding 0–2 transient `worldmonitor-seeder-*` containers)
`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`.
**Re-verified 2026-11-06 — the 2026-11-05 "the seeder does not exist" finding was WRONG, and the cause is a
+7 -7
View File
@@ -3,8 +3,8 @@ title: Known Hosts and Paths
type: infrastructure
status: partial
created: 2026-07-22
updated: 2026-11-05
verified_on: 2026-11-05
updated: 2026-11-07
verified_on: 2026-11-07
confidence: medium
tags: [infrastructure, hosts, containers, paths]
sources: []
@@ -22,8 +22,8 @@ sources: []
| `custom:omni-101` | OmniRoute proxy (CT101 node `omniroute-1`) — **the live default provider** | `http://100.78.63.20:20128` | Yes (live probe 2026-10-26) |
| CT201 | Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.77` (LAN) | Yes (live probe 2026-10-30 — LAN address corrected from `192.168.178.166`) |
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net`, `100.118.5.51` (Tailscale) | Yes |
| PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-10-26) |
| PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-10-26) |
| PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-11-07 — 17 LXCs + 4 VMs) |
| PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-11-07 — 20 LXCs + 2 VMs) |
| CT450 (hermesagent) | Hermes Agent **v0.22.5** (2026.9.24), local `bf0203a794a0` — **same build as CT460**; hermes-webui, hermes-ui (Vite). ⚠ a stale **root-level** `/usr/local/bin/hermes` on CT450 still reports **v0.19.0** — the live agent is the `hermes`-user install under `~/.hermes/hermes-agent` | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes (live probe 2026-10-30 — version corrected from `v0.22.3`, build from `5b147ab036d6` to `bf0203a794a0`) |
## Project Paths
@@ -34,7 +34,7 @@ sources: []
| `/home/hermes/workspace` | Workspace | default terminal cwd |
| `/home/hermes/.local/bin/hermes` | Hermes binary | |
| `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` |
| `/home/hermes/.hermes/skills/` | Skills dir | **302 skills across 68 categories** (`.archive/` excluded — 46 archived; re-verified 2026-11-05) |
| `/home/hermes/.hermes/skills/` | Skills dir | **302 skills across 68 categories** (`.archive/` excluded — 46 archived; re-verified 2026-11-07) |
| `/home/hermes/.hermes/cron/` | Cron jobs | **29 jobs — 26 enabled / 3 disabled** (re-verified 2026-10-31 00:3x). ⚠ Superseded: the 2026-10-30 03:3x reading was 25 / 4 — the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure, but **that pause lasted only until 10:02:48**, when its long-running slot closed `completed`, incident `ea79a2_d93c3acdc4f2` auto-closed `resolved`, and the job returned to `enabled: true` / `last_status: ok` / streak 0. **CT460 itself was then dark for 5 h 30 m on 2026-10-30** (PVE `pve` rebooted; host boot 16:24:00, CT460 16:28:00, scheduler resumed 16:37:21 — ≈39 cron slots lost, 0 failures) → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] |
| `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ |
| `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 |
@@ -59,8 +59,8 @@ sources: []
- `/home/hermes/workspace/readlater/docker-compose.yml`
## Homelab References
- **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-10-26 (17 LXCs)
- **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-10-26 (19 LXCs). Guest IDs collide between the two hosts
- **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-11-07 (**17 LXCs** + 4 VMs; boot 2026-10-30 16:24:00)
- **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-11-07 (**20 LXCs** + 2 VMs; boot 2026-11-05 21:19:52). ⚠ **Corrected 2026-11-07:** the previous figure of 19 LXCs omitted **CT107 (`mem0`)**, which is running — see [[infrastructure/proxmox]]. Guest IDs collide between the two hosts
- **CT600 (`omniroute`):** OmniRoute server, `100.88.81.19` — **running and live**; `omniroute` is the hostname that resolves to it
- **CT601 (`omni`):** the old CT601 OmniRoute guest **no longer exists** on either hypervisor (`pct config 601` → no such config on both `pve` and `m93p`). The orphaned `omni` Tailscale peer (`100.93.204.4`) has since **expired out of the tailnet entirely** — as of the 2026-10-30 live probe the tailnet contains **no** peer named `omni` and no peer at `100.93.204.4`, so the old "offline 69d" description no longer applies. The only OmniRoute nodes left are CT600 (`omniroute`) and CT101 (`omniroute-1`).
- **CT101 (`omniroute`, m93p):** second OmniRoute node, `100.78.63.20` — running
+8 -8
View File
@@ -3,8 +3,8 @@ title: Network Map
type: infrastructure
status: active
created: 2026-07-22
updated: 2026-11-06
verified_on: 2026-11-06
updated: 2026-11-07
verified_on: 2026-11-07
confidence: high
tags: [infrastructure, network]
sources: [raw/configs/hermes-config-sanitized.txt, live-probe:ct460-ss+tailscale-serve-status-2026-11-05]
@@ -22,9 +22,9 @@ Where Hermes and its dependencies run, and how traffic reaches them.
- **This container (hermes, ct460):** `192.168.178.129` (DHCP, eth0)
### Tailscale (`kangaroo-eel.ts.net`)
_Peer IPs below re-checked 2026-11-06 against `tailscale status` on ct460; `webserver`, `clawtest`, `hermesagent`, `docker`, `omniroute` all matched the values listed. Full 52-row node table with live states lives in [[infrastructure/tailscale]]._
_Peer IPs below re-checked 2026-11-07 against `tailscale status` on ct460; `webserver`, `clawtest`, `hermesagent`, `docker`, `omniroute` all matched the values listed. Full 52-row node table with live states lives in [[infrastructure/tailscale]]._
- **pve:** `100.96.100.82`
- **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.0), discovered 2026-10-14.** See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host.
- **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.10), discovered 2026-10-14.** ⚠ Corrected 2026-11-07: this line still read *"9.2.0"* although both hosts run the identical `pve-manager 9.2.10` build (live `dpkg -l pve-manager` on both). See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host.
- **hermes (ct460):** `100.118.5.51`
- **hermesagent (ct450, the original/legacy Hermes agent dashboard):** `100.98.205.112`
- **omniroute (ct600):** resolves via `omniroute.kangaroo-eel.ts.net` → `100.88.81.19`
@@ -68,7 +68,7 @@ _Peer IPs below re-checked 2026-11-06 against `tailscale status` on ct460; `webs
Loopback-only helpers (not routable): `49347` hermes gateway, `42013` Paseo, `42647` + `38187` opencode.
## Tailscale Serve / Funnel (verified 2026-11-06)
## Tailscale Serve / Funnel (verified 2026-11-07)
`tailscale serve status` on ct460 reports 14 proxies: 12 under `https://hermes.kangaroo-eel.ts.net` (tailnet only), one tailnet route on `:9443`, and one Funnel route on `:8443` (public).
@@ -80,12 +80,12 @@ Loopback-only helpers (not routable): `49347` hermes gateway, `42013` Paseo, `42
| `/wiki` | `100.118.5.51:9120` | Agent Estate Wiki |
| `/webui` | `127.0.0.1:8787` | hermes-webui |
| `/ignis` | `127.0.0.1:8080` | Ignis |
| `/gitea` | `100.82.100.128:3000` | Gitea (ct207 tailnet IP) |
| `/gitea` | `100.82.100.128:3000` | Gitea — **CT221 on `m93p`**, not ct207 (⚠ **corrected 2026-11-07**: this line wrongly said *"ct207 tailnet IP"*; live `pct exec 221 -- tailscale ip -4` → `100.82.100.128`, and ct207's own IP is `100.82.40.23`). The target answers **HTTP 200**. |
| `/desktop` | `127.0.0.1:6081` | desktop VNC (noVNC) |
| `/vnc-camofox` | `127.0.0.1:6080` | Camofox VNC |
| `/evening-grid.html` | `127.0.0.1:9099/evening-grid.html` | jobs dashboard page |
| `/agent-estate` | `127.0.0.1:8769` | **DEAD — nothing listening on 8769 (re-confirmed 2026-11-06)** |
| `/synergy-prep` | `127.0.0.1:9123` | **DEAD — nothing listening on 9123 (re-confirmed 2026-11-06)** |
| `/agent-estate` | `127.0.0.1:8769` | **DEAD — nothing listening on 8769 (re-confirmed 2026-11-07)** |
| `/synergy-prep` | `127.0.0.1:9123` | **DEAD — nothing listening on 9123 (re-confirmed 2026-11-07)** |
- `:9443` (tailnet only) — `/` → `127.0.0.1:5173`.
- Funnel `:8443` (public) — `/webhook` → `127.0.0.1:8085/webhook` (AgentMail webhook receiver).
+35 -23
View File
@@ -3,8 +3,8 @@ title: Proxmox VE
type: infrastructure
status: active
created: 2026-07-26
updated: 2026-11-01
verified_on: 2026-11-01
updated: 2026-11-07
verified_on: 2026-11-07
confidence: high
tags: [proxmox, virtualization, infrastructure, homelab]
---
@@ -21,7 +21,8 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
- **Primary user:** `Anthony@pam` (full admin)
- **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve`
## Live LXC Inventory (verified 2026-10-31 via `pct list`)
## Live LXC Inventory (verified 2026-11-07 via `pct list`)
Re-verified 2026-11-07 03:2x AWST (stocktake): **live `pct list` on `pve` still returns exactly the 17 LXCs in the table below** (CT255 `mattermost` `running`, CT209 `cloudreve` and CT555 `elementsynapse` `stopped`, CT600 `omniroute` `running`, and **no CT601 on either hypervisor**), and **`qm list` on `pve` still returns exactly the 4 VMs** (100 `homeassistant`, 101 `guest-vps`, 110 `omarchy`, 282 `ubuntu-server-xfce` — all running; **no 310, no 403, no 401**). Both `pve-manager` packages are still **9.2.10** on **both** hosts. The **one correction** was on the `m93p` side: see the `m93p` section below — **CT107 (`mem0`) was missing**, so the m93p guest count was one short.
Re-verified 2026-11-01 03:5x AWST (stocktake): **`pct list` on `pve` and `qm list` on `pve` both still match the tables below exactly** — 17 LXCs and 4 VMs on `pve`, with CT255 `mattermost` still `running`, CT209 `cloudreve` and CT555 `elementsynapse` still `stopped`, and **no CT601 on either hypervisor** (`/etc/pve/lxc/601.conf` absent). **`m93p` re-confirmed**: `pve-manager/9.2.10` (same build as `pve`), `qm list` returns **VM 103 `cachyos-m93p` stopped** and **VM 106 `openmausbot-vm` running**, and `pct exec 1001 -- docker ps` returns `sshwifty` (CT1001). Storage percentages were refreshed against live `pvesm status` on both hosts (see the Storage section). **Note:** `m93p` SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout).
@@ -103,7 +104,7 @@ as a hypervisor.
- **Hostname:** `m93p` · **Tailscale:** `100.104.51.98`
- **PVE version:** **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, kernel `7.0.14-12-pve`) — **corrected 2026-10-31**. The previous entry read *"9.2.0"* and claimed `m93p` was *"materially newer than pve's 8.x"*. Both hosts now run the **identical `pve-manager 9.2.10` build**; only the kernel differs (`7.0.14-12-pve` vs `7.0.14-5-pve`). The hosts are **version-identical, not version-skewed**.
- **Uptime:** **up 1 day, 12:02** at 2026-11-01 03:56 AWST (live `uptime`) — the host **rebooted ~2026-10-30 16:0x**, consistent with the CT1264 openmausbot outage below. Load average was **7.66 / 12.80 / 11.32** (1/5/15 min) at that moment — elevated, worth watching
- **Uptime:** **up 1 day, 7:18** at 2026-11-07 04:38 AWST (live `uptime -s` → **boot 2026-11-05 21:19:52**; `uptime` → `up 1 day, 7:18`, load average **0.76 / 1.08 / 1.24** — normal). ⚠ **Corrected 2026-11-07:** the previous entry read *"up 1 day, 12:02 at 2026-11-01 03:56 AWST — the host rebooted ~2026-10-30 16:0x"*. The **reboot itself is still real** (`pve` booted **2026-10-30 16:24:00** per `uptime -s`, and the CT1264 openmausbot outage below follows from it), but the **uptime figure and the implied boot date were wrong** — the host has since rebooted again on **2026-11-05 21:19:52**, so it was up only **1 day, 7:18** when probed, not 12 h. The `pve` host's own boot time is **2026-10-30 16:24:00** (up **7 days, 12:14** at probe time)
- **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf`
### LXC Inventory (verified 2026-10-31 via `pct list` on m93p)
@@ -115,6 +116,7 @@ as a hypervisor.
| 102 | vaultwarden | running | Vaultwarden password manager |
| 104 | rclone | running | Rclone |
| 105 | conduit | running | **Conduit** — iPhone calendar / health / location / contacts bridge (see [[systems/messaging-integrations]]). **Added 2026-10-31** — this guest was missing from the 2026-10-14 table despite running |
| 107 | mem0 | running | **mem0 MCP server** (`mem0-mcp.service` → `:8890`), Tailscale `100.73.94.53` (`mem0` peer) — **added 2026-11-07**, this guest was missing from the table despite running |
| 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** |
| 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one |
| 500 | litellm | running | LiteLLM proxy |
@@ -130,8 +132,15 @@ as a hypervisor.
| 1263 | social-dashboard | running | Social dashboard |
| 1264 | openmausbot | **stopped** | OpenMausBot — **stopped**, which is why the `openmausbot` Tailscale peer is offline 9d. **Added 2026-10-31** — was missing from the 2026-10-14 table. The companion **VM106 `openmausbot-vm` is running** and holds the separate `openmausbot-vm` peer (`100.77.107.30`) |
**Re-verified 2026-10-31:** live `pct list` on `m93p` returns **19 LXCs — 18 running, 1 stopped
(CT998 nightbridge) plus CT1264 openmausbot stopped**. The 2026-10-14 table listed only **17**, omitting
**Re-verified 2026-11-07 03:2x AWST (stocktake):** live `pct list` on `m93p` returns **20 LXCs — 18 running,
2 stopped (CT998 `nightbridge` and CT1264 `openmausbot`)**. ⚠ **The count was one short**: the previous
figure of 19 omitted **CT107 (`mem0`)**, a live running guest confirmed from inside the container
(`pct exec 107 -- tailscale ip -4` → `100.73.94.53`, `systemctl is-active tailscaled` → `active`,
`mem0-mcp.service` → `active`). CT107 is now in the table above.
**Re-verified 2026-10-31:** the earlier reading of **19 LXCs — 18 running, 1 stopped
(CT998 nightbridge) plus CT1264 openmausbot stopped** was itself already one short for the same reason
(CT107 `mem0` was not counted). The 2026-10-14 table listed only **17**, omitting
two live guests: **CT105 (`conduit`)** and **CT1264 (`openmausbot`)**. Note that **VMIDs 103, 220 and 555
do not exist on `m93p`** (`/etc/pve/lxc/{103,220,555}.conf` all absent) — CT103 (cliproxyapi),
CT220 (caddy) and CT555 (elementsynapse) are on **`pve`** only, and `pve`'s CT555 is **stopped** while
@@ -162,36 +171,39 @@ tailnet. → [[systems/omniroute]], [[infrastructure/hosts]]
| Datastore | Type | Status | Total | Used | Used % |
|---|---|---|---|---|---|
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 700 GiB | **76.43 %** (`pvesm`) |
| `SSD1` | lvmthin | active | 932 GiB | 599 GiB | **64.32 %** |
| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.0 TiB | **82.77 %** |
| `local` | dir | active | 67.7 GiB | 54.0 GiB | **79.66 %** |
| `local-lvm` | lvmthin | active | 141 GiB | 19.4 GiB | **13.75 %** |
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 717 GiB | **78.29 %** (`pvesm`) |
| `SSD1` | lvmthin | active | 931 GiB | 610 GiB | **65.46 %** |
| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
| `local` | dir | active | 67.7 GiB | 55.3 GiB | **81.62 %** |
| `local-lvm` | lvmthin | active | 141 GiB | 20.1 GiB | **14.25 %** |
**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 %**, ~169 GiB free. `fred-qnap` is now
**82.45 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.0 TiB used) — the share was
**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 % (2026-11-01) → 78.29 % (2026-11-07)**, ~152 GiB free. `fred-qnap` is now
**84.21 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.3 TiB used) — the share was
remounted over NFS rather than SMB.
**Storage re-verified 2026-11-07 03:2x AWST** against live `pvesm status` on both hosts — every
percentage on both tables below was refreshed.
**`iMacHDD` no longer appears in `pvesm status` on `pve`** — the 2026-10-14 table listed it as an
inactive `cifs` entry. The iMac shares now live on **`m93p`** as `cifs-anthonyimac` and `cifs-imachdd`
(both active, 60.25 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing.
(both active, 76.21 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing.
### Host `m93p` (100.104.51.98)
| Datastore | Type | Status | Total | Used | Used % |
|---|---|---|---|---|---|
| `cifs-anthonyimac` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `cifs-imachdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `local` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `local-hdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** |
| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** |
| `nfs-ssd0-39` | dir | active | 916 GiB | 700 GiB | **76.43 %** |
| `omb-internal` | dir | active | 93.9 GiB | 54.4 GiB | **57.92 %** |
| `cifs-anthonyimac` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `cifs-imachdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `local` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `local-hdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
| `nfs-ssd0-39` | dir | active | 916 GiB | 717 GiB | **78.29 %** |
| `omb-internal` | dir | active | 93.9 GiB | 60.7 GiB | **64.66 %** |
**`m93p` storage was not documented before 2026-10-31.** Note that `m93p` exposes the **same backing
volumes as `pve`** under different names — `nfs-proxmox-31`/`nfs-proxmox-54` mirror `pve`'s `fred-qnap`
(both 82.45 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 76.43 %). The iMac shares that were
(both 84.21 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 78.29 %). The iMac shares that were
`iMacHDD` on `pve` are `cifs-anthonyimac` / `cifs-imachdd` on `m93p`.
**Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`).
+16 -15
View File
@@ -3,8 +3,8 @@ title: Tailscale
type: infrastructure
status: active
created: 2026-07-26
updated: 2026-11-06
verified_on: 2026-11-06
updated: 2026-11-07
verified_on: 2026-11-07
confidence: high
tags: [tailscale, vpn, networking, infrastructure]
---
@@ -19,7 +19,8 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
## Tailscale Node Inventory (verified 2026-11-06)
## Tailscale Node Inventory (verified 2026-11-07)
Re-verified 2026-11-07 03:1x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 9 offline-age drifts, no IP changes and no missing peers.** **`octop` came back online**: the peer record reads `Online: true` with `LastSeen 0001-01-01` and `tailscale ping 100.92.163.122` → **`pong from octop via DERP(syd) in 98ms`**, so the row moves offline → online. It remains **not an LXC or VM on either hypervisor** (live `pct list` / `qm list` on both `pve` and `m93p` list no such guest) — an **orphaned tailnet peer**, so its return is not an estate change. The **9 offline ages** were refreshed (+1d each): cloudreve 104d → **105d**; conduit 60d → **61d**; elementsynapse 107d → **108d**; freshrss-new 80d → **81d**; octoagent 4d → **5d**; omarchybook 15d → **16d**; omarchymac 4d → **5d**; openmausbot 15d → **16d**; ubuntu-server-xfce 76d → **77d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) **17 guests** and `m93p` (100.104.51.98) **20 guests + 2 VMs** — ⚠ **the m93p guest count on this page was one short**: live `pct list` on `m93p` returns **20 LXCs** (the extra is **CT107 `mem0`**, `100.73.94.53`, `mem0-mcp.service` active — confirmed from inside the guest), not 19; CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT998 `nightbridge` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). The route table still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; **both DEAD targets re-confirmed dead** — `ss -ltn` still shows **no listener on 8769** (`/agent-estate`) and **none on 9123** (`/synergy-prep`) — while every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` answers **HTTP 200**.
Re-verified 2026-11-06 03:0x–03:2x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 8 offline-age drifts, no IP changes and no missing peers.** **`anthonys-mac-mini` came back online**: the peer record reads `Online: true` and `tailscale ping 100.76.60.37` → `pong from anthonys-mac-mini via DERP(syd) in 103ms`, so the row moves offline → online (client device, not an estate host). The **8 offline ages** were refreshed (+1d each): cloudreve 103d → **104d**; elementsynapse 106d → **107d**; freshrss-new 79d → **80d**; octoagent 3d → **4d**; octop 1d → **2d**; omarchybook 14d → **15d**; omarchymac 3d → **4d**; openmausbot 14d → **15d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). `m93p` re-pinged → `pong from m93p via 192.168.178.250:41641 in 3ms` (online, peer record agrees), and `octop` re-pinged → **timeout** (genuinely offline, matching its `LastSeen`). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) 17 guests and `m93p` (100.104.51.98) 19 guests + 2 VMs — CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). ⚠ **Two CT460 Serve routes are dead but still present in `tailscale serve status`**: `/agent-estate` → `127.0.0.1:8769` and `/synergy-prep` → `127.0.0.1:9123` have **no listener** (live `ss -ltn` probe) and therefore serve 502 — both rows are now marked DEAD here, matching [[infrastructure/network-map]] which had already recorded them as dead on 2026-11-05. The route table itself still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` is reachable.
Full node list from `tailscale status` — **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec <vmid> -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
@@ -44,14 +45,14 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| anthonys-mac-mini | 100.76.60.37 | online — **returned 2026-11-06** (peer record `Online: true`, `tailscale ping` → `pong from anthonys-mac-mini (100.76.60.37) via DERP(syd) in 103ms`; client device, not an estate host) |
| clawtest | 100.122.189.81 | online — CT333 (pve) |
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
| cloudreve | 100.111.187.7 | offline 104d — CT209 stopped |
| conduit | 100.81.79.87 | offline 60d — last seen 2026-09-06 01:32Z |
| cloudreve | 100.111.187.7 | offline 105d — CT209 stopped |
| conduit | 100.81.79.87 | offline 61d — last seen 2026-09-06 01:32Z |
| docker | 100.96.150.69 | online — CT200 (pve) |
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
| elementsynapse | 100.75.250.37 | offline 107d — CT555 stopped |
| elementsynapse | 100.75.250.37 | offline 108d — CT555 stopped |
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
| freshrss-new | 100.110.180.115 | offline 80d |
| freshrss-new | 100.110.180.115 | offline 81d |
| gitea | 100.82.100.128 | online — CT221 (m93p) |
| guest-vps | 100.117.14.84 | online |
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
@@ -65,15 +66,15 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
| obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) |
| octoagent | 100.117.10.68 | offline 4d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
| octop | 100.92.163.122 | offline 2d — last seen 2026-11-03 09:03Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
| octoagent | 100.117.10.68 | offline 5d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
| octop | 100.92.163.122 | online — **returned 2026-11-07** (peer record `Online: true`, `tailscale ping` → `pong from octop via DERP(syd) in 98ms`); still **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
| ollama | 100.108.44.119 | online — CT205 (pve) |
| omarchy | 100.73.227.103 | online |
| omarchybook | 100.107.9.100 | offline 15d |
| omarchymac | 100.126.173.115 | offline 4d — last seen 2026-11-01 11:13Z |
| omarchybook | 100.107.9.100 | offline 16d |
| omarchymac | 100.126.173.115 | offline 5d — last seen 2026-11-01 11:13Z |
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
| openmausbot | 100.91.214.91 | offline 15d — CT1264 (m93p) stopped |
| openmausbot | 100.91.214.91 | offline 16d — CT1264 (m93p) stopped |
| openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) |
| openweb | 100.69.230.13 | online — CT245 (m93p) |
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
@@ -86,7 +87,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
| springroll | 100.100.153.125 | online |
| springroll-browser | 100.109.221.91 | online |
| ubuntu-server-xfce | 100.101.26.104 | offline 76d — last seen 2026-08-20 19:36Z |
| ubuntu-server-xfce | 100.101.26.104 | offline 77d — last seen 2026-08-20 19:36Z |
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
| webserver | 100.82.40.23 | online — CT207 (pve) |
@@ -106,8 +107,8 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| `/gitea` | `http://100.82.100.128:3000` | Gitea |
| `/desktop` | `http://127.0.0.1:6081` | CUA noVNC |
| `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC |
| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep — ⚠ **DEAD: nothing listening on 9123** (live `ss -ltn` probe 2026-11-06; the route still exists in `tailscale serve status` but serves 502) |
| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) — ⚠ **DEAD: nothing listening on 8769** (live `ss -ltn` probe 2026-11-06; route still present in `tailscale serve status` but serves 502) |
| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep — ⚠ **DEAD: nothing listening on 9123** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07**; the route still exists in `tailscale serve status` but serves 502) |
| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) — ⚠ **DEAD: nothing listening on 8769** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07**; route still present in `tailscale serve status` but serves 502) |
| `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid |
Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 route table above matched live exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; no drift).
+104
View File
@@ -2226,3 +2226,107 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
[[current-state]] (+frontmatter, +Recent Changes block), this index (+frontmatter, +summary, +incident
bullet), [[runbooks/job-radar]] (+concurrency subsection), this log, and the new incident page. Vault
mirror `/home/hermes/wiki` updated (both conventions).
## 2026-11-07 03:0x AWST - stocktake (wiki-stocktake-maintenance)
- **Script:** `wiki-maintain.sh` (repo, canonical) run clean — skills-index regenerated (**302 skills /
68 categories**), freshness report written to `raw/stocktake/freshness-20261107.txt`, Quartz rebuilt
(**769 files**) and deployed to `public/`, `hermes-wiki-serve.service` restarted.
- **Freshness:** **0 stale, 125 fresh, 3 no-date of 128 pages** — no page is past the 21-day threshold.
(`log.md` has no `updated` date and lands in the no-date bucket, which is correct for an append-only log.)
**Freshness is not accuracy**, so a live-vs-page diff was run against every page carrying live claims.
- **Tailscale (infrastructure/tailscale.md):** table rebuilt from `tailscale status --json`. **1 state
transition — `octop` (`100.92.163.122`) is back online** (peer record `Online: true`; `tailscale ping` →
`pong from octop via DERP(syd) in 98ms`); it is still **no LXC or VM on either hypervisor**, so it is an
orphaned tailnet peer, not an estate change. **9 offline ages refreshed (+1d each).** All **51 live IPv4
peers** match the table's IPs exactly — **no missing peers, no orphan rows, no IP mismatches** (52 rows =
51 peers + `hermes`; the ~23 IPv6-only `funnel-ingress-node` peers excluded by design). The CT460 Serve
route table still matches `tailscale serve status` **exactly (12 paths on :443)**; **both DEAD targets
re-confirmed dead** (no listener on `8769` `/agent-estate`, none on `9123` `/synergy-prep`), every other
Serve target has a live listener, and `/gitea` (`100.82.100.128:3000`) answers **HTTP 200**.
- **Proxmox (infrastructure/proxmox.md):** ⚠ **`m93p` runs 20 LXCs, not 19 — CT107 `mem0` was missing**
(running; `mem0-mcp.service` active on `:8890`; `pct exec 107 -- tailscale ip -4` → `100.73.94.53`).
Added to the table; the 2026-10-31 reading of 19 was already short for the same reason. Live now reads
**20 LXCs — 18 running, 2 stopped** (CT998, CT1264). `pve` still matches exactly: **17 LXCs + 4 VMs**.
Both hosts run the identical **`pve-manager 9.2.10`** build.
- **⚠ `m93p` uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*; live
`uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18**, load **0.76 / 1.08 / 1.24** — normal).
The 2026-10-30 reboot was real but belonged to **`pve`** (boot **2026-10-30 16:24:00**).
- **Storage refreshed on both hosts** against live `pvesm status`: `pve` `ssd0` **76.43 → 78.29 %**
(~152 GiB free), `SSD1` **64.32 → 65.46 %**, `fred-qnap` **82.77 → 84.21 %**, `local` **79.66 → 81.62 %**,
`local-lvm` **13.75 → 14.25 %**; `m93p` iMac shares + `local` **60.25 → 76.21 %**, `nfs-proxmox-*`
**82.78 → 84.21 %**, `nfs-ssd0-39` **76.43 → 78.29 %**, `omb-internal` **57.92 → 64.66 %**.
- **Docker (infrastructure/docker-services.md):** set-wise diff against live `docker ps` on CT200/201/333 —
CT200 **61 running / 68 total** exact, CT201 **20 / 26** exact, CT333 **8 / 9** exact, **but CT333's
enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name
**`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected. CT460 `ignis` running.
- **Cross-page corrections:** [[infrastructure/network-map]] `/gitea` was annotated *"Gitea (ct207 tailnet
IP)"* — it is **CT221 on `m93p`** (ct207 is `100.82.40.23`); also its `m93p` PVE version read *9.2.0*
(both hosts are 9.2.10). [[agents/hermes-chaos]] read *"`v0.22.3` … `5b147ab036d6` — the same build as
CT460"*, which contradicted itself: live is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to
CT460's (`hermes update --check` reports **v0.23.0** available). [[agents/hermes-production]]'s
*"2 cron jobs still pin `custom:omni`"* is **no longer true** — live `jobs.json` has **0** on the bare key
and **21** on `custom:omni-101`.
- **Fleet re-verified unchanged:** CT460 **29 jobs (26 enabled / 3 disabled)**; CT450 **25 jobs
(13 enabled / 12 disabled)** with **9 shared names** — confirmed by a live set-diff of both `jobs.json`.
A keyed `deepseek-paid-smart` chat probe against `omniroute-1` (`100.78.63.20:20128`) returns **HTTP 200**;
`config.yaml` still defaults to `custom:omni-101`.
- **Pages touched:** [[infrastructure/tailscale]], [[infrastructure/proxmox]], [[infrastructure/hosts]],
[[infrastructure/network-map]], [[infrastructure/docker-services]], [[agents/hermes-production]],
[[agents/hermes-chaos]], [[current-state]], [[index]], this log. No new incident page — no incident was
found; every drift above was a documentation drift corrected against a live probe.
## 2026-11-07 05:1x AWST - incident resolution: Gitea config backup push dead since 2026-08-08
- **Anthony's ask:** "fix this" on the failing backup alert — with a pointer to check Notion for the API key.
- **[[incidents/2026-08-08-gitea-config-backup-push-dead]] is now `status: resolved`.** The nightly
`e27da6c34bbb` config backup had been committing locally and failing to push **every night since
2026-08-08** — **92 commits** stranded — while its cron row read `last_status: ok`.
- **Four independent root causes, all confirmed against live systems:**
1. **No Gitea credential.** `origin` was HTTPS with no userinfo; the global `credential.helper=store`
file `~/.git-credentials` held only a `github.com` entry. The token in the Notion vault was
**valid** and byte-identical to the one the wiki repo carries in its remote URL — it simply was
never wired into the backup repo. Fixed with a host-scoped `git credential approve` entry (not
embedded in the URL); `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`.
2. **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's
`success = (returncode == 0)` recorded `ok` on every failed night. The scheduler was correct;
**the script was the liar**. This is why the outage ran 92 days without a job-level red.
3. **A 1.76 GB `checkpoints/` backlog** in the unpushed range (13 blobs > 100 MB) — junk that
`.gitignore` already excluded for *new* commits but that history still carried.
4. **Both secondary legs broken.** The `github` remote did not exist, and the fallback pushed a
branch's **full ancestry**, which contains a 7.45 GB tarball blob — so GitHub's 100 MB file
limit could never accept it. The AgentMail alert called a `python3 send_email.py` that no longer
exists, via a `chaos-agent@agentmail.to` inbox that no longer exists.
- **Backlog drained:** `git filter-repo --partial --refs origin/main..main --path checkpoints
--invert-paths --prune-empty never --force` stripped `checkpoints/` from the **unpushed range only**
(`origin/main` untouched, push stayed a fast-forward), then `checkpoints/` was untracked at the tip.
Push set **~2 GB → 264.8 MB**; 93 commits pushed in 11.6 s; the 7.45 GB tarball was already on
`origin/main` and never resent. Pre-rewrite history preserved under tag
`backup/pre-rewrite-20261107-032658`. **Token leakage scan over the HEAD tree, the working tree and
every unpushed blob: clean.**
- **Script rewritten** (`~/.hermes/config-backup/backup.sh`, via OpenCode + a correction pass): exits
**0 only on verified success**; stamps `.last_backup_ok` **only** after remote verification; decides
the push by **remote-vs-local** rather than by working-tree changes (the exact mechanism that created
the backlog); guards every `git ls-remote` with `|| true` so an unreachable remote reaches
`notify_failure` instead of dying at exit 128 *before* the alert; pushes GitHub as a **parentless
orphan snapshot** (`git commit-tree` from `HEAD^{tree}` — no history, no >100 MB blob, HEAD/branch/
index untouched) as a warning-only secondary target; and reports Telegram/AgentMail delivery
**honestly** (`curl --fail` + `"ok":true`; the real `agentmail` CLI, key read from `.env` at runtime
and passed via the environment, never argv).
- **Verified live 2026-11-07:** real run exit **0** in 47 s; `git rev-parse HEAD` ==
`git ls-remote origin refs/heads/main` == `a50a5170c1c0e2af5235fa63a95a732e2d780703`;
`rev-list --count origin/main..main` = **0**; Gitea API `main` = `a50a5170c1c0`, updated
`2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub
mirror now carries a **0-parent** orphan snapshot `faed9fb5c677` (30,086 files, largest blob 23.5 MB)
with `master` untouched at `c99c94e6b13b` — **no data lost**, since GitHub's tree was a strict subset
of local HEAD (only `BACKUP-INDEX.md` added). AgentMail and Telegram alert paths each exercised
directly (AgentMail self-test to the agent's own inbox; Telegram `getMe`, no message sent to Anthony).
Sandbox E2E: unreachable remote → exit 1 + alert + health **not** stamped; local-ahead-with-clean-tree
→ pushed + verified; already-up-to-date → no push, exit 0.
- **Pages touched:** [[incidents/2026-08-08-gitea-config-backup-push-dead]] (status → resolved, full
Resolution section added), [[infrastructure/backups]] (⚠ → ✅, re-verified 2026-11-07),
[[systems/scheduled-tasks]] (job row → RESOLVED), this log.
- **Note — unrelated pre-existing state:** the 03:00 `wiki-stocktake-maintenance` slot **failed** with
`RuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership`
(failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are
included in this commit so the tree is not left dirty.
+7 -7
View File
@@ -7,8 +7,8 @@ Threshold: >21 days since `updated`
## FRESH (125)
- [ 18d] CONTRIBUTING.md
- [ 12d] agents/hermes-chaos.md
- [ 12d] agents/hermes-production.md
- [ 0d] agents/hermes-chaos.md
- [ 0d] agents/hermes-production.md
- [ 18d] agents/index.md
- [ 9d] agents/nanobot.md
- [ 18d] agents/open-claw.md
@@ -77,12 +77,12 @@ Threshold: >21 days since `updated`
- [ 0d] index.md
- [ 3d] infrastructure/backups.md
- [ 18d] infrastructure/credentials-map.md
- [ 1d] infrastructure/docker-services.md
- [ 0d] infrastructure/docker-services.md
- [ 18d] infrastructure/domains-and-tunnels.md
- [ 2d] infrastructure/hosts.md
- [ 1d] infrastructure/network-map.md
- [ 6d] infrastructure/proxmox.md
- [ 1d] infrastructure/tailscale.md
- [ 0d] infrastructure/hosts.md
- [ 0d] infrastructure/network-map.md
- [ 0d] infrastructure/proxmox.md
- [ 0d] infrastructure/tailscale.md
- [ 3d] queries/retained-research-answers.md
- [ 13d] reports/communities-wa-social-media-audit-2026.md
- [ 6d] reports/uber-eats/2026-07.md
+1 -1
View File
@@ -215,7 +215,7 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text |
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken. ⚠ **The 2026-10-17 03:00 slot failed** on the same `omni-101` 503 — the first miss since the **2026-10-15** storm slot (which died with `RuntimeError: Connection error.`; the 10-16 slot succeeded). `raw/stocktake/freshness-20261017.txt` was not written and `systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 slot succeeded, so `freshness-20261016.txt` exists); the **2026-10-18 03:00** slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident `1a8e90_3c9c1d573cde` `detected`. Note the ledger tags this row `failure_type: unknown` even though the execution carries an explicit 503 message |
| 🦝 Hermes Config Backup to Gitea (Daily) ⚠ **push has failed every night since 2026-08-08 — 75 local commits unpushed, job still reads `ok`** → [[incidents/2026-08-08-gitea-config-backup-push-dead]] | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
| 🦝 Hermes Config Backup to Gitea (Daily) ✅ **RESOLVED 2026-11-07** — push live again, 92-commit backlog drained and pushed, script hardened to exit non-zero on failure so the job row can no longer read `ok` while the push is dead. Live run exit 0; `origin/main` == local HEAD (`a50a5170c1c0`); `rev-list --count origin/main..main` = 0. → [[incidents/2026-08-08-gitea-config-backup-push-dead]] | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
## Removed Jobs
| Name | Removed | Reason |