diff --git a/agents/hermes-chaos.md b/agents/hermes-chaos.md index bad6928..bd2df73 100644 --- a/agents/hermes-chaos.md +++ b/agents/hermes-chaos.md @@ -3,8 +3,8 @@ title: Hermes Chaos type: system status: active created: 2026-08-15 -updated: 2026-10-26 -verified_on: 2026-10-26 +updated: 2026-11-07 +verified_on: 2026-11-07 confidence: high tags: [hermes, chaos, ct450, experimental, agent] sources: [live-probe, config-inspection] @@ -28,10 +28,10 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod ## Current State - **Status:** Active (experimental) -- **Hermes version:** `v0.22.3` (2026.9.24), local `5b147ab036d6` — **the same build as CT460** (updated 2026-10-21). **0 commits behind `origin/main`** (verified 2026-10-26; the old "3 commits behind" claim is stale), though `hermes update --check` reports stable **v0.22.5** available. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]]. +- **Hermes version:** `v0.22.5` (2026.9.24), local `bf0203a794a0` — **the same build as CT460** (updated 2026-10-21). ⚠ **Corrected 2026-11-07:** this line still read *"`v0.22.3` … local `5b147ab036d6`"* while asserting it was *"the same build as CT460"* — it was not. Live `hermes --version` on CT450 now returns **`v0.22.5` … local `bf0203a794a0`**, byte-identical to CT460's own `hermes --version`, so the *same build* claim is finally true. `hermes update --check` reports stable **v0.23.0** (`2bd378ed1b8d`) available. Python 3.14.9. See [[incidents/2026-10-03-ct450-hermes-update]]. - **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]]. - **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01) -- **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` 2026-10-26). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460) +- **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` **2026-11-07**; unchanged). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460) — both confirmed by a live set-diff of the two `jobs.json` files on 2026-11-07 - **Production counterparts:** CT460 has **28 jobs / 25 enabled** — the two fleets no longer share a job count, and the CT450 duplicates are paused ## Providers & Routing diff --git a/agents/hermes-production.md b/agents/hermes-production.md index bc71abe..36155cb 100644 --- a/agents/hermes-production.md +++ b/agents/hermes-production.md @@ -3,8 +3,8 @@ title: Hermes Production type: system status: active created: 2026-08-15 -updated: 2026-10-26 -verified_on: 2026-10-26 +updated: 2026-11-07 +verified_on: 2026-11-07 confidence: high tags: [hermes, production, ct460, agent] sources: [live-probe, config-inspection] @@ -43,7 +43,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn ## Providers & Routing -- **Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, CT101/m93p) — verified live 2026-10-26. The `custom:omni` provider key does not exist in `custom_providers`, but **2 cron jobs still pin it** (see [[systems/scheduled-tasks]]) +- **Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, CT101/m93p) — verified live 2026-10-26, **re-verified 2026-11-07** (live `config.yaml` `model.default.provider: custom:omni-101`; a keyed `deepseek-paid-smart` chat probe against `100.78.63.20:20128` returns **HTTP 200**). The `custom:omni` provider key does not exist in `custom_providers`. ⚠ **Corrected 2026-11-07:** the old claim that *"2 cron jobs still pin `custom:omni`"* is **no longer true** — live `jobs.json` shows **no job pinned to the bare `custom:omni`**; **all 21 provider-pinned jobs use `custom:omni-101`** (see [[systems/scheduled-tasks]]) - **Default model:** `smart-route` (previously `freellmapi-auto`) - **Fallback providers** (live chain, re-verified 2026-10-26): `nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6_1t`. ⚠ The **first** entry is the dead `nous` credential — a fallover tries it first - **Auxiliary models:** all eight `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) are **`no-think/freeapi/auto` on `custom:omni-101`** with a `freeapi/deepseek-v4-flash` → `kg/kilo-auto/free` fallback chain (moved 2026-10-25 19:13) @@ -74,7 +74,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn | Aspect | Production (CT460) | Chaos (CT450) | |--------|-------------------|---------------| | Purpose | Stable daily operations | Experimental/testing | -| Cron jobs | 28 (25 enabled / 3 disabled) | 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460 | +| Cron jobs | **29 (26 enabled / 3 disabled)** (re-verified 2026-11-07) | 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460 (re-verified 2026-11-07) | | Config changes | Requires verification | Safe to experiment | | Memory | Production Mnemosyne | Separate state | diff --git a/current-state.md b/current-state.md index 13fd9e8..0239de1 100644 --- a/current-state.md +++ b/current-state.md @@ -14,6 +14,45 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv # Current State Updated: 2026-11-07 +## Recent Changes (2026-11-07, stocktake; live probes 2026-11-07 03:1x-03:2x AWST) + +- **The tailnet lost a peer and the m93p guest list lost a guest — both corrections came from live probes, not the age scan.** + [[infrastructure/tailscale]] was rebuilt from `tailscale status --json`: **`octop` (`100.92.163.122`) is back + online** (peer record `Online: true`, `tailscale ping` → `pong from octop via DERP(syd) in 98ms`; it remains + **no LXC or VM on either hypervisor** — an orphaned tailnet peer), and the **9 remaining offline ages** were + refreshed (+1d each). All **51 live IPv4 peers** match the table's IPs exactly, with no missing peers and no + orphan rows (**52 rows** = 51 peers + `hermes` itself). The CT460 Serve route table still matches + `tailscale serve status` **exactly (12 paths on :443)**; both previously-flagged dead targets were + **re-confirmed dead** — still no listener on `8769` (`/agent-estate`) or `9123` (`/synergy-prep`). +- **⚠ `m93p` actually runs 20 LXCs, not 19 — CT107 (`mem0`) was missing from the Proxmox page.** + Live `pct list` on `m93p` returns **20 LXCs — 18 running, 2 stopped** (CT998 `nightbridge`, CT1264 + `openmausbot`). The omitted guest is **CT107 `mem0`** (`mem0-mcp.service` active on `:8890`, + `pct exec 107 -- tailscale ip -4` → `100.73.94.53`). Added to [[infrastructure/proxmox]]; the + **2026-10-31 reading of 19 was already short for the same reason**. `pve` still returns exactly + **17 LXCs + 4 VMs**, and both hosts still run the identical `pve-manager 9.2.10` build. +- **⚠ `m93p`'s uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*. + Live `uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18** at probe time, load + **0.76 / 1.08 / 1.24** — normal). The 2026-10-30 reboot was real but belonged to **`pve`** + (boot **2026-10-30 16:24:00**), not `m93p`. +- **Storage percentages refreshed on both hosts** against live `pvesm status` — `pve`'s `ssd0` moved + **76.43 % → 78.29 %** (~152 GiB free), `fred-qnap` **82.77 % → 84.21 %**, `local` **79.66 % → 81.62 %**; + on `m93p` the iMac shares and `local` moved **60.25 % → 76.21 %** and `omb-internal` + **57.92 % → 64.66 %**. +- **Docker inventories diffed set-wise against live `docker ps` on all three hosts.** CT200 matches + **61 running / 68 total** exactly; CT201 matches **20 / 26** exactly; CT333 matches **8 / 9** — but its + **enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name is + **`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected in [[infrastructure/docker-services]]. +- **Two small cross-page corrections.** [[infrastructure/network-map]]'s `/gitea` route was annotated + *"Gitea (ct207 tailnet IP)"* — it is **CT221 on `m93p`** (`100.82.100.128`; ct207's own IP is + `100.82.40.23`); the target answers **HTTP 200**. [[agents/hermes-chaos]]'s version line read + *"`v0.22.3` … local `5b147ab036d6` — the same build as CT460"*, which contradicted itself: live + `hermes --version` on CT450 is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to CT460's. + [[agents/hermes-production]]'s claim that *2 cron jobs still pin `custom:omni`* is **no longer true** — + live `jobs.json` shows **0** jobs on the bare key and **21** on `custom:omni-101`. +- **Cron fleet re-verified unchanged:** **29 jobs (26 enabled / 3 disabled)**, matching the nightly audit + hours earlier. CT450's fleet is **25 jobs (13 enabled / 12 disabled)** with **9 shared names** — confirmed + by a live set-diff of both `jobs.json` files. → [[systems/scheduled-tasks]] + ## Recent Changes (2026-11-07, nightly audit; live probes 2026-11-07 00:3x-00:5x AWST) - **A red row with no alert: the Job Radar's 11:30 slot was discarded as stale while a concurrent fire diff --git a/incidents/2026-08-08-gitea-config-backup-push-dead.md b/incidents/2026-08-08-gitea-config-backup-push-dead.md index c1ea6c0..d862a92 100644 --- a/incidents/2026-08-08-gitea-config-backup-push-dead.md +++ b/incidents/2026-08-08-gitea-config-backup-push-dead.md @@ -1,18 +1,23 @@ --- -title: Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 While the Job Reads `ok` +title: Hermes Config Backup to Gitea — Push Was Dead 2026-08-08 → 2026-11-07 While the Job Read `ok` type: incident -status: active +status: resolved created: 2026-10-21 -updated: 2026-10-22 -verified_on: 2026-10-22 -review_after: 2026-10-29 +updated: 2026-11-07 +resolved_on: 2026-11-07 +verified_on: 2026-11-07 +review_after: 2027-02-07 confidence: high tags: [backup, gitea, cron, git, green-vs-actual] -sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki] +sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki, gitea-api, github-api, sandbox-e2e-test] --- # Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08 +> **RESOLVED 2026-11-07.** The push is live again, the backlog is drained, and the script now exits +> non-zero on failure so the cron row cannot lie. See [Resolution 2026-11-07](#resolution-2026-11-07) +> at the foot of this page for the full record. The original investigation is preserved below. + ## Headline The daily **Hermes Config Backup to Gitea** job (`e27da6c34bbb`, `0 11 * * *`, no-agent, `hermes-config-backup.sh`) @@ -103,3 +108,92 @@ change as well the *only* off-box copy path was this dead push → [[infrastruct - [[infrastructure/backups]] - [[systems/scheduled-tasks]] - [[systems/hermes-agent]] + +--- + +# Resolution 2026-11-07 + +Anthony asked to fix the failing push (and to check the Notion vault for the API key). Resolved in +one session: the push is live, the backlog is drained, and the script can no longer report a false green. + +## Root causes (four, all independent) + +1. **No Gitea credential.** `origin` was HTTPS with no userinfo, and the global `credential.helper=store` + file `~/.git-credentials` held only a `github.com` entry — so the cron worker could not authenticate + non-interactively. The token in the Notion vault was valid and identical to the one the wiki repo + carries in its remote URL; it simply was never wired into the backup repo. +2. **`backup.sh` had no `exit` statement at all.** It always returned 0, so the scheduler + (`success = (returncode == 0)` in `cron/scheduler_script.py`) recorded `last_status: ok` on every + failed night. This is why the outage went unnoticed for 92 days — it was a *false green*, not a + missing signal. (The incident's earlier note blamed the scheduler; the scheduler was correct. The + script was the liar.) +3. **A 1.76 GB `checkpoints/` backlog** sat in the unpushed range — 13 blobs over 100 MB, all junk + that `.gitignore` already excluded for new commits but that history still carried. +4. **Two broken secondary legs.** The `github` remote did not exist, and the fallback + `_github_clean_push` pushed a branch's full ancestry — which contains a 7.45 GB tarball blob, so + GitHub's 100 MB file limit could never accept it. The AgentMail alert called a + `python3 send_email.py` that no longer exists, via a `chaos-agent@agentmail.to` inbox that no + longer exists. + +## Fix applied + +- **Credential:** stored a host-scoped Gitea entry via `git credential approve` (not embedded in the + remote URL). `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`. +- **Backlog:** `git filter-repo --partial --refs origin/main..main --path checkpoints --invert-paths + --prune-empty never --force` stripped `checkpoints/` from the unpushed range only (so `origin/main` + was untouched and the push stayed a fast-forward), then `checkpoints/` was untracked at the tip. + Pre-rewrite history is preserved under the tag `backup/pre-rewrite-20261107-032658`. + **Push set: ~2 GB → 264.8 MB.** The 7.45 GB tarball was already on `origin/main` and was never resent. +- **Script hardening** (`~/.hermes/config-backup/backup.sh`, rewritten via OpenCode + a correction pass): + - exits **0 only on verified success**, `1` otherwise — the cron row can no longer lie; + - `.last_backup_ok` is stamped **only** after the remote is confirmed to hold the commit; + - the push decision compares **remote vs local**, not "were there new working-tree changes" — a + local-ahead-of-remote with a clean tree now still pushes (this is exactly the mechanism that + created the 92-commit backlog); + - every `git ls-remote` is guarded with `|| true`, so an unreachable remote reaches the alert path + instead of killing the script at exit 128 before `notify_failure` fires; + - the GitHub leg now pushes a **parentless orphan snapshot** built from `HEAD^{tree}` via + `git commit-tree` — no history, no >100 MB blob, HEAD/branch/index untouched — and is a + warning-only secondary target that can never fail the job; + - the AgentMail leg uses the real CLI (`agentmail inboxes:messages send --inbox-id `) with the + key read from `.env` at runtime and passed via the environment, never argv; + - the Telegram leg uses `curl --fail` and requires `"ok":true` in the body, so it reports `sent` + only on a real delivery. + +## Verification (live, 2026-11-07 AWST) + +| Check | Result | +|---|---| +| Real `backup.sh` run | exit **0**, 47 s | +| `git rev-parse HEAD` vs `git ls-remote origin refs/heads/main` | identical — `a50a5170c1c0e2af5235fa63a95a732e2d780703` | +| `git rev-list --count origin/main..main` | **0** (was 92) | +| Gitea API repo `Anthony/hermes-config-backup` | HTTP 200, default branch `main`, updated `2026-11-07T05:10:46+08:00`, 8.1 MB | +| Gitea API `branches/main` | `a50a5170c1c0` — `🦝 nightly backup: 2026-11-07 05:10:33 AWST` | +| `.last_backup_ok` | refreshed to the run timestamp (was stale from 2026-11-06 11:00) | +| GitHub `main` | `faed9fb5c677`, **0 parents** (orphan snapshot confirmed), 30,086 files, largest blob 23.5 MB | +| GitHub `master` branch | untouched at `c99c94e6b13b` | +| GitHub force-push data loss | none — GitHub's tree was a strict **subset** of local HEAD (only `BACKUP-INDEX.md` added) | +| Token leakage scan (HEAD tree, working tree, all unpushed blobs) | **clean** — token absent everywhere | +| AgentMail leg (self-test to the agent's own inbox) | delivered, `message_id` returned | +| Telegram leg (`getMe`, no message sent to Anthony) | `"ok":true` detected → would report `sent` | +| Sandbox E2E: unreachable remote | exit **1**, `notify_failure` fired, health file **not** stamped | +| Sandbox E2E: local ahead of remote, clean working tree | pushed + verified, exit **0**, health stamped | +| Sandbox E2E: already up to date | no push, exit **0**, health stamped | + +## Verification command set (for the next audit) + +``` +git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s' +git -C ~/.hermes/config-backup rev-list --count origin/main..main +tail -20 ~/.hermes/config-backup/backup.log +tail -3 ~/.hermes/config-backup/.backup_history +``` + +## Still open (not part of this incident) + +- Gitea host SSH login is denied for `anthonymau` — free disk headroom on the Gitea box is still + unverified (the Gitea API reports repo size, not host free space). +- `hermes-gateway-runtime-health.service` still fails every 5 min because the Mattermost host + `100.98.116.14` has been offline since ~Sept 2026. Separate incident. +- The `checkpoints/` directory is still excluded from backups by policy; the live snapshots remain + at `~/.hermes/checkpoints/` on-box only. diff --git a/index.md b/index.md index dfd0f6c..bcdb41a 100644 --- a/index.md +++ b/index.md @@ -28,7 +28,7 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi > **2026-11-04 (nightly audit) — the ledger raised its own red row:** the **11-03 fleet settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) — **unclosed incident rows 0 → 1**, non-clean `last_status` **1 → 2**, deliveries **7 rows / 7 `delivered`**. The single failure is the **Job Radar Stage 1** (`dca8482e4f76`), which ran **07:00:33 → 13:28:52 (6 h 28 m 18 s)** and died on **`RuntimeError: stream active timeout`** — a **new terminal signature**, not the iteration cap — after going **silent for 65 m 21 s**; its incident `dca848_dfd4bea84f2a` (`timeout`) is **`alerted` and open** and the notice **was delivered** (13:28:51.741724) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. That answers the 11-02 page's open question: **output is suppressed when the run ends `completed`, delivered when it ends `failed`** — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so **2026-11-02's and 2026-11-03's candidate sets are both unrecovered**. ⚠ **The 03:00 stocktake overran for a second day** (`1a8e90ea2271`, 03:00:30 → 10:41:13 = **7 h 40 m 43 s**, `suppressed`, nothing delivered, artefacts uncommitted again). **New page:** [[systems/hermes-hq-push]] (v0.9.27, enabled; manifest says it **replaces `dispatch-push`**, and no trace of the old plugin survives). > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered -> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. +> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.10), was discovered during the 2026-10-14 stocktake. > Guest IDs collide between the two hosts. ## Start here @@ -125,8 +125,8 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi - [[infrastructure/backups]] — hermes backup CLI, retention, and gaps ### Networking -- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 9.2.10 + `m93p` 9.2.10, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 76.43 % / ~169 GiB free, 2026-10-31) -- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **50-node** tailnet inventory (re-verified 2026-10-31: `mattermost` back **online** after ~40 days; 7 offline ages refreshed) +- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 9.2.10 + `m93p` 9.2.10, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 78.29 % / ~152 GiB free, 2026-11-07) +- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **52-node** tailnet inventory (re-verified 2026-11-07: `octop` back **online**; 9 offline ages refreshed; **CT107 `mem0`** added to the m93p guest list) - [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly ## Concepts diff --git a/infrastructure/backups.md b/infrastructure/backups.md index 7581b88..8d30d68 100644 --- a/infrastructure/backups.md +++ b/infrastructure/backups.md @@ -3,8 +3,8 @@ title: Backups type: infrastructure status: active created: 2026-07-22 -updated: 2026-11-04 -verified_on: 2026-11-04 +updated: 2026-11-07 +verified_on: 2026-11-07 confidence: medium tags: [infrastructure, backups] sources: [raw/configs/hermes-config-sanitized.txt] @@ -54,7 +54,7 @@ How Hermes's own configuration/state gets backed up, and what's missing. local snapshot and no off-box copy** — the Gitea mirror below has been dead since 2026-08-08. → [[systems/hermes-agent]] -## ⚠ Scheduled config backup — the Gitea mirror has been dead since 2026-08-08 +## ✅ Scheduled config backup — RESOLVED 2026-11-07 (was dead since 2026-08-08) `e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its @@ -66,31 +66,41 @@ helper — so the non-interactive push dies on `--- Backup complete: FAIL ---`. Full evidence and the proposed fix: [[incidents/2026-08-08-gitea-config-backup-push-dead]]. -**Consequence for this page: everything above describes a topology that has not carried `~/.hermes` off the box -since 2026-08-07.** The 2026-07-22 full zip and the wiki/authoring repos remain the only recent off-box copies. -Telegram failure notices *have* been firing daily (last: 2026-10-21 11:00:14, `message_id 8449`); the AgentMail -notification step failed. +> **RESOLVED 2026-11-07.** The 92-commit backlog is drained and pushed to Gitea, and the script has been +> hardened so the cron row can no longer report a false green. Full record: +> [[incidents/2026-08-08-gitea-config-backup-push-dead#Resolution 2026-11-07]]. -**Re-verified 2026-11-04 — still dead, and the drift is now 89 commits.** Live `git -C ~/.hermes/config-backup`: -`origin` = `https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` (**unchanged** — still HTTPS with -no userinfo and no credential helper, so the credential root cause has not been addressed); `rev-list --count -origin/main..HEAD` = **89**; `log origin/main -1` = **2026-08-07 11:00:47 +0800 1e1eb208** (`🦝 nightly backup: -2026-08-07 11:00:29 AWST`) — i.e. **nothing has reached Gitea in 89 days**. `backup.log` confirms the nightly -failure is current, not historical: the last two runs read -`[2026-11-02 11:00:32 AWST] ✗ Push to Gitea FAILED` / `--- Backup complete: FAIL ---` and -`[2026-11-03 11:00:31 AWST] ✗ Push to Gitea FAILED` / `--- Backup complete: FAIL ---`. -**A second, separate push failure is now also present:** the script attempts a `github` remote that **does not -exist** — `fatal: 'github' does not appear to be a git repository` → `✗ Push to GitHub FAILED`. So both off-box -targets are broken; the GitHub leg is a missing-remote error rather than a credential error, and would need -`git remote add github …` with credentials before it can work at all. -**The cron row still lies:** `e27da6c34bbb` reads `enabled: true, last_status: ok` in `jobs.json` while every -nightly execution has failed the push — the classic green-row/dead-artifact pattern. +**What was wrong (four independent causes):** (1) no Gitea credential — `origin` was HTTPS and +`~/.git-credentials` held only a `github.com` entry, so the non-interactive push could not authenticate; +(2) **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's +`success = (returncode == 0)` recorded `ok` on every failed night (the scheduler was correct; the script +was the liar); (3) a 1.76 GB `checkpoints/` backlog sat in the unpushed range, 13 blobs over 100 MB; +(4) both secondary legs were broken — the `github` remote did not exist, and its fallback pushed a +branch's full ancestry containing a 7.45 GB tarball blob that GitHub's 100 MB limit could never accept. -**Full-backup gap unchanged:** the one full `hermes backup` zip on disk is still -`/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, 22 Jul) — **now 105 days old**. No new full zip has -been taken. `crontab -l -u hermes` is still empty and there are still **0 user systemd timers**, so nothing -schedules the full zip; only `e5ed215fc91c` (**Mnemosyne DB Backup**, weekly `0 3 * * 0`) and `e27da6c34bbb` -(config backup, daily) exist in the cron fleet, and neither takes a full `hermes backup`. +**What was done:** host-scoped Gitea credential stored via `git credential approve`; `checkpoints/` +stripped from the unpushed range with `git filter-repo --partial` (push set ~2 GB → 264.8 MB, pre-rewrite +history preserved under tag `backup/pre-rewrite-20261107-032658`); the script rewritten to exit non-zero on +failure, stamp `.last_backup_ok` only after remote verification, decide the push by **remote-vs-local** +rather than by working-tree changes, guard every `ls-remote` with `|| true` so an unreachable remote still +reaches the alert path, push GitHub as a **parentless orphan snapshot**, and report Telegram/AgentMail +delivery honestly (`curl --fail` + `"ok":true`; real `agentmail` CLI). + +**Re-verified 2026-11-07 — live and green.** Real run exit **0** in 47 s; `git rev-parse HEAD` and +`git ls-remote origin refs/heads/main` both `a50a5170c1c0e2af5235fa63a95a732e2d780703`; +`rev-list --count origin/main..main` = **0**; Gitea API confirms `main` at `a50a5170c1c0`, updated +`2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub mirror +now carries a parentless orphan snapshot (`faed9fb5c677`, 0 parents, 30,086 files, largest blob 23.5 MB) — +no data lost, since GitHub's tree was a strict subset of local HEAD. The AgentMail and Telegram alert +paths were each exercised directly and report honestly. + +**Consequence for this page: the topology above carries `~/.hermes` off the box again.** The one full +`hermes backup` zip on disk is still `/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, 22 Jul) — +**now 108 days old**, and no new full zip has been taken. `crontab -l -u hermes` is still empty and there +are still **0 user systemd timers**, so nothing schedules the full zip; only `e5ed215fc91c` +(**Mnemosyne DB Backup**, weekly `0 3 * * 0`) and `e27da6c34bbb` (config backup, daily) exist in the cron +fleet, and neither takes a full `hermes backup`. The nightly config backup is a *config* mirror, not a +full-home backup — the gap on the full zip is unchanged. ## Relevant runbooks - [[runbooks/backup-wiki]] diff --git a/infrastructure/docker-services.md b/infrastructure/docker-services.md index 0eb642d..881c8dc 100644 --- a/infrastructure/docker-services.md +++ b/infrastructure/docker-services.md @@ -3,8 +3,8 @@ title: Docker Services type: infrastructure status: mixed created: 2026-07-22 -updated: 2026-11-06 -verified_on: 2026-11-06 +updated: 2026-11-07 +verified_on: 2026-11-07 confidence: high tags: [docker, homelab, infrastructure] sources: [raw/configs/hermes-config-sanitized.txt, live-probe] @@ -20,13 +20,13 @@ Docker runs on **four hosts**, and CT460's socket is not readable by the agent u | Host | Access | Containers running | |---|---|---| -| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** running / 68 total — main Docker host (**re-verified 2026-11-06**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) | -| CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** running / 26 total — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**re-verified 2026-11-05**) | -| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total — `flamboyant_mahavira`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → `flamboyant_mahavira` (2026-11-04) → **`nostalgic_keldysh`** (2026-11-06) — all `ghcr.io/github/github-mcp-server`, the live one created 2026-11-05T22:15:57Z | +| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** running / 68 total — main Docker host (**re-verified 2026-11-07**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) | +| CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** running / 26 total — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**re-verified 2026-11-07** — set-wise diff against live `docker ps`: no additions, no removals) | +| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total (**re-verified 2026-11-07**) — `nostalgic_keldysh`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → `flamboyant_mahavira` (2026-11-04) → **`nostalgic_keldysh`** (2026-11-06) — all `ghcr.io/github/github-mcp-server`, the live one created 2026-11-05T22:15:57Z. ⚠ **Corrected 2026-11-07:** the enumerated list still opened with `flamboyant_mahavira`, which is **not running** — live `docker ps` on CT333 returns **`nostalgic_keldysh`** (created 2026-11-05 22:15:57, `Up 29 hours`), `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1` (8 running), plus `qdrant` **exited** (9 total) | | CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** running / 9 total — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented | | CT460 `hermes` | `pct exec 460 -- docker ps` **as root** from the PVE host — socket is `root:docker srw-rw----`, the `hermes` user is not in group `docker` (its own `docker ps` → `permission denied while trying to connect to the docker API`) | `ignis` — **running** (`Up 5 days`, `ignis-subpath:0.8.9`, `0.0.0.0:8080->8080/tcp`), re-verified 2026-11-05 | -## CT200 live inventory (re-verified 2026-11-06, 61 running / 68 total, excluding 0–2 transient `worldmonitor-seeder-*` containers) +## CT200 live inventory (re-verified 2026-11-07, 61 running / 68 total, excluding 0–2 transient `worldmonitor-seeder-*` containers) `8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`. **Re-verified 2026-11-06 — the 2026-11-05 "the seeder does not exist" finding was WRONG, and the cause is a diff --git a/infrastructure/hosts.md b/infrastructure/hosts.md index 8323d0c..a833c1e 100644 --- a/infrastructure/hosts.md +++ b/infrastructure/hosts.md @@ -3,8 +3,8 @@ title: Known Hosts and Paths type: infrastructure status: partial created: 2026-07-22 -updated: 2026-11-05 -verified_on: 2026-11-05 +updated: 2026-11-07 +verified_on: 2026-11-07 confidence: medium tags: [infrastructure, hosts, containers, paths] sources: [] @@ -22,8 +22,8 @@ sources: [] | `custom:omni-101` | OmniRoute proxy (CT101 node `omniroute-1`) — **the live default provider** | `http://100.78.63.20:20128` | Yes (live probe 2026-10-26) | | CT201 | Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.77` (LAN) | Yes (live probe 2026-10-30 — LAN address corrected from `192.168.178.166`) | | CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net`, `100.118.5.51` (Tailscale) | Yes | -| PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-10-26) | -| PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-10-26) | +| PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-11-07 — 17 LXCs + 4 VMs) | +| PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-11-07 — 20 LXCs + 2 VMs) | | CT450 (hermesagent) | Hermes Agent **v0.22.5** (2026.9.24), local `bf0203a794a0` — **same build as CT460**; hermes-webui, hermes-ui (Vite). ⚠ a stale **root-level** `/usr/local/bin/hermes` on CT450 still reports **v0.19.0** — the live agent is the `hermes`-user install under `~/.hermes/hermes-agent` | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes (live probe 2026-10-30 — version corrected from `v0.22.3`, build from `5b147ab036d6` to `bf0203a794a0`) | ## Project Paths @@ -34,7 +34,7 @@ sources: [] | `/home/hermes/workspace` | Workspace | default terminal cwd | | `/home/hermes/.local/bin/hermes` | Hermes binary | | | `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` | -| `/home/hermes/.hermes/skills/` | Skills dir | **302 skills across 68 categories** (`.archive/` excluded — 46 archived; re-verified 2026-11-05) | +| `/home/hermes/.hermes/skills/` | Skills dir | **302 skills across 68 categories** (`.archive/` excluded — 46 archived; re-verified 2026-11-07) | | `/home/hermes/.hermes/cron/` | Cron jobs | **29 jobs — 26 enabled / 3 disabled** (re-verified 2026-10-31 00:3x). ⚠ Superseded: the 2026-10-30 03:3x reading was 25 / 4 — the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure, but **that pause lasted only until 10:02:48**, when its long-running slot closed `completed`, incident `ea79a2_d93c3acdc4f2` auto-closed `resolved`, and the job returned to `enabled: true` / `last_status: ok` / streak 0. **CT460 itself was then dark for 5 h 30 m on 2026-10-30** (PVE `pve` rebooted; host boot 16:24:00, CT460 16:28:00, scheduler resumed 16:37:21 — ≈39 cron slots lost, 0 failures) → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] | | `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ | | `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 | @@ -59,8 +59,8 @@ sources: [] - `/home/hermes/workspace/readlater/docker-compose.yml` ## Homelab References -- **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-10-26 (17 LXCs) -- **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-10-26 (19 LXCs). Guest IDs collide between the two hosts +- **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-11-07 (**17 LXCs** + 4 VMs; boot 2026-10-30 16:24:00) +- **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-11-07 (**20 LXCs** + 2 VMs; boot 2026-11-05 21:19:52). ⚠ **Corrected 2026-11-07:** the previous figure of 19 LXCs omitted **CT107 (`mem0`)**, which is running — see [[infrastructure/proxmox]]. Guest IDs collide between the two hosts - **CT600 (`omniroute`):** OmniRoute server, `100.88.81.19` — **running and live**; `omniroute` is the hostname that resolves to it - **CT601 (`omni`):** the old CT601 OmniRoute guest **no longer exists** on either hypervisor (`pct config 601` → no such config on both `pve` and `m93p`). The orphaned `omni` Tailscale peer (`100.93.204.4`) has since **expired out of the tailnet entirely** — as of the 2026-10-30 live probe the tailnet contains **no** peer named `omni` and no peer at `100.93.204.4`, so the old "offline 69d" description no longer applies. The only OmniRoute nodes left are CT600 (`omniroute`) and CT101 (`omniroute-1`). - **CT101 (`omniroute`, m93p):** second OmniRoute node, `100.78.63.20` — running diff --git a/infrastructure/network-map.md b/infrastructure/network-map.md index fcff18a..0ecd860 100644 --- a/infrastructure/network-map.md +++ b/infrastructure/network-map.md @@ -3,8 +3,8 @@ title: Network Map type: infrastructure status: active created: 2026-07-22 -updated: 2026-11-06 -verified_on: 2026-11-06 +updated: 2026-11-07 +verified_on: 2026-11-07 confidence: high tags: [infrastructure, network] sources: [raw/configs/hermes-config-sanitized.txt, live-probe:ct460-ss+tailscale-serve-status-2026-11-05] @@ -22,9 +22,9 @@ Where Hermes and its dependencies run, and how traffic reaches them. - **This container (hermes, ct460):** `192.168.178.129` (DHCP, eth0) ### Tailscale (`kangaroo-eel.ts.net`) -_Peer IPs below re-checked 2026-11-06 against `tailscale status` on ct460; `webserver`, `clawtest`, `hermesagent`, `docker`, `omniroute` all matched the values listed. Full 52-row node table with live states lives in [[infrastructure/tailscale]]._ +_Peer IPs below re-checked 2026-11-07 against `tailscale status` on ct460; `webserver`, `clawtest`, `hermesagent`, `docker`, `omniroute` all matched the values listed. Full 52-row node table with live states lives in [[infrastructure/tailscale]]._ - **pve:** `100.96.100.82` -- **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.0), discovered 2026-10-14.** See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host. +- **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.10), discovered 2026-10-14.** ⚠ Corrected 2026-11-07: this line still read *"9.2.0"* although both hosts run the identical `pve-manager 9.2.10` build (live `dpkg -l pve-manager` on both). See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host. - **hermes (ct460):** `100.118.5.51` - **hermesagent (ct450, the original/legacy Hermes agent dashboard):** `100.98.205.112` - **omniroute (ct600):** resolves via `omniroute.kangaroo-eel.ts.net` → `100.88.81.19` @@ -68,7 +68,7 @@ _Peer IPs below re-checked 2026-11-06 against `tailscale status` on ct460; `webs Loopback-only helpers (not routable): `49347` hermes gateway, `42013` Paseo, `42647` + `38187` opencode. -## Tailscale Serve / Funnel (verified 2026-11-06) +## Tailscale Serve / Funnel (verified 2026-11-07) `tailscale serve status` on ct460 reports 14 proxies: 12 under `https://hermes.kangaroo-eel.ts.net` (tailnet only), one tailnet route on `:9443`, and one Funnel route on `:8443` (public). @@ -80,12 +80,12 @@ Loopback-only helpers (not routable): `49347` hermes gateway, `42013` Paseo, `42 | `/wiki` | `100.118.5.51:9120` | Agent Estate Wiki | | `/webui` | `127.0.0.1:8787` | hermes-webui | | `/ignis` | `127.0.0.1:8080` | Ignis | -| `/gitea` | `100.82.100.128:3000` | Gitea (ct207 tailnet IP) | +| `/gitea` | `100.82.100.128:3000` | Gitea — **CT221 on `m93p`**, not ct207 (⚠ **corrected 2026-11-07**: this line wrongly said *"ct207 tailnet IP"*; live `pct exec 221 -- tailscale ip -4` → `100.82.100.128`, and ct207's own IP is `100.82.40.23`). The target answers **HTTP 200**. | | `/desktop` | `127.0.0.1:6081` | desktop VNC (noVNC) | | `/vnc-camofox` | `127.0.0.1:6080` | Camofox VNC | | `/evening-grid.html` | `127.0.0.1:9099/evening-grid.html` | jobs dashboard page | -| `/agent-estate` | `127.0.0.1:8769` | **DEAD — nothing listening on 8769 (re-confirmed 2026-11-06)** | -| `/synergy-prep` | `127.0.0.1:9123` | **DEAD — nothing listening on 9123 (re-confirmed 2026-11-06)** | +| `/agent-estate` | `127.0.0.1:8769` | **DEAD — nothing listening on 8769 (re-confirmed 2026-11-07)** | +| `/synergy-prep` | `127.0.0.1:9123` | **DEAD — nothing listening on 9123 (re-confirmed 2026-11-07)** | - `:9443` (tailnet only) — `/` → `127.0.0.1:5173`. - Funnel `:8443` (public) — `/webhook` → `127.0.0.1:8085/webhook` (AgentMail webhook receiver). diff --git a/infrastructure/proxmox.md b/infrastructure/proxmox.md index 970da28..2290214 100644 --- a/infrastructure/proxmox.md +++ b/infrastructure/proxmox.md @@ -3,8 +3,8 @@ title: Proxmox VE type: infrastructure status: active created: 2026-07-26 -updated: 2026-11-01 -verified_on: 2026-11-01 +updated: 2026-11-07 +verified_on: 2026-11-07 confidence: high tags: [proxmox, virtualization, infrastructure, homelab] --- @@ -21,7 +21,8 @@ Hypervisor hosting all VMs and LXC containers for the homelab. - **Primary user:** `Anthony@pam` (full admin) - **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve` -## Live LXC Inventory (verified 2026-10-31 via `pct list`) +## Live LXC Inventory (verified 2026-11-07 via `pct list`) +Re-verified 2026-11-07 03:2x AWST (stocktake): **live `pct list` on `pve` still returns exactly the 17 LXCs in the table below** (CT255 `mattermost` `running`, CT209 `cloudreve` and CT555 `elementsynapse` `stopped`, CT600 `omniroute` `running`, and **no CT601 on either hypervisor**), and **`qm list` on `pve` still returns exactly the 4 VMs** (100 `homeassistant`, 101 `guest-vps`, 110 `omarchy`, 282 `ubuntu-server-xfce` — all running; **no 310, no 403, no 401**). Both `pve-manager` packages are still **9.2.10** on **both** hosts. The **one correction** was on the `m93p` side: see the `m93p` section below — **CT107 (`mem0`) was missing**, so the m93p guest count was one short. Re-verified 2026-11-01 03:5x AWST (stocktake): **`pct list` on `pve` and `qm list` on `pve` both still match the tables below exactly** — 17 LXCs and 4 VMs on `pve`, with CT255 `mattermost` still `running`, CT209 `cloudreve` and CT555 `elementsynapse` still `stopped`, and **no CT601 on either hypervisor** (`/etc/pve/lxc/601.conf` absent). **`m93p` re-confirmed**: `pve-manager/9.2.10` (same build as `pve`), `qm list` returns **VM 103 `cachyos-m93p` stopped** and **VM 106 `openmausbot-vm` running**, and `pct exec 1001 -- docker ps` returns `sshwifty` (CT1001). Storage percentages were refreshed against live `pvesm status` on both hosts (see the Storage section). **Note:** `m93p` SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout). @@ -103,7 +104,7 @@ as a hypervisor. - **Hostname:** `m93p` · **Tailscale:** `100.104.51.98` - **PVE version:** **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, kernel `7.0.14-12-pve`) — **corrected 2026-10-31**. The previous entry read *"9.2.0"* and claimed `m93p` was *"materially newer than pve's 8.x"*. Both hosts now run the **identical `pve-manager 9.2.10` build**; only the kernel differs (`7.0.14-12-pve` vs `7.0.14-5-pve`). The hosts are **version-identical, not version-skewed**. -- **Uptime:** **up 1 day, 12:02** at 2026-11-01 03:56 AWST (live `uptime`) — the host **rebooted ~2026-10-30 16:0x**, consistent with the CT1264 openmausbot outage below. Load average was **7.66 / 12.80 / 11.32** (1/5/15 min) at that moment — elevated, worth watching +- **Uptime:** **up 1 day, 7:18** at 2026-11-07 04:38 AWST (live `uptime -s` → **boot 2026-11-05 21:19:52**; `uptime` → `up 1 day, 7:18`, load average **0.76 / 1.08 / 1.24** — normal). ⚠ **Corrected 2026-11-07:** the previous entry read *"up 1 day, 12:02 at 2026-11-01 03:56 AWST — the host rebooted ~2026-10-30 16:0x"*. The **reboot itself is still real** (`pve` booted **2026-10-30 16:24:00** per `uptime -s`, and the CT1264 openmausbot outage below follows from it), but the **uptime figure and the implied boot date were wrong** — the host has since rebooted again on **2026-11-05 21:19:52**, so it was up only **1 day, 7:18** when probed, not 12 h. The `pve` host's own boot time is **2026-10-30 16:24:00** (up **7 days, 12:14** at probe time) - **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf` ### LXC Inventory (verified 2026-10-31 via `pct list` on m93p) @@ -115,6 +116,7 @@ as a hypervisor. | 102 | vaultwarden | running | Vaultwarden password manager | | 104 | rclone | running | Rclone | | 105 | conduit | running | **Conduit** — iPhone calendar / health / location / contacts bridge (see [[systems/messaging-integrations]]). **Added 2026-10-31** — this guest was missing from the 2026-10-14 table despite running | +| 107 | mem0 | running | **mem0 MCP server** (`mem0-mcp.service` → `:8890`), Tailscale `100.73.94.53` (`mem0` peer) — **added 2026-11-07**, this guest was missing from the table despite running | | 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** | | 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one | | 500 | litellm | running | LiteLLM proxy | @@ -130,8 +132,15 @@ as a hypervisor. | 1263 | social-dashboard | running | Social dashboard | | 1264 | openmausbot | **stopped** | OpenMausBot — **stopped**, which is why the `openmausbot` Tailscale peer is offline 9d. **Added 2026-10-31** — was missing from the 2026-10-14 table. The companion **VM106 `openmausbot-vm` is running** and holds the separate `openmausbot-vm` peer (`100.77.107.30`) | -**Re-verified 2026-10-31:** live `pct list` on `m93p` returns **19 LXCs — 18 running, 1 stopped -(CT998 nightbridge) plus CT1264 openmausbot stopped**. The 2026-10-14 table listed only **17**, omitting +**Re-verified 2026-11-07 03:2x AWST (stocktake):** live `pct list` on `m93p` returns **20 LXCs — 18 running, +2 stopped (CT998 `nightbridge` and CT1264 `openmausbot`)**. ⚠ **The count was one short**: the previous +figure of 19 omitted **CT107 (`mem0`)**, a live running guest confirmed from inside the container +(`pct exec 107 -- tailscale ip -4` → `100.73.94.53`, `systemctl is-active tailscaled` → `active`, +`mem0-mcp.service` → `active`). CT107 is now in the table above. + +**Re-verified 2026-10-31:** the earlier reading of **19 LXCs — 18 running, 1 stopped +(CT998 nightbridge) plus CT1264 openmausbot stopped** was itself already one short for the same reason +(CT107 `mem0` was not counted). The 2026-10-14 table listed only **17**, omitting two live guests: **CT105 (`conduit`)** and **CT1264 (`openmausbot`)**. Note that **VMIDs 103, 220 and 555 do not exist on `m93p`** (`/etc/pve/lxc/{103,220,555}.conf` all absent) — CT103 (cliproxyapi), CT220 (caddy) and CT555 (elementsynapse) are on **`pve`** only, and `pve`'s CT555 is **stopped** while @@ -162,36 +171,39 @@ tailnet. → [[systems/omniroute]], [[infrastructure/hosts]] | Datastore | Type | Status | Total | Used | Used % | |---|---|---|---|---|---| -| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 700 GiB | **76.43 %** (`pvesm`) | -| `SSD1` | lvmthin | active | 932 GiB | 599 GiB | **64.32 %** | -| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.0 TiB | **82.77 %** | -| `local` | dir | active | 67.7 GiB | 54.0 GiB | **79.66 %** | -| `local-lvm` | lvmthin | active | 141 GiB | 19.4 GiB | **13.75 %** | +| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 717 GiB | **78.29 %** (`pvesm`) | +| `SSD1` | lvmthin | active | 931 GiB | 610 GiB | **65.46 %** | +| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.3 TiB | **84.21 %** | +| `local` | dir | active | 67.7 GiB | 55.3 GiB | **81.62 %** | +| `local-lvm` | lvmthin | active | 141 GiB | 20.1 GiB | **14.25 %** | -**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 %**, ~169 GiB free. `fred-qnap` is now -**82.45 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.0 TiB used) — the share was +**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 % (2026-11-01) → 78.29 % (2026-11-07)**, ~152 GiB free. `fred-qnap` is now +**84.21 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.3 TiB used) — the share was remounted over NFS rather than SMB. +**Storage re-verified 2026-11-07 03:2x AWST** against live `pvesm status` on both hosts — every +percentage on both tables below was refreshed. + **`iMacHDD` no longer appears in `pvesm status` on `pve`** — the 2026-10-14 table listed it as an inactive `cifs` entry. The iMac shares now live on **`m93p`** as `cifs-anthonyimac` and `cifs-imachdd` -(both active, 60.25 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing. +(both active, 76.21 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing. ### Host `m93p` (100.104.51.98) | Datastore | Type | Status | Total | Used | Used % | |---|---|---|---|---|---| -| `cifs-anthonyimac` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % | -| `cifs-imachdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % | -| `local` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % | -| `local-hdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % | -| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** | -| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** | -| `nfs-ssd0-39` | dir | active | 916 GiB | 700 GiB | **76.43 %** | -| `omb-internal` | dir | active | 93.9 GiB | 54.4 GiB | **57.92 %** | +| `cifs-anthonyimac` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % | +| `cifs-imachdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % | +| `local` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % | +| `local-hdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % | +| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** | +| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** | +| `nfs-ssd0-39` | dir | active | 916 GiB | 717 GiB | **78.29 %** | +| `omb-internal` | dir | active | 93.9 GiB | 60.7 GiB | **64.66 %** | **`m93p` storage was not documented before 2026-10-31.** Note that `m93p` exposes the **same backing volumes as `pve`** under different names — `nfs-proxmox-31`/`nfs-proxmox-54` mirror `pve`'s `fred-qnap` -(both 82.45 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 76.43 %). The iMac shares that were +(both 84.21 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 78.29 %). The iMac shares that were `iMacHDD` on `pve` are `cifs-anthonyimac` / `cifs-imachdd` on `m93p`. **Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`). diff --git a/infrastructure/tailscale.md b/infrastructure/tailscale.md index 9cabb2e..f73d322 100644 --- a/infrastructure/tailscale.md +++ b/infrastructure/tailscale.md @@ -3,8 +3,8 @@ title: Tailscale type: infrastructure status: active created: 2026-07-26 -updated: 2026-11-06 -verified_on: 2026-11-06 +updated: 2026-11-07 +verified_on: 2026-11-07 confidence: high tags: [tailscale, vpn, networking, infrastructure] --- @@ -19,7 +19,8 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access - **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`) - **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`) -## Tailscale Node Inventory (verified 2026-11-06) +## Tailscale Node Inventory (verified 2026-11-07) +Re-verified 2026-11-07 03:1x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 9 offline-age drifts, no IP changes and no missing peers.** **`octop` came back online**: the peer record reads `Online: true` with `LastSeen 0001-01-01` and `tailscale ping 100.92.163.122` → **`pong from octop via DERP(syd) in 98ms`**, so the row moves offline → online. It remains **not an LXC or VM on either hypervisor** (live `pct list` / `qm list` on both `pve` and `m93p` list no such guest) — an **orphaned tailnet peer**, so its return is not an estate change. The **9 offline ages** were refreshed (+1d each): cloudreve 104d → **105d**; conduit 60d → **61d**; elementsynapse 107d → **108d**; freshrss-new 80d → **81d**; octoagent 4d → **5d**; omarchybook 15d → **16d**; omarchymac 4d → **5d**; openmausbot 15d → **16d**; ubuntu-server-xfce 76d → **77d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) **17 guests** and `m93p` (100.104.51.98) **20 guests + 2 VMs** — ⚠ **the m93p guest count on this page was one short**: live `pct list` on `m93p` returns **20 LXCs** (the extra is **CT107 `mem0`**, `100.73.94.53`, `mem0-mcp.service` active — confirmed from inside the guest), not 19; CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT998 `nightbridge` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). The route table still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; **both DEAD targets re-confirmed dead** — `ss -ltn` still shows **no listener on 8769** (`/agent-estate`) and **none on 9123** (`/synergy-prep`) — while every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` answers **HTTP 200**. Re-verified 2026-11-06 03:0x–03:2x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 8 offline-age drifts, no IP changes and no missing peers.** **`anthonys-mac-mini` came back online**: the peer record reads `Online: true` and `tailscale ping 100.76.60.37` → `pong from anthonys-mac-mini via DERP(syd) in 103ms`, so the row moves offline → online (client device, not an estate host). The **8 offline ages** were refreshed (+1d each): cloudreve 103d → **104d**; elementsynapse 106d → **107d**; freshrss-new 79d → **80d**; octoagent 3d → **4d**; octop 1d → **2d**; omarchybook 14d → **15d**; omarchymac 3d → **4d**; openmausbot 14d → **15d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). `m93p` re-pinged → `pong from m93p via 192.168.178.250:41641 in 3ms` (online, peer record agrees), and `octop` re-pinged → **timeout** (genuinely offline, matching its `LastSeen`). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) 17 guests and `m93p` (100.104.51.98) 19 guests + 2 VMs — CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). ⚠ **Two CT460 Serve routes are dead but still present in `tailscale serve status`**: `/agent-estate` → `127.0.0.1:8769` and `/synergy-prep` → `127.0.0.1:9123` have **no listener** (live `ss -ltn` probe) and therefore serve 502 — both rows are now marked DEAD here, matching [[infrastructure/network-map]] which had already recorded them as dead on 2026-11-05. The route table itself still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` is reachable. Full node list from `tailscale status` — **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age. Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route. @@ -44,14 +45,14 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | anthonys-mac-mini | 100.76.60.37 | online — **returned 2026-11-06** (peer record `Online: true`, `tailscale ping` → `pong from anthonys-mac-mini (100.76.60.37) via DERP(syd) in 103ms`; client device, not an estate host) | | clawtest | 100.122.189.81 | online — CT333 (pve) | | cliproxyapi | 100.83.112.94 | online — CT103 (pve) | -| cloudreve | 100.111.187.7 | offline 104d — CT209 stopped | -| conduit | 100.81.79.87 | offline 60d — last seen 2026-09-06 01:32Z | +| cloudreve | 100.111.187.7 | offline 105d — CT209 stopped | +| conduit | 100.81.79.87 | offline 61d — last seen 2026-09-06 01:32Z | | docker | 100.96.150.69 | online — CT200 (pve) | | docker-01 | 100.121.12.31 | online — CT1001 (m93p) | -| elementsynapse | 100.75.250.37 | offline 107d — CT555 stopped | +| elementsynapse | 100.75.250.37 | offline 108d — CT555 stopped | | freellmapi | 100.127.208.75 | online — CT1006 (m93p) | | freshrss | 100.83.60.93 | online — CT1260 (m93p) | -| freshrss-new | 100.110.180.115 | offline 80d | +| freshrss-new | 100.110.180.115 | offline 81d | | gitea | 100.82.100.128 | online — CT221 (m93p) | | guest-vps | 100.117.14.84 | online | | hermesagent | 100.98.205.112 | online — CT450 (pve) | @@ -65,15 +66,15 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | nextcloud | 100.90.102.19 | online — CT270 (pve) | | ntfy | 100.67.153.116 | online — CT1241 (m93p) | | obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) | -| octoagent | 100.117.10.68 | offline 4d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | -| octop | 100.92.163.122 | offline 2d — last seen 2026-11-03 09:03Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | +| octoagent | 100.117.10.68 | offline 5d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | +| octop | 100.92.163.122 | online — **returned 2026-11-07** (peer record `Online: true`, `tailscale ping` → `pong from octop via DERP(syd) in 98ms`); still **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | | ollama | 100.108.44.119 | online — CT205 (pve) | | omarchy | 100.73.227.103 | online | -| omarchybook | 100.107.9.100 | offline 15d | -| omarchymac | 100.126.173.115 | offline 4d — last seen 2026-11-01 11:13Z | +| omarchybook | 100.107.9.100 | offline 16d | +| omarchymac | 100.126.173.115 | offline 5d — last seen 2026-11-01 11:13Z | | omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute | | omniroute-1 | 100.78.63.20 | online — CT101 (m93p) | -| openmausbot | 100.91.214.91 | offline 15d — CT1264 (m93p) stopped | +| openmausbot | 100.91.214.91 | offline 16d — CT1264 (m93p) stopped | | openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) | | openweb | 100.69.230.13 | online — CT245 (m93p) | | paseo | 100.72.188.93 | online — CT1261 (m93p) | @@ -86,7 +87,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | social-dashboard | 100.70.183.37 | online — CT1263 (m93p) | | springroll | 100.100.153.125 | online | | springroll-browser | 100.109.221.91 | online | -| ubuntu-server-xfce | 100.101.26.104 | offline 76d — last seen 2026-08-20 19:36Z | +| ubuntu-server-xfce | 100.101.26.104 | offline 77d — last seen 2026-08-20 19:36Z | | vaultwarden | 100.123.225.90 | online — CT102 (m93p) | | webserver | 100.82.40.23 | online — CT207 (pve) | @@ -106,8 +107,8 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | `/gitea` | `http://100.82.100.128:3000` | Gitea | | `/desktop` | `http://127.0.0.1:6081` | CUA noVNC | | `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC | -| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep — ⚠ **DEAD: nothing listening on 9123** (live `ss -ltn` probe 2026-11-06; the route still exists in `tailscale serve status` but serves 502) | -| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) — ⚠ **DEAD: nothing listening on 8769** (live `ss -ltn` probe 2026-11-06; route still present in `tailscale serve status` but serves 502) | +| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep — ⚠ **DEAD: nothing listening on 9123** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07**; the route still exists in `tailscale serve status` but serves 502) | +| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) — ⚠ **DEAD: nothing listening on 8769** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07**; route still present in `tailscale serve status` but serves 502) | | `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid | Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 route table above matched live exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; no drift). diff --git a/log.md b/log.md index b1dc1d9..52c0f49 100644 --- a/log.md +++ b/log.md @@ -2226,3 +2226,107 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted* [[current-state]] (+frontmatter, +Recent Changes block), this index (+frontmatter, +summary, +incident bullet), [[runbooks/job-radar]] (+concurrency subsection), this log, and the new incident page. Vault mirror `/home/hermes/wiki` updated (both conventions). +## 2026-11-07 03:0x AWST - stocktake (wiki-stocktake-maintenance) + +- **Script:** `wiki-maintain.sh` (repo, canonical) run clean — skills-index regenerated (**302 skills / + 68 categories**), freshness report written to `raw/stocktake/freshness-20261107.txt`, Quartz rebuilt + (**769 files**) and deployed to `public/`, `hermes-wiki-serve.service` restarted. +- **Freshness:** **0 stale, 125 fresh, 3 no-date of 128 pages** — no page is past the 21-day threshold. + (`log.md` has no `updated` date and lands in the no-date bucket, which is correct for an append-only log.) + **Freshness is not accuracy**, so a live-vs-page diff was run against every page carrying live claims. +- **Tailscale (infrastructure/tailscale.md):** table rebuilt from `tailscale status --json`. **1 state + transition — `octop` (`100.92.163.122`) is back online** (peer record `Online: true`; `tailscale ping` → + `pong from octop via DERP(syd) in 98ms`); it is still **no LXC or VM on either hypervisor**, so it is an + orphaned tailnet peer, not an estate change. **9 offline ages refreshed (+1d each).** All **51 live IPv4 + peers** match the table's IPs exactly — **no missing peers, no orphan rows, no IP mismatches** (52 rows = + 51 peers + `hermes`; the ~23 IPv6-only `funnel-ingress-node` peers excluded by design). The CT460 Serve + route table still matches `tailscale serve status` **exactly (12 paths on :443)**; **both DEAD targets + re-confirmed dead** (no listener on `8769` `/agent-estate`, none on `9123` `/synergy-prep`), every other + Serve target has a live listener, and `/gitea` (`100.82.100.128:3000`) answers **HTTP 200**. +- **Proxmox (infrastructure/proxmox.md):** ⚠ **`m93p` runs 20 LXCs, not 19 — CT107 `mem0` was missing** + (running; `mem0-mcp.service` active on `:8890`; `pct exec 107 -- tailscale ip -4` → `100.73.94.53`). + Added to the table; the 2026-10-31 reading of 19 was already short for the same reason. Live now reads + **20 LXCs — 18 running, 2 stopped** (CT998, CT1264). `pve` still matches exactly: **17 LXCs + 4 VMs**. + Both hosts run the identical **`pve-manager 9.2.10`** build. +- **⚠ `m93p` uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*; live + `uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18**, load **0.76 / 1.08 / 1.24** — normal). + The 2026-10-30 reboot was real but belonged to **`pve`** (boot **2026-10-30 16:24:00**). +- **Storage refreshed on both hosts** against live `pvesm status`: `pve` `ssd0` **76.43 → 78.29 %** + (~152 GiB free), `SSD1` **64.32 → 65.46 %**, `fred-qnap` **82.77 → 84.21 %**, `local` **79.66 → 81.62 %**, + `local-lvm` **13.75 → 14.25 %**; `m93p` iMac shares + `local` **60.25 → 76.21 %**, `nfs-proxmox-*` + **82.78 → 84.21 %**, `nfs-ssd0-39` **76.43 → 78.29 %**, `omb-internal` **57.92 → 64.66 %**. +- **Docker (infrastructure/docker-services.md):** set-wise diff against live `docker ps` on CT200/201/333 — + CT200 **61 running / 68 total** exact, CT201 **20 / 26** exact, CT333 **8 / 9** exact, **but CT333's + enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name + **`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected. CT460 `ignis` running. +- **Cross-page corrections:** [[infrastructure/network-map]] `/gitea` was annotated *"Gitea (ct207 tailnet + IP)"* — it is **CT221 on `m93p`** (ct207 is `100.82.40.23`); also its `m93p` PVE version read *9.2.0* + (both hosts are 9.2.10). [[agents/hermes-chaos]] read *"`v0.22.3` … `5b147ab036d6` — the same build as + CT460"*, which contradicted itself: live is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to + CT460's (`hermes update --check` reports **v0.23.0** available). [[agents/hermes-production]]'s + *"2 cron jobs still pin `custom:omni`"* is **no longer true** — live `jobs.json` has **0** on the bare key + and **21** on `custom:omni-101`. +- **Fleet re-verified unchanged:** CT460 **29 jobs (26 enabled / 3 disabled)**; CT450 **25 jobs + (13 enabled / 12 disabled)** with **9 shared names** — confirmed by a live set-diff of both `jobs.json`. + A keyed `deepseek-paid-smart` chat probe against `omniroute-1` (`100.78.63.20:20128`) returns **HTTP 200**; + `config.yaml` still defaults to `custom:omni-101`. +- **Pages touched:** [[infrastructure/tailscale]], [[infrastructure/proxmox]], [[infrastructure/hosts]], + [[infrastructure/network-map]], [[infrastructure/docker-services]], [[agents/hermes-production]], + [[agents/hermes-chaos]], [[current-state]], [[index]], this log. No new incident page — no incident was + found; every drift above was a documentation drift corrected against a live probe. + + +## 2026-11-07 05:1x AWST - incident resolution: Gitea config backup push dead since 2026-08-08 + +- **Anthony's ask:** "fix this" on the failing backup alert — with a pointer to check Notion for the API key. +- **[[incidents/2026-08-08-gitea-config-backup-push-dead]] is now `status: resolved`.** The nightly + `e27da6c34bbb` config backup had been committing locally and failing to push **every night since + 2026-08-08** — **92 commits** stranded — while its cron row read `last_status: ok`. +- **Four independent root causes, all confirmed against live systems:** + 1. **No Gitea credential.** `origin` was HTTPS with no userinfo; the global `credential.helper=store` + file `~/.git-credentials` held only a `github.com` entry. The token in the Notion vault was + **valid** and byte-identical to the one the wiki repo carries in its remote URL — it simply was + never wired into the backup repo. Fixed with a host-scoped `git credential approve` entry (not + embedded in the URL); `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`. + 2. **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's + `success = (returncode == 0)` recorded `ok` on every failed night. The scheduler was correct; + **the script was the liar**. This is why the outage ran 92 days without a job-level red. + 3. **A 1.76 GB `checkpoints/` backlog** in the unpushed range (13 blobs > 100 MB) — junk that + `.gitignore` already excluded for *new* commits but that history still carried. + 4. **Both secondary legs broken.** The `github` remote did not exist, and the fallback pushed a + branch's **full ancestry**, which contains a 7.45 GB tarball blob — so GitHub's 100 MB file + limit could never accept it. The AgentMail alert called a `python3 send_email.py` that no longer + exists, via a `chaos-agent@agentmail.to` inbox that no longer exists. +- **Backlog drained:** `git filter-repo --partial --refs origin/main..main --path checkpoints + --invert-paths --prune-empty never --force` stripped `checkpoints/` from the **unpushed range only** + (`origin/main` untouched, push stayed a fast-forward), then `checkpoints/` was untracked at the tip. + Push set **~2 GB → 264.8 MB**; 93 commits pushed in 11.6 s; the 7.45 GB tarball was already on + `origin/main` and never resent. Pre-rewrite history preserved under tag + `backup/pre-rewrite-20261107-032658`. **Token leakage scan over the HEAD tree, the working tree and + every unpushed blob: clean.** +- **Script rewritten** (`~/.hermes/config-backup/backup.sh`, via OpenCode + a correction pass): exits + **0 only on verified success**; stamps `.last_backup_ok` **only** after remote verification; decides + the push by **remote-vs-local** rather than by working-tree changes (the exact mechanism that created + the backlog); guards every `git ls-remote` with `|| true` so an unreachable remote reaches + `notify_failure` instead of dying at exit 128 *before* the alert; pushes GitHub as a **parentless + orphan snapshot** (`git commit-tree` from `HEAD^{tree}` — no history, no >100 MB blob, HEAD/branch/ + index untouched) as a warning-only secondary target; and reports Telegram/AgentMail delivery + **honestly** (`curl --fail` + `"ok":true`; the real `agentmail` CLI, key read from `.env` at runtime + and passed via the environment, never argv). +- **Verified live 2026-11-07:** real run exit **0** in 47 s; `git rev-parse HEAD` == + `git ls-remote origin refs/heads/main` == `a50a5170c1c0e2af5235fa63a95a732e2d780703`; + `rev-list --count origin/main..main` = **0**; Gitea API `main` = `a50a5170c1c0`, updated + `2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub + mirror now carries a **0-parent** orphan snapshot `faed9fb5c677` (30,086 files, largest blob 23.5 MB) + with `master` untouched at `c99c94e6b13b` — **no data lost**, since GitHub's tree was a strict subset + of local HEAD (only `BACKUP-INDEX.md` added). AgentMail and Telegram alert paths each exercised + directly (AgentMail self-test to the agent's own inbox; Telegram `getMe`, no message sent to Anthony). + Sandbox E2E: unreachable remote → exit 1 + alert + health **not** stamped; local-ahead-with-clean-tree + → pushed + verified; already-up-to-date → no push, exit 0. +- **Pages touched:** [[incidents/2026-08-08-gitea-config-backup-push-dead]] (status → resolved, full + Resolution section added), [[infrastructure/backups]] (⚠ → ✅, re-verified 2026-11-07), + [[systems/scheduled-tasks]] (job row → RESOLVED), this log. +- **Note — unrelated pre-existing state:** the 03:00 `wiki-stocktake-maintenance` slot **failed** with + `RuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership` + (failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are + included in this commit so the tree is not left dirty. diff --git a/raw/stocktake/freshness-20261107.txt b/raw/stocktake/freshness-20261107.txt index 6cd20fd..4627a37 100644 --- a/raw/stocktake/freshness-20261107.txt +++ b/raw/stocktake/freshness-20261107.txt @@ -7,8 +7,8 @@ Threshold: >21 days since `updated` ## FRESH (125) - [ 18d] CONTRIBUTING.md -- [ 12d] agents/hermes-chaos.md -- [ 12d] agents/hermes-production.md +- [ 0d] agents/hermes-chaos.md +- [ 0d] agents/hermes-production.md - [ 18d] agents/index.md - [ 9d] agents/nanobot.md - [ 18d] agents/open-claw.md @@ -77,12 +77,12 @@ Threshold: >21 days since `updated` - [ 0d] index.md - [ 3d] infrastructure/backups.md - [ 18d] infrastructure/credentials-map.md -- [ 1d] infrastructure/docker-services.md +- [ 0d] infrastructure/docker-services.md - [ 18d] infrastructure/domains-and-tunnels.md -- [ 2d] infrastructure/hosts.md -- [ 1d] infrastructure/network-map.md -- [ 6d] infrastructure/proxmox.md -- [ 1d] infrastructure/tailscale.md +- [ 0d] infrastructure/hosts.md +- [ 0d] infrastructure/network-map.md +- [ 0d] infrastructure/proxmox.md +- [ 0d] infrastructure/tailscale.md - [ 3d] queries/retained-research-answers.md - [ 13d] reports/communities-wa-social-media-audit-2026.md - [ 6d] reports/uber-eats/2026-07.md diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 9009fe9..53f451c 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -215,7 +215,7 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne | weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 | | Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text | | wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken. ⚠ **The 2026-10-17 03:00 slot failed** on the same `omni-101` 503 — the first miss since the **2026-10-15** storm slot (which died with `RuntimeError: Connection error.`; the 10-16 slot succeeded). `raw/stocktake/freshness-20261017.txt` was not written and `systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 slot succeeded, so `freshness-20261016.txt` exists); the **2026-10-18 03:00** slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident `1a8e90_3c9c1d573cde` `detected`. Note the ledger tags this row `failure_type: unknown` even though the execution carries an explicit 503 message | -| 🦝 Hermes Config Backup to Gitea (Daily) ⚠ **push has failed every night since 2026-08-08 — 75 local commits unpushed, job still reads `ok`** → [[incidents/2026-08-08-gitea-config-backup-push-dead]] | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent | +| 🦝 Hermes Config Backup to Gitea (Daily) ✅ **RESOLVED 2026-11-07** — push live again, 92-commit backlog drained and pushed, script hardened to exit non-zero on failure so the job row can no longer read `ok` while the push is dead. Live run exit 0; `origin/main` == local HEAD (`a50a5170c1c0`); `rev-list --count origin/main..main` = 0. → [[incidents/2026-08-08-gitea-config-backup-push-dead]] | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent | ## Removed Jobs | Name | Removed | Reason |