resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly
- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo - Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved) - Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB - Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification, decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree, honest AgentMail/Telegram delivery reporting - Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed - Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated - Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with 'lost its durable fire claim ownership' and left the tree dirty)
This commit is contained in:
@@ -3,8 +3,8 @@ title: Hermes Chaos
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-15
|
created: 2026-08-15
|
||||||
updated: 2026-10-26
|
updated: 2026-11-07
|
||||||
verified_on: 2026-10-26
|
verified_on: 2026-11-07
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, chaos, ct450, experimental, agent]
|
tags: [hermes, chaos, ct450, experimental, agent]
|
||||||
sources: [live-probe, config-inspection]
|
sources: [live-probe, config-inspection]
|
||||||
@@ -28,10 +28,10 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
|
|||||||
## Current State
|
## Current State
|
||||||
|
|
||||||
- **Status:** Active (experimental)
|
- **Status:** Active (experimental)
|
||||||
- **Hermes version:** `v0.22.3` (2026.9.24), local `5b147ab036d6` — **the same build as CT460** (updated 2026-10-21). **0 commits behind `origin/main`** (verified 2026-10-26; the old "3 commits behind" claim is stale), though `hermes update --check` reports stable **v0.22.5** available. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]].
|
- **Hermes version:** `v0.22.5` (2026.9.24), local `bf0203a794a0` — **the same build as CT460** (updated 2026-10-21). ⚠ **Corrected 2026-11-07:** this line still read *"`v0.22.3` … local `5b147ab036d6`"* while asserting it was *"the same build as CT460"* — it was not. Live `hermes --version` on CT450 now returns **`v0.22.5` … local `bf0203a794a0`**, byte-identical to CT460's own `hermes --version`, so the *same build* claim is finally true. `hermes update --check` reports stable **v0.23.0** (`2bd378ed1b8d`) available. Python 3.14.9. See [[incidents/2026-10-03-ct450-hermes-update]].
|
||||||
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
|
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
|
||||||
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
|
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
|
||||||
- **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` 2026-10-26). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460)
|
- **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` **2026-11-07**; unchanged). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460) — both confirmed by a live set-diff of the two `jobs.json` files on 2026-11-07
|
||||||
- **Production counterparts:** CT460 has **28 jobs / 25 enabled** — the two fleets no longer share a job count, and the CT450 duplicates are paused
|
- **Production counterparts:** CT460 has **28 jobs / 25 enabled** — the two fleets no longer share a job count, and the CT450 duplicates are paused
|
||||||
|
|
||||||
## Providers & Routing
|
## Providers & Routing
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Hermes Production
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-15
|
created: 2026-08-15
|
||||||
updated: 2026-10-26
|
updated: 2026-11-07
|
||||||
verified_on: 2026-10-26
|
verified_on: 2026-11-07
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, production, ct460, agent]
|
tags: [hermes, production, ct460, agent]
|
||||||
sources: [live-probe, config-inspection]
|
sources: [live-probe, config-inspection]
|
||||||
@@ -43,7 +43,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
|
|
||||||
## Providers & Routing
|
## Providers & Routing
|
||||||
|
|
||||||
- **Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, CT101/m93p) — verified live 2026-10-26. The `custom:omni` provider key does not exist in `custom_providers`, but **2 cron jobs still pin it** (see [[systems/scheduled-tasks]])
|
- **Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, CT101/m93p) — verified live 2026-10-26, **re-verified 2026-11-07** (live `config.yaml` `model.default.provider: custom:omni-101`; a keyed `deepseek-paid-smart` chat probe against `100.78.63.20:20128` returns **HTTP 200**). The `custom:omni` provider key does not exist in `custom_providers`. ⚠ **Corrected 2026-11-07:** the old claim that *"2 cron jobs still pin `custom:omni`"* is **no longer true** — live `jobs.json` shows **no job pinned to the bare `custom:omni`**; **all 21 provider-pinned jobs use `custom:omni-101`** (see [[systems/scheduled-tasks]])
|
||||||
- **Default model:** `smart-route` (previously `freellmapi-auto`)
|
- **Default model:** `smart-route` (previously `freellmapi-auto`)
|
||||||
- **Fallback providers** (live chain, re-verified 2026-10-26): `nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6_1t`. ⚠ The **first** entry is the dead `nous` credential — a fallover tries it first
|
- **Fallback providers** (live chain, re-verified 2026-10-26): `nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6_1t`. ⚠ The **first** entry is the dead `nous` credential — a fallover tries it first
|
||||||
- **Auxiliary models:** all eight `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) are **`no-think/freeapi/auto` on `custom:omni-101`** with a `freeapi/deepseek-v4-flash` → `kg/kilo-auto/free` fallback chain (moved 2026-10-25 19:13)
|
- **Auxiliary models:** all eight `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) are **`no-think/freeapi/auto` on `custom:omni-101`** with a `freeapi/deepseek-v4-flash` → `kg/kilo-auto/free` fallback chain (moved 2026-10-25 19:13)
|
||||||
@@ -74,7 +74,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
| Aspect | Production (CT460) | Chaos (CT450) |
|
| Aspect | Production (CT460) | Chaos (CT450) |
|
||||||
|--------|-------------------|---------------|
|
|--------|-------------------|---------------|
|
||||||
| Purpose | Stable daily operations | Experimental/testing |
|
| Purpose | Stable daily operations | Experimental/testing |
|
||||||
| Cron jobs | 28 (25 enabled / 3 disabled) | 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460 |
|
| Cron jobs | **29 (26 enabled / 3 disabled)** (re-verified 2026-11-07) | 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460 (re-verified 2026-11-07) |
|
||||||
| Config changes | Requires verification | Safe to experiment |
|
| Config changes | Requires verification | Safe to experiment |
|
||||||
| Memory | Production Mnemosyne | Separate state |
|
| Memory | Production Mnemosyne | Separate state |
|
||||||
|
|
||||||
|
|||||||
@@ -14,6 +14,45 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
|
|||||||
# Current State
|
# Current State
|
||||||
Updated: 2026-11-07
|
Updated: 2026-11-07
|
||||||
|
|
||||||
|
## Recent Changes (2026-11-07, stocktake; live probes 2026-11-07 03:1x-03:2x AWST)
|
||||||
|
|
||||||
|
- **The tailnet lost a peer and the m93p guest list lost a guest — both corrections came from live probes, not the age scan.**
|
||||||
|
[[infrastructure/tailscale]] was rebuilt from `tailscale status --json`: **`octop` (`100.92.163.122`) is back
|
||||||
|
online** (peer record `Online: true`, `tailscale ping` → `pong from octop via DERP(syd) in 98ms`; it remains
|
||||||
|
**no LXC or VM on either hypervisor** — an orphaned tailnet peer), and the **9 remaining offline ages** were
|
||||||
|
refreshed (+1d each). All **51 live IPv4 peers** match the table's IPs exactly, with no missing peers and no
|
||||||
|
orphan rows (**52 rows** = 51 peers + `hermes` itself). The CT460 Serve route table still matches
|
||||||
|
`tailscale serve status` **exactly (12 paths on :443)**; both previously-flagged dead targets were
|
||||||
|
**re-confirmed dead** — still no listener on `8769` (`/agent-estate`) or `9123` (`/synergy-prep`).
|
||||||
|
- **⚠ `m93p` actually runs 20 LXCs, not 19 — CT107 (`mem0`) was missing from the Proxmox page.**
|
||||||
|
Live `pct list` on `m93p` returns **20 LXCs — 18 running, 2 stopped** (CT998 `nightbridge`, CT1264
|
||||||
|
`openmausbot`). The omitted guest is **CT107 `mem0`** (`mem0-mcp.service` active on `:8890`,
|
||||||
|
`pct exec 107 -- tailscale ip -4` → `100.73.94.53`). Added to [[infrastructure/proxmox]]; the
|
||||||
|
**2026-10-31 reading of 19 was already short for the same reason**. `pve` still returns exactly
|
||||||
|
**17 LXCs + 4 VMs**, and both hosts still run the identical `pve-manager 9.2.10` build.
|
||||||
|
- **⚠ `m93p`'s uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*.
|
||||||
|
Live `uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18** at probe time, load
|
||||||
|
**0.76 / 1.08 / 1.24** — normal). The 2026-10-30 reboot was real but belonged to **`pve`**
|
||||||
|
(boot **2026-10-30 16:24:00**), not `m93p`.
|
||||||
|
- **Storage percentages refreshed on both hosts** against live `pvesm status` — `pve`'s `ssd0` moved
|
||||||
|
**76.43 % → 78.29 %** (~152 GiB free), `fred-qnap` **82.77 % → 84.21 %**, `local` **79.66 % → 81.62 %**;
|
||||||
|
on `m93p` the iMac shares and `local` moved **60.25 % → 76.21 %** and `omb-internal`
|
||||||
|
**57.92 % → 64.66 %**.
|
||||||
|
- **Docker inventories diffed set-wise against live `docker ps` on all three hosts.** CT200 matches
|
||||||
|
**61 running / 68 total** exactly; CT201 matches **20 / 26** exactly; CT333 matches **8 / 9** — but its
|
||||||
|
**enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name is
|
||||||
|
**`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected in [[infrastructure/docker-services]].
|
||||||
|
- **Two small cross-page corrections.** [[infrastructure/network-map]]'s `/gitea` route was annotated
|
||||||
|
*"Gitea (ct207 tailnet IP)"* — it is **CT221 on `m93p`** (`100.82.100.128`; ct207's own IP is
|
||||||
|
`100.82.40.23`); the target answers **HTTP 200**. [[agents/hermes-chaos]]'s version line read
|
||||||
|
*"`v0.22.3` … local `5b147ab036d6` — the same build as CT460"*, which contradicted itself: live
|
||||||
|
`hermes --version` on CT450 is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to CT460's.
|
||||||
|
[[agents/hermes-production]]'s claim that *2 cron jobs still pin `custom:omni`* is **no longer true** —
|
||||||
|
live `jobs.json` shows **0** jobs on the bare key and **21** on `custom:omni-101`.
|
||||||
|
- **Cron fleet re-verified unchanged:** **29 jobs (26 enabled / 3 disabled)**, matching the nightly audit
|
||||||
|
hours earlier. CT450's fleet is **25 jobs (13 enabled / 12 disabled)** with **9 shared names** — confirmed
|
||||||
|
by a live set-diff of both `jobs.json` files. → [[systems/scheduled-tasks]]
|
||||||
|
|
||||||
## Recent Changes (2026-11-07, nightly audit; live probes 2026-11-07 00:3x-00:5x AWST)
|
## Recent Changes (2026-11-07, nightly audit; live probes 2026-11-07 00:3x-00:5x AWST)
|
||||||
|
|
||||||
- **A red row with no alert: the Job Radar's 11:30 slot was discarded as stale while a concurrent fire
|
- **A red row with no alert: the Job Radar's 11:30 slot was discarded as stale while a concurrent fire
|
||||||
|
|||||||
@@ -1,18 +1,23 @@
|
|||||||
---
|
---
|
||||||
title: Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 While the Job Reads `ok`
|
title: Hermes Config Backup to Gitea — Push Was Dead 2026-08-08 → 2026-11-07 While the Job Read `ok`
|
||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: resolved
|
||||||
created: 2026-10-21
|
created: 2026-10-21
|
||||||
updated: 2026-10-22
|
updated: 2026-11-07
|
||||||
verified_on: 2026-10-22
|
resolved_on: 2026-11-07
|
||||||
review_after: 2026-10-29
|
verified_on: 2026-11-07
|
||||||
|
review_after: 2027-02-07
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [backup, gitea, cron, git, green-vs-actual]
|
tags: [backup, gitea, cron, git, green-vs-actual]
|
||||||
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki]
|
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki, gitea-api, github-api, sandbox-e2e-test]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08
|
# Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08
|
||||||
|
|
||||||
|
> **RESOLVED 2026-11-07.** The push is live again, the backlog is drained, and the script now exits
|
||||||
|
> non-zero on failure so the cron row cannot lie. See [Resolution 2026-11-07](#resolution-2026-11-07)
|
||||||
|
> at the foot of this page for the full record. The original investigation is preserved below.
|
||||||
|
|
||||||
## Headline
|
## Headline
|
||||||
|
|
||||||
The daily **Hermes Config Backup to Gitea** job (`e27da6c34bbb`, `0 11 * * *`, no-agent, `hermes-config-backup.sh`)
|
The daily **Hermes Config Backup to Gitea** job (`e27da6c34bbb`, `0 11 * * *`, no-agent, `hermes-config-backup.sh`)
|
||||||
@@ -103,3 +108,92 @@ change as well the *only* off-box copy path was this dead push → [[infrastruct
|
|||||||
- [[infrastructure/backups]]
|
- [[infrastructure/backups]]
|
||||||
- [[systems/scheduled-tasks]]
|
- [[systems/scheduled-tasks]]
|
||||||
- [[systems/hermes-agent]]
|
- [[systems/hermes-agent]]
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# Resolution 2026-11-07
|
||||||
|
|
||||||
|
Anthony asked to fix the failing push (and to check the Notion vault for the API key). Resolved in
|
||||||
|
one session: the push is live, the backlog is drained, and the script can no longer report a false green.
|
||||||
|
|
||||||
|
## Root causes (four, all independent)
|
||||||
|
|
||||||
|
1. **No Gitea credential.** `origin` was HTTPS with no userinfo, and the global `credential.helper=store`
|
||||||
|
file `~/.git-credentials` held only a `github.com` entry — so the cron worker could not authenticate
|
||||||
|
non-interactively. The token in the Notion vault was valid and identical to the one the wiki repo
|
||||||
|
carries in its remote URL; it simply was never wired into the backup repo.
|
||||||
|
2. **`backup.sh` had no `exit` statement at all.** It always returned 0, so the scheduler
|
||||||
|
(`success = (returncode == 0)` in `cron/scheduler_script.py`) recorded `last_status: ok` on every
|
||||||
|
failed night. This is why the outage went unnoticed for 92 days — it was a *false green*, not a
|
||||||
|
missing signal. (The incident's earlier note blamed the scheduler; the scheduler was correct. The
|
||||||
|
script was the liar.)
|
||||||
|
3. **A 1.76 GB `checkpoints/` backlog** sat in the unpushed range — 13 blobs over 100 MB, all junk
|
||||||
|
that `.gitignore` already excluded for new commits but that history still carried.
|
||||||
|
4. **Two broken secondary legs.** The `github` remote did not exist, and the fallback
|
||||||
|
`_github_clean_push` pushed a branch's full ancestry — which contains a 7.45 GB tarball blob, so
|
||||||
|
GitHub's 100 MB file limit could never accept it. The AgentMail alert called a
|
||||||
|
`python3 send_email.py` that no longer exists, via a `chaos-agent@agentmail.to` inbox that no
|
||||||
|
longer exists.
|
||||||
|
|
||||||
|
## Fix applied
|
||||||
|
|
||||||
|
- **Credential:** stored a host-scoped Gitea entry via `git credential approve` (not embedded in the
|
||||||
|
remote URL). `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`.
|
||||||
|
- **Backlog:** `git filter-repo --partial --refs origin/main..main --path checkpoints --invert-paths
|
||||||
|
--prune-empty never --force` stripped `checkpoints/` from the unpushed range only (so `origin/main`
|
||||||
|
was untouched and the push stayed a fast-forward), then `checkpoints/` was untracked at the tip.
|
||||||
|
Pre-rewrite history is preserved under the tag `backup/pre-rewrite-20261107-032658`.
|
||||||
|
**Push set: ~2 GB → 264.8 MB.** The 7.45 GB tarball was already on `origin/main` and was never resent.
|
||||||
|
- **Script hardening** (`~/.hermes/config-backup/backup.sh`, rewritten via OpenCode + a correction pass):
|
||||||
|
- exits **0 only on verified success**, `1` otherwise — the cron row can no longer lie;
|
||||||
|
- `.last_backup_ok` is stamped **only** after the remote is confirmed to hold the commit;
|
||||||
|
- the push decision compares **remote vs local**, not "were there new working-tree changes" — a
|
||||||
|
local-ahead-of-remote with a clean tree now still pushes (this is exactly the mechanism that
|
||||||
|
created the 92-commit backlog);
|
||||||
|
- every `git ls-remote` is guarded with `|| true`, so an unreachable remote reaches the alert path
|
||||||
|
instead of killing the script at exit 128 before `notify_failure` fires;
|
||||||
|
- the GitHub leg now pushes a **parentless orphan snapshot** built from `HEAD^{tree}` via
|
||||||
|
`git commit-tree` — no history, no >100 MB blob, HEAD/branch/index untouched — and is a
|
||||||
|
warning-only secondary target that can never fail the job;
|
||||||
|
- the AgentMail leg uses the real CLI (`agentmail inboxes:messages send --inbox-id <addr>`) with the
|
||||||
|
key read from `.env` at runtime and passed via the environment, never argv;
|
||||||
|
- the Telegram leg uses `curl --fail` and requires `"ok":true` in the body, so it reports `sent`
|
||||||
|
only on a real delivery.
|
||||||
|
|
||||||
|
## Verification (live, 2026-11-07 AWST)
|
||||||
|
|
||||||
|
| Check | Result |
|
||||||
|
|---|---|
|
||||||
|
| Real `backup.sh` run | exit **0**, 47 s |
|
||||||
|
| `git rev-parse HEAD` vs `git ls-remote origin refs/heads/main` | identical — `a50a5170c1c0e2af5235fa63a95a732e2d780703` |
|
||||||
|
| `git rev-list --count origin/main..main` | **0** (was 92) |
|
||||||
|
| Gitea API repo `Anthony/hermes-config-backup` | HTTP 200, default branch `main`, updated `2026-11-07T05:10:46+08:00`, 8.1 MB |
|
||||||
|
| Gitea API `branches/main` | `a50a5170c1c0` — `🦝 nightly backup: 2026-11-07 05:10:33 AWST` |
|
||||||
|
| `.last_backup_ok` | refreshed to the run timestamp (was stale from 2026-11-06 11:00) |
|
||||||
|
| GitHub `main` | `faed9fb5c677`, **0 parents** (orphan snapshot confirmed), 30,086 files, largest blob 23.5 MB |
|
||||||
|
| GitHub `master` branch | untouched at `c99c94e6b13b` |
|
||||||
|
| GitHub force-push data loss | none — GitHub's tree was a strict **subset** of local HEAD (only `BACKUP-INDEX.md` added) |
|
||||||
|
| Token leakage scan (HEAD tree, working tree, all unpushed blobs) | **clean** — token absent everywhere |
|
||||||
|
| AgentMail leg (self-test to the agent's own inbox) | delivered, `message_id` returned |
|
||||||
|
| Telegram leg (`getMe`, no message sent to Anthony) | `"ok":true` detected → would report `sent` |
|
||||||
|
| Sandbox E2E: unreachable remote | exit **1**, `notify_failure` fired, health file **not** stamped |
|
||||||
|
| Sandbox E2E: local ahead of remote, clean working tree | pushed + verified, exit **0**, health stamped |
|
||||||
|
| Sandbox E2E: already up to date | no push, exit **0**, health stamped |
|
||||||
|
|
||||||
|
## Verification command set (for the next audit)
|
||||||
|
|
||||||
|
```
|
||||||
|
git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
|
||||||
|
git -C ~/.hermes/config-backup rev-list --count origin/main..main
|
||||||
|
tail -20 ~/.hermes/config-backup/backup.log
|
||||||
|
tail -3 ~/.hermes/config-backup/.backup_history
|
||||||
|
```
|
||||||
|
|
||||||
|
## Still open (not part of this incident)
|
||||||
|
|
||||||
|
- Gitea host SSH login is denied for `anthonymau` — free disk headroom on the Gitea box is still
|
||||||
|
unverified (the Gitea API reports repo size, not host free space).
|
||||||
|
- `hermes-gateway-runtime-health.service` still fails every 5 min because the Mattermost host
|
||||||
|
`100.98.116.14` has been offline since ~Sept 2026. Separate incident.
|
||||||
|
- The `checkpoints/` directory is still excluded from backups by policy; the live snapshots remain
|
||||||
|
at `~/.hermes/checkpoints/` on-box only.
|
||||||
|
|||||||
@@ -28,7 +28,7 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
|
|||||||
> **2026-11-04 (nightly audit) — the ledger raised its own red row:** the **11-03 fleet settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) — **unclosed incident rows 0 → 1**, non-clean `last_status` **1 → 2**, deliveries **7 rows / 7 `delivered`**. The single failure is the **Job Radar Stage 1** (`dca8482e4f76`), which ran **07:00:33 → 13:28:52 (6 h 28 m 18 s)** and died on **`RuntimeError: stream active timeout`** — a **new terminal signature**, not the iteration cap — after going **silent for 65 m 21 s**; its incident `dca848_dfd4bea84f2a` (`timeout`) is **`alerted` and open** and the notice **was delivered** (13:28:51.741724) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. That answers the 11-02 page's open question: **output is suppressed when the run ends `completed`, delivered when it ends `failed`** — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so **2026-11-02's and 2026-11-03's candidate sets are both unrecovered**. ⚠ **The 03:00 stocktake overran for a second day** (`1a8e90ea2271`, 03:00:30 → 10:41:13 = **7 h 40 m 43 s**, `suppressed`, nothing delivered, artefacts uncommitted again). **New page:** [[systems/hermes-hq-push]] (v0.9.27, enabled; manifest says it **replaces `dispatch-push`**, and no trace of the old plugin survives).
|
> **2026-11-04 (nightly audit) — the ledger raised its own red row:** the **11-03 fleet settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) — **unclosed incident rows 0 → 1**, non-clean `last_status` **1 → 2**, deliveries **7 rows / 7 `delivered`**. The single failure is the **Job Radar Stage 1** (`dca8482e4f76`), which ran **07:00:33 → 13:28:52 (6 h 28 m 18 s)** and died on **`RuntimeError: stream active timeout`** — a **new terminal signature**, not the iteration cap — after going **silent for 65 m 21 s**; its incident `dca848_dfd4bea84f2a` (`timeout`) is **`alerted` and open** and the notice **was delivered** (13:28:51.741724) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. That answers the 11-02 page's open question: **output is suppressed when the run ends `completed`, delivered when it ends `failed`** — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so **2026-11-02's and 2026-11-03's candidate sets are both unrecovered**. ⚠ **The 03:00 stocktake overran for a second day** (`1a8e90ea2271`, 03:00:30 → 10:41:13 = **7 h 40 m 43 s**, `suppressed`, nothing delivered, artefacts uncommitted again). **New page:** [[systems/hermes-hq-push]] (v0.9.27, enabled; manifest says it **replaces `dispatch-push`**, and no trace of the old plugin survives).
|
||||||
>
|
>
|
||||||
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
||||||
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
|
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.10), was discovered during the 2026-10-14 stocktake.
|
||||||
> Guest IDs collide between the two hosts.
|
> Guest IDs collide between the two hosts.
|
||||||
|
|
||||||
## Start here
|
## Start here
|
||||||
@@ -125,8 +125,8 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
|
|||||||
- [[infrastructure/backups]] — hermes backup CLI, retention, and gaps
|
- [[infrastructure/backups]] — hermes backup CLI, retention, and gaps
|
||||||
|
|
||||||
### Networking
|
### Networking
|
||||||
- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 9.2.10 + `m93p` 9.2.10, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 76.43 % / ~169 GiB free, 2026-10-31)
|
- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 9.2.10 + `m93p` 9.2.10, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 78.29 % / ~152 GiB free, 2026-11-07)
|
||||||
- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **50-node** tailnet inventory (re-verified 2026-10-31: `mattermost` back **online** after ~40 days; 7 offline ages refreshed)
|
- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **52-node** tailnet inventory (re-verified 2026-11-07: `octop` back **online**; 9 offline ages refreshed; **CT107 `mem0`** added to the m93p guest list)
|
||||||
- [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly
|
- [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly
|
||||||
|
|
||||||
## Concepts
|
## Concepts
|
||||||
|
|||||||
+36
-26
@@ -3,8 +3,8 @@ title: Backups
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-04
|
updated: 2026-11-07
|
||||||
verified_on: 2026-11-04
|
verified_on: 2026-11-07
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [infrastructure, backups]
|
tags: [infrastructure, backups]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -54,7 +54,7 @@ How Hermes's own configuration/state gets backed up, and what's missing.
|
|||||||
local snapshot and no off-box copy** — the Gitea mirror below has been dead since 2026-08-08.
|
local snapshot and no off-box copy** — the Gitea mirror below has been dead since 2026-08-08.
|
||||||
→ [[systems/hermes-agent]]
|
→ [[systems/hermes-agent]]
|
||||||
|
|
||||||
## ⚠ Scheduled config backup — the Gitea mirror has been dead since 2026-08-08
|
## ✅ Scheduled config backup — RESOLVED 2026-11-07 (was dead since 2026-08-08)
|
||||||
|
|
||||||
`e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits
|
`e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits
|
||||||
locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its
|
locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its
|
||||||
@@ -66,31 +66,41 @@ helper — so the non-interactive push dies on
|
|||||||
`--- Backup complete: FAIL ---`. Full evidence and the proposed fix:
|
`--- Backup complete: FAIL ---`. Full evidence and the proposed fix:
|
||||||
[[incidents/2026-08-08-gitea-config-backup-push-dead]].
|
[[incidents/2026-08-08-gitea-config-backup-push-dead]].
|
||||||
|
|
||||||
**Consequence for this page: everything above describes a topology that has not carried `~/.hermes` off the box
|
> **RESOLVED 2026-11-07.** The 92-commit backlog is drained and pushed to Gitea, and the script has been
|
||||||
since 2026-08-07.** The 2026-07-22 full zip and the wiki/authoring repos remain the only recent off-box copies.
|
> hardened so the cron row can no longer report a false green. Full record:
|
||||||
Telegram failure notices *have* been firing daily (last: 2026-10-21 11:00:14, `message_id 8449`); the AgentMail
|
> [[incidents/2026-08-08-gitea-config-backup-push-dead#Resolution 2026-11-07]].
|
||||||
notification step failed.
|
|
||||||
|
|
||||||
**Re-verified 2026-11-04 — still dead, and the drift is now 89 commits.** Live `git -C ~/.hermes/config-backup`:
|
**What was wrong (four independent causes):** (1) no Gitea credential — `origin` was HTTPS and
|
||||||
`origin` = `https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` (**unchanged** — still HTTPS with
|
`~/.git-credentials` held only a `github.com` entry, so the non-interactive push could not authenticate;
|
||||||
no userinfo and no credential helper, so the credential root cause has not been addressed); `rev-list --count
|
(2) **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's
|
||||||
origin/main..HEAD` = **89**; `log origin/main -1` = **2026-08-07 11:00:47 +0800 1e1eb208** (`🦝 nightly backup:
|
`success = (returncode == 0)` recorded `ok` on every failed night (the scheduler was correct; the script
|
||||||
2026-08-07 11:00:29 AWST`) — i.e. **nothing has reached Gitea in 89 days**. `backup.log` confirms the nightly
|
was the liar); (3) a 1.76 GB `checkpoints/` backlog sat in the unpushed range, 13 blobs over 100 MB;
|
||||||
failure is current, not historical: the last two runs read
|
(4) both secondary legs were broken — the `github` remote did not exist, and its fallback pushed a
|
||||||
`[2026-11-02 11:00:32 AWST] ✗ Push to Gitea FAILED` / `--- Backup complete: FAIL ---` and
|
branch's full ancestry containing a 7.45 GB tarball blob that GitHub's 100 MB limit could never accept.
|
||||||
`[2026-11-03 11:00:31 AWST] ✗ Push to Gitea FAILED` / `--- Backup complete: FAIL ---`.
|
|
||||||
**A second, separate push failure is now also present:** the script attempts a `github` remote that **does not
|
|
||||||
exist** — `fatal: 'github' does not appear to be a git repository` → `✗ Push to GitHub FAILED`. So both off-box
|
|
||||||
targets are broken; the GitHub leg is a missing-remote error rather than a credential error, and would need
|
|
||||||
`git remote add github …` with credentials before it can work at all.
|
|
||||||
**The cron row still lies:** `e27da6c34bbb` reads `enabled: true, last_status: ok` in `jobs.json` while every
|
|
||||||
nightly execution has failed the push — the classic green-row/dead-artifact pattern.
|
|
||||||
|
|
||||||
**Full-backup gap unchanged:** the one full `hermes backup` zip on disk is still
|
**What was done:** host-scoped Gitea credential stored via `git credential approve`; `checkpoints/`
|
||||||
`/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, 22 Jul) — **now 105 days old**. No new full zip has
|
stripped from the unpushed range with `git filter-repo --partial` (push set ~2 GB → 264.8 MB, pre-rewrite
|
||||||
been taken. `crontab -l -u hermes` is still empty and there are still **0 user systemd timers**, so nothing
|
history preserved under tag `backup/pre-rewrite-20261107-032658`); the script rewritten to exit non-zero on
|
||||||
schedules the full zip; only `e5ed215fc91c` (**Mnemosyne DB Backup**, weekly `0 3 * * 0`) and `e27da6c34bbb`
|
failure, stamp `.last_backup_ok` only after remote verification, decide the push by **remote-vs-local**
|
||||||
(config backup, daily) exist in the cron fleet, and neither takes a full `hermes backup`.
|
rather than by working-tree changes, guard every `ls-remote` with `|| true` so an unreachable remote still
|
||||||
|
reaches the alert path, push GitHub as a **parentless orphan snapshot**, and report Telegram/AgentMail
|
||||||
|
delivery honestly (`curl --fail` + `"ok":true`; real `agentmail` CLI).
|
||||||
|
|
||||||
|
**Re-verified 2026-11-07 — live and green.** Real run exit **0** in 47 s; `git rev-parse HEAD` and
|
||||||
|
`git ls-remote origin refs/heads/main` both `a50a5170c1c0e2af5235fa63a95a732e2d780703`;
|
||||||
|
`rev-list --count origin/main..main` = **0**; Gitea API confirms `main` at `a50a5170c1c0`, updated
|
||||||
|
`2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub mirror
|
||||||
|
now carries a parentless orphan snapshot (`faed9fb5c677`, 0 parents, 30,086 files, largest blob 23.5 MB) —
|
||||||
|
no data lost, since GitHub's tree was a strict subset of local HEAD. The AgentMail and Telegram alert
|
||||||
|
paths were each exercised directly and report honestly.
|
||||||
|
|
||||||
|
**Consequence for this page: the topology above carries `~/.hermes` off the box again.** The one full
|
||||||
|
`hermes backup` zip on disk is still `/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, 22 Jul) —
|
||||||
|
**now 108 days old**, and no new full zip has been taken. `crontab -l -u hermes` is still empty and there
|
||||||
|
are still **0 user systemd timers**, so nothing schedules the full zip; only `e5ed215fc91c`
|
||||||
|
(**Mnemosyne DB Backup**, weekly `0 3 * * 0`) and `e27da6c34bbb` (config backup, daily) exist in the cron
|
||||||
|
fleet, and neither takes a full `hermes backup`. The nightly config backup is a *config* mirror, not a
|
||||||
|
full-home backup — the gap on the full zip is unchanged.
|
||||||
|
|
||||||
## Relevant runbooks
|
## Relevant runbooks
|
||||||
- [[runbooks/backup-wiki]]
|
- [[runbooks/backup-wiki]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Docker Services
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: mixed
|
status: mixed
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-06
|
updated: 2026-11-07
|
||||||
verified_on: 2026-11-06
|
verified_on: 2026-11-07
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [docker, homelab, infrastructure]
|
tags: [docker, homelab, infrastructure]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt, live-probe]
|
sources: [raw/configs/hermes-config-sanitized.txt, live-probe]
|
||||||
@@ -20,13 +20,13 @@ Docker runs on **four hosts**, and CT460's socket is not readable by the agent u
|
|||||||
|
|
||||||
| Host | Access | Containers running |
|
| Host | Access | Containers running |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** running / 68 total — main Docker host (**re-verified 2026-11-06**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) |
|
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** running / 68 total — main Docker host (**re-verified 2026-11-07**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) |
|
||||||
| CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** running / 26 total — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**re-verified 2026-11-05**) |
|
| CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** running / 26 total — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**re-verified 2026-11-07** — set-wise diff against live `docker ps`: no additions, no removals) |
|
||||||
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total — `flamboyant_mahavira`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → `flamboyant_mahavira` (2026-11-04) → **`nostalgic_keldysh`** (2026-11-06) — all `ghcr.io/github/github-mcp-server`, the live one created 2026-11-05T22:15:57Z |
|
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total (**re-verified 2026-11-07**) — `nostalgic_keldysh`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → `flamboyant_mahavira` (2026-11-04) → **`nostalgic_keldysh`** (2026-11-06) — all `ghcr.io/github/github-mcp-server`, the live one created 2026-11-05T22:15:57Z. ⚠ **Corrected 2026-11-07:** the enumerated list still opened with `flamboyant_mahavira`, which is **not running** — live `docker ps` on CT333 returns **`nostalgic_keldysh`** (created 2026-11-05 22:15:57, `Up 29 hours`), `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1` (8 running), plus `qdrant` **exited** (9 total) |
|
||||||
| CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** running / 9 total — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented |
|
| CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** running / 9 total — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented |
|
||||||
| CT460 `hermes` | `pct exec 460 -- docker ps` **as root** from the PVE host — socket is `root:docker srw-rw----`, the `hermes` user is not in group `docker` (its own `docker ps` → `permission denied while trying to connect to the docker API`) | `ignis` — **running** (`Up 5 days`, `ignis-subpath:0.8.9`, `0.0.0.0:8080->8080/tcp`), re-verified 2026-11-05 |
|
| CT460 `hermes` | `pct exec 460 -- docker ps` **as root** from the PVE host — socket is `root:docker srw-rw----`, the `hermes` user is not in group `docker` (its own `docker ps` → `permission denied while trying to connect to the docker API`) | `ignis` — **running** (`Up 5 days`, `ignis-subpath:0.8.9`, `0.0.0.0:8080->8080/tcp`), re-verified 2026-11-05 |
|
||||||
|
|
||||||
## CT200 live inventory (re-verified 2026-11-06, 61 running / 68 total, excluding 0–2 transient `worldmonitor-seeder-*` containers)
|
## CT200 live inventory (re-verified 2026-11-07, 61 running / 68 total, excluding 0–2 transient `worldmonitor-seeder-*` containers)
|
||||||
`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`.
|
`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`.
|
||||||
|
|
||||||
**Re-verified 2026-11-06 — the 2026-11-05 "the seeder does not exist" finding was WRONG, and the cause is a
|
**Re-verified 2026-11-06 — the 2026-11-05 "the seeder does not exist" finding was WRONG, and the cause is a
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Known Hosts and Paths
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: partial
|
status: partial
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-05
|
updated: 2026-11-07
|
||||||
verified_on: 2026-11-05
|
verified_on: 2026-11-07
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [infrastructure, hosts, containers, paths]
|
tags: [infrastructure, hosts, containers, paths]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -22,8 +22,8 @@ sources: []
|
|||||||
| `custom:omni-101` | OmniRoute proxy (CT101 node `omniroute-1`) — **the live default provider** | `http://100.78.63.20:20128` | Yes (live probe 2026-10-26) |
|
| `custom:omni-101` | OmniRoute proxy (CT101 node `omniroute-1`) — **the live default provider** | `http://100.78.63.20:20128` | Yes (live probe 2026-10-26) |
|
||||||
| CT201 | Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.77` (LAN) | Yes (live probe 2026-10-30 — LAN address corrected from `192.168.178.166`) |
|
| CT201 | Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.77` (LAN) | Yes (live probe 2026-10-30 — LAN address corrected from `192.168.178.166`) |
|
||||||
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net`, `100.118.5.51` (Tailscale) | Yes |
|
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net`, `100.118.5.51` (Tailscale) | Yes |
|
||||||
| PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-10-26) |
|
| PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-11-07 — 17 LXCs + 4 VMs) |
|
||||||
| PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-10-26) |
|
| PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-11-07 — 20 LXCs + 2 VMs) |
|
||||||
| CT450 (hermesagent) | Hermes Agent **v0.22.5** (2026.9.24), local `bf0203a794a0` — **same build as CT460**; hermes-webui, hermes-ui (Vite). ⚠ a stale **root-level** `/usr/local/bin/hermes` on CT450 still reports **v0.19.0** — the live agent is the `hermes`-user install under `~/.hermes/hermes-agent` | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes (live probe 2026-10-30 — version corrected from `v0.22.3`, build from `5b147ab036d6` to `bf0203a794a0`) |
|
| CT450 (hermesagent) | Hermes Agent **v0.22.5** (2026.9.24), local `bf0203a794a0` — **same build as CT460**; hermes-webui, hermes-ui (Vite). ⚠ a stale **root-level** `/usr/local/bin/hermes` on CT450 still reports **v0.19.0** — the live agent is the `hermes`-user install under `~/.hermes/hermes-agent` | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes (live probe 2026-10-30 — version corrected from `v0.22.3`, build from `5b147ab036d6` to `bf0203a794a0`) |
|
||||||
|
|
||||||
## Project Paths
|
## Project Paths
|
||||||
@@ -34,7 +34,7 @@ sources: []
|
|||||||
| `/home/hermes/workspace` | Workspace | default terminal cwd |
|
| `/home/hermes/workspace` | Workspace | default terminal cwd |
|
||||||
| `/home/hermes/.local/bin/hermes` | Hermes binary | |
|
| `/home/hermes/.local/bin/hermes` | Hermes binary | |
|
||||||
| `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` |
|
| `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` |
|
||||||
| `/home/hermes/.hermes/skills/` | Skills dir | **302 skills across 68 categories** (`.archive/` excluded — 46 archived; re-verified 2026-11-05) |
|
| `/home/hermes/.hermes/skills/` | Skills dir | **302 skills across 68 categories** (`.archive/` excluded — 46 archived; re-verified 2026-11-07) |
|
||||||
| `/home/hermes/.hermes/cron/` | Cron jobs | **29 jobs — 26 enabled / 3 disabled** (re-verified 2026-10-31 00:3x). ⚠ Superseded: the 2026-10-30 03:3x reading was 25 / 4 — the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure, but **that pause lasted only until 10:02:48**, when its long-running slot closed `completed`, incident `ea79a2_d93c3acdc4f2` auto-closed `resolved`, and the job returned to `enabled: true` / `last_status: ok` / streak 0. **CT460 itself was then dark for 5 h 30 m on 2026-10-30** (PVE `pve` rebooted; host boot 16:24:00, CT460 16:28:00, scheduler resumed 16:37:21 — ≈39 cron slots lost, 0 failures) → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] |
|
| `/home/hermes/.hermes/cron/` | Cron jobs | **29 jobs — 26 enabled / 3 disabled** (re-verified 2026-10-31 00:3x). ⚠ Superseded: the 2026-10-30 03:3x reading was 25 / 4 — the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure, but **that pause lasted only until 10:02:48**, when its long-running slot closed `completed`, incident `ea79a2_d93c3acdc4f2` auto-closed `resolved`, and the job returned to `enabled: true` / `last_status: ok` / streak 0. **CT460 itself was then dark for 5 h 30 m on 2026-10-30** (PVE `pve` rebooted; host boot 16:24:00, CT460 16:28:00, scheduler resumed 16:37:21 — ≈39 cron slots lost, 0 failures) → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] |
|
||||||
| `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ |
|
| `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ |
|
||||||
| `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 |
|
| `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 |
|
||||||
@@ -59,8 +59,8 @@ sources: []
|
|||||||
- `/home/hermes/workspace/readlater/docker-compose.yml`
|
- `/home/hermes/workspace/readlater/docker-compose.yml`
|
||||||
|
|
||||||
## Homelab References
|
## Homelab References
|
||||||
- **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-10-26 (17 LXCs)
|
- **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-11-07 (**17 LXCs** + 4 VMs; boot 2026-10-30 16:24:00)
|
||||||
- **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-10-26 (19 LXCs). Guest IDs collide between the two hosts
|
- **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-11-07 (**20 LXCs** + 2 VMs; boot 2026-11-05 21:19:52). ⚠ **Corrected 2026-11-07:** the previous figure of 19 LXCs omitted **CT107 (`mem0`)**, which is running — see [[infrastructure/proxmox]]. Guest IDs collide between the two hosts
|
||||||
- **CT600 (`omniroute`):** OmniRoute server, `100.88.81.19` — **running and live**; `omniroute` is the hostname that resolves to it
|
- **CT600 (`omniroute`):** OmniRoute server, `100.88.81.19` — **running and live**; `omniroute` is the hostname that resolves to it
|
||||||
- **CT601 (`omni`):** the old CT601 OmniRoute guest **no longer exists** on either hypervisor (`pct config 601` → no such config on both `pve` and `m93p`). The orphaned `omni` Tailscale peer (`100.93.204.4`) has since **expired out of the tailnet entirely** — as of the 2026-10-30 live probe the tailnet contains **no** peer named `omni` and no peer at `100.93.204.4`, so the old "offline 69d" description no longer applies. The only OmniRoute nodes left are CT600 (`omniroute`) and CT101 (`omniroute-1`).
|
- **CT601 (`omni`):** the old CT601 OmniRoute guest **no longer exists** on either hypervisor (`pct config 601` → no such config on both `pve` and `m93p`). The orphaned `omni` Tailscale peer (`100.93.204.4`) has since **expired out of the tailnet entirely** — as of the 2026-10-30 live probe the tailnet contains **no** peer named `omni` and no peer at `100.93.204.4`, so the old "offline 69d" description no longer applies. The only OmniRoute nodes left are CT600 (`omniroute`) and CT101 (`omniroute-1`).
|
||||||
- **CT101 (`omniroute`, m93p):** second OmniRoute node, `100.78.63.20` — running
|
- **CT101 (`omniroute`, m93p):** second OmniRoute node, `100.78.63.20` — running
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Network Map
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-06
|
updated: 2026-11-07
|
||||||
verified_on: 2026-11-06
|
verified_on: 2026-11-07
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [infrastructure, network]
|
tags: [infrastructure, network]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt, live-probe:ct460-ss+tailscale-serve-status-2026-11-05]
|
sources: [raw/configs/hermes-config-sanitized.txt, live-probe:ct460-ss+tailscale-serve-status-2026-11-05]
|
||||||
@@ -22,9 +22,9 @@ Where Hermes and its dependencies run, and how traffic reaches them.
|
|||||||
- **This container (hermes, ct460):** `192.168.178.129` (DHCP, eth0)
|
- **This container (hermes, ct460):** `192.168.178.129` (DHCP, eth0)
|
||||||
|
|
||||||
### Tailscale (`kangaroo-eel.ts.net`)
|
### Tailscale (`kangaroo-eel.ts.net`)
|
||||||
_Peer IPs below re-checked 2026-11-06 against `tailscale status` on ct460; `webserver`, `clawtest`, `hermesagent`, `docker`, `omniroute` all matched the values listed. Full 52-row node table with live states lives in [[infrastructure/tailscale]]._
|
_Peer IPs below re-checked 2026-11-07 against `tailscale status` on ct460; `webserver`, `clawtest`, `hermesagent`, `docker`, `omniroute` all matched the values listed. Full 52-row node table with live states lives in [[infrastructure/tailscale]]._
|
||||||
- **pve:** `100.96.100.82`
|
- **pve:** `100.96.100.82`
|
||||||
- **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.0), discovered 2026-10-14.** See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host.
|
- **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.10), discovered 2026-10-14.** ⚠ Corrected 2026-11-07: this line still read *"9.2.0"* although both hosts run the identical `pve-manager 9.2.10` build (live `dpkg -l pve-manager` on both). See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host.
|
||||||
- **hermes (ct460):** `100.118.5.51`
|
- **hermes (ct460):** `100.118.5.51`
|
||||||
- **hermesagent (ct450, the original/legacy Hermes agent dashboard):** `100.98.205.112`
|
- **hermesagent (ct450, the original/legacy Hermes agent dashboard):** `100.98.205.112`
|
||||||
- **omniroute (ct600):** resolves via `omniroute.kangaroo-eel.ts.net` → `100.88.81.19`
|
- **omniroute (ct600):** resolves via `omniroute.kangaroo-eel.ts.net` → `100.88.81.19`
|
||||||
@@ -68,7 +68,7 @@ _Peer IPs below re-checked 2026-11-06 against `tailscale status` on ct460; `webs
|
|||||||
|
|
||||||
Loopback-only helpers (not routable): `49347` hermes gateway, `42013` Paseo, `42647` + `38187` opencode.
|
Loopback-only helpers (not routable): `49347` hermes gateway, `42013` Paseo, `42647` + `38187` opencode.
|
||||||
|
|
||||||
## Tailscale Serve / Funnel (verified 2026-11-06)
|
## Tailscale Serve / Funnel (verified 2026-11-07)
|
||||||
|
|
||||||
`tailscale serve status` on ct460 reports 14 proxies: 12 under `https://hermes.kangaroo-eel.ts.net` (tailnet only), one tailnet route on `:9443`, and one Funnel route on `:8443` (public).
|
`tailscale serve status` on ct460 reports 14 proxies: 12 under `https://hermes.kangaroo-eel.ts.net` (tailnet only), one tailnet route on `:9443`, and one Funnel route on `:8443` (public).
|
||||||
|
|
||||||
@@ -80,12 +80,12 @@ Loopback-only helpers (not routable): `49347` hermes gateway, `42013` Paseo, `42
|
|||||||
| `/wiki` | `100.118.5.51:9120` | Agent Estate Wiki |
|
| `/wiki` | `100.118.5.51:9120` | Agent Estate Wiki |
|
||||||
| `/webui` | `127.0.0.1:8787` | hermes-webui |
|
| `/webui` | `127.0.0.1:8787` | hermes-webui |
|
||||||
| `/ignis` | `127.0.0.1:8080` | Ignis |
|
| `/ignis` | `127.0.0.1:8080` | Ignis |
|
||||||
| `/gitea` | `100.82.100.128:3000` | Gitea (ct207 tailnet IP) |
|
| `/gitea` | `100.82.100.128:3000` | Gitea — **CT221 on `m93p`**, not ct207 (⚠ **corrected 2026-11-07**: this line wrongly said *"ct207 tailnet IP"*; live `pct exec 221 -- tailscale ip -4` → `100.82.100.128`, and ct207's own IP is `100.82.40.23`). The target answers **HTTP 200**. |
|
||||||
| `/desktop` | `127.0.0.1:6081` | desktop VNC (noVNC) |
|
| `/desktop` | `127.0.0.1:6081` | desktop VNC (noVNC) |
|
||||||
| `/vnc-camofox` | `127.0.0.1:6080` | Camofox VNC |
|
| `/vnc-camofox` | `127.0.0.1:6080` | Camofox VNC |
|
||||||
| `/evening-grid.html` | `127.0.0.1:9099/evening-grid.html` | jobs dashboard page |
|
| `/evening-grid.html` | `127.0.0.1:9099/evening-grid.html` | jobs dashboard page |
|
||||||
| `/agent-estate` | `127.0.0.1:8769` | **DEAD — nothing listening on 8769 (re-confirmed 2026-11-06)** |
|
| `/agent-estate` | `127.0.0.1:8769` | **DEAD — nothing listening on 8769 (re-confirmed 2026-11-07)** |
|
||||||
| `/synergy-prep` | `127.0.0.1:9123` | **DEAD — nothing listening on 9123 (re-confirmed 2026-11-06)** |
|
| `/synergy-prep` | `127.0.0.1:9123` | **DEAD — nothing listening on 9123 (re-confirmed 2026-11-07)** |
|
||||||
|
|
||||||
- `:9443` (tailnet only) — `/` → `127.0.0.1:5173`.
|
- `:9443` (tailnet only) — `/` → `127.0.0.1:5173`.
|
||||||
- Funnel `:8443` (public) — `/webhook` → `127.0.0.1:8085/webhook` (AgentMail webhook receiver).
|
- Funnel `:8443` (public) — `/webhook` → `127.0.0.1:8085/webhook` (AgentMail webhook receiver).
|
||||||
|
|||||||
+35
-23
@@ -3,8 +3,8 @@ title: Proxmox VE
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-26
|
created: 2026-07-26
|
||||||
updated: 2026-11-01
|
updated: 2026-11-07
|
||||||
verified_on: 2026-11-01
|
verified_on: 2026-11-07
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [proxmox, virtualization, infrastructure, homelab]
|
tags: [proxmox, virtualization, infrastructure, homelab]
|
||||||
---
|
---
|
||||||
@@ -21,7 +21,8 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
|
|||||||
- **Primary user:** `Anthony@pam` (full admin)
|
- **Primary user:** `Anthony@pam` (full admin)
|
||||||
- **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve`
|
- **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve`
|
||||||
|
|
||||||
## Live LXC Inventory (verified 2026-10-31 via `pct list`)
|
## Live LXC Inventory (verified 2026-11-07 via `pct list`)
|
||||||
|
Re-verified 2026-11-07 03:2x AWST (stocktake): **live `pct list` on `pve` still returns exactly the 17 LXCs in the table below** (CT255 `mattermost` `running`, CT209 `cloudreve` and CT555 `elementsynapse` `stopped`, CT600 `omniroute` `running`, and **no CT601 on either hypervisor**), and **`qm list` on `pve` still returns exactly the 4 VMs** (100 `homeassistant`, 101 `guest-vps`, 110 `omarchy`, 282 `ubuntu-server-xfce` — all running; **no 310, no 403, no 401**). Both `pve-manager` packages are still **9.2.10** on **both** hosts. The **one correction** was on the `m93p` side: see the `m93p` section below — **CT107 (`mem0`) was missing**, so the m93p guest count was one short.
|
||||||
|
|
||||||
Re-verified 2026-11-01 03:5x AWST (stocktake): **`pct list` on `pve` and `qm list` on `pve` both still match the tables below exactly** — 17 LXCs and 4 VMs on `pve`, with CT255 `mattermost` still `running`, CT209 `cloudreve` and CT555 `elementsynapse` still `stopped`, and **no CT601 on either hypervisor** (`/etc/pve/lxc/601.conf` absent). **`m93p` re-confirmed**: `pve-manager/9.2.10` (same build as `pve`), `qm list` returns **VM 103 `cachyos-m93p` stopped** and **VM 106 `openmausbot-vm` running**, and `pct exec 1001 -- docker ps` returns `sshwifty` (CT1001). Storage percentages were refreshed against live `pvesm status` on both hosts (see the Storage section). **Note:** `m93p` SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout).
|
Re-verified 2026-11-01 03:5x AWST (stocktake): **`pct list` on `pve` and `qm list` on `pve` both still match the tables below exactly** — 17 LXCs and 4 VMs on `pve`, with CT255 `mattermost` still `running`, CT209 `cloudreve` and CT555 `elementsynapse` still `stopped`, and **no CT601 on either hypervisor** (`/etc/pve/lxc/601.conf` absent). **`m93p` re-confirmed**: `pve-manager/9.2.10` (same build as `pve`), `qm list` returns **VM 103 `cachyos-m93p` stopped** and **VM 106 `openmausbot-vm` running**, and `pct exec 1001 -- docker ps` returns `sshwifty` (CT1001). Storage percentages were refreshed against live `pvesm status` on both hosts (see the Storage section). **Note:** `m93p` SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout).
|
||||||
|
|
||||||
@@ -103,7 +104,7 @@ as a hypervisor.
|
|||||||
|
|
||||||
- **Hostname:** `m93p` · **Tailscale:** `100.104.51.98`
|
- **Hostname:** `m93p` · **Tailscale:** `100.104.51.98`
|
||||||
- **PVE version:** **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, kernel `7.0.14-12-pve`) — **corrected 2026-10-31**. The previous entry read *"9.2.0"* and claimed `m93p` was *"materially newer than pve's 8.x"*. Both hosts now run the **identical `pve-manager 9.2.10` build**; only the kernel differs (`7.0.14-12-pve` vs `7.0.14-5-pve`). The hosts are **version-identical, not version-skewed**.
|
- **PVE version:** **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, kernel `7.0.14-12-pve`) — **corrected 2026-10-31**. The previous entry read *"9.2.0"* and claimed `m93p` was *"materially newer than pve's 8.x"*. Both hosts now run the **identical `pve-manager 9.2.10` build**; only the kernel differs (`7.0.14-12-pve` vs `7.0.14-5-pve`). The hosts are **version-identical, not version-skewed**.
|
||||||
- **Uptime:** **up 1 day, 12:02** at 2026-11-01 03:56 AWST (live `uptime`) — the host **rebooted ~2026-10-30 16:0x**, consistent with the CT1264 openmausbot outage below. Load average was **7.66 / 12.80 / 11.32** (1/5/15 min) at that moment — elevated, worth watching
|
- **Uptime:** **up 1 day, 7:18** at 2026-11-07 04:38 AWST (live `uptime -s` → **boot 2026-11-05 21:19:52**; `uptime` → `up 1 day, 7:18`, load average **0.76 / 1.08 / 1.24** — normal). ⚠ **Corrected 2026-11-07:** the previous entry read *"up 1 day, 12:02 at 2026-11-01 03:56 AWST — the host rebooted ~2026-10-30 16:0x"*. The **reboot itself is still real** (`pve` booted **2026-10-30 16:24:00** per `uptime -s`, and the CT1264 openmausbot outage below follows from it), but the **uptime figure and the implied boot date were wrong** — the host has since rebooted again on **2026-11-05 21:19:52**, so it was up only **1 day, 7:18** when probed, not 12 h. The `pve` host's own boot time is **2026-10-30 16:24:00** (up **7 days, 12:14** at probe time)
|
||||||
- **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf`
|
- **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf`
|
||||||
|
|
||||||
### LXC Inventory (verified 2026-10-31 via `pct list` on m93p)
|
### LXC Inventory (verified 2026-10-31 via `pct list` on m93p)
|
||||||
@@ -115,6 +116,7 @@ as a hypervisor.
|
|||||||
| 102 | vaultwarden | running | Vaultwarden password manager |
|
| 102 | vaultwarden | running | Vaultwarden password manager |
|
||||||
| 104 | rclone | running | Rclone |
|
| 104 | rclone | running | Rclone |
|
||||||
| 105 | conduit | running | **Conduit** — iPhone calendar / health / location / contacts bridge (see [[systems/messaging-integrations]]). **Added 2026-10-31** — this guest was missing from the 2026-10-14 table despite running |
|
| 105 | conduit | running | **Conduit** — iPhone calendar / health / location / contacts bridge (see [[systems/messaging-integrations]]). **Added 2026-10-31** — this guest was missing from the 2026-10-14 table despite running |
|
||||||
|
| 107 | mem0 | running | **mem0 MCP server** (`mem0-mcp.service` → `:8890`), Tailscale `100.73.94.53` (`mem0` peer) — **added 2026-11-07**, this guest was missing from the table despite running |
|
||||||
| 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** |
|
| 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** |
|
||||||
| 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one |
|
| 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one |
|
||||||
| 500 | litellm | running | LiteLLM proxy |
|
| 500 | litellm | running | LiteLLM proxy |
|
||||||
@@ -130,8 +132,15 @@ as a hypervisor.
|
|||||||
| 1263 | social-dashboard | running | Social dashboard |
|
| 1263 | social-dashboard | running | Social dashboard |
|
||||||
| 1264 | openmausbot | **stopped** | OpenMausBot — **stopped**, which is why the `openmausbot` Tailscale peer is offline 9d. **Added 2026-10-31** — was missing from the 2026-10-14 table. The companion **VM106 `openmausbot-vm` is running** and holds the separate `openmausbot-vm` peer (`100.77.107.30`) |
|
| 1264 | openmausbot | **stopped** | OpenMausBot — **stopped**, which is why the `openmausbot` Tailscale peer is offline 9d. **Added 2026-10-31** — was missing from the 2026-10-14 table. The companion **VM106 `openmausbot-vm` is running** and holds the separate `openmausbot-vm` peer (`100.77.107.30`) |
|
||||||
|
|
||||||
**Re-verified 2026-10-31:** live `pct list` on `m93p` returns **19 LXCs — 18 running, 1 stopped
|
**Re-verified 2026-11-07 03:2x AWST (stocktake):** live `pct list` on `m93p` returns **20 LXCs — 18 running,
|
||||||
(CT998 nightbridge) plus CT1264 openmausbot stopped**. The 2026-10-14 table listed only **17**, omitting
|
2 stopped (CT998 `nightbridge` and CT1264 `openmausbot`)**. ⚠ **The count was one short**: the previous
|
||||||
|
figure of 19 omitted **CT107 (`mem0`)**, a live running guest confirmed from inside the container
|
||||||
|
(`pct exec 107 -- tailscale ip -4` → `100.73.94.53`, `systemctl is-active tailscaled` → `active`,
|
||||||
|
`mem0-mcp.service` → `active`). CT107 is now in the table above.
|
||||||
|
|
||||||
|
**Re-verified 2026-10-31:** the earlier reading of **19 LXCs — 18 running, 1 stopped
|
||||||
|
(CT998 nightbridge) plus CT1264 openmausbot stopped** was itself already one short for the same reason
|
||||||
|
(CT107 `mem0` was not counted). The 2026-10-14 table listed only **17**, omitting
|
||||||
two live guests: **CT105 (`conduit`)** and **CT1264 (`openmausbot`)**. Note that **VMIDs 103, 220 and 555
|
two live guests: **CT105 (`conduit`)** and **CT1264 (`openmausbot`)**. Note that **VMIDs 103, 220 and 555
|
||||||
do not exist on `m93p`** (`/etc/pve/lxc/{103,220,555}.conf` all absent) — CT103 (cliproxyapi),
|
do not exist on `m93p`** (`/etc/pve/lxc/{103,220,555}.conf` all absent) — CT103 (cliproxyapi),
|
||||||
CT220 (caddy) and CT555 (elementsynapse) are on **`pve`** only, and `pve`'s CT555 is **stopped** while
|
CT220 (caddy) and CT555 (elementsynapse) are on **`pve`** only, and `pve`'s CT555 is **stopped** while
|
||||||
@@ -162,36 +171,39 @@ tailnet. → [[systems/omniroute]], [[infrastructure/hosts]]
|
|||||||
|
|
||||||
| Datastore | Type | Status | Total | Used | Used % |
|
| Datastore | Type | Status | Total | Used | Used % |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 700 GiB | **76.43 %** (`pvesm`) |
|
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 717 GiB | **78.29 %** (`pvesm`) |
|
||||||
| `SSD1` | lvmthin | active | 932 GiB | 599 GiB | **64.32 %** |
|
| `SSD1` | lvmthin | active | 931 GiB | 610 GiB | **65.46 %** |
|
||||||
| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.0 TiB | **82.77 %** |
|
| `fred-qnap` | **nfs** | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
|
||||||
| `local` | dir | active | 67.7 GiB | 54.0 GiB | **79.66 %** |
|
| `local` | dir | active | 67.7 GiB | 55.3 GiB | **81.62 %** |
|
||||||
| `local-lvm` | lvmthin | active | 141 GiB | 19.4 GiB | **13.75 %** |
|
| `local-lvm` | lvmthin | active | 141 GiB | 20.1 GiB | **14.25 %** |
|
||||||
|
|
||||||
**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 %**, ~169 GiB free. `fred-qnap` is now
|
**`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 % (2026-11-01) → 78.29 % (2026-11-07)**, ~152 GiB free. `fred-qnap` is now
|
||||||
**82.45 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.0 TiB used) — the share was
|
**84.21 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.3 TiB used) — the share was
|
||||||
remounted over NFS rather than SMB.
|
remounted over NFS rather than SMB.
|
||||||
|
|
||||||
|
**Storage re-verified 2026-11-07 03:2x AWST** against live `pvesm status` on both hosts — every
|
||||||
|
percentage on both tables below was refreshed.
|
||||||
|
|
||||||
**`iMacHDD` no longer appears in `pvesm status` on `pve`** — the 2026-10-14 table listed it as an
|
**`iMacHDD` no longer appears in `pvesm status` on `pve`** — the 2026-10-14 table listed it as an
|
||||||
inactive `cifs` entry. The iMac shares now live on **`m93p`** as `cifs-anthonyimac` and `cifs-imachdd`
|
inactive `cifs` entry. The iMac shares now live on **`m93p`** as `cifs-anthonyimac` and `cifs-imachdd`
|
||||||
(both active, 60.25 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing.
|
(both active, 76.21 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing.
|
||||||
|
|
||||||
### Host `m93p` (100.104.51.98)
|
### Host `m93p` (100.104.51.98)
|
||||||
|
|
||||||
| Datastore | Type | Status | Total | Used | Used % |
|
| Datastore | Type | Status | Total | Used | Used % |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| `cifs-anthonyimac` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
|
| `cifs-anthonyimac` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
|
||||||
| `cifs-imachdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
|
| `cifs-imachdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
|
||||||
| `local` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
|
| `local` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
|
||||||
| `local-hdd` | dir | active | 93.9 GiB | 56.6 GiB | 60.25 % |
|
| `local-hdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
|
||||||
| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** |
|
| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
|
||||||
| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.0 TiB | **82.78 %** |
|
| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
|
||||||
| `nfs-ssd0-39` | dir | active | 916 GiB | 700 GiB | **76.43 %** |
|
| `nfs-ssd0-39` | dir | active | 916 GiB | 717 GiB | **78.29 %** |
|
||||||
| `omb-internal` | dir | active | 93.9 GiB | 54.4 GiB | **57.92 %** |
|
| `omb-internal` | dir | active | 93.9 GiB | 60.7 GiB | **64.66 %** |
|
||||||
|
|
||||||
**`m93p` storage was not documented before 2026-10-31.** Note that `m93p` exposes the **same backing
|
**`m93p` storage was not documented before 2026-10-31.** Note that `m93p` exposes the **same backing
|
||||||
volumes as `pve`** under different names — `nfs-proxmox-31`/`nfs-proxmox-54` mirror `pve`'s `fred-qnap`
|
volumes as `pve`** under different names — `nfs-proxmox-31`/`nfs-proxmox-54` mirror `pve`'s `fred-qnap`
|
||||||
(both 82.45 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 76.43 %). The iMac shares that were
|
(both 84.21 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 78.29 %). The iMac shares that were
|
||||||
`iMacHDD` on `pve` are `cifs-anthonyimac` / `cifs-imachdd` on `m93p`.
|
`iMacHDD` on `pve` are `cifs-anthonyimac` / `cifs-imachdd` on `m93p`.
|
||||||
|
|
||||||
**Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`).
|
**Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`).
|
||||||
|
|||||||
+16
-15
@@ -3,8 +3,8 @@ title: Tailscale
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-26
|
created: 2026-07-26
|
||||||
updated: 2026-11-06
|
updated: 2026-11-07
|
||||||
verified_on: 2026-11-06
|
verified_on: 2026-11-07
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [tailscale, vpn, networking, infrastructure]
|
tags: [tailscale, vpn, networking, infrastructure]
|
||||||
---
|
---
|
||||||
@@ -19,7 +19,8 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
|
|||||||
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
|
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
|
||||||
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
|
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
|
||||||
|
|
||||||
## Tailscale Node Inventory (verified 2026-11-06)
|
## Tailscale Node Inventory (verified 2026-11-07)
|
||||||
|
Re-verified 2026-11-07 03:1x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 9 offline-age drifts, no IP changes and no missing peers.** **`octop` came back online**: the peer record reads `Online: true` with `LastSeen 0001-01-01` and `tailscale ping 100.92.163.122` → **`pong from octop via DERP(syd) in 98ms`**, so the row moves offline → online. It remains **not an LXC or VM on either hypervisor** (live `pct list` / `qm list` on both `pve` and `m93p` list no such guest) — an **orphaned tailnet peer**, so its return is not an estate change. The **9 offline ages** were refreshed (+1d each): cloudreve 104d → **105d**; conduit 60d → **61d**; elementsynapse 107d → **108d**; freshrss-new 80d → **81d**; octoagent 4d → **5d**; omarchybook 15d → **16d**; omarchymac 4d → **5d**; openmausbot 15d → **16d**; ubuntu-server-xfce 76d → **77d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) **17 guests** and `m93p` (100.104.51.98) **20 guests + 2 VMs** — ⚠ **the m93p guest count on this page was one short**: live `pct list` on `m93p` returns **20 LXCs** (the extra is **CT107 `mem0`**, `100.73.94.53`, `mem0-mcp.service` active — confirmed from inside the guest), not 19; CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT998 `nightbridge` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). The route table still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; **both DEAD targets re-confirmed dead** — `ss -ltn` still shows **no listener on 8769** (`/agent-estate`) and **none on 9123** (`/synergy-prep`) — while every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` answers **HTTP 200**.
|
||||||
Re-verified 2026-11-06 03:0x–03:2x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 8 offline-age drifts, no IP changes and no missing peers.** **`anthonys-mac-mini` came back online**: the peer record reads `Online: true` and `tailscale ping 100.76.60.37` → `pong from anthonys-mac-mini via DERP(syd) in 103ms`, so the row moves offline → online (client device, not an estate host). The **8 offline ages** were refreshed (+1d each): cloudreve 103d → **104d**; elementsynapse 106d → **107d**; freshrss-new 79d → **80d**; octoagent 3d → **4d**; octop 1d → **2d**; omarchybook 14d → **15d**; omarchymac 3d → **4d**; openmausbot 14d → **15d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). `m93p` re-pinged → `pong from m93p via 192.168.178.250:41641 in 3ms` (online, peer record agrees), and `octop` re-pinged → **timeout** (genuinely offline, matching its `LastSeen`). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) 17 guests and `m93p` (100.104.51.98) 19 guests + 2 VMs — CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). ⚠ **Two CT460 Serve routes are dead but still present in `tailscale serve status`**: `/agent-estate` → `127.0.0.1:8769` and `/synergy-prep` → `127.0.0.1:9123` have **no listener** (live `ss -ltn` probe) and therefore serve 502 — both rows are now marked DEAD here, matching [[infrastructure/network-map]] which had already recorded them as dead on 2026-11-05. The route table itself still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` is reachable.
|
Re-verified 2026-11-06 03:0x–03:2x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 8 offline-age drifts, no IP changes and no missing peers.** **`anthonys-mac-mini` came back online**: the peer record reads `Online: true` and `tailscale ping 100.76.60.37` → `pong from anthonys-mac-mini via DERP(syd) in 103ms`, so the row moves offline → online (client device, not an estate host). The **8 offline ages** were refreshed (+1d each): cloudreve 103d → **104d**; elementsynapse 106d → **107d**; freshrss-new 79d → **80d**; octoagent 3d → **4d**; octop 1d → **2d**; omarchybook 14d → **15d**; omarchymac 3d → **4d**; openmausbot 14d → **15d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). `m93p` re-pinged → `pong from m93p via 192.168.178.250:41641 in 3ms` (online, peer record agrees), and `octop` re-pinged → **timeout** (genuinely offline, matching its `LastSeen`). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) 17 guests and `m93p` (100.104.51.98) 19 guests + 2 VMs — CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). ⚠ **Two CT460 Serve routes are dead but still present in `tailscale serve status`**: `/agent-estate` → `127.0.0.1:8769` and `/synergy-prep` → `127.0.0.1:9123` have **no listener** (live `ss -ltn` probe) and therefore serve 502 — both rows are now marked DEAD here, matching [[infrastructure/network-map]] which had already recorded them as dead on 2026-11-05. The route table itself still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` is reachable.
|
||||||
Full node list from `tailscale status` — **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
|
Full node list from `tailscale status` — **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
|
||||||
Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec <vmid> -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
|
Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec <vmid> -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
|
||||||
@@ -44,14 +45,14 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
|||||||
| anthonys-mac-mini | 100.76.60.37 | online — **returned 2026-11-06** (peer record `Online: true`, `tailscale ping` → `pong from anthonys-mac-mini (100.76.60.37) via DERP(syd) in 103ms`; client device, not an estate host) |
|
| anthonys-mac-mini | 100.76.60.37 | online — **returned 2026-11-06** (peer record `Online: true`, `tailscale ping` → `pong from anthonys-mac-mini (100.76.60.37) via DERP(syd) in 103ms`; client device, not an estate host) |
|
||||||
| clawtest | 100.122.189.81 | online — CT333 (pve) |
|
| clawtest | 100.122.189.81 | online — CT333 (pve) |
|
||||||
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
|
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
|
||||||
| cloudreve | 100.111.187.7 | offline 104d — CT209 stopped |
|
| cloudreve | 100.111.187.7 | offline 105d — CT209 stopped |
|
||||||
| conduit | 100.81.79.87 | offline 60d — last seen 2026-09-06 01:32Z |
|
| conduit | 100.81.79.87 | offline 61d — last seen 2026-09-06 01:32Z |
|
||||||
| docker | 100.96.150.69 | online — CT200 (pve) |
|
| docker | 100.96.150.69 | online — CT200 (pve) |
|
||||||
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
|
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
|
||||||
| elementsynapse | 100.75.250.37 | offline 107d — CT555 stopped |
|
| elementsynapse | 100.75.250.37 | offline 108d — CT555 stopped |
|
||||||
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
|
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
|
||||||
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
|
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
|
||||||
| freshrss-new | 100.110.180.115 | offline 80d |
|
| freshrss-new | 100.110.180.115 | offline 81d |
|
||||||
| gitea | 100.82.100.128 | online — CT221 (m93p) |
|
| gitea | 100.82.100.128 | online — CT221 (m93p) |
|
||||||
| guest-vps | 100.117.14.84 | online |
|
| guest-vps | 100.117.14.84 | online |
|
||||||
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
|
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
|
||||||
@@ -65,15 +66,15 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
|||||||
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
|
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
|
||||||
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
|
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
|
||||||
| obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) |
|
| obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) |
|
||||||
| octoagent | 100.117.10.68 | offline 4d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
|
| octoagent | 100.117.10.68 | offline 5d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
|
||||||
| octop | 100.92.163.122 | offline 2d — last seen 2026-11-03 09:03Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
|
| octop | 100.92.163.122 | online — **returned 2026-11-07** (peer record `Online: true`, `tailscale ping` → `pong from octop via DERP(syd) in 98ms`); still **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
|
||||||
| ollama | 100.108.44.119 | online — CT205 (pve) |
|
| ollama | 100.108.44.119 | online — CT205 (pve) |
|
||||||
| omarchy | 100.73.227.103 | online |
|
| omarchy | 100.73.227.103 | online |
|
||||||
| omarchybook | 100.107.9.100 | offline 15d |
|
| omarchybook | 100.107.9.100 | offline 16d |
|
||||||
| omarchymac | 100.126.173.115 | offline 4d — last seen 2026-11-01 11:13Z |
|
| omarchymac | 100.126.173.115 | offline 5d — last seen 2026-11-01 11:13Z |
|
||||||
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
|
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
|
||||||
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
|
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
|
||||||
| openmausbot | 100.91.214.91 | offline 15d — CT1264 (m93p) stopped |
|
| openmausbot | 100.91.214.91 | offline 16d — CT1264 (m93p) stopped |
|
||||||
| openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) |
|
| openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) |
|
||||||
| openweb | 100.69.230.13 | online — CT245 (m93p) |
|
| openweb | 100.69.230.13 | online — CT245 (m93p) |
|
||||||
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
|
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
|
||||||
@@ -86,7 +87,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
|||||||
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
|
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
|
||||||
| springroll | 100.100.153.125 | online |
|
| springroll | 100.100.153.125 | online |
|
||||||
| springroll-browser | 100.109.221.91 | online |
|
| springroll-browser | 100.109.221.91 | online |
|
||||||
| ubuntu-server-xfce | 100.101.26.104 | offline 76d — last seen 2026-08-20 19:36Z |
|
| ubuntu-server-xfce | 100.101.26.104 | offline 77d — last seen 2026-08-20 19:36Z |
|
||||||
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
|
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
|
||||||
| webserver | 100.82.40.23 | online — CT207 (pve) |
|
| webserver | 100.82.40.23 | online — CT207 (pve) |
|
||||||
|
|
||||||
@@ -106,8 +107,8 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
|||||||
| `/gitea` | `http://100.82.100.128:3000` | Gitea |
|
| `/gitea` | `http://100.82.100.128:3000` | Gitea |
|
||||||
| `/desktop` | `http://127.0.0.1:6081` | CUA noVNC |
|
| `/desktop` | `http://127.0.0.1:6081` | CUA noVNC |
|
||||||
| `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC |
|
| `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC |
|
||||||
| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep — ⚠ **DEAD: nothing listening on 9123** (live `ss -ltn` probe 2026-11-06; the route still exists in `tailscale serve status` but serves 502) |
|
| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep — ⚠ **DEAD: nothing listening on 9123** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07**; the route still exists in `tailscale serve status` but serves 502) |
|
||||||
| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) — ⚠ **DEAD: nothing listening on 8769** (live `ss -ltn` probe 2026-11-06; route still present in `tailscale serve status` but serves 502) |
|
| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) — ⚠ **DEAD: nothing listening on 8769** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07**; route still present in `tailscale serve status` but serves 502) |
|
||||||
| `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid |
|
| `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid |
|
||||||
|
|
||||||
Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 route table above matched live exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; no drift).
|
Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 route table above matched live exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; no drift).
|
||||||
|
|||||||
@@ -2226,3 +2226,107 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
|
|||||||
[[current-state]] (+frontmatter, +Recent Changes block), this index (+frontmatter, +summary, +incident
|
[[current-state]] (+frontmatter, +Recent Changes block), this index (+frontmatter, +summary, +incident
|
||||||
bullet), [[runbooks/job-radar]] (+concurrency subsection), this log, and the new incident page. Vault
|
bullet), [[runbooks/job-radar]] (+concurrency subsection), this log, and the new incident page. Vault
|
||||||
mirror `/home/hermes/wiki` updated (both conventions).
|
mirror `/home/hermes/wiki` updated (both conventions).
|
||||||
|
## 2026-11-07 03:0x AWST - stocktake (wiki-stocktake-maintenance)
|
||||||
|
|
||||||
|
- **Script:** `wiki-maintain.sh` (repo, canonical) run clean — skills-index regenerated (**302 skills /
|
||||||
|
68 categories**), freshness report written to `raw/stocktake/freshness-20261107.txt`, Quartz rebuilt
|
||||||
|
(**769 files**) and deployed to `public/`, `hermes-wiki-serve.service` restarted.
|
||||||
|
- **Freshness:** **0 stale, 125 fresh, 3 no-date of 128 pages** — no page is past the 21-day threshold.
|
||||||
|
(`log.md` has no `updated` date and lands in the no-date bucket, which is correct for an append-only log.)
|
||||||
|
**Freshness is not accuracy**, so a live-vs-page diff was run against every page carrying live claims.
|
||||||
|
- **Tailscale (infrastructure/tailscale.md):** table rebuilt from `tailscale status --json`. **1 state
|
||||||
|
transition — `octop` (`100.92.163.122`) is back online** (peer record `Online: true`; `tailscale ping` →
|
||||||
|
`pong from octop via DERP(syd) in 98ms`); it is still **no LXC or VM on either hypervisor**, so it is an
|
||||||
|
orphaned tailnet peer, not an estate change. **9 offline ages refreshed (+1d each).** All **51 live IPv4
|
||||||
|
peers** match the table's IPs exactly — **no missing peers, no orphan rows, no IP mismatches** (52 rows =
|
||||||
|
51 peers + `hermes`; the ~23 IPv6-only `funnel-ingress-node` peers excluded by design). The CT460 Serve
|
||||||
|
route table still matches `tailscale serve status` **exactly (12 paths on :443)**; **both DEAD targets
|
||||||
|
re-confirmed dead** (no listener on `8769` `/agent-estate`, none on `9123` `/synergy-prep`), every other
|
||||||
|
Serve target has a live listener, and `/gitea` (`100.82.100.128:3000`) answers **HTTP 200**.
|
||||||
|
- **Proxmox (infrastructure/proxmox.md):** ⚠ **`m93p` runs 20 LXCs, not 19 — CT107 `mem0` was missing**
|
||||||
|
(running; `mem0-mcp.service` active on `:8890`; `pct exec 107 -- tailscale ip -4` → `100.73.94.53`).
|
||||||
|
Added to the table; the 2026-10-31 reading of 19 was already short for the same reason. Live now reads
|
||||||
|
**20 LXCs — 18 running, 2 stopped** (CT998, CT1264). `pve` still matches exactly: **17 LXCs + 4 VMs**.
|
||||||
|
Both hosts run the identical **`pve-manager 9.2.10`** build.
|
||||||
|
- **⚠ `m93p` uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*; live
|
||||||
|
`uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18**, load **0.76 / 1.08 / 1.24** — normal).
|
||||||
|
The 2026-10-30 reboot was real but belonged to **`pve`** (boot **2026-10-30 16:24:00**).
|
||||||
|
- **Storage refreshed on both hosts** against live `pvesm status`: `pve` `ssd0` **76.43 → 78.29 %**
|
||||||
|
(~152 GiB free), `SSD1` **64.32 → 65.46 %**, `fred-qnap` **82.77 → 84.21 %**, `local` **79.66 → 81.62 %**,
|
||||||
|
`local-lvm` **13.75 → 14.25 %**; `m93p` iMac shares + `local` **60.25 → 76.21 %**, `nfs-proxmox-*`
|
||||||
|
**82.78 → 84.21 %**, `nfs-ssd0-39` **76.43 → 78.29 %**, `omb-internal` **57.92 → 64.66 %**.
|
||||||
|
- **Docker (infrastructure/docker-services.md):** set-wise diff against live `docker ps` on CT200/201/333 —
|
||||||
|
CT200 **61 running / 68 total** exact, CT201 **20 / 26** exact, CT333 **8 / 9** exact, **but CT333's
|
||||||
|
enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name
|
||||||
|
**`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected. CT460 `ignis` running.
|
||||||
|
- **Cross-page corrections:** [[infrastructure/network-map]] `/gitea` was annotated *"Gitea (ct207 tailnet
|
||||||
|
IP)"* — it is **CT221 on `m93p`** (ct207 is `100.82.40.23`); also its `m93p` PVE version read *9.2.0*
|
||||||
|
(both hosts are 9.2.10). [[agents/hermes-chaos]] read *"`v0.22.3` … `5b147ab036d6` — the same build as
|
||||||
|
CT460"*, which contradicted itself: live is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to
|
||||||
|
CT460's (`hermes update --check` reports **v0.23.0** available). [[agents/hermes-production]]'s
|
||||||
|
*"2 cron jobs still pin `custom:omni`"* is **no longer true** — live `jobs.json` has **0** on the bare key
|
||||||
|
and **21** on `custom:omni-101`.
|
||||||
|
- **Fleet re-verified unchanged:** CT460 **29 jobs (26 enabled / 3 disabled)**; CT450 **25 jobs
|
||||||
|
(13 enabled / 12 disabled)** with **9 shared names** — confirmed by a live set-diff of both `jobs.json`.
|
||||||
|
A keyed `deepseek-paid-smart` chat probe against `omniroute-1` (`100.78.63.20:20128`) returns **HTTP 200**;
|
||||||
|
`config.yaml` still defaults to `custom:omni-101`.
|
||||||
|
- **Pages touched:** [[infrastructure/tailscale]], [[infrastructure/proxmox]], [[infrastructure/hosts]],
|
||||||
|
[[infrastructure/network-map]], [[infrastructure/docker-services]], [[agents/hermes-production]],
|
||||||
|
[[agents/hermes-chaos]], [[current-state]], [[index]], this log. No new incident page — no incident was
|
||||||
|
found; every drift above was a documentation drift corrected against a live probe.
|
||||||
|
|
||||||
|
|
||||||
|
## 2026-11-07 05:1x AWST - incident resolution: Gitea config backup push dead since 2026-08-08
|
||||||
|
|
||||||
|
- **Anthony's ask:** "fix this" on the failing backup alert — with a pointer to check Notion for the API key.
|
||||||
|
- **[[incidents/2026-08-08-gitea-config-backup-push-dead]] is now `status: resolved`.** The nightly
|
||||||
|
`e27da6c34bbb` config backup had been committing locally and failing to push **every night since
|
||||||
|
2026-08-08** — **92 commits** stranded — while its cron row read `last_status: ok`.
|
||||||
|
- **Four independent root causes, all confirmed against live systems:**
|
||||||
|
1. **No Gitea credential.** `origin` was HTTPS with no userinfo; the global `credential.helper=store`
|
||||||
|
file `~/.git-credentials` held only a `github.com` entry. The token in the Notion vault was
|
||||||
|
**valid** and byte-identical to the one the wiki repo carries in its remote URL — it simply was
|
||||||
|
never wired into the backup repo. Fixed with a host-scoped `git credential approve` entry (not
|
||||||
|
embedded in the URL); `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`.
|
||||||
|
2. **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's
|
||||||
|
`success = (returncode == 0)` recorded `ok` on every failed night. The scheduler was correct;
|
||||||
|
**the script was the liar**. This is why the outage ran 92 days without a job-level red.
|
||||||
|
3. **A 1.76 GB `checkpoints/` backlog** in the unpushed range (13 blobs > 100 MB) — junk that
|
||||||
|
`.gitignore` already excluded for *new* commits but that history still carried.
|
||||||
|
4. **Both secondary legs broken.** The `github` remote did not exist, and the fallback pushed a
|
||||||
|
branch's **full ancestry**, which contains a 7.45 GB tarball blob — so GitHub's 100 MB file
|
||||||
|
limit could never accept it. The AgentMail alert called a `python3 send_email.py` that no longer
|
||||||
|
exists, via a `chaos-agent@agentmail.to` inbox that no longer exists.
|
||||||
|
- **Backlog drained:** `git filter-repo --partial --refs origin/main..main --path checkpoints
|
||||||
|
--invert-paths --prune-empty never --force` stripped `checkpoints/` from the **unpushed range only**
|
||||||
|
(`origin/main` untouched, push stayed a fast-forward), then `checkpoints/` was untracked at the tip.
|
||||||
|
Push set **~2 GB → 264.8 MB**; 93 commits pushed in 11.6 s; the 7.45 GB tarball was already on
|
||||||
|
`origin/main` and never resent. Pre-rewrite history preserved under tag
|
||||||
|
`backup/pre-rewrite-20261107-032658`. **Token leakage scan over the HEAD tree, the working tree and
|
||||||
|
every unpushed blob: clean.**
|
||||||
|
- **Script rewritten** (`~/.hermes/config-backup/backup.sh`, via OpenCode + a correction pass): exits
|
||||||
|
**0 only on verified success**; stamps `.last_backup_ok` **only** after remote verification; decides
|
||||||
|
the push by **remote-vs-local** rather than by working-tree changes (the exact mechanism that created
|
||||||
|
the backlog); guards every `git ls-remote` with `|| true` so an unreachable remote reaches
|
||||||
|
`notify_failure` instead of dying at exit 128 *before* the alert; pushes GitHub as a **parentless
|
||||||
|
orphan snapshot** (`git commit-tree` from `HEAD^{tree}` — no history, no >100 MB blob, HEAD/branch/
|
||||||
|
index untouched) as a warning-only secondary target; and reports Telegram/AgentMail delivery
|
||||||
|
**honestly** (`curl --fail` + `"ok":true`; the real `agentmail` CLI, key read from `.env` at runtime
|
||||||
|
and passed via the environment, never argv).
|
||||||
|
- **Verified live 2026-11-07:** real run exit **0** in 47 s; `git rev-parse HEAD` ==
|
||||||
|
`git ls-remote origin refs/heads/main` == `a50a5170c1c0e2af5235fa63a95a732e2d780703`;
|
||||||
|
`rev-list --count origin/main..main` = **0**; Gitea API `main` = `a50a5170c1c0`, updated
|
||||||
|
`2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub
|
||||||
|
mirror now carries a **0-parent** orphan snapshot `faed9fb5c677` (30,086 files, largest blob 23.5 MB)
|
||||||
|
with `master` untouched at `c99c94e6b13b` — **no data lost**, since GitHub's tree was a strict subset
|
||||||
|
of local HEAD (only `BACKUP-INDEX.md` added). AgentMail and Telegram alert paths each exercised
|
||||||
|
directly (AgentMail self-test to the agent's own inbox; Telegram `getMe`, no message sent to Anthony).
|
||||||
|
Sandbox E2E: unreachable remote → exit 1 + alert + health **not** stamped; local-ahead-with-clean-tree
|
||||||
|
→ pushed + verified; already-up-to-date → no push, exit 0.
|
||||||
|
- **Pages touched:** [[incidents/2026-08-08-gitea-config-backup-push-dead]] (status → resolved, full
|
||||||
|
Resolution section added), [[infrastructure/backups]] (⚠ → ✅, re-verified 2026-11-07),
|
||||||
|
[[systems/scheduled-tasks]] (job row → RESOLVED), this log.
|
||||||
|
- **Note — unrelated pre-existing state:** the 03:00 `wiki-stocktake-maintenance` slot **failed** with
|
||||||
|
`RuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership`
|
||||||
|
(failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are
|
||||||
|
included in this commit so the tree is not left dirty.
|
||||||
|
|||||||
@@ -7,8 +7,8 @@ Threshold: >21 days since `updated`
|
|||||||
## FRESH (125)
|
## FRESH (125)
|
||||||
|
|
||||||
- [ 18d] CONTRIBUTING.md
|
- [ 18d] CONTRIBUTING.md
|
||||||
- [ 12d] agents/hermes-chaos.md
|
- [ 0d] agents/hermes-chaos.md
|
||||||
- [ 12d] agents/hermes-production.md
|
- [ 0d] agents/hermes-production.md
|
||||||
- [ 18d] agents/index.md
|
- [ 18d] agents/index.md
|
||||||
- [ 9d] agents/nanobot.md
|
- [ 9d] agents/nanobot.md
|
||||||
- [ 18d] agents/open-claw.md
|
- [ 18d] agents/open-claw.md
|
||||||
@@ -77,12 +77,12 @@ Threshold: >21 days since `updated`
|
|||||||
- [ 0d] index.md
|
- [ 0d] index.md
|
||||||
- [ 3d] infrastructure/backups.md
|
- [ 3d] infrastructure/backups.md
|
||||||
- [ 18d] infrastructure/credentials-map.md
|
- [ 18d] infrastructure/credentials-map.md
|
||||||
- [ 1d] infrastructure/docker-services.md
|
- [ 0d] infrastructure/docker-services.md
|
||||||
- [ 18d] infrastructure/domains-and-tunnels.md
|
- [ 18d] infrastructure/domains-and-tunnels.md
|
||||||
- [ 2d] infrastructure/hosts.md
|
- [ 0d] infrastructure/hosts.md
|
||||||
- [ 1d] infrastructure/network-map.md
|
- [ 0d] infrastructure/network-map.md
|
||||||
- [ 6d] infrastructure/proxmox.md
|
- [ 0d] infrastructure/proxmox.md
|
||||||
- [ 1d] infrastructure/tailscale.md
|
- [ 0d] infrastructure/tailscale.md
|
||||||
- [ 3d] queries/retained-research-answers.md
|
- [ 3d] queries/retained-research-answers.md
|
||||||
- [ 13d] reports/communities-wa-social-media-audit-2026.md
|
- [ 13d] reports/communities-wa-social-media-audit-2026.md
|
||||||
- [ 6d] reports/uber-eats/2026-07.md
|
- [ 6d] reports/uber-eats/2026-07.md
|
||||||
|
|||||||
@@ -215,7 +215,7 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
|
|||||||
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
|
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
|
||||||
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text |
|
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text |
|
||||||
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken. ⚠ **The 2026-10-17 03:00 slot failed** on the same `omni-101` 503 — the first miss since the **2026-10-15** storm slot (which died with `RuntimeError: Connection error.`; the 10-16 slot succeeded). `raw/stocktake/freshness-20261017.txt` was not written and `systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 slot succeeded, so `freshness-20261016.txt` exists); the **2026-10-18 03:00** slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident `1a8e90_3c9c1d573cde` `detected`. Note the ledger tags this row `failure_type: unknown` even though the execution carries an explicit 503 message |
|
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken. ⚠ **The 2026-10-17 03:00 slot failed** on the same `omni-101` 503 — the first miss since the **2026-10-15** storm slot (which died with `RuntimeError: Connection error.`; the 10-16 slot succeeded). `raw/stocktake/freshness-20261017.txt` was not written and `systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 slot succeeded, so `freshness-20261016.txt` exists); the **2026-10-18 03:00** slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident `1a8e90_3c9c1d573cde` `detected`. Note the ledger tags this row `failure_type: unknown` even though the execution carries an explicit 503 message |
|
||||||
| 🦝 Hermes Config Backup to Gitea (Daily) ⚠ **push has failed every night since 2026-08-08 — 75 local commits unpushed, job still reads `ok`** → [[incidents/2026-08-08-gitea-config-backup-push-dead]] | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
| 🦝 Hermes Config Backup to Gitea (Daily) ✅ **RESOLVED 2026-11-07** — push live again, 92-commit backlog drained and pushed, script hardened to exit non-zero on failure so the job row can no longer read `ok` while the push is dead. Live run exit 0; `origin/main` == local HEAD (`a50a5170c1c0`); `rev-list --count origin/main..main` = 0. → [[incidents/2026-08-08-gitea-config-backup-push-dead]] | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
||||||
|
|
||||||
## Removed Jobs
|
## Removed Jobs
|
||||||
| Name | Removed | Reason |
|
| Name | Removed | Reason |
|
||||||
|
|||||||
Reference in New Issue
Block a user