Anthony/agent-estate-wiki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

incident: SFTPGo outage 2026-09-02 — CT299 tailscaled death + stale lxc-attach recovery

Browse Source 
- New incident page documenting CT299 SFTPGo outage (02:18 AWST)
- Root cause: swap exhaustion (43Gi, 100%) killed tailscaled; 5 stale lxc-attach PIDs blocked pct start
- Resolution: killed PIDs 3037951/3334928/3352203/3402511/4004148, pct stop/start 299
- Updated current-state.md with Recent Changes (2026-09-02), bumped updated to 2026-09-04
- Updated log.md with incident entry
- Updated index.md last-updated date

No secrets written. Verified end-to-end: tailscale direct, SFTPGo WebAdmin HTTP 401, SFTP banner SSH-2.0-SFTPGo_2.7.0.
This commit is contained in:
Tony0410
2026-09-04 00:05:32 +08:00
parent 1d42c6bc83
commit 77250ea8c9
13 changed files with 746 additions and 507 deletions
Download Patch File Download Diff File Expand all files Collapse all files
log.md
+70 -16
View File
@@ -177,21 +177,75 @@
- **Updated [[index]]** — last-updated date 2026-08-10 → 2026-08-13.
- No new pages created; all changes applied to existing pages. No secrets written.
## [2026-08-14] update | Nightly wiki audit — OmniRoute factory reset, Guanaco update, GPT Luna research
- **OmniRoute factory reset documented:** CT601 database wiped clean (0 combos, 0 providers, 0 API keys, 115 built-in auto entries). Backup at `/opt/omniroute/factory-reset-backup-20260814-225524`. Providers re-added from Notion vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter (all passed live tests). OpenAI keys invalid — removed. Multiple free combos broken by upstream model ID changes (OpenRouter Poolside dash→dot, Groq llama-4-scout retired). Default model changed from `auto/best-free` to `resilient-free`. Working free combos: `free-lkgp`, `mega-free`, `smart-route`. See [[systems/omniroute]].
- **Guanaco router documented:** CT205 `ollama` container — Guanaco v0.8.11 → v0.8.14. DSML tag leakage fixes (tool-call formatting no longer leaks raw syntax), dashboard performance fix. Created [[systems/guanaco]].
- **GPT-5.6 Luna research documented:** Community consensus — Luna cleaner/polished; DeepSeek V4 Flash more persistent/cheaper. Provider route matters more than model name. Luna for planning/review, DeepSeek for execution. Updated [[systems/model-providers]].
- **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-14), bumped `updated`/`verified_on`
- **Updated [[systems/model-providers]]** — provider table (`omniroute` → `omni`), Guanaco entry, GPT-5.6 Luna entry, default model changed to `resilient-free`, bumped dates
- **Created [[systems/guanaco]]** — new page for Guanaco LLM proxy on CT205
- **Updated [[current-state]]** — added Recent Changes (2026-08-14) with all seven items, bumped `updated`
- **Updated [[infrastructure/proxmox]]** — CT205 description updated with Guanaco proxy
- **Updated [[infrastructure/hosts]]** — bumped `updated` date
- **Updated [[index]]** — last-updated date → 2026-08-14, added Guanaco link to Core systems
## [2026-08-15] update | Nightly wiki audit — Skills index drift fix, agent pages added, scheduled-tasks updated
- **Skills index drift incident:** `systems/skills-index.md` claimed 66 skills when 330 were actually installed (4x undercount). Page was hand-written once on 2026-07-22/25 and never regenerated. Fixed by regenerating from live skill tree, creating `regenerate-skills-index.sh` script, and adding cron job `1a8e90ea2271` (daily 03:00 AWST) to auto-regenerate and commit to Gitea. See [[incidents/2026-08-15-skills-index-drift]].
- **Updated [[systems/skills-index]]** — added auto-regeneration notice, updated counts (66→330), bumped `updated`/`verified_on` to 2026-08-15.
- **Updated [[systems/scheduled-tasks]]** — added Skills Stock-Take Regeneration job, bumped count (28→29), updated dates.
- **Updated [[systems/hermes-agent]]** — skills count (66→330), cron jobs count (25→29), added Mattermost gateway warning note, updated dates.
- **Updated [[index.md]]** — added Incidents section with link to skills drift incident, updated last-updated date to 2026-08-15.
- **Created [[incidents/2026-08-15-skills-index-drift]]** — new incident page documenting the 4x undercount, root cause (static docs for mutable inventory), resolution (auto-regeneration), and prevention (daily cron).
- No secrets written.
## [2026-08-16] maintenance | Stocktake + scheduled-tasks cleanup
- Ran `wiki-maintain.sh` content regeneration + freshness audit (36 stale / 34 fresh / 7 no-date of 77 pages)
- Rebuilt Quartz deploy (201 HTML files)
- Updated `systems/scheduled-tasks` to 29 active jobs after verifying live `hermes cron list` and adding wiki-stocktake-maintenance
- Cleaned duplicate rows from the scheduled-tasks table
## [2026-08-17] update | Nightly wiki audit — Nextcloud ncdownloader crash
- **New page:** `systems/nextcloud.md` — documented Nextcloud CT270: version 33.0.2.2, apps inventory, ncdownloader incompatibility incident, access details, related pages.
- **Updated [[current-state]]** — added Recent Changes (2026-08-17) section with ncdownloader fix details.
- **Updated [[infrastructure/proxmox]]** — CT270 entry now shows Nextcloud version and ncdownloader status.
- **Updated [[index.md]]** — added Nextcloud link to Core systems, bumped last-updated to 2026-08-17.
- No secrets written.
## [2026-08-16] update | Nightly wiki audit — OmniRoute capacity pressure, Agent Zero sluggishness, Hermes verification quirk
- **OmniRoute upstream capacity pressure documented:** `ServiceUnavailableError: Structurally heavy chat request capacity is busy` reported across agents via OmniRoute. Root cause: upstream OpenAI capacity pressure (recent incident Aug 11, now resolved). OmniRoute's context compression not always applied before sending oversized requests, amplifying the issue. Diagnostic requirements recorded.
- **Agent Zero sluggishness confirmed as orchestration overhead:** GPT-5.6 Luna is fast — the latency comes from Agent Zero's orchestration layer (large system prompts, multi-step reasoning, memory lookups, sub-agent spawning). GPT-5.6 Luna confirmed as real model (lightweight/fast tier, launched Jul 9, 2026).
- **Hermes model verification quirk documented:** Custom endpoint verification can fail with "could not reach" when the endpoint returns HTTP 402 (auth required) during unauthenticated transient checks. Config `http://omni:20128/v1` is correct — `/v1` is required.
- **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-16) section, bumped `updated`/`verified_on` to 2026-08-16.
- **Updated [[current-state]]** — added Recent Changes (2026-08-16) section, bumped `updated` to 2026-08-16.
- **Updated [[index.md]]** — bumped last-updated date to 2026-08-16.
- No new pages created. No secrets written.
## [2026-08-18] incident | Mnemosyne capture pipeline broken (active)
- **New page:** `incidents/2026-08-18-mnemosyne-capture-pipeline` — working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39, ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. MEMDEBUG logging recommended. Config verified clean (native YAML types). Still active as of verification.
- **Updated [[systems/mnemosyne]]** — added 2026-08-18 incident to History and Evidence, bumped `updated`/`verified_on` to 2026-08-18, added cross-reference to Related.
- **Updated [[current-state]]** — added Recent Changes (2026-08-18) section with capture pipeline incident, bumped `updated` to 2026-08-18.
- **Updated [[index.md]]** — added 2026-08-18 incident to Incidents section, bumped last-updated to 2026-08-18.
- **Marked [[incidents/2026-08-01-mnemosyne-capture-failure]]** — status changed from `active` to `resolved` (that earlier incident was fixed).
- No secrets written.
## [2026-08-23] update | Nightly wiki audit — Cron failures, wiki-stocktake drift, Mnemosyne watchdog history
- **Cron job failures documented (Weekly Review session):** Three distinct failure patterns identified in past 7 days:
- `wiki-stocktake-maintenance` (1a8e90ea2271) failing ×4 (Aug 20–23) — config drift skip: model changed from `cx/gpt-5.6-luna` → `go-cheap`/`opencode-go/gpt-5.6-luna`, job unpinned. Requires explicit provider/model pinning.
- CT221 gitea miner watchdog (d5516d0a5147) failing ×6+ (Aug 19) — SSH to PVE (root@100.96.100.82) failed repeatedly; CT 221 status = UNKNOWN, never inspected.
- Mnemosyne Watchdog (96eb95ce12f2) major incident (Aug 19 early AM) — no consolidation for 36–41.5+ hours (last seen Aug 17 07:15). Watchdog cycled for ~2 hours before stopping.
- **Updated [[systems/scheduled-tasks]]** — added Known Issues section with cron failure patterns, bumped `updated`/`verified_on` to 2026-08-23.
- **Updated [[systems/mnemosyne]]** — added Mnemosyne Watchdog Aug 19 incident to History section, bumped `updated`/`verified_on` to 2026-08-23.
- **Updated [[current-state]]** — added Recent Changes (2026-08-23) section with cron failures summary.
- **Updated [[index.md]]** — bumped last-updated date to 2026-08-23.
- No secrets written.
## [2026-08-19] update | Nightly wiki audit — Ignis deployment, Command Code provider, Tavili search status
- **New page:** `systems/ignis.md` — documented Ignis browser-native Obsidian (Docker `ignis-subpath:0.8.9`, port 8080, Tailscale Serve at `hermes.kangaroo-eel.ts.net/ignis`). Volume mount: `/home/hermes/wiki/agent-estate-wiki:/vaults/wiki`. Verified live: container running, Tailscale route active.
- **Updated [[systems/model-providers]]** — added `command-code` provider (`https://api.commandcode.ai/provider/v1`) to Current Provider Stack. Bumped `updated`/`verified_on` to 2026-08-19.
- **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-19) section documenting Command Code provider, `deepseek-flash-mix` combo, and Tavili search proxy status. Bumped `updated`/`verified_on` to 2026-08-19.
- **Updated [[infrastructure/docker-services]]** — added Ignis compose file and service definition.
- **Updated [[infrastructure/hosts]]** — added Ignis to CT460 row.
- **Updated [[systems/obsidian]]** — added Ignis as browser interface, corrected vault path to `agent-estate-wiki/`, added Quartz static site reference.
- **Updated [[current-state]]** — added Recent Changes (2026-08-19) section with Ignis deployment, Command Code provider, and Tavili search status.
- **Updated [[index.md]]** — added Ignis link to Core systems, bumped last-updated to 2026-08-19.
- No secrets written.
## [2026-08-26] update | Nightly wiki audit — Cron job reconciliation
- **Updated [[systems/scheduled-tasks]]** — reconciled live `hermes cron list` (27 jobs) against wiki table (29 entries). Removed: `Workday start reminder` (no longer in live list), `CT 221 Miner Watchdog` (job `d5516d0a5147` removed; CT221 status unknown), `Review Agent Estate Phase One` (one-time job completed 2026-08-08). Added: `weekly-save-restricted-bot-restart` (`0 3 * * 0`, local, no-agent). Updated count from 29 to 27.
- **Updated [[current-state]]** — added Recent Changes (2026-08-26) section documenting cron reconciliation and wiki-stocktake-maintenance drift persistence.
- **Updated [[index.md]]** — bumped last-updated to 2026-08-26, corrected job count reference (29 → 27).
- **Known Issues cleanup** — removed stale CT221 miner watchdog SSH failure entry (job no longer exists).
- No secrets written.
## [2026-09-02] incident | SFTPGo outage on CT299
- **New page:** `incidents/2026-09-02-sftpgo-outage.md` — documented CT299 SFTPGo outage (02:18 AWST). Root cause: CT299 swap exhausted (43Gi used, 100%), `tailscaled` died; PVE `pct start` failed due to 5 stale `lxc-attach` processes pinning cgroup. Killed PIDs 3037951/3334928/3352203/3402511/4004148, `pct stop 299 --skiplock` + `pct start 299`. Verified end-to-end: direct Tailscale connection, SFTPGo WebAdmin HTTP 401, SFTP banner `SSH-2.0-SFTPGo_2.7.0`.
- **Updated [[current-state]]** — added Recent Changes (2026-09-02) section, bumped `updated`/`verified_on` to 2026-09-04.
- **Updated [[infrastructure/proxmox]]** — CT299 entry already documented incident; no change needed.
- No secrets written.
## [2026-08-28] update | Department of Communities role and social-media audit
- **New page:** `reports/communities-wa-social-media-audit-2026` — durable strategic summary of Anthony's supplied public-source audit for his Department of Communities Social Media Officer role.
- **Updated [[systems/current-profile]]** — recorded current employment context and linked the report.
- **Updated [[index.md]]** — added the report to the Reports section and refreshed the last-updated date.
- No secrets written.