snapshot: preserve central wiki state 2026-08-15 (7 modified + 1 new file)

2026-08-15 22:11:30 +08:00
commit 92e8028bb8
71 changed files with 9897 additions and 0 deletions
@@ -0,0 +1,94 @@
+---
+title: Backup Wiki
+type: runbook
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+last_tested: 2026-07-22
+confidence: high
+tags: [runbook, wiki, backup]
+sources: []
+---
+
+# Backup Wiki
+
+## Purpose
+Create a durable backup of the wiki so recoverable copies exist before mutating changes.
+
+## Symptoms that match this runbook
+- Before a big restructuring, page rewrite, or history migration
+- Weekly or scheduled backup reminder
+- Suspected accidental deletion or corruption
+
+## Prerequisites
+- Wiki path: `/home/hermes/wiki`
+- Git is available
+- Backup destination is writable
+
+## Procedure
+
+### 1. Sanity-check repo state
+```bash
+cd /home/hermes/wiki && git status --short && git log --oneline -3
+```
+
+### 2. Commit any open changes first
+```bash
+cd /home/hermes/wiki
+git add -A
+git commit -m "backup-baseline: $(date +%Y-%m-%d)"
+```
+
+### 3. Export a timestamped copy outside the repo
+```bash
+BACKUP_DIR=/home/hermes/wiki-backups
+mkdir -p "$BACKUP_DIR"
+BACKUP="$BACKUP_DIR/wiki-$(date +%Y%m%d-%H%M%S).tar.gz"
+tar -czf "$BACKUP" -C /home/hermes wiki
+echo "Wrote $BACKUP"
+```
+
+### 4. Optional: push to a Git remote if configured
+```bash
+cd /home/hermes/wiki
+git remote -v
+git push --all
+git push --tags
+```
+If no remote is configured, skip this step and rely on the tarball.
+
+### 5. Rotate old backups if needed
+Keep N most recent backups:
+```bash
+ls -1t "$BACKUP_DIR"/wiki-*.tar.gz | tail -n +6 | xargs -r rm
+```
+
+## Verification
+- `git status --short` shows a clean tree
+- Tarball exists and is non-empty:
+```bash
+stat "$BACKUP"
+tar -tzf "$BACKUP" | head -20
+```
+
+## Rollback
+- Restore from tarball:
+```bash
+BACKUP=$(ls -1t /home/hermes/wiki-backups/wiki-*.tar.gz | head -1)
+rm -rf /home/hermes/wiki-restore
+mkdir -p /home/hermes/wiki-restore
+tar -xzf "$BACKUP" -C /home/hermes wiki-restore --strip-components=1
+```
+- Verify restored content before replacing live wiki
+
+## Notes
+- This runbook intentionally does not write secrets or token dumps into backups
+- If you need a full Hermes config+data backup, use `hermes backup` in addition to this wiki-only backup
+
+## Last tested
+2026-07-22
+
+## Related
+- [[runbooks/update-hermes-safely]]
+- [[current-state]]
@@ -0,0 +1,84 @@
+---
+title: Diagnose Failed Docker Service
+type: runbook
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+last_tested: 2026-07-22
+confidence: medium
+tags: [runbook, docker, homelab]
+sources: []
+---
+
+# Diagnose Failed Docker Service
+
+## Purpose
+Diagnose container or compose issues on hosts where Docker is present.
+
+## Symptoms that match this runbook
+- Service reports container exit/failed state
+- `docker ps` does not show an expected container
+- Compose stack reports unhealthy status
+
+## Prerequisites
+- `docker` installed and accessible to the user
+- Docker daemon running
+- Compose file path known for the affected project
+
+## Procedure
+
+### 1. Check Docker daemon state
+```bash
+systemctl status docker
+```
+
+### 2. List containers
+```bash
+docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
+```
+
+### 3. Inspect logs for the failing container
+```bash
+docker logs --tail 100 <container-name>
+```
+
+### 4. If using compose, check compose state
+```bash
+cd /home/hermes/<project>/docker-compose
+docker compose ps
+docker compose logs --tail 100 <service-name>
+```
+
+### 5. Restart the failed container only
+```bash
+docker restart <container-name>
+```
+
+### 6. If restart fails, investigate restart policy and image
+```bash
+docker inspect --format "{{.HostConfig.RestartPolicy}}" <container-name>
+docker inspect --format "{{.Config.Image}}" <container-name>
+```
+
+## Verification
+- `docker ps` shows the container `Up` or `healthy`
+- Service or WebUI dependent on the container is reachable again
+- Logs show clean startup messages
+
+## Rollback
+- Do not delete volumes unless confirmed not in use
+- If a container recreate is required, preserve volume mappings and env files
+- If uncertain, collect logs first and ask before replacement
+
+## Notes
+- This runbook is generic. This current Hermes host does not have Docker installed as of 2026-07-22
+- Docker-related paths must be confirmed before executing compose commands
+- Do not invent container names or project paths
+
+## Last tested
+2026-07-22 on system where Docker runtime is absent; read-only steps verified, restart steps not executed
+
+## Related
+- [[docker-services]]
+- [[hosts]]
@@ -0,0 +1,128 @@
+---
+title: Hermes Gateway Resource Resilience
+type: runbook
+status: active
+created: 2026-07-25
+updated: 2026-07-25
+review_after: 2026-10-23
+verified_on: 2026-07-25
+version_applies_to: Proxmox CT460
+confidence: high
+tags: [hermes, infrastructure, runbook]
+sources: [live systemd and cgroup v2 state on CT460]
+---
+
+# Hermes Gateway Resource Resilience
+
+## Purpose
+
+Keep Telegram and the Hermes gateway available when the browser or Web UI
+causes severe memory pressure inside CT460.
+
+The observed failure was real: the CT460 memory cgroup OOM-killed a large Web
+UI process and a Chrome child. Because the gateway previously used
+`OOMPolicy=stop`, the Chrome child death caused systemd to stop the entire
+gateway until `Restart=always` brought it back.
+
+## Active controls
+
+### Gateway
+
+File:
+`/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf`
+
+```ini
+[Service]
+OOMPolicy=continue
+MemoryLow=512M
+CPUWeight=1000
+```
+
+The gateway has no `MemoryHigh` or `MemoryMax`. It receives reclaim protection
+and maximum relative CPU weight, while `OOMPolicy=continue` prevents an
+OOM-killed browser child from stopping the whole unit. If the main gateway
+process exits, its existing `Restart=always` policy still restarts it.
+
+### Camofox
+
+File:
+`/home/hermes/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`
+
+```ini
+[Service]
+MemoryHigh=1G
+MemoryMax=1536M
+MemorySwapMax=768M
+OOMPolicy=kill
+```
+
+### Hermes Web UI
+
+File:
+`/etc/systemd/system/hermes-webui.service.d/resource-containment.conf`
+
+```ini
+[Service]
+MemoryHigh=1G
+MemoryMax=1536M
+MemorySwapMax=1G
+OOMPolicy=kill
+```
+
+Camofox and Web UI are allowed to restart under extreme growth instead of
+consuming most of the container and taking Telegram down with them.
+
+## I/O priority limitation
+
+Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
+is changed and verified. CT460 exposes the I/O controller at its top cgroup,
+but it is not delegated into the container's systemd service tree. Systemd
+accepts the property, but no live `io.weight` control is created.
+
+## Health check
+
+```bash
+systemctl is-active hermes-webui.service
+runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
+  systemctl --user is-active camofox-browser.service hermes-gateway.service
+```
+
+Check effective limits:
+
+```bash
+systemctl show hermes-webui.service \
+  -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
+runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
+  systemctl --user show hermes-gateway.service \
+  -p OOMPolicy -p MemoryLow -p MemoryHigh -p MemoryMax -p CPUWeight
+runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
+  systemctl --user show camofox-browser.service \
+  -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
+```
+
+Expected gateway values include `OOMPolicy=continue`,
+`MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and
+`CPUWeight=1000`.
+
+## Backups and rollback
+
+Original source units are backed up at:
+
+`/home/hermes/.hermes/backups/resource-hardening-20260725/`
+
+To roll back, remove only these three drop-ins:
+
+- `resource-resilience.conf` for `hermes-gateway.service`
+- `resource-containment.conf` for `camofox-browser.service`
+- `resource-containment.conf` for `hermes-webui.service`
+
+Then reload the system and user systemd managers and restart only those three
+services. The original source units were never modified.
+
+## Related
+
+- [[systems/hermes-agent]]
+- [[systems/messaging-integrations]]
+- [[systems/browser-backend]]
+- [[runbooks/restart-hermes]]
+- [[systems/tool-search]]
@@ -0,0 +1,248 @@
+---
+title: Daily Marketing Job Radar Operations and Recovery
+type: runbook
+status: active
+created: 2026-07-27
+updated: 2026-07-28
+review_after: 2026-10-25
+verified_on: 2026-07-28
+version_applies_to: Hermes Agent cron dca8482e4f76
+confidence: high
+tags: [automation, cron, jobs, notion, dashboard, runbook]
+sources: [live-cron-output, live-notion-api, live-dashboard-api, hermes-docs]
+---
+
+# Daily Marketing Job Radar Operations and Recovery
+
+## Purpose
+
+Run a verified Western Australian marketing-job search, deduplicate against the complete Notion board, create only fresh direct adverts, score new roles against Anthony's resume, automatically produce cover letters for strong matches, and expose the results through the Jobs Radar dashboard.
+
+## Production configuration
+
+| Setting | Value |
+|---|---|
+| Cron job | `dca8482e4f76` — Daily Marketing Job Radar |
+| Schedule | `0 7 * * 1-5` (07:00 AWST weekdays) |
+| Delivery | Telegram group `-1004321904721`, topic `1030` |
+| Skill | `job-search-automation` |
+| Primary model | `openai-codex / gpt-5.6-terra` |
+| First fallback | `opencode-go / mimo-v2.5` |
+| Dashboard | `https://hermes.kangaroo-eel.ts.net/jobs/` |
+| Dashboard backend | `127.0.0.1:9099` |
+| Notion database | `0f90ba2b-8b10-4d02-a62c-6f692d5b1168` |
+
+Cron jobs inherit the profile-level `fallback_providers` chain; Hermes does not currently store a separate fallback chain in each cron record. MiMo v2.5 is first in the global chain, so Terra fails over to it before the older fallbacks.
+
+## Cover letters are non-negotiable
+
+A successful radar run includes the cover-letter stage.
+
+- Audit **every Strong Match added today**, including pages created by an interrupted earlier run.
+- A valid result requires a canonical rendered `.html` URL in Notion, not plain text and not the dashboard root.
+- Production template files:
+  - `/home/hermes/.hermes/dashboard/templates/cover-letter.html`
+  - `/home/hermes/.hermes/dashboard/templates/Anthony_Martin_Cover_Letter_Template.docx`
+- The old upload-cache location is not authoritative and may be evicted.
+- Required nonempty artifacts: `.txt`, `.json`, `.html`, `.docx`, `.pdf`.
+- The HTML must contain zero unresolved `{{...}}` tokens and return HTTP 200.
+- Retry generation once when an artifact, Notion link or HTTP check fails.
+- A clean run requires `MISSING_COVER_LETTER_COUNT=0`. Any remaining missing letter must be reported prominently with its Notion page ID; it must never be silently omitted.
+
+### Live audit — 27 July 2026
+
+- Same-day strong matches: **1**
+- Verified canonical HTML cover letters: **1**
+- Missing cover letters: **0**
+- The Market Creations Agency strong match has all five artifact formats, zero unresolved template tokens and a live HTTP 200 HTML link.
+
+## Required production pipeline
+
+1. Obtain the live AWST date and calculate a seven-day freshness cutoff.
+2. Search SEEK, LinkedIn, Indeed, WA Government Jobs and direct employer sites.
+3. Open and verify every individual advert. Search snippets are discovery evidence only.
+4. Reject stale, expired, cached, interstate, non-marketing, salary-guide, search/category and aggregator pages.
+5. Reject **all Pacific Energy roles**. Anthony was made redundant there in July 2026; old Pacific Energy adverts must never be presented as opportunities.
+6. Accept only direct individual advert URLs:
+   - SEEK `/job/<numeric-id>`
+   - LinkedIn `/jobs/view/...<numeric-id>`
+   - Indeed `/viewjob?jk=<real-id>`
+   - direct employer career pages
+   - individual WA Government role pages
+7. A WA Government slug page is valid without `AdvertID` in its URL when the page proves the exact role and agency, WA location, unique job/pool reference, future closing date and active **Apply Now** function.
+8. Query the complete Notion database with live pagination. Deduplicate by exact URL, normalized company plus role, and materially similar company-role combinations.
+9. Create only verified nonduplicates and require returned Notion page IDs.
+10. Refresh `POST http://127.0.0.1:9099/api/refresh`, then confirm every page and direct URL through `/api/jobs`.
+11. Score newly created roles against `/home/hermes/.hermes/radar-ref/resumes/current_resume_from_website.txt` using the documented 100-point rubric. Strong Match starts at 70.
+12. For each new strong match, run:
+
+```bash
+/home/hermes/.hermes/hermes-agent/venv/bin/python \
+  /home/hermes/.hermes/scripts/regenerate_cover_letter.py \
+  <NOTION_PAGE_ID> \
+  "Generate the initial tailored cover letter using the verified job description. Preserve Anthony's factual employment history and use the canonical teal Montserrat template."
+```
+
+13. Verify nonempty `.txt`, `.json`, `.html`, `.docx` and `.pdf` artifacts; no unresolved `{{...}}` tokens; and a live `.html` Notion Cover Letter URL.
+14. Report only facts produced by tools in that run. Never estimate counts or fabricate Notion writes.
+
+## Dashboard behaviour
+
+Source: `/home/hermes/.hermes/dashboard/index.html`  
+Backend: `/home/hermes/.hermes/scripts/job_dashboard_server.py`
+
+### Layout (top to bottom)
+1. **Hero header** — RADAR logo, name, dateline
+2. **Market Intel panel** — expandable salary/location/source stats from `/api/market-intel`
+3. **KPI strip** — clickable filter cards: Total Jobs, Backlog, Applied, Phone Screen, Interviews, Offers, Closing Soon
+4. **Filter bar** — text search, view toggles (table/cards), Refresh button
+5. **Stats strip** — This Week, Stale 14d+, Strong Match, Past Close, Applied Rate
+6. **Pipeline Funnel** — clickable per-status breakdown with bar chart
+7. **Job table/cards** — sortable columns, inline status pills, cover letter links, edit actions
+
+### Column sorting
+Click any column header (Status, Company, Role, Match, Salary, Age, Closes, Updated) to sort ascending; click again for descending. Arrow indicator shows current direction.
+
+### Status management
+- **Bulk update**: select jobs via checkboxes → pick status from dropdown → Apply. Sends `{ids, updates: {status}}` to `POST /api/jobs/bulk-update`.
+- **Single edit**: click ✎ → modal with Status, Priority, Role, Salary, Closing Date, URL, Notes, Cover Letter, Attachments. Saves to Notion via `POST /api/jobs/<id>/update`.
+- **Funnel/status filters**: clicking a status in the funnel or KPI strip filters the table to that status.
+
+### Rich text notes
+- Write/Preview editor in the edit modal (monospace textarea + markdown preview)
+- Supports: bold `**`, italic `*`, links `[text](url)`, bullet lists `- item`
+- Syncs to Notion `Notes / JD Summary` rich_text property on every save
+
+### Document attachments
+- Attach any URL (documents, links, files) to a job entry
+- Stored as **bookmark blocks** on the Notion page body (not a property)
+- Lazy-loaded when edit modal opens: `GET /api/jobs/<id>/attachments`
+- Add: `POST /api/jobs/<id>/attachments` with `{name, url}`
+- Delete: `POST /api/jobs/<id>/attachments/delete` with `{block_id}`
+- Visible in Notion when you open the page
+
+### Cover letter integration
+- Regenerate button (🔄) on each row calls `POST /api/jobs/<id>/cover-letter/regenerate`
+- CL URL stored in Notion `Cover Letter` property and rendered as clickable link in table
+- All 5 artifact formats: TXT, JSON, HTML, DOCX, PDF
+
+### Market Intel
+- Expandable panel at top of page
+- Data from `GET /api/market-intel`: total tracked, salary stats (avg/median/min/max), counts
+- Lazy-loaded on expand; also refreshes on ↺ Refresh
+
+### API endpoints
+| Endpoint | Method | Purpose |
+|---|---|---|
+| `/api/jobs` | GET | All jobs (cached, paginated from Notion) |
+| `/api/stats` | GET | Aggregate statistics |
+| `/api/refresh` | POST | Force Notion cache refresh |
+| `/api/market-intel` | GET | Salary/location/source analytics |
+| `/api/jobs/<id>/update` | POST | Update single job properties |
+| `/api/jobs/<id>/attachments` | GET | Fetch page block attachments |
+| `/api/jobs/<id>/attachments` | POST | Add bookmark attachment to page |
+| `/api/jobs/<id>/attachments/delete` | POST | Remove block from page |
+| `/api/jobs/<id>/cover-letter/regenerate` | POST | Regenerate cover letter |
+| `/api/jobs/<id>/match` | GET | Match analysis for a job |
+| `/api/jobs/bulk-update` | POST | Update multiple jobs at once |
+| `/api/jobs/create` | POST | Create new job in Notion |
+
+- Company names and job titles with a valid `URL` render as blue external links and open the direct advert in a new tab.
+- URL values are protocol-validated; only `http:` and `https:` are rendered as links.
+- KPI/stat cards are one-click filters with an active highlight:
+  - Total Jobs
+  - Backlog
+  - Applied
+  - Interviews
+  - Offers
+  - Closing Soon
+  - This Week
+  - Stale
+  - Strong Matches
+  - Past Closing
+  - Application Rate
+- Funnel status rows are also one-click exact-status filters.
+- Existing status editing remains through the job edit modal and bulk-action controls, which write back to Notion through `/api/update`.
+- The frontend filter/link test harness is `/tmp/radar_dashboard_test.js` (temporary; recreate if absent).
+
+## 27 July 2026 incident
+
+### Symptoms
+
+- The morning cron returned old and irrelevant listings, including the stale Pacific Energy Marketing Specialist advert associated with Anthony's redundancy.
+- Aggregator mirrors, generic SEEK salary pages and category pages were treated as job adverts.
+- Counts and quality signals were presented without a demonstrated live Notion query/write trail.
+- The dashboard had valid URLs for almost every record, but the frontend never rendered `job.url`; users could not open the actual adverts.
+- The earlier prompt explicitly narrowed the task so far that it accidentally prohibited the documented strong-match cover-letter stage.
+
+### Recovery
+
+- Archived all **16** records created by the contaminated morning run: 16 successful, zero failures.
+- Replaced the cron prompt with deterministic freshness, source, verification, Notion, scoring, exclusion and cover-letter gates.
+- Pinned the cron primary to `gpt-5.6-terra`.
+- Added and directly probed `opencode-go / mimo-v2.5`; it returned `MIMO_FALLBACK_OK`, then was inserted first in the inherited fallback chain.
+- Repaired `regenerate_cover_letter.py`:
+  - absolute Hermes paths rather than `~` expansion
+  - one correct command dispatcher rather than two conflicting `__main__` blocks
+- Repaired dashboard direct links and one-click quick filters.
+- The corrected Terra run created four verified fresh direct LinkedIn roles. One scored 75 and produced all five cover-letter artifact formats.
+- A follow-up run queried 166 live Notion records across two pages, verified three current direct adverts, and correctly skipped all three as duplicates. No Pacific Energy records were added.
+
+### Final verification
+
+- Dashboard API: 166 records
+- Records created on 27 July after cleanup: 4
+- Pacific Energy records created on 27 July: 0
+- Records with direct URLs: 164 of 166
+- Strong-match cover letter artifact sizes were nonzero for all five formats
+- Unresolved template tokens: 0
+- Dashboard, API and generated cover-letter URLs returned HTTP 200
+- Synthetic dashboard tests passed all quick-filter categories, unsafe-URL rejection and rendered job-link checks
+
+## Tailscale route recovery found during verification
+
+The dashboard's local service was healthy while the tailnet URL failed because the CT460 `:443` Serve table had been cleared. Running the original atomic helper restored Serve but also erased the public `:8443` Funnel route.
+
+`/home/hermes/.hermes/scripts/tailscale-serve-apply.sh` now atomically restores both:
+
+- tailnet-only `:443`: `/`, `/jobs`, `/desktop`, `/vnc-camofox`, `/evening-grid.html`
+- public `:8443`: `/webhook` → `http://127.0.0.1:8085/webhook`
+
+Important: `tailscale serve reset` clears Funnel state too. Never restore only the 443 routes and assume 8443 survived.
+
+## Verification commands
+
+```bash
+# Cron configuration
+hermes cron list
+
+# Fallback order
+hermes fallback list
+
+# Dashboard/API
+curl -I https://hermes.kangaroo-eel.ts.net/jobs/
+curl -sS https://hermes.kangaroo-eel.ts.net/jobs/api/jobs | jq '.jobs | length'
+
+# Local service
+ss -ltnp 'sport = :9099'
+curl -sS http://127.0.0.1:9099/api/jobs | jq '.jobs | length'
+
+# Route table
+tailscale serve status
+
+# Atomic route rebuild
+/home/hermes/.hermes/scripts/tailscale-serve-apply.sh --dry-run
+/home/hermes/.hermes/scripts/tailscale-serve-apply.sh
+```
+
+## Backups from the incident
+
+- `/home/hermes/workspace/index.html.radar-links-backup-20260727`
+- `/home/hermes/workspace/jobs.json.radar-prompt-backup-20260727`
+- `/home/hermes/workspace/config.yaml.pre-mimo-fallback-20260727`
+
+## Related
+
+- [[systems/scheduled-tasks]]
+- [[infrastructure/tailscale]]
+- [[systems/model-providers]]
@@ -0,0 +1,93 @@
+---
+title: Multiplexer Setup (Multi-Profile Gateway)
+type: runbook
+status: active
+created: 2026-07-28
+updated: 2026-07-28
+verified_on: 2026-07-28
+confidence: high
+tags: [hermes, multiplexer, telegram, profiles, gateway]
+---
+
+# Multiplexer Setup
+
+## What It Is
+
+One gateway process serving multiple Hermes profiles through a single Telegram bot. Messages are routed to the correct profile by `profile_routes` (chat_id → profile).
+
+## Architecture
+
+```
+Telegram Bot (single token)
+         │
+         ▼
+   Default Gateway (multiplex_profiles: true)
+         │
+         ├── profile_routes: chat_id → profile
+         │
+         ├── default profile (Rhino chat: -1004321904721)
+         └── ops profile (Tusk ops: -1003914987043, -1003932503629)
+```
+
+## Key Config Locations
+
+### Default profile: `~/.hermes/config.yaml`
+
+```yaml
+gateway:
+  multiplex_profiles: true
+  profile_routes:
+    - name: ops-group
+      platform: telegram
+      chat_id: '-1003914987043'
+      profile: ops
+    - name: ops-group-2
+      platform: telegram
+      chat_id: '-1003932503629'
+      profile: ops
+
+telegram:
+  allowed_chats: '-1004321904721,-1003914987043,-1003932503629'
+  group_allowed_chats: '-1004321904721,-1003914987043,-1003932503629'
+```
+
+### Ops profile: `~/.hermes/profiles/ops/config.yaml`
+
+```yaml
+telegram:
+  allowed_chats:
+    - '-1003914987043'
+    - '-1003932503629'
+  group_allowed_chats:
+    - '-1003914987043'
+    - '-1003932503629'
+
+platforms:
+  telegram:
+    enabled: false   # ← DISABLED — multiplexer handles this
+```
+
+## The Silent-Drop Pitfall
+
+The `allowed_chats` gate runs on the **default profile's adapter** BEFORE `profile_routes` stamps `source.profile`. If a chat is only authorized in the secondary profile's config but not the default's, messages from that chat are silently dropped.
+
+**Moral:** When adding a new chat for a secondary profile, add it to BOTH profiles' `allowed_chats`.
+
+## Adding a New Group
+
+1. Add the chat_id to the default profile's `profile_routes`
+2. Add the chat_id to the ops profile's `allowed_chats` and `group_allowed_chats`
+3. **Add the chat_id to the default profile's `allowed_chats` and `group_allowed_chats`** (comma-separated list)
+4. Restart the gateway: `systemctl --user restart hermes-gateway`
+
+## Restarting
+
+From an SSH shell (outside the agent process):
+```bash
+systemctl --user restart hermes-gateway
+```
+
+## Related
+
+- [[hermes-agent]]
+- [[messaging-integrations]]
@@ -0,0 +1,103 @@
+---
+title: Provider Health Check
+type: runbook
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+last_tested: 2026-07-22
+confidence: high
+tags: [runbook, providers, health, routing]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# Provider Health Check
+
+## Purpose
+Verify Hermes can reach LiteLLM, OmniRoute, OpenRouter-style fallbacks, and the WebUI, and report which paths are healthy or degraded.
+
+## Symptoms that match this runbook
+- Models list is slow or blank in Hermes
+- Fallback model retries happen immediately
+- Chat returns 5xx or gateway errors
+
+## Prerequisites
+- Network access from this host to `litellm:4000`, `omniroute:20128`, `opencode.ai`, `api.telegram.org`, Telegram fallback IPs
+- Hermes config slocated at `~/.hermes/config.yaml`
+
+## Procedure
+
+### 1. Check local Hermes services
+```bash
+hermes gateway status
+hermes dashboard status
+```
+
+### 2. Check main provider reachability
+```bash
+python3 -c "import urllib.request; urllib.request.urlopen('http://litellm:4000/v1/models', timeout=4)"
+```
+Current install behavior: LiteLLM responds but requires auth, so HTTP 401 is expected from an unauthenticated check.
+
+### 3. Check OmniRoute reachability
+```bash
+python3 -c "
+import urllib.request, json, yaml
+with open('/home/hermes/.hermes/config.yaml') as f:
+    cfg = yaml.safe_load(f)
+key = [p['api_key'] for p in cfg.get('custom_providers',[]) if p.get('name')=='omniroute'][0]
+req = urllib.request.Request('http://omniroute:20128/v1/models', headers={'Authorization': f'Bearer {key}'})
+with urllib.request.urlopen(req, timeout=5) as r:
+    data = json.loads(r.read())
+    print('OmniRoute OK, models:', len(data.get('data', [])))
+"
+```
+Current verified result: reachable and returns a non-empty model catalog.
+
+### 4. Check fallback provider endpoints
+```bash
+curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://opencode.ai/zen/v1/models
+curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://inference-api.nousresearch.com/v1/models
+curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://ollama.com/v1/models
+```
+Any non-200 here means that fallback path is currently offline.
+
+### 5. Check WebUI health
+```bash
+curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8787/health
+```
+Expected: `200`
+
+### 6. Check gateway platform reachability
+Tail the gateway journal for errors:
+```bash
+journalctl --user -u hermes-gateway.service -n 100 --no-pager
+```
+Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or Email fetch.
+
+## Verification
+- Gateway status shows `active (running)`.
+- LiteLLM 401 from unauthenticated check is expected.
+- OmniRoute returns model count > 0.
+- Fallback URLs return 200 or expected auth codes.
+- WebUI `/health` returns `200`.
+- Gateway journal shows no red alarm pattern.
+
+## Current install references
+- Hermes: v0.19.0, git install: `/home/hermes/.hermes/hermes-agent`
+- LiteLLM base URL: `http://litellm:4000/v1`
+- OmniRoute base URL: `http://omniroute:20128/v1`
+- OmniRoute direct IP: `100.88.81.19:20128`
+- WebUI: `http://0.0.0.0:8787/health`
+
+## Rollback
+- This read-only runbook has no rollback side effects
+- If you change fallback providers during remediation, record and revert via `hermes fallback list` + previous wiki state
+
+## Last tested
+2026-07-22
+
+## Related
+- [[model-providers]]
+- [[messaging-integrations]]
+- [[restart-hermes]]
@@ -0,0 +1,20 @@
+---
+title: Recover a Docker service
+type: runbook
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+tags: [runbook, docker, recovery]
+sources: []
+confidence: high
+---
+# Recover Docker service
+## Procedure
+1. Identify the container.
+2. Inspect status and last 100 log lines.
+3. If unhealthy, restart then re-check.
+4. If still failing, inspect image and mounts.
+## Verification
+Container shows healthy after recovery probe reboots.
+## Last successfully used
+2026-07-22
@@ -0,0 +1,79 @@
+---
+title: Restart Browser Helper
+type: runbook
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+last_tested: 2026-07-22
+confidence: high
+tags: [runbook, browser, cua-driver, camofox]
+sources: []
+---
+
+# Restart Browser Helper
+
+## Purpose
+Restart the computer-use browser stack that Hermes uses for web automation, screenshots, and VNC/noVNC access.
+
+## Symptoms that match this runbook
+- Browser screenshots are failing
+- `cua-driver` MCP tools return errors
+- VNC/noVNC is unreachable on `5901` / `6081`
+
+## Prerequisites
+- Display helper already running: `Xvfb :99`, x11vnc on `127.0.0.1:5901`, noVNC at `127.0.0.1:6081`
+- `cua-driver` binary installed at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver`
+- Browser VM/desktop helper is expected but not mandatory
+
+## Procedure
+
+### 1. Inspect current browser helpers
+```bash
+ps aux | grep -E "cua-driver serve|x11vnc|fluxbox|vnc-watcher" | grep -v grep
+```
+
+### 2. Stop cua-driver only
+```bash
+pkill -f "cua-driver serve"
+```
+
+### 3. Start cua-driver
+```bash
+/home/hermes/.local/bin/cua-driver serve
+```
+Run in background or a tmux/screen session if needed.
+
+### 4. Verify ports remain available
+```bash
+nc -z 127.0.0.1 5901 && echo "x11vnc ok" || echo "x11vnc missing"
+curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:6081/
+```
+
+### 5. Restart full browser VM stack only if step 4 fails
+```bash
+pkill -f "Xvfb :99"
+pkill -f "x11vnc"
+pkill -f "fluxbox"
+/home/hermes/.local/bin/cua-driver serve
+```
+
+## Verification
+- `cua-driver serve` process is present in `ps`
+- `5901` and `6081` are reachable
+- No zombie accumulation in browser/desktop processes
+
+## Rollback
+- If browser automation is not mission-critical, skip full VM restart and continue without it
+- Do not stop gateway or dashboard unless necessary
+
+## Notes
+- This runbook assumes the standalone desktop helper is running separately from Hermes core
+- The browser backend path can work without browser automation; this stack is optional depending on task
+
+## Last tested
+2026-07-22
+
+## Related
+- [[browser-backend]]
+- [[restart-hermes]]
@@ -0,0 +1,96 @@
+---
+title: Restart Hermes
+type: runbook
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+last_tested: 2026-07-22
+confidence: high
+tags: [runbook, hermes, restart]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# Restart Hermes
+
+## Purpose
+Restart the Hermes agent, gateway, dashboard, and browser helper processes cleanly.
+
+## Symptoms that match this runbook
+- Hermes stops responding in chat or WebUI
+- Tools fail mid-session but the agent is otherwise running
+- After an update or config change
+
+## Prerequisites
+- User account `hermes` with access to systemd --user
+- No active backups in progress
+- If using a browser session, warn the user before restarting dashboards
+
+## Procedure
+
+### 1. Identify the active Hermes processes
+```bash
+ps aux | grep -E "hermes_cli.main (gateway|dashboard) run|hermes-webui/server.py|cua-driver serve|x11vnc.*5901" | grep -v grep
+```
+Expected active pieces on this install:
+- gateway: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run`
+- dashboard: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main dashboard ...`
+- webui: `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py`
+- camofox/cua-driver: `/home/hermes/.local/bin/cua-driver serve`
+- x11vnc: `-display :99 ... 5901`
+
+### 2. Restart the gateway
+```bash
+hermes gateway restart
+```
+
+### 3. Restart the dashboard if needed
+```bash
+hermes dashboard restart
+```
+If that fails, stop and rerun:
+```bash
+hermes dashboard stop
+hermes dashboard --host 0.0.0.0 --port 9119 --no-open
+```
+
+### 4. Restart the browser helper if needed
+```bash
+pkill -f "cua-driver serve"
+/home/hermes/.local/bin/cua-driver serve
+```
+
+### 5. Restart the WebUI if needed
+Find its PID, then restart standalone:
+```bash
+pkill -f "hermes-webui/server.py"
+/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py
+```
+
+## Verification
+```bash
+hermes gateway status
+hermes dashboard status
+ss -tlnp | grep -E "8787|9119|5901|6081"
+ps aux | grep -E "gateway run|dashboard|cua-driver serve|server.py" | grep -v grep
+```
+Current known-good ports after verified restart:
+- gateway: active via `hermes-gateway.service`
+- dashboard: `0.0.0.0:9119`
+- webui: `0.0.0.0:8787`
+- VNC: `5901`
+- noVNC: `6081`
+
+## Rollback
+- If restart fails, review recent config changes and consider restoring `config.yaml.bak.*`
+- Do not delete old config files without explicit confirmation
+- Re-run backup first if you plan to rollback config
+
+## Notes
+- Last tested: 2026-07-22 on debian 13, user-mode systemd
+- WebUI logs: `/home/hermes/.hermes/webui/bootstrap-8787.log`
+- Defunct `<defunct>` processes in process list are expected zombie shells from prior runs and do not require action unless they pile up
+
+## Related
+- [[runbooks/backup-wiki]]
+- [[runbooks/update-hermes-safely]]
@@ -0,0 +1,27 @@
+---
+title: Restore OpenRouter connectivity
+type: runbook
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+tags: [runbook, openrouter, recovery]
+sources: []
+confidence: high
+---
+# Restore OpenRouter connectivity
+## Symptoms
+- TLS connection hangs
+- Model requests time out
+- Other providers still work
+## Procedure
+1. Check session activity; ensure previous HTTPS calls completed.
+2. Review last WARP errors.
+3. Restart the Hermes process with the GoOp tool.
+4. Send a small test completion request.
+5. Confirm tool calling works.
+## Verification
+A test prompt returns successfully through OpenRouter.
+## Escalation
+If the failure persists, check DNS, certificate expiry and provider status.
+## Last successfully used
+2026-06-22
@@ -0,0 +1,84 @@
+---
+title: Test OpenRouter
+type: runbook
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+last_tested: 2026-07-22
+confidence: high
+tags: [runbook, openrouter, providers, fallback]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# Test OpenRouter
+
+## Purpose
+Test whether OpenRouter is available to Hermes and whether fallback providers are actually callable in this installation.
+
+## Symptoms that match this runbook
+- Hermes reports fallback model errors
+- `hermes fallback list` shows expected providers, but actual requests fail
+- User wants to confirm OpenRouter is usable without relying on catalog names
+
+## Prerequisites
+- Hermes CLI available: `/home/hermes/.local/bin/hermes`
+- Internet egress for opencode.ai, inference-api.nousresearch.com, ollama.com, kilo.ai, litellm:4000
+
+## Procedure
+
+### 1. List fallback providers
+```bash
+hermes fallback list
+```
+Expected active fallbacks on this install:
+1. `deepseek-v4-flash-free` via opencode-zen
+2. `stepfun/step-3.7-flash:free` via nous
+3. `qwen3.5:397b` via ollama-cloud
+4. `poolside/laguna-m.1:free` via kilocode
+5. `or-inclusionai_ling-2_6-1t` via custom/litellm
+
+### 2. Test OpenRouter-style direct access if present
+```bash
+curl -s -o /dev/null -w "%{http_code}" --max-time 8 https://openrouter.ai/api/v1/models
+```
+Current install state as of last run: OpenRouter is not configured as a standalone provider in Hermes, so this is an optional confirmation step only.
+
+### 3. Test each verified fallback URL
+```bash
+for url in \
+  "https://opencode.ai/zen/v1/models" \
+  "https://inference-api.nousresearch.com/v1/models" \
+  "https://ollama.com/v1/models" \
+  "https://api.kilo.ai/api/gateway/models"; do
+  printf "%s -> " "$url"
+  curl -s -o /dev/null -w "%{http_code}" --max-time 8 "$url"
+  echo
+done
+```
+
+### 4. Check Hermes fallback config reflects reality
+```bash
+grep -A5 "^fallback:" ~/.hermes/config.yaml
+```
+Verify provider names and URLs in config match what the user wants to use.
+
+## Verification
+- `hermes fallback list` shows the exact same providers as the live config
+- Each URL returns an HTTP response within timeout
+- No leftover undefined OpenRouter provider entries exist in `custom_providers`
+
+## Rollback
+- No system changes in this runbook
+- If a provider looks wrong but is part of live chat behavior, treat it like a config issue, not a transient health failure
+
+## Notes
+- In this installation, OpenRouter is not a live first-class provider in `custom_providers`
+- This runbook tests confirmed fallback paths, not assumed ones
+
+## Last tested
+2026-07-22
+
+## Related
+- [[provider-health]]
+- [[model-providers]]
@@ -0,0 +1,26 @@
+---
+title: Troubleshoot Tool Loop
+type: runbook
+status: unresolved
+created: 2026-07-22
+updated: 2026-07-22
+review_after: 2026-08-22
+confidence: low
+tags: [runbook, hermes]
+sources: []
+---
+
+> **Stub — structure only.** Headings exist but content has not been verified/written yet. Flagged during 2026-07-22 wiki audit; fill in with verified facts or mark deprecated if no longer relevant.
+
+# Troubleshoot Tool Loop
+## Symptoms
+Agent loops on one tool or endpoint.
+## Procedure
+1. Inspect recent Hermes agent logs.
+2. Identify the looping tool path.
+3. Restart worker or flush stuck request.
+4. Re-run limited test scenario.
+## Verification
+Agent handles the same scenario without repeating.
+## Last successfully used
+2026-07-22
@@ -0,0 +1,106 @@
+---
+title: Update Hermes safely
+type: runbook
+status: active
+created: 2026-07-22
+updated: 2026-07-22
+verified_on: 2026-07-22
+last_tested: 2026-07-22
+confidence: high
+tags: [runbook, hermes, update]
+sources: [raw/configs/hermes-config-sanitized.txt]
+---
+
+# Update Hermes safely
+
+## Purpose
+Update Hermes to a newer release without losing configuration, skills, or sessions.
+
+## Symptoms that match this runbook
+- User wants to move to a newer Hermes version
+- `hermes version` shows an older build than expected
+- Update-related errors after manual changes in the install directory
+
+## Prerequisites
+- Hermes install method: git-based install at `/home/hermes/.hermes/hermes-agent`
+- Working Python virtualenv at `/home/hermes/.hermes/hermes-agent/venv`
+- Available disk space in Hermes install and backup paths
+- No active Hermes chat/webui shell issues before starting
+
+## Procedure
+
+### 1. Create a backup first
+```bash
+hermes backup --quick --label pre-update-$(date +%Y%m%d)
+```
+If you need a full backup instead:
+```bash
+hermes backup --label pre-update-$(date +%Y%m%d)
+```
+
+### 2. Check whether an update is available
+```bash
+hermes update --check
+```
+
+### 3. Review configuration version and backups
+```bash
+python3 -c "import yaml; print(yaml.safe_load(open('/home/hermes/.hermes/config.yaml')).get('_config_version'))"
+ls -la ~/.hermes/config.yaml.bak*
+```
+
+### 4. Run the update with noninteractive flags
+```bash
+cd /home/hermes/.hermes/hermes-agent
+git status --short
+hermes update --yes --branch main
+```
+
+### 5. Verify the new version
+```bash
+hermes version
+```
+
+### 6. Restart the required Hermes pieces
+```bash
+hermes gateway status
+hermes dashboard status
+hermes gateway restart
+hermes dashboard restart
+```
+
+### 7. Sanity check health
+```bash
+hermes gateway status
+curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8787/health
+```
+
+## Verification
+- `hermes version` shows a newer version than before
+- `hermes gateway status` is active
+- WebUI `/health` returns `200`
+- Providers reachable with [[provider-health]]
+
+## Rollback
+- Rollback Hermes config:
+```bash
+ls -lt ~/.hermes/config.yaml.bak*
+cp ~/.hermes/config.yaml.bak.<timestamp> ~/.hermes/config.yaml
+hermes gateway restart
+hermes dashboard restart
+```
+- Rollback entire Hermes install requires git revert or reinstall from previous checkout
+- If update warns about config migration, revert the backup and re-run manually
+
+## Notes
+- Hermes version on this install: v0.19.0 (2026.7.20)
+- Install directory: `/home/hermes/.hermes/hermes-agent`
+- Update preserves venv by default unless `--force-venv` is used
+
+## Last tested
+2026-07-22 on debian 13, git-based Hermes install
+
+## Related
+- [[restart-hermes]]
+- [[backup-wiki]]
+- [[provider-health]]