- freshness: 36 stale / 93 fresh / 3 no-date -> 0 stale / 129 fresh / 3 no-date of 132 (same-day report regenerated after the corrections, per pattern 13) - skills: 299 skills / 67 categories (count unchanged; generator arithmetic re-verified by hand against the live tree: 300 SKILL.md files - 1 nested duplicate = 299) - runbooks/test-openrouter: live fallback chain rewritten (primary back to smart-route/custom:omni-101; mimo-v2.5, stepfun, qwen3.5/ollama-cloud out; guacano + kilo-reflection-beam-free in); ollama.com dropped from the URL test list; guacano/litellm noted as internal + key-gated - comparisons/hermes-vs-nanobot: the 2026-10-20 'CT333 is zeroclaw in Proxmox naming' claim was itself wrong - pct list and hostname both return clawtest - incidents/2026-10-22-nous: refresh-token failure gone from logs; new signature 404 insufficient_credits_for_paid_model from auxiliary.pulse_memory (ninth slot, back on nous, paid model) - open item revised - incidents/2026-08-29-gmail: email crons now ok but root cause unfixed 74 days (himalaya message read still in fetch_newsletters.py; goimapnotify back up PID 441; monitor still paused) - 30 further pages re-verified live and de-flagged with dated probe notes (ct207 ingress, CT205 guanaco v0.8.15, CT270 nextcloud 33.0.2.2, headless stack, hermes-ui both hosts, systemd units + cgroup limits, terminal/tool_search/mnemosyne/custom_providers config blocks, ops profile still deleted, hosts/pve guest lists) - log.md addendum + index.md date bump; no page deleted, no secrets written
100 lines
3.3 KiB
Markdown
100 lines
3.3 KiB
Markdown
---
|
|
title: Diagnose Failed Docker Service
|
|
type: runbook
|
|
status: active
|
|
created: 2026-07-22
|
|
updated: 2026-11-11
|
|
verified_on: 2026-11-11
|
|
last_tested: 2026-10-20
|
|
confidence: medium
|
|
tags: [runbook, docker, homelab]
|
|
sources: []
|
|
---
|
|
|
|
# Diagnose Failed Docker Service
|
|
|
|
## Purpose
|
|
Diagnose container or compose issues on hosts where Docker is present.
|
|
|
|
## Symptoms that match this runbook
|
|
- Service reports container exit/failed state
|
|
- `docker ps` does not show an expected container
|
|
- Compose stack reports unhealthy status
|
|
|
|
## Prerequisites
|
|
- `docker` installed and accessible to the user
|
|
- Docker daemon running
|
|
- Compose file path known for the affected project
|
|
|
|
> **Important scoping note, verified 2026-10-20: the `hermes` user is NOT in the `docker` group on
|
|
> CT460.** `systemctl is-active docker` → `active`, but `docker ps` as `hermes` returns
|
|
> **`permission denied while trying to connect to the docker API at unix:///var/run/docker.sock`**.
|
|
> Most Docker services in this estate run on **other hosts** (runtipi CT201, cloudreve CT209,
|
|
> LiteLLM CT500 on m93p, etc.) where the invoking account does have socket access.
|
|
> So: on CT460, either use `sudo docker …` or SSH to the host that actually owns the container.
|
|
> Note also that `systems/homepage-dashboard.md` now records the CT201 stack as **degraded** —
|
|
> the main dashboard container is gone — which is exactly the symptom this runbook would be used for.
|
|
|
|
> **Scoping note re-confirmed 2026-11-11.** `id -nG hermes` still contains **no `docker` group**
|
|
> membership, so on CT460 `docker ps` as `hermes` still fails with
|
|
> `permission denied while trying to connect to the docker API at unix:///var/run/docker.sock`.
|
|
> The rule stands: on this box use `sudo docker …`, or SSH/`pct exec` to the host that owns the
|
|
> container. (No container claims on this page were re-verified in this pass.)
|
|
|
|
## Procedure
|
|
|
|
### 1. Check Docker daemon state
|
|
```bash
|
|
systemctl status docker
|
|
```
|
|
|
|
### 2. List containers
|
|
```bash
|
|
docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
|
|
```
|
|
|
|
### 3. Inspect logs for the failing container
|
|
```bash
|
|
docker logs --tail 100 <container-name>
|
|
```
|
|
|
|
### 4. If using compose, check compose state
|
|
```bash
|
|
cd /home/hermes/<project>/docker-compose
|
|
docker compose ps
|
|
docker compose logs --tail 100 <service-name>
|
|
```
|
|
|
|
### 5. Restart the failed container only
|
|
```bash
|
|
docker restart <container-name>
|
|
```
|
|
|
|
### 6. If restart fails, investigate restart policy and image
|
|
```bash
|
|
docker inspect --format "{{.HostConfig.RestartPolicy}}" <container-name>
|
|
docker inspect --format "{{.Config.Image}}" <container-name>
|
|
```
|
|
|
|
## Verification
|
|
- `docker ps` shows the container `Up` or `healthy`
|
|
- Service or WebUI dependent on the container is reachable again
|
|
- Logs show clean startup messages
|
|
|
|
## Rollback
|
|
- Do not delete volumes unless confirmed not in use
|
|
- If a container recreate is required, preserve volume mappings and env files
|
|
- If uncertain, collect logs first and ask before replacement
|
|
|
|
## Notes
|
|
- This runbook is generic. This current Hermes host does not have Docker installed as of 2026-07-22
|
|
- Docker-related paths must be confirmed before executing compose commands
|
|
- Do not invent container names or project paths
|
|
|
|
## Last tested
|
|
2026-07-22 on system where Docker runtime is absent; read-only steps verified, restart steps not executed
|
|
|
|
## Related
|
|
- [[docker-services]]
|
|
- [[hosts]]
|