- freshness: 36 stale / 93 fresh / 3 no-date -> 0 stale / 129 fresh / 3 no-date of 132 (same-day report regenerated after the corrections, per pattern 13) - skills: 299 skills / 67 categories (count unchanged; generator arithmetic re-verified by hand against the live tree: 300 SKILL.md files - 1 nested duplicate = 299) - runbooks/test-openrouter: live fallback chain rewritten (primary back to smart-route/custom:omni-101; mimo-v2.5, stepfun, qwen3.5/ollama-cloud out; guacano + kilo-reflection-beam-free in); ollama.com dropped from the URL test list; guacano/litellm noted as internal + key-gated - comparisons/hermes-vs-nanobot: the 2026-10-20 'CT333 is zeroclaw in Proxmox naming' claim was itself wrong - pct list and hostname both return clawtest - incidents/2026-10-22-nous: refresh-token failure gone from logs; new signature 404 insufficient_credits_for_paid_model from auxiliary.pulse_memory (ninth slot, back on nous, paid model) - open item revised - incidents/2026-08-29-gmail: email crons now ok but root cause unfixed 74 days (himalaya message read still in fetch_newsletters.py; goimapnotify back up PID 441; monitor still paused) - 30 further pages re-verified live and de-flagged with dated probe notes (ct207 ingress, CT205 guanaco v0.8.15, CT270 nextcloud 33.0.2.2, headless stack, hermes-ui both hosts, systemd units + cgroup limits, terminal/tool_search/mnemosyne/custom_providers config blocks, ops profile still deleted, hosts/pve guest lists) - log.md addendum + index.md date bump; no page deleted, no secrets written
4.6 KiB
title, type, status, created, updated, review_after, verified_on, confidence, tags, sources
| title | type | status | created | updated | review_after | verified_on | confidence | tags | sources | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Troubleshoot Tool Loop | runbook | unresolved | 2026-07-22 | 2026-11-11 | 2026-11-19 | 2026-11-11 | medium |
|
Filled in 2026-10-20 from live configuration, live process state and two recorded incidents. The "stub" note below is retained as history.
Troubleshoot Tool Loop
Symptoms
- The agent repeats the same tool call with identical arguments and makes no progress.
- The agent repeats itself in the response stream (no tool involved) — the model loops on its own output and Hermes cuts the stream.
These are two different faults with two different detectors. Do not assume one covers the other.
1. Tool-call loop — detected by tool_loop_guardrails
Symptoms
Agent loops on one tool or endpoint. Hermes emits a warning naming the tool.
Configuration (verified live 2026-10-20, ~/.hermes/config.yaml)
tool_loop_guardrails:
warnings_enabled: true
hard_stop_enabled: false
warn_after:
exact_failure: 2
same_tool_failure: 3
idempotent_no_progress: 2
hard_stop_after:
exact_failure: 5
same_tool_failure: 8
idempotent_no_progress: 5
hard_stop_enabled: falseis the key fact. Hermes will warn about a stuck tool loop but will not force-stop itself. A genuinely stuck loop only gets a warning — a human (or the messaging integration) is the real backstop. See concepts/tool-calling.
Procedure
- Inspect recent Hermes agent logs:
~/.hermes/logs/agent.log*(grep for the tool name). - Identify the looping tool path. Common culprits in this estate: a browser tool against a dead Camofox service, a terminal tool against a stale persistent shell, or a webhook that returns 200 with an empty body.
- Check the dependency, not just the agent. On this estate the most frequent tool-loop cause was an upstream service being down while the tool still returned a response. Verify with the relevant runbook: runbooks/restart-browser, runbooks/restart-hermes, runbooks/provider-health.
- If the loop is a stuck worker, restart it:
systemctl --user restart hermes-gateway.service. - Re-run a limited test scenario and confirm the guard no longer fires.
Verification
Agent handles the same scenario without repeating, and no tool_loop_guardrails warning appears
in agent.log.
2. Model-output (degenerate stream) loop — detected by the stream guard
Symptoms
The run delivers a canned non-answer such as "Your request was not processed. Send it again
if you still want me to carry it out.", and then fails with
RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.
This is the 2026-10-18 case — see
incidents/2026-10-18-degenerate-stream-canned-substitute
Procedure
- Read the run's
errorin~/.hermes/cron/executions.db— the phraserepetition loopidentifies this fault. - Check
delivery_outcomeon the same row. In the 2026-10-18 incident the row wasdelivery_outcome: delivered— the canned non-answer reached Telegram before the run failed. A delivered message is not evidence of delivered content. - Confirm the provider is healthy: the 2026-10-18 case rode the same
custom:omni-101alias that produced the 2026-10-14/15 connection-refused storm and the 2026-10-16/17 and 2026-10-19 resource-pressure 503 bursts. See runbooks/provider-health and incidents/2026-10-16-omniroute-503-resource-pressure. - Do not lower
tool_loop_guardrailsthresholds for this fault — they watch tool calls, and the degenerate-stream guard is a separate mechanism.
Verification
The next scheduled run of the job completes with status: completed and error: null, and the
delivered message contains real content.
Last successfully used
2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the procedure has
not itself been executed end-to-end in a live loop, so confidence was raised only to medium.
Guardrail config re-read 2026-11-11 — byte-for-byte unchanged.
tool_loop_guardrails:warnings_enabled: true,hard_stop_enabled: false,warn_after {exact_failure: 2, same_tool_failure: 3, idempotent_no_progress: 2},hard_stop_after {exact_failure: 5, same_tool_failure: 8, idempotent_no_progress: 5}.hard_stop_enabled: falsetherefore still means a stuck tool loop only ever warns.