--- title: Troubleshoot Tool Loop type: runbook status: unresolved created: 2026-07-22 updated: 2026-11-11 review_after: 2026-11-19 verified_on: 2026-11-11 confidence: medium tags: [runbook, hermes] sources: [] --- > **Filled in 2026-10-20** from live configuration, live process state and two recorded incidents. > The "stub" note below is retained as history. # Troubleshoot Tool Loop ## Symptoms - The agent repeats the **same tool call** with identical arguments and makes no progress. - The agent repeats **itself in the response stream** (no tool involved) — the model loops on its own output and Hermes cuts the stream. **These are two different faults with two different detectors.** Do not assume one covers the other. ## 1. Tool-call loop — detected by `tool_loop_guardrails` ### Symptoms Agent loops on one tool or endpoint. Hermes emits a warning naming the tool. ### Configuration (verified live 2026-10-20, `~/.hermes/config.yaml`) ```yaml tool_loop_guardrails: warnings_enabled: true hard_stop_enabled: false warn_after: exact_failure: 2 same_tool_failure: 3 idempotent_no_progress: 2 hard_stop_after: exact_failure: 5 same_tool_failure: 8 idempotent_no_progress: 5 ``` > **`hard_stop_enabled: false` is the key fact.** Hermes will *warn* about a stuck tool loop but > will **not** force-stop itself. A genuinely stuck loop only gets a warning — a human (or the > messaging integration) is the real backstop. See [[concepts/tool-calling]]. ### Procedure 1. Inspect recent Hermes agent logs: `~/.hermes/logs/agent.log*` (grep for the tool name). 2. Identify the looping tool path. Common culprits in this estate: a browser tool against a dead Camofox service, a terminal tool against a stale persistent shell, or a webhook that returns 200 with an empty body. 3. **Check the dependency, not just the agent.** On this estate the most frequent tool-loop cause was an upstream service being down while the tool still returned a response. Verify with the relevant runbook: [[runbooks/restart-browser]], [[runbooks/restart-hermes]], [[runbooks/provider-health]]. 4. If the loop is a stuck worker, restart it: `systemctl --user restart hermes-gateway.service`. 5. Re-run a limited test scenario and confirm the guard no longer fires. ### Verification Agent handles the same scenario without repeating, and no `tool_loop_guardrails` warning appears in `agent.log`. ## 2. Model-output (degenerate stream) loop — detected by the stream guard ### Symptoms The run **delivers a canned non-answer** such as *"Your request was not processed. Send it again if you still want me to carry it out."*, and then fails with `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` ### This is the 2026-10-18 case — see [[incidents/2026-10-18-degenerate-stream-canned-substitute]] ### Procedure 1. Read the run's `error` in `~/.hermes/cron/executions.db` — the phrase `repetition loop` identifies this fault. 2. **Check `delivery_outcome` on the same row.** In the 2026-10-18 incident the row was `delivery_outcome: delivered` — the canned non-answer reached Telegram *before* the run failed. **A delivered message is not evidence of delivered content.** 3. Confirm the provider is healthy: the 2026-10-18 case rode the same `custom:omni-101` alias that produced the 2026-10-14/15 connection-refused storm and the 2026-10-16/17 and 2026-10-19 resource-pressure 503 bursts. See [[runbooks/provider-health]] and [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 4. Do **not** lower `tool_loop_guardrails` thresholds for this fault — they watch tool calls, and the degenerate-stream guard is a separate mechanism. ### Verification The next scheduled run of the job completes with `status: completed` and `error: null`, and the delivered message contains real content. ## Last successfully used 2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the *procedure* has not itself been executed end-to-end in a live loop, so `confidence` was raised only to `medium`. > **Guardrail config re-read 2026-11-11 — byte-for-byte unchanged.** > `tool_loop_guardrails`: `warnings_enabled: true`, `hard_stop_enabled: false`, > `warn_after {exact_failure: 2, same_tool_failure: 3, idempotent_no_progress: 2}`, > `hard_stop_after {exact_failure: 5, same_tool_failure: 8, idempotent_no_progress: 5}`. > `hard_stop_enabled: false` therefore still means a stuck tool loop only ever *warns*. ## Related - [[concepts/tool-calling]] - [[incidents/2026-10-18-degenerate-stream-canned-substitute]] - [[runbooks/provider-health]] - [[runbooks/restart-hermes]]