Files
agent-estate-wiki/runbooks/troubleshoot-tool-loop.md
T

4.2 KiB

title, type, status, created, updated, review_after, verified_on, confidence, tags, sources
title type status created updated review_after verified_on confidence tags sources
Troubleshoot Tool Loop runbook unresolved 2026-07-22 2026-10-20 2026-11-19 2026-10-20 medium
runbook
hermes

Filled in 2026-10-20 from live configuration, live process state and two recorded incidents. The "stub" note below is retained as history.

Troubleshoot Tool Loop

Symptoms

  • The agent repeats the same tool call with identical arguments and makes no progress.
  • The agent repeats itself in the response stream (no tool involved) — the model loops on its own output and Hermes cuts the stream.

These are two different faults with two different detectors. Do not assume one covers the other.

1. Tool-call loop — detected by tool_loop_guardrails

Symptoms

Agent loops on one tool or endpoint. Hermes emits a warning naming the tool.

Configuration (verified live 2026-10-20, ~/.hermes/config.yaml)

tool_loop_guardrails:
  warnings_enabled: true
  hard_stop_enabled: false
  warn_after:
    exact_failure: 2
    same_tool_failure: 3
    idempotent_no_progress: 2
  hard_stop_after:
    exact_failure: 5
    same_tool_failure: 8
    idempotent_no_progress: 5

hard_stop_enabled: false is the key fact. Hermes will warn about a stuck tool loop but will not force-stop itself. A genuinely stuck loop only gets a warning — a human (or the messaging integration) is the real backstop. See concepts/tool-calling.

Procedure

  1. Inspect recent Hermes agent logs: ~/.hermes/logs/agent.log* (grep for the tool name).
  2. Identify the looping tool path. Common culprits in this estate: a browser tool against a dead Camofox service, a terminal tool against a stale persistent shell, or a webhook that returns 200 with an empty body.
  3. Check the dependency, not just the agent. On this estate the most frequent tool-loop cause was an upstream service being down while the tool still returned a response. Verify with the relevant runbook: runbooks/restart-browser, runbooks/restart-hermes, runbooks/provider-health.
  4. If the loop is a stuck worker, restart it: systemctl --user restart hermes-gateway.service.
  5. Re-run a limited test scenario and confirm the guard no longer fires.

Verification

Agent handles the same scenario without repeating, and no tool_loop_guardrails warning appears in agent.log.

2. Model-output (degenerate stream) loop — detected by the stream guard

Symptoms

The run delivers a canned non-answer such as "Your request was not processed. Send it again if you still want me to carry it out.", and then fails with RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.

This is the 2026-10-18 case — see

incidents/2026-10-18-degenerate-stream-canned-substitute

Procedure

  1. Read the run's error in ~/.hermes/cron/executions.db — the phrase repetition loop identifies this fault.
  2. Check delivery_outcome on the same row. In the 2026-10-18 incident the row was delivery_outcome: delivered — the canned non-answer reached Telegram before the run failed. A delivered message is not evidence of delivered content.
  3. Confirm the provider is healthy: the 2026-10-18 case rode the same custom:omni-101 alias that produced the 2026-10-14/15 connection-refused storm and the 2026-10-16/17 and 2026-10-19 resource-pressure 503 bursts. See runbooks/provider-health and incidents/2026-10-16-omniroute-503-resource-pressure.
  4. Do not lower tool_loop_guardrails thresholds for this fault — they watch tool calls, and the degenerate-stream guard is a separate mechanism.

Verification

The next scheduled run of the job completes with status: completed and error: null, and the delivered message contains real content.

Last successfully used

2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the procedure has not itself been executed end-to-end in a live loop, so confidence was raised only to medium.