Maxxwell by Rindler
Writing

Monitoring and Debugging an AI Agent Swarm

2026-09-08

When you're running more than a couple of AI coding agents, the bottleneck is no longer model output. It's your ability to see what's happening and intervene.

When you're running more than a couple of AI coding agents, the bottleneck is no longer model output. It's your ability to see what's happening and intervene before something stupid lands on main.

This tutorial walks through a concrete setup for monitoring and debugging a multi-agent coding swarm in real time: logging, tracing, and inspecting decisions; spotting misaligned goals, loops, and conflicting edits before they ship.

If you want the wider theory first, read the pillar piece "AI coding agent orchestration: the complete guide for multi-agent development" - this one stays practical.

What you'll build

By the end you'll have:

You can do this with whatever agent stack you already use: Claude Code, Codex, Cursor, homegrown scripts. I'll show examples using OpenTelemetry-compatible tracing plus a local orchestrator window (Maxxwell), but the patterns apply generically.

Step 0: Prerequisites and baseline setup

Before wiring up observability, clean up the basics.

Prerequisites:

Baseline safety:

  1. Isolate work per agent.
    • Use separate git worktrees so parallel edits don't collide.
    • Claude Code explicitly recommends this pattern; copy it:
   git worktree add ../feature-login feature/login
   git worktree add ../feature-billing feature/billing
  1. Tag each agent session.
    • Give each worker a clear task name and repo path.
    • This becomes the key for tracing and dashboarding later.

If you're already running Maxxwell, you get per-session state labels and orchestration out of the box, but the tracing pieces are still worth doing.

Step 1: Instrument your agents for structured logging

You can't debug a swarm with raw chat logs and screenshots. Start by making every agent request an event you can query.

1.1 Decide your logging schema

You want one log line per agent action with at least:

Example JSON log:

{
  "ts": "2026-09-03T16:20:31.123Z",
  "agent_id": "worker-3",
  "task_id": "feature-login",
  "request_id": "req-9f3a",
  "phase": "edit",
  "goal": "Add password reset endpoint",
  "files_touched": ["services/auth.py", "routes/auth_reset.py"],
  "status": "started"
}

1.2 Hook your GenAI client

If you’re using an SDK that already integrates with OpenTelemetry’s GenAI conventions, enable it.

Example with a hypothetical Python client:

from genai_client import Client
from otel_genai import instrument_genai

instrument_genai()  # wires spans for prompts, tool calls, etc.

client = Client(api_key=API_KEY)

def run_agent_step(agent_id, task_id, phase, prompt):
    with tracer.start_as_current_span("agent.step") as span:
        span.set_attribute("genai.agent.id", agent_id)
        span.set_attribute("genai.task.id", task_id)
        span.set_attribute("genai.phase", phase)
        resp = client.chat(prompt)
        return resp

OpenTelemetry’s GenAI repo now has conventions for client calls, tools, evaluation, and MCP. Using them means your traces will plug into Honeycomb/Langfuse/LangSmith/Datadog without a bespoke schema.

Step 2: Trace agent workflows, not just prompts

You don’t just care about “what did the model say?”. You care about “what did the workflow actually do?”. That’s where traces and spans matter.

2.1 Model each agent as a trace tree

Treat a human task as a root span and each agent step as a child.

Example:

with tracer.start_as_current_span("task.feature-login") as root:
    root.set_attribute("genai.task.description", "Implement login + reset flow")

    run_agent_step("worker-1", "feature-login", "plan", plan_prompt)
    run_agent_step("worker-1", "feature-login", "edit", edit_prompt)
    run_agent_step("worker-1", "feature-login", "test", test_prompt)

In Honeycomb or Langfuse, this gives you an end-to-end trace per feature with clear visibility into how many agent steps it took, where latency is, and where errors originate.

2.2 Capture tool calls and file edits as spans

Use the GenAI tool-execution conventions to mark structured actions.

Whenever an agent:

Example for file writes:

def write_file(agent_id, path, new_contents):
    with tracer.start_as_current_span("agent.write_file") as span:
        span.set_attribute("genai.agent.id", agent_id)
        span.set_attribute("code.file.path", path)
        span.set_attribute("code.change.hash", hash(new_contents))
        # actual write
        with open(path, "w") as f:
            f.write(new_contents)

Now you can query “which files did agent worker-3 touch today?” or “show me all write_file spans for feature-login”. This is how you trace file-level edits across agents without diffing the whole repo manually.

Step 3: Build a real-time agent session dashboard

Logs and traces are only useful if you can see them while the swarm is running.

3.1 Define session states that matter

OpenAI’s Symphony write-up notes engineers get uncomfortable managing more than 3-5 sessions. Past that, you need a dashboard.

Track at least these states per agent session:

In Maxxwell, each agent lane carries a readable state derived from real terminal sessions: working, idle, waiting, blocked, done, dead, or not heard from, with a “possibly stalled” overlay. You still control what happens; it just surfaces the state.

3.2 Implement a simple terminal dashboard

If you’re DIY-ing this, a basic TUI can go a long way.

Structure:

Pseudo-code for the state:

def compute_state(session):
    if session.dead:
        return "dead"
    if session.blocked:
        return "blocked"
    if session.waiting_on_human:
        return "waiting_on_you"
    if now - session.last_span_time > STALL_THRESHOLD:
        return "possibly_stalled"
    if session.active_span:
        return "working"
    return "idle"

Run this loop every few seconds and redraw the table. Now you can answer “which of these is stuck and which is just slow?” without clicking eight windows.

Maxxwell already does this as a desktop app: one window, all sessions, each marked with what it’s doing. If your orchestration layer is homegrown, copy the state model.

Step 4: Inspect agent reasoning steps and decisions

You need decision provenance: not just that an agent changed a file, but why.

4.1 Log reasoning chunks separately from final answers

When the agent produces a plan or a chain-of-thought (whether explicit or via system tools), treat it as its own artifact.

For each step:

Example log line:

{
  "ts": "2026-09-03T16:22:11.001Z",
  "agent_id": "worker-2",
  "task_id": "feature-billing",
  "reasoning_id": "r-42",
  "decision_summary": "Will add new endpoint to billing_api.py and update tests in test_billing_api.py",
  "constraints": ["do not modify payments_gateway.py", "all tests must pass"]
}

This gives you an audit trail of “what the agent thought it was doing” when you inspect weird behavior later.

4.2 Build an inspection command per session

Expose a CLI or UI control to show recent decisions for a given agent:

agent-inspect --agent worker-2 --last 10

Outputs something like:

[16:22:11] plan: add endpoint + tests (constraints: no payments_gateway.py)
[16:22:35] edit: billing_api.py, test_billing_api.py
[16:23:02] test: pytest tests/billing
[16:23:18] error: tests failed, retrying with updated fixture

In Maxxwell, the orchestrator seat functions as this inspection layer: it’s itself a coding-agent session started from a written brief, and you talk to it instead of to twelve terminals. It can summarize what each worker landed, what it decided, and what’s waiting on you.

Step 5: Detect misaligned goals early

“Building the wrong thing for twenty minutes” is usually a goal alignment problem, not a model capability issue.

5.1 Attach explicit goals to traces

Make the human brief a first-class attribute:

Example:

root.set_attribute("genai.task.goal", "Implement password reset without touching login UI")
root.set_attribute("genai.task.scope", "backend/auth service only")
root.set_attribute("genai.task.done_criteria", "tests green + endpoint documented")

When you inspect a trace that went sideways, you can compare each span’s goal and files_touched with the root goal.

5.2 Run a simple goal alignment check

Add a periodic check that:

Pseudo-code:

def check_goal_alignment(task):
    allowed = set(task.scope_files)
    for span in recent_spans(task.id):
        touched = set(span.attributes.get("files_touched", []))
        if not touched.issubset(allowed):
            alert("goal_misalignment", task.id, span)

This won’t auto-correct drift - and shouldn’t, if you care about safety - but it will surface misalignment while it’s still cheap to stop.

Maxxwell stays deliberately on this side of the line: it conducts the swarm, it does not silently re-aim or restart work. Anything that changes the fleet drafts a command you send yourself.

Step 6: Catch infinite loops and stalled sessions

A common failure mode in agent swarms is an agent quietly looping: retrying the same failing command or test forever.

6.1 Define obvious loop patterns

Mark a session as "possibly stalled" when:

Example pattern:

def detect_loops(spans):
    errors = [s.attributes.get("error.message") for s in spans]
    if len(errors) >= 3 and len(set(errors[-3:])) == 1:
        return True
    return False

Use that to set the session state to possibly_stalled and send an alert.

6.2 Wire basic alerts

You don’t need a full-blown NOC; a couple of simple triggers help:

Examples:

agent-alerts --on possibly_stalled --notify slack:#agent-watch
agent-alerts --on long_edit --threshold 600 --notify email:[email protected]

This is where real-time monitoring earns its keep: you see loops as they form, not when your CPU graph spikes.

Step 7: Resolve conflicting edits between agents

Parallelism is great until two agents edit the same file differently.

7.1 Use worktrees and merge queues

You already saw worktrees in Step 0. Add a merge queue pattern:

Example:

# agent branches
git checkout -b agent-worker-1 feature/login
# agent does work, commits

# merge queue process
git checkout staging
for branch in $(cat agent-branches.txt); do
  git merge --no-ff "$branch" || handle_conflict "$branch"
  pytest || handle_failure "$branch"
done

This is the same pattern ctx and Claude Code push: isolate parallel work and let conflicts appear in a controlled replay.

7.2 Tie merge conflicts back to agent traces

When a merge conflict happens, record:

If you’ve wired spans and logs as above, you can query:

That turns a messy conflict into a structured investigation: you see who did what, why, and in what order.

Step 8: Conduct the swarm from a single orchestrator

Past a handful of sessions, you don’t want to talk to twelve terminals. You want one conductor seat.

8.1 Brief an orchestrator agent

Create a dedicated orchestrator session with its own task:

You can build this yourself as another coding agent that sees meta-data instead of raw repo access, or you can use a tool like Maxxwell that gives you an orchestrator seat out of the box.

In Maxxwell:

8.2 Keep the person in control

Whatever orchestrator you use, keep one constraint: the person presses enter.

In Maxxwell, fleet controls draft rather than act. A control that would change the swarm writes a fully formed, unsent sentence into the composer; it never runs the command.

In a DIY setup, mimic that posture:

That’s how you get leverage without silent breakage: orchestration, not autopilot.

FAQ: common questions about monitoring agent swarms

How do I monitor AI coding agents in real time without a full observability stack?

Start with structured logging and a simple TUI.

watch -n 2 'jq -r "select(.agent_id==\"worker-1\")" agent.log'

This gives you a basic real-time view while you decide which tracing backend to adopt.

How can I debug a multi-agent coding swarm that seems stuck?

Look for:

Use your inspection command (agent-inspect) or an orchestrator like Maxxwell to see recent actions and decide yourself whether to stop, re-aim, or kill a worker.

What’s the best way to inspect agent reasoning steps safely?

Log decisions separately from code edits.

Then, when something odd happens, you query reasoning logs to understand intent and compare it with the actual file-level edits.

How do I prevent agents from modifying the wrong files?

Use a combination of scope constraints and alignment checks:

Isolation via git worktrees plus these checks catches most misfires before they reach main.

Do I need a tool like Maxxwell if I already have tmux and scripts?

If you’re only running one or two sessions, probably not.

Once you hit the OpenAI/Symphony discomfort zone (3-5+ sessions) and spend your day context switching, a conductor window starts to pay for itself.

Maxxwell’s value is in the orchestration layer: one window with per-session state, an orchestrator seat that speaks your language, and controls that draft rather than act. If your tmux setup gives you that with equal clarity and less maintenance, stick with it. Otherwise, it’s worth trying on a personal repo.