Maxxwell by Rindler
Writing

Instrumenting Agent Fleets with OpenTelemetry

2026-09-26

Most teams adopt agents one developer at a time. The real pain starts when several people have fleets running and nobody can see what’s actually happening.

Most teams adopt agents one developer at a time. The real pain starts when several people have fleets running and nobody can see what’s actually happening.

This tutorial shows how to add metrics, logs, and traces around Maxxwell so a lead can answer three questions from a single view:

The focus is practical: shell commands, OpenTelemetry wiring, and a simple “agent workflow health” dashboard you can evolve.

What we’ll build

By the end, you’ll have:

We’ll use:

This assumes you already know your way around Maxxwell and run multiple coding agents daily. If you’re still in “single Claude tab” land, this is overkill.

For more conceptual background on "agent-native" workflows, see the companion piece Agent-native development: a working definition - this tutorial is the wiring, that article is the mental model.

Why agent observability matters more than model quality

Surveys say adoption is ahead of trust:

In other words: you already run agents, they already ship code, and nobody fully trusts them.

At that point, the bottleneck isn’t “write more code”, it’s:

Google Cloud, Dynatrace and OpenTelemetry all say the same thing: agent systems need multi-signal observability - logs, metrics, and traces - not just console dumps.

Maxxwell already gives you an operational surface:

This tutorial just instruments that surface.

Step 0 - Prerequisites and mental model

You’ll need:

We’ll treat Maxxwell as the control plane, and our telemetry sidecar as:

The core pattern works for any agent orchestrator (CommandSlate, Helmor, The Cog, your own tmux zoo). Maxxwell just happens to make the fleet topology explicit.

Step 1 - Decide what to measure first

Borrowing from Fiddler’s agentic KPIs and OpenTelemetry’s AI guidance, we’ll start with:

Per-session metrics

Fleet-wide metrics

Logs

Traces

You can add more later, but this gets you a useful workflow health dashboard in a day.

Step 2 - Tap into session and worker output

Maxxwell runs your workers as real CLI sessions. For observability, treat them like any other process:

If you drive workers from a shell entrypoint, a simple pattern is (adapt the command to your own launcher):

# Example: start a worker session with logs redirected
/usr/local/bin/claude-code \
  > logs/feature-123-api-refactor.out 2> logs/feature-123-api-refactor.err &

On desktop, you can instead:

The invariant you want: every session has a stable ID, and you can find all its logs on disk.

Step 3 - Set up a minimal OpenTelemetry sidecar

We’ll build an agent-otel-sidecar that:

Install the basics (Node.js example):

npm init -y
npm install \
  @opentelemetry/api \
  @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/exporter-metrics-otlp-http \
  @opentelemetry/exporter-logs-otlp-http \
  chokidar

Create otel.ts:

// otel.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-http';
import { diag, DiagConsoleLogger, DiagLogLevel } from '@opentelemetry/api';

diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.INFO);

const traceExporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT,
});

const metricExporter = new OTLPMetricExporter({
  url: process.env.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT,
});

const logExporter = new OTLPLogExporter({
  url: process.env.OTEL_EXPORTER_OTLP_LOGS_ENDPOINT,
});

export const sdk = new NodeSDK({
  traceExporter,
  metricExporter,
  logExporter,
  serviceName: 'maxxwell-fleet',
});

export async function startOtel() {
  await sdk.start();
}

export async function shutdownOtel() {
  await sdk.shutdown();
}

Wire it into index.ts:

// index.ts
import { startOtel, shutdownOtel } from './otel';
import { startLogWatcher } from './log-watcher';

async function main() {
  await startOtel();
  await startLogWatcher();

  process.on('SIGINT', async () => {
    await shutdownOtel();
    process.exit(0);
  });
}

main().catch(async (err) => {
  console.error(err);
  await shutdownOtel();
  process.exit(1);
});

Point OTEL_EXPORTER_OTLP_*_ENDPOINT at your collector, run the sidecar, and you’re ready to emit telemetry.

Step 4 - Turn worker logs into structured events

Assume each worker log line is either:

If you can, make your workers log structured events like:

{"ts":"2026-09-05T12:34:56Z","level":"info","sessionId":"feature-123","event":"model_call","tool":"claude-code","tokens_in":512,"tokens_out":1632}

Then write a simple watcher:

// log-watcher.ts
import chokidar from 'chokidar';
import fs from 'fs';
import readline from 'readline';
import { context, trace, metrics, logs } from '@opentelemetry/api';

const tracer = trace.getTracer('maxxwell-fleet');
const meter = metrics.getMeter('maxxwell-fleet');
const logger = logs.getLogger('maxxwell-fleet');

const tokenCounter = meter.createCounter('agent_tokens_total', {
  description: 'Total tokens used by agents',
});

const sessionErrorCounter = meter.createCounter('agent_session_errors_total');

export async function startLogWatcher() {
  const watcher = chokidar.watch('logs/*.out', { persistent: true });

  watcher.on('add', (path) => tailFile(path));
}

function tailFile(path: string) {
  const stream = fs.createReadStream(path, { encoding: 'utf8' });
  const rl = readline.createInterface({ input: stream });

  rl.on('line', (line) => handleLogLine(path, line));
}

function handleLogLine(path: string, line: string) {
  let evt: any = { raw: line };
  try {
    evt = JSON.parse(line);
  } catch {
    // fall back to raw
  }

  const sessionId = evt.sessionId || inferSessionIdFromPath(path);

  if (evt.event === 'model_call') {
    tokenCounter.add(evt.tokens_in ?? 0, {
      direction: 'in',
      tool: evt.tool,
      sessionId,
    });
    tokenCounter.add(evt.tokens_out ?? 0, {
      direction: 'out',
      tool: evt.tool,
      sessionId,
    });
  }

  if (evt.level === 'error') {
    sessionErrorCounter.add(1, { sessionId, tool: evt.tool });
  }

  const span = tracer.startSpan('agent.log_event', undefined, context.active());
  span.setAttributes({
    'agent.session_id': sessionId,
    'agent.event': evt.event || 'log',
    'agent.tool': evt.tool || 'unknown',
  });
  span.end();

  logger.emit({
    body: line,
    attributes: {
      'agent.session_id': sessionId,
      'agent.tool': evt.tool || 'unknown',
      'agent.level': evt.level || 'info',
    },
  });
}

function inferSessionIdFromPath(path: string): string {
  return path.split('/').pop()?.replace('.out', '') ?? 'unknown';
}

Now every worker log line becomes:

Step 5 - Emit session state metrics from Maxxwell

Maxxwell already tracks per-session state (working, idle, waiting on you, etc.). You want that state in your metrics backend.

Assume you have some way to dump session state as JSON (a shell helper you write, or whatever your launcher exposes) that prints something like:

[
  {"id":"feature-123","tool":"claude-code","state":"working"},
  {"id":"bug-456","tool":"codex","state":"waiting_on_you"}
]

You can scrape this every N seconds from the sidecar and update gauges.

Add to log-watcher.ts or a new module:

import { exec } from 'child_process';

const sessionStateGauge = meter.createUpDownCounter('agent_session_state', {
  description: 'Session state encoded as integer per session',
});

const STATE_MAP: Record<string, number> = {
  working: 1,
  idle: 0,
  waiting_on_you: 2,
  blocked: 3,
  done: 4,
  dead: 5,
};

export function startSessionPoller() {
  setInterval(() => {
    exec('./dump-sessions.sh', (err, stdout) => {
      if (err) return;
      try {
        const sessions = JSON.parse(stdout);
        sessions.forEach((s: any) => {
          const v = STATE_MAP[s.state] ?? -1;
          sessionStateGauge.add(v, {
            sessionId: s.id,
            tool: s.tool,
            state: s.state,
          });
        });
      } catch {}
    });
  }, 5000);
}

Call startSessionPoller() from main().

Your metrics backend can now graph:

This is where Maxxwell pays off compared to DIY tmux: states are first-class, not inferred from log silence.

Step 6 - Add workflow-level traces from the orchestrator

The orchestrator seat in Maxxwell is itself a coding-agent session started from a brief. You talk to it instead of a dozen terminals; it fans work out to workers.

You want one root span per workflow (per orchestrator brief) and child spans per worker lane.

Pattern:

Example span wiring in handleLogLine:

const workflowId = evt.workflowId;

if (evt.event === 'workflow_started') {
  const span = tracer.startSpan('workflow', {
    attributes: {
      'workflow.id': workflowId,
      'workflow.summary': evt.summary,
    },
  });
  span.end();
}

if (evt.event === 'worker_assigned') {
  const ctx = trace.setSpan(context.active(), tracer.startSpan('workflow.assign_worker', {
    attributes: {
      'workflow.id': workflowId,
      'agent.session_id': sessionId,
      'agent.tool': evt.tool,
    },
  }));

  // later events for this session can use ctx to link to the workflow
}

In your tracing UI, you’ll see:

This is how you answer, “Why did this feature take three hours yesterday?” with evidence instead of Slack archaeology.

This view of workflow spans and session states over time makes it obvious which lanes are healthy, stalled, or waiting on human input.

Step 7 - Build a simple agent workflow health dashboard

Once telemetry flows, build a minimum viable dashboard. Most tools (Grafana, Honeycomb, Dynatrace) can do this in 15-30 minutes.

Start with:

  1. Session state panel
    • Timeseries of agent_session_state by state.
    • Alert when waiting_on_you sessions > N for more than M minutes.
  1. Token and cost panel
    • agent_tokens_total by tool and direction.
    • Basic cost estimate: tokens * price; Sonar expects AI-assisted code share to hit 65% by 2027, so this will only grow.
  1. Error and stall panel
    • agent_session_errors_total by tool.
    • A log table filtered to state = 'blocked' or 'dead'.
  1. Workflow traces panel
    • List recent workflow spans, with duration and outcome.
    • Drill-down trace view for any workflow to see agent handoffs.

This gives a tech lead one screen that answers:

From there, you can iterate into SLOs: “80% of workflows finish in < 10 minutes of agent time, 95% have no unreviewed errors,” etc.

Step 8 - Avoid the common observability traps

A few hard-earned lessons from agent observability docs (Google, Microsoft, OTel) and production teams:

Maxxwell’s own posture helps: it never acts without you (fleet controls draft rather than act), and it exposes uncertain state as “not heard from” instead of “probably working”. Your telemetry should do the same - better gaps than lies.

Maxxwell vs other orchestration tools, observability-wise

Other multi-agent tools are doing interesting things here:

Maxxwell’s angle is different:

That makes Maxxwell a good fit for agent-native development where the control plane is explicit but the autopilot is still you.

FAQ

How do I detect bottlenecks in agent sessions?

Use the combination of:

Alert on patterns like “session in waiting_on_you > 15 minutes” or “workflow duration > 4x median”.

Can I get team-wide visibility without sending code to a third party?

Yes. Maxxwell itself is local and doesn’t require any server.

For observability, you can:

That gives you a fleet health dashboard without shipping source.

How do I monitor sessions that are waiting on human input?

Expose a dedicated metric:

Populate it from Maxxwell’s session state poller whenever state === 'waiting_on_you'.

Then build a panel showing all such sessions, and an alert when their count or age crosses a threshold. This tells a lead where to focus attention.

What’s the difference between logs and traces in this setup?

In this Maxxwell setup:

You’ll usually:

Does Maxxwell automatically correct drifting agents?

No. Maxxwell conducts; it doesn’t autopilot.

It gives you:

You still own decisions like re-aiming a session that’s quietly building the wrong thing. Observability just makes those situations visible earlier.