Debugging & Observability

Your agent ran a cron job at 3 AM. Something went wrong. The output was weird. How do you figure out what happened? This chapter gives you the debugging and observability toolkit.

The 3 Pillars of Agent Observability

Common Agent Failures & How to Debug Them

🔌 Observability by Platform

# View session history
openclaw sessions list --active 60

# Check cron job runs
openclaw cron runs --job "Trading Plan" --limit 5

# View session logs
openclaw sessions history --session <key> --include-tools

# Check gateway status
openclaw status

# Monitor in real-time
# Add a daily self-diagnostic cron:
openclaw cron add --name "Daily Health Check" \
  --cron "0 22 * * *" --session isolated \
  --message "Run a self-diagnostic:
  1. Check all cron jobs ran today (list runs)
  2. Check for any errors in recent sessions
  3. Report: tasks completed, tasks failed, total cost
  Post summary to Discord."
  --model "haiku" --announce

import anthropic
import json
from datetime import datetime

client = anthropic.Anthropic()

def logged_completion(prompt, model="claude-sonnet-4-20250514"):
    """Wrapper that logs every API call"""
    start = datetime.now()
    
    response = client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    )
    
    log_entry = {
        "timestamp": start.isoformat(),
        "model": model,
        "input_tokens": response.usage.input_tokens,
        "output_tokens": response.usage.output_tokens,
        "latency_ms": (datetime.now() - start).total_seconds() * 1000,
        "prompt_preview": prompt[:100],
        "stop_reason": response.stop_reason,
    }
    
    with open("logs/agent.jsonl", "a") as f:
        f.write(json.dumps(log_entry) + "\n")
    
    return response

The "Morning After" Checklist

Quick daily review checklist:

1. Did all scheduled cron jobs run? ✅/❌
2. Any error messages in logs? ✅/❌
3. Token usage within budget? ✅/❌
4. Output quality acceptable? ✅/❌
5. Any unexpected behaviors? ✅/❌

If all ✅ → Great, move on.
If any ❌ → Debug using the failure patterns above.

The "Time Travel" Debug Technique

The most powerful debugging technique for agents: reproduce the exact conditions. When something goes wrong at 3 AM, you need to see what the agent saw.

# Step 1: Find the failing run
openclaw cron runs --job "Trading Plan" --limit 1

# Step 2: Check what files existed at that time
git log --oneline --until="2026-02-22T06:00:00" -5

# Step 3: Check what the agent's memory looked like
git show HEAD~2:memory/2026-02-21.md

# Step 4: Re-run with the same context
openclaw cron run --job "Trading Plan" --force
# This re-runs the exact same prompt in a fresh session

# Step 5: Compare outputs
# Old output (from logs) vs new output (from re-run)
# If they match → the input was the problem
# If they differ → the model was non-deterministic (use temperature 0)

Building Your Dashboard

After a month of running agents, you'll want a dashboard. Here's the minimal viable monitoring setup:

openclaw cron add \
  --name "Agent Health Dashboard" \
  --cron "0 22 * * *" \
  --session isolated \
  --message "End-of-day health check:

1. List all cron jobs and their last run status
2. Count total sessions today
3. Estimate today's API cost
4. Check for any errors in logs
5. Compare today's output quality to baseline

Format:
📊 **Agent Health — [Date]**
- Cron jobs: [X/Y ran successfully]
- Sessions: [N total]
- Est. cost: $[amount]
- Errors: [count] [brief description if any]
- Quality: [✅ Good / ⚠️ Check needed / ❌ Issues found]

If all green, keep it to 5 lines max." \
  --model "haiku" --announce \
  --channel discord --to "channel:YOUR_ID"

The Debugging Flowchart

When your agent does something weird, follow this exact sequence:

Observable Agent Architecture

The best debugging setup is one where you can see everything without adding debug code. Build observability in from day one:

• 📝 Input logging — save every prompt sent to the model (with timestamps)
• 📝 Output logging — save every response received
• 📝 Tool call logging — which tools were called, with what args, and what they returned
• 📝 Decision logging — why the agent chose action A over action B
• 📝 Cost logging — tokens used per request (catches runaway costs early)

The "Replay" Technique

The most powerful debugging technique: replay the exact same input and see if you get the same output. If you do, the bug is deterministic (probably a prompt issue). If you don't, the bug is stochastic (probably a temperature or context window issue).

This is why input logging matters so much. Without the exact input, you can't replay. And without replay, you're guessing.