Introducing unbox-ai: See Where Your Agent's Tokens Actually Go

Every time our QA agent misbehaved, the debugging session started the same way: export the trace, open a 10 MB JSON file, and scroll. Every generation in an agent run resends the full conversation, so a 16-generation trace holds hundreds of duplicated messages, and the answer to "why was this run slow" or "why did this run cost that much" is buried under sixteen copies of the same system prompt.

We got tired of it and built unbox-ai. It started as an internal tool for debugging our agent traces. It's now open source, MIT licensed, and on npm:

That one command starts a local server and opens a viewer with a token treemap, a latency waterfall, and a deduplicated conversation view. No account, no upload, nothing leaves your machine.

The problem: agents pay for the same tokens over and over

An agent loop is a sequence of model calls where each call includes everything that came before: the system prompt, every tool definition, and the whole conversation so far. The model's actual output is a sliver on top.

Here is a real trace from our QA agent, summarized by the CLI:

Read those totals again. The run sent 310,300 input tokens to produce 2,500 output tokens. That is a 124:1 ratio, and it is normal. In a typical run around 90% of input tokens are the system prompt and tool definitions, paid again on every request. 98% of the model time in this trace was spent waiting for the model to read the prompt, not to write the answer.

None of this is visible in the raw JSON. It's the first thing you see in unbox-ai.

What the viewer shows

The viewer has three main pieces:

• Context treemap. A bundle-analyzer-style map of input tokens, attributed to the system prompt, each tool definition, and each conversation message. Toggle between per-generation and cumulative (size times how often it was resent), and between tokens and cost. This is where you find the tool description that quietly costs you 4k tokens on every single call.
• Timeline waterfall. Latency per generation with TTFT marks, token counts, and cost. Slow runs stop being a mystery; you see exactly which generation stalled and whether the time went to prompt processing or output.
• Generation detail. Only the messages that are new since the previous generation, with tool calls paired to their results. The sixteen copies of the system prompt collapse into one. Raw JSON is one click away when you need it.

unbox-ai view a.trace.json b.trace.json opens several files (any mix of formats) as one run list, with a tab per file. Dropping another .json anywhere in the window opens it too.

Token attribution is estimated (character-proportional, scaled to the reported per-generation totals) and labeled as such in the UI.

Live view for AI SDK apps

If you build on the Vercel AI SDK, unbox-ai devtools is a drop-in replacement for the @ai-sdk/devtools viewer. Instrument your app exactly as the AI SDK documents it:

Then instead of npx @ai-sdk/devtools, run:

Every generateText and streamText call streams into the viewer live. Each run gets its own entry in the sidebar, nested agent runs (tools that call the AI SDK again) indent under their parent, and the viewer follows the newest run until you pin one. You watch the treemap and waterfall grow while your agent runs, which is a very different experience from exporting a trace after the fact.

The same binary works for your coding agent

Here is the part we use daily. The viewer is for humans; the CLI is for agents. Every command is read-only, caps its output, and prints the exact follow-up command that returns anything it truncated. When stdout is not a TTY, unbox-ai trace.json prints the summary instead of starting a server, so an agent never spawns one by accident. That makes it safe to allowlist wholesale.

So when a run fails, we don't paste JSON into Claude Code. We say "figure out why this run failed" and the agent runs summary, drills into the suspicious generation with event, greps the messages, and follows the printed get pointers to fetch full values. Bounded output means it never blows its own context window doing so; the irony of a trace debugging tool flooding an agent's context was not lost on us.

There is a ready-made agent skill with the full workflow:

Or skip setup entirely and drop two lines in your AGENTS.md telling the agent to use npx unbox-ai for trace files. The viewer meets the CLI halfway too: a "copy for agent" button in the header copies the exact summary command for the run on screen, so you can hand your agent whatever you're currently looking at.

Compare is for prompt and model A/B

unbox-ai compare a.json b.json answers "what changed and what did it buy". It prints token, cost, time, and cache deltas between two runs, then a line diff of the system prompt and the added, removed, and changed tool definitions.

We lean on this constantly. Trim a tool description, run the same test, compare. Swap the model, compare. Two runs from one devtools database work too: unbox-ai compare db.json --run 0 --run 1.

Trace formats

unbox-ai auto-detects three formats today:

• Gateway exports: JSON with events[] of generations carrying metrics, tools, and cumulative message snapshots. Conversation resets and multi-agent interleaving show up as segments.
• opencode session exports: adapted automatically, with real cache read/write tokens and per-tool execution times.
• AI SDK devtools databases (.devtools/generations.json): adapted automatically and served live by unbox-ai devtools.

There is no universal AI-trace standard yet; the closest candidates (OpenTelemetry GenAI conventions, OpenInference, OpenLLMetry) are all span-based. Adapters are deliberately small: implement detect and adapt in one file and register it. If your stack exports traces in another shape, this is the contribution we want most.

Try it

If you run agents in production, you have trace files sitting around right now. Point the viewer at one:

Our bet is that the first treemap surprises you. It surprised us, and we wrote the agent.

The code is at github.com/tester-army/unbox-ai. Issues and adapter PRs welcome.