bitmcp · eval

Measuring how LLM agents actually use your MCP server

Bitmovin Observability · Klagenfurt Claude vs Codex Price: Free (MIT)

The Match

Which agent actually uses your MCP server?

You shipped an MCP server. But do agents pick the right tool for the question — every time, or 7 runs out of 10? bitmcp-eval settles it: same prompts, same server, two agents, scored.

By the Observability Team · Filed under Model-Context-Protocol · Reading time 2 min

A recording proxy sits in front of your server and the agent is pointed at it. Every JSON-RPC message is forwarded byte-for-byte while a copy notes each tools/call — the tool, its arguments, the result, whether it errored, and how long it took. Nothing on your server changes.

Each test case is run again and again, because agents don't behave the same way twice. What you get back isn't a single lucky green check — it's a pass rate with the spread laid bare, per agent, so "it seemed to work" becomes a number you can stand behind.

The verdict need not be mechanical. An optional LLM judge reads the request, the recorded calls and the full conversation, then files its own semantic opinion — pass, fail or uncertain — and it never overrules the tool-based count. The disagreements are the interesting rows.

Two agents ship today — claude, confined to the server under test, and codex, run in a neutral sandbox with its escapes flagged. Run one, or run both and read the head-to-head below.

Final — Standings6 iterations each

Test case	Claude	Codex
current weather lookup	6/6	6/6
compare two forecasts	6/6	4/6
list supported cities	5/6	4/6
Pass rate	94%	78%

Winner

claude

Spread

+16

Iterations

Illustrative run · bundled demo weather server · your mileage varies by prompt

How the match is scored

A recording proxy, not a rewrite

Your prompts

YAML test cases naming the tools you expect the agent to call.

The agent

claude or codex, run headlessly, once per iteration.

The proxy

Forwards MCP traffic byte-for-byte and records every tools/call.

The verdict

Recorded calls validated vs. expectations → live table + HTML report.

Operates on the transport layer over MCP StreamableHTTP — zero changes to the server under test, local or remote, headers or OAuth handled for you.

Get in the game

Two minutes against the demo server

Terminal — install & run

# build the monorepo
corepack enable
yarn install && yarn build

# term 1 — demo server on :3210
yarn demo-server

# term 2 — run the eval
yarn start -c examples/eval.yaml

testcases/current-weather.yaml

name: current weather lookup
prompt: 'What is the weather in Vienna?'
expectedTools:
  - get_current_weather

# every listed tool must be called
# successfully, once per iteration —
# point mcp.url at your own server
# and write your own cases next.

Scouting report

What's on the roster

Transport-level proxy

Records every tools/call with arguments, result, error state and duration. No changes to the server under test.

Claude & codex

Run one or both. The suite executes once per agent with per-agent pass rates — a true head-to-head.

Iterations & spread

Each case runs N times because agents are non-deterministic. You measure the spread, not one run.

OAuth, handled

A 401 is auto-detected; the proxy runs auth-code + PKCE and refreshes tokens. Dynamic client registration needs no config.

Optional LLM judge

A semantic verdict per iteration from any OpenAI-compatible endpoint. It flags disagreements; it never overrules the count.

Live TUI & HTML report

Watch runs live, then share a self-contained report with each iteration's tool calls and full conversation.

The Final Word

Put your own server on the board.

★ Star on GitHub Read the docs git clone →

MCP EVAL — an open-source broadsheet published by Bitmovin, Inc. · MIT. GitHub · License · bitmovin.com