You built an MCP server, but you can only guess how agents actually use it. bitmcp-eval runs your prompts through a real agent, records the traffic through a proxy, and compares the tool calls it sees with the ones you expected.
bitmcp-eval starts a proxy in front of your server and points the agent at it. Every message is forwarded byte-for-byte while a copy records each tool call.
YAML test cases naming the tools you expect the agent to call.
claude or codex, run headlessly, once per iteration.
Forwards MCP traffic and records every tools/call.
Recorded calls checked against expectations → live TUI + report.
The proxy works on the transport layer (MCP StreamableHTTP), so your server needs no changes at all. It can run locally or remote, and the proxy takes care of auth headers and OAuth.
Install the CLI from npm to evaluate your own MCP server, or clone the repo to try it against the bundled demo weather server first.
# install the CLI from npm npm install -g @bitmcp-eval/cli bitmcp-eval -c your-eval.yaml # or run it once, without installing npx @bitmcp-eval/cli -c your-eval.yaml
Want to see it work first? Clone the repo and run the two-minute demo against the bundled weather server:
# install & build the monorepo corepack enable yarn install yarn build # terminal 1: demo MCP server on :3210 yarn demo-server # terminal 2: run the evaluation yarn start -c examples/eval.yaml
✓ current weather lookup 2/2 passed ✓ compare two forecasts 2/2 passed ✓ list supported cities 2/2 passed 3 test cases · 6 iterations 6 passed · 0 failed · 100%
Then point mcp.url at your own server and write your test cases. Full setup in the
README.
What you get out of a run, and what the harness handles for you.
Records every tools/call with name, arguments, result, error state and duration. The server
under test stays unchanged.
Run one agent or both. The suite executes once per agent and the report shows pass rates for each.
Agents are non-deterministic, so each test case runs several times and the report shows how much the behavior varies.
When the server responds with a 401, the proxy runs the auth-code + PKCE flow, caches the
token and refreshes it during long runs. Servers with dynamic client registration need no config.
An LLM reviews each iteration and adds a second, semantic verdict. Works with any OpenAI-compatible endpoint, and the report highlights iterations where it disagrees with the tool-call result.
Watch runs live, then share a self-contained report with each iteration's tool calls and full conversation.
Point it at your server, write a handful of test cases, and see how agents actually use your tools.