Run v1 of a support agent: 6 cases, 5 trials each.
evalloop circles every failing trial and names the check.
Fix the agent. Run the exact same suite again.
Every trial passes. Now it is safe to merge.
evalloop runs your agent many times against real cases, grades every tool call and every answer, and blocks the merge when behaviour gets worse.
Run v1 of a support agent: 6 cases, 5 trials each.
evalloop circles every failing trial and names the check.
Fix the agent. Run the exact same suite again.
Every trial passes. Now it is safe to merge.
The same support agent, on Groq (gpt-oss-120b). 141 trials, 0 API errors, run on 17 Nov 2026. A single run would have looked fine.
Each square is one run, in the order it finished. One run of one model: results move between runs, which is exactly why you measure it this way. Full table. Run it yourself with python examples/benchmark.py.
Keep your agent as it is. Pick how you built it, copy three snippets, and you have tests that run in CI.
One package, no runtime dependencies.
$ pip install evalloop-agents
$ pip install "evalloop-agents[agents]"
$ pip install "evalloop-agents[langgraph]"
Each case is an input plus the checks it must pass.
# evals.py from evalloop import Case, agent, tool from evalloop import tool_called, grounded @tool() def lookup_order(order_id): ... @agent def support(text): ... # your code, returns the answer cases = [ Case("track", "Where is order #1043?", [tool_called("lookup_order"), grounded()]), ]
# evals.py from agents import Agent from evalloop import Case, tool_called, grounded from evalloop.frameworks import openai_agents_agent bot = Agent(name="support", tools=[lookup_order], instructions="...") support = openai_agents_agent(bot) cases = [ Case("track", "Where is order #1043?", [tool_called("lookup_order"), grounded()]), ]
# evals.py from evalloop import Case, tool_called, grounded from evalloop.frameworks import langgraph_agent from my_app import graph # compiled StateGraph support = langgraph_agent(graph) cases = [ Case("track", "Where is order #1043?", [tool_called("lookup_order"), grounded()]), ]
Five trials per case, one HTML report, exit code 1 on regressions.
$ evalloop run evals:cases \
--agent evals:support \
--trials 5 --html report.html \
--store runs --baseline last \
--fail-on-regression
This is the report evalloop writes, with numbers from a real run of the demo agent. Open any case to see the exact check that failed and the trace behind it.
The demo agent is simulated so this page needs no API key. The numbers come from an actual evalloop run: 6 cases x 5 trials per version.
A case that passes 3 of 5 times usually breaks for one reason. evalloop explain compares the runs that passed with the runs that failed and shows where they split, with counts.
$ evalloop explain run.json refund_over_limit_escalates 3/5 passed (flaky) symptom check tool_called(escalate_to_human) failed likely cause skipped escalate_to_human [2 of 2 failing runs, 0 of 3 passing runs] likely cause called issue_refund, which passing runs do not call [2 of 2 failing runs, 0 of 3 passing runs] first split trial 3 vs passing trial 1, tool call 2: expected escalate_to_human, got issue_refund
lookup_order(order_id="1044")lookup_order(order_id="1044")escalate_to_human(order_id="1044", reason="over refund limit")issue_refund(order_id="1044", amount=899.0)"...above my refund limit, so I escalated it to a human agent.""Done. I refunded $899.00 for order 1044."Demo agent. The HTML report shows this diff for every failing case.
It looks for skipped or extra tools, changed arguments, a different order, tool errors, loops and slow runs. Comparing models instead? evalloop compare a.json b.json puts reliability, latency and cost side by side.
Each one is a check you attach to a case. Checks read the full trace, so you test what the agent did, not only what it said. Hover a card.
Every case runs k times. A case counts as reliable only if all k pass, and flaky cases are marked.
trials=5, pass^kAny amount, date or id not seen in the input or a tool result fails.
grounded()Test whole conversations, turn by turn.
Conversation([...])Eight built-in attacks hidden in tool results. Fails if a planted secret reaches the user.
Case(poison=...), no_secret_leak()Typos, caps and filler words. Ids stay intact.
variants(case)Which tools ran, with which arguments, in what order.
tool_order(), tool_args()Same tool, same arguments, again and again.
no_loop(), max_steps()A dollar budget per run.
cost_under(0.01)p50 and p95 latency per run.
latency_under(2000)Emails, phones, cards and keys in the answer.
no_pii()For tone and policy, with any model as the judge.
judge(criteria, ask)Any function of the trace is a check.
custom()Press the button and evalloop runs six cases against a built-in mock model in your browser. Want a real model? Open "Use my own model" and paste a key.
Every model call and tool call is saved with its arguments, output, time and tokens. Use the decorators, a framework adapter, or import message logs you already have.
from evalloop import from_openai_messages trace = from_openai_messages(logged_messages) trace.tool_calls # [lookup_order(order_id="1043"), ...]
Each case runs several trials. You get the pass rate with a 95% confidence interval, pass^k reliability, and flaky cases marked.
refund_small_order PASS PASS PASS FAIL PASS 80% FAIL grounded(numbers): 59.0 not in any source pass rate 33% (95% CI 19% to 51%) reliability 0%
Runs are saved. A two-proportion test tells a real regression apart from bad luck, so a single unlucky trial does not block you.
refund_small_order 100% to 40% regressed p = 0.01 unknown_order 80% to 60% within noise
In GitHub Actions, evalloop exits with code 1, writes JUnit and a Markdown summary, and uploads the HTML report.
# .github/workflows/agent-evals.yml (the key step)
- run: |
evalloop run evals:cases --agent evals:support \
--trials 5 --workers 4 \
--store runs --baseline last --alpha 0.2 \
--fail-on-regression --fail-under 0.9 \
--html report.html --junit junit.xml \
--markdown "$GITHUB_STEP_SUMMARY"
Agents do not do the same thing twice. A test that passes once tells you very little, and most of the bugs I care about only show up on the third or fourth try. So I built the tool I wanted: run every case many times, check what the agent actually did, and stop the merge when it gets worse.
It is open source and still young. If you ship agents, I would really like to hear what breaks for you.