The problem
Replacing a production model with a cheaper one is usually decided by a demo, a public benchmark, or the price sheet. None of those replay the traffic the system actually receives, and none of them tell you how much of the resulting score belongs to the judge rather than the candidate.
What I built
A multi-provider, multi-judge evaluation harness spanning OpenAI, Anthropic, Gemini and DeepSeek, with 7 evaluation arms that replay real production conversations against the incumbent. The design is deliberately adversarial to itself: judges cross-judge each other reciprocally, and a self-preference probe checks whether a model favours its own outputs.
How it works
Each arm pins a candidate, a judge and a prompt contract, then replays the same conversation set through all of them. Because arms share inputs, the differences between arms are attributable — a scoring change can be traced to the judge, the candidate or the parse layer rather than to a mixture of all three. Verdicts are parsed to a structured schema, and parse failures are counted rather than silently discarded.
Evidence
The harness cleared a production migration off Claude 3.5 Sonnet to a candidate ~12× cheaper per token — a decision made on replayed evaluation results, not on price.
Its more useful output was a warning about the method itself. Swapping the judge moved the pass rate 37.4 points on byte-identical outputs (82.7% → 45.3%), against a sampling error of ±9–11, and 37.6% of judge verdicts failed to parse. Judge choice and output reliability, not the candidate model, dominated the result — which means any single-judge evaluation of this kind is measuring at least as much of its judge as of its subject.
What I owned
I built the harness, the arm design, the cross-judging scheme and the self-preference probe, and wrote the analysis that produced the judge-variance finding.
Stack
Python · OpenAI · Anthropic · Gemini · DeepSeek · LiteLLM gateway · Langfuse tracing