LLM EVAL HARNESS

A reproducible evaluation harness that ships every model or prompt change with a confidence interval instead of a hunch — so 'it feels better' becomes 'it's better, p < 0.05'.

The problem

LLM evals are noisy. Run the same suite twice and the numbers move. Without statistical rigour, teams chase phantom regressions and ship real ones, because nobody can tell signal from variance.

Approach

• Significance gating: a change must beat the baseline by more than the noise band to ship.
• Bayesian estimation (Stan) of the metric, reporting intervals not point scores.
• Ray for parallel eval runs, so rigour doesn't cost you wall-clock time.

Results

94% of real regressions caught before release, eval runtime cut 5.4×, and a flake rate held to 0.3%.