Pairwise evaluation

/pairwise ih.va.lyoo'ay.shuhn/Comparing two candidate outputs for the same input and selecting the better one (often with an LLM judge). It's especially useful when absolute scoring is hard to calibrate.(noun)

“We ran pairwise evals to pick the best prompt variant.”

Related Evaluation terms

• Absolute scoring
• Agent
• Alignment
• Annotation schema
• Baseline
• Baseline experiment
• Benchmark
• Calibration
• CI/CD integration
• Coherence
• Confidence interval
• Eval
• Eval harness
• Eval leakage
• Experiment
• Factuality
• Failure mode
• Faithfulness
• Feedback signal
• Groundedness
• Hallucination
• Inter-annotator agreement (IAA)
• LLM-as-a-judge
• Loop
• Model comparison
• Multimodal
• Non-determinism
• Offline evaluation
• Pass@k
• Playground
• Quality gate
• RAG (retrieval-augmented generation)
• RAG evaluation
• Reference-based scoring
• Reference-free scoring
• Regression testing
• Release criteria
• Remote evaluation
• Rubric
• Safety
• Score distribution
• Scorer
• Semantic failure
• Signal-to-noise ratio
• Task (eval task)
• Toxicity score