# Non-determinism

The property of agents where the same input can produce different outputs across identical requests. (noun)

## Why it matters

Non-determinism is the fundamental reason agents need evals rather than just testing. When the same input can produce different outputs on every run, a single passing test tells you almost nothing. You need to measure quality across distributions of outputs, track score trends over time, and build datasets large enough to capture the variance. This also changes how you think about regressions. A prompt change might improve average quality while making edge cases worse, but this won't be identified without running your eval suite across enough examples to see the distribution shift.

“Because of non-determinism, we evaluate changes on distributions of scores, not one-off outputs.”

## Customer example

Notion embraced non-determinism by moving ~70 engineers beyond "vibe checks" to systematic evals from feedback and traces, so the team can ship quickly even as agent behaviors and outcomes vary across runs.

## Related Evaluation terms

- [Absolute scoring](/content/encyclopedia/absolute-scoring/index.html)
- [Agent](/content/encyclopedia/agent/index.html)
- [Alignment](/content/encyclopedia/alignment/index.html)
- [Annotation schema](/content/encyclopedia/annotation-schema/index.html)
- [Baseline](/content/encyclopedia/baseline/index.html)
- [Baseline experiment](/content/encyclopedia/baseline-experiment/index.html)
- [Benchmark](/content/encyclopedia/benchmark/index.html)
- [Calibration](/content/encyclopedia/calibration/index.html)
- [CI/CD integration](/content/encyclopedia/ci-cd-integration/index.html)
- [Coherence](/content/encyclopedia/coherence/index.html)
- [Confidence interval](/content/encyclopedia/confidence-interval/index.html)
- [Eval](/content/encyclopedia/eval/index.html)
- [Eval harness](/content/encyclopedia/eval-harness/index.html)
- [Eval leakage](/content/encyclopedia/eval-leakage/index.html)
- [Experiment](/content/encyclopedia/experiment/index.html)
- [Factuality](/content/encyclopedia/factuality/index.html)
- [Failure mode](/content/encyclopedia/failure-mode/index.html)
- [Faithfulness](/content/encyclopedia/faithfulness/index.html)
- [Feedback signal](/content/encyclopedia/feedback-signal/index.html)
- [Groundedness](/content/encyclopedia/groundedness/index.html)
- [Hallucination](/content/encyclopedia/hallucination/index.html)
- [Inter-annotator agreement (IAA)](/content/encyclopedia/inter-annotator-agreement-iaa/index.html)
- [LLM-as-a-judge](/content/encyclopedia/llm-as-a-judge/index.html)
- [Loop](/content/encyclopedia/loop/index.html)
- [Model comparison](/content/encyclopedia/model-comparison/index.html)
- [Multimodal](/content/encyclopedia/multimodal/index.html)
- [Offline evaluation](/content/encyclopedia/offline-evaluation/index.html)
- [Pairwise evaluation](/content/encyclopedia/pairwise-evaluation/index.html)
- [Pass@k](/content/encyclopedia/pass-at-k/index.html)
- [Playground](/content/encyclopedia/playground/index.html)
- [Quality gate](/content/encyclopedia/quality-gate/index.html)
- [RAG (retrieval-augmented generation)](/content/encyclopedia/rag-retrieval-augmented-generation/index.html)
- [RAG evaluation](/content/encyclopedia/rag-evaluation/index.html)
- [Reference-based scoring](/content/encyclopedia/reference-based-scoring/index.html)
- [Reference-free scoring](/content/encyclopedia/reference-free-scoring/index.html)
- [Regression testing](/content/encyclopedia/regression-testing/index.html)
- [Release criteria](/content/encyclopedia/release-criteria/index.html)
- [Remote evaluation](/content/encyclopedia/remote-evaluation/index.html)
- [Rubric](/content/encyclopedia/rubric/index.html)
- [Safety](/content/encyclopedia/safety/index.html)
- [Score distribution](/content/encyclopedia/score-distribution/index.html)
- [Scorer](/content/encyclopedia/scorer/index.html)
- [Semantic failure](/content/encyclopedia/semantic-failure/index.html)
- [Signal-to-noise ratio](/content/encyclopedia/signal-to-noise-ratio/index.html)
- [Task (eval task)](/content/encyclopedia/task-eval-task/index.html)
- [Toxicity score](/content/encyclopedia/toxicity-score/index.html)