Dataset format for system and user prompts - Braintrust

Documentation Index

Fetch the complete documentation index at: /docs/llms.txt

Use this file to discover all available pages before exploring further.

Applies to:

Summary

Goal: Structure dataset rows with varying system and user prompts for multi-model comparison in Playground.Features: Datasets with chat message arrays, Playground model comparison, role-based message formatting.

Configuration Steps

Step 1: Structure dataset rows with chat message arrays

Each dataset row should contain an array with system and user messages using role-based formatting.

{
  "input": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"}
  ]
}

Step 2: Add rows with varying system prompts

Create multiple rows where system prompts differ while maintaining the same structure.

{
  "input": [
    {"role": "system", "content": "You are a creative writer."},
    {"role": "user", "content": "Write a short story."}
  ]
}

Step 3: Enable “Append dataset messages” in Playground

In Playground Advanced settings, enable “Append dataset messages” and set the path to your messages array field (e.g., input).

Step 4: Configure model comparison

Add multiple model configurations in Playground to test each dataset row across all models with their respective system prompts.

Key Notes