Compare Kimi K3 and DeepSeek V4 - Evals - Braintrust

Open models keep improving, and it should be easy to test candidate models against your own prompts, datasets, and production traces. Kimi K3 and DeepSeek V4 Flash are now available as built-in models, joining GLM-5.2. You can run all three in playgrounds, prompts, and scorers without setting up another inference provider or managing separate API keys, then call the same models through the gateway when you deploy.

Built-in models are available on every plan. On Starter, you can try all three if your organization signed up with a work email or has a card on file. Usage draws from your monthly model credits, then continues at pay-as-you-go rates.

Evaluate before you switch

Compare Kimi K3 and DeepSeek V4 Flash against your current model on the quality, response time, and token usage that matter for your agents. Run an eval on representative data, inspect the results side by side, and trace the model in production in one place.

As an example, we ran all three models side by side on 327 dialogs from the hard scaffolding task in MathTutorBench. An LLM judge scored every response and token usage and response time was recorded from the same run.

Each dialog contains a math problem and a conversation between a student and tutor, and the model continues the dialog with a teacher response that advances the student's reasoning without giving away the answer. The judge compares that response against the human teacher response from the dataset, producing a 0–1 teaching quality score where 0.50 means the judge rated the two equally. Every model ran at high and none reasoning effort, three trials per dialog, with response caching disabled, so the same dataset and scorer produce one comparable set of results for all three.

One dialog, three responses

One item gives the model this problem:

Julia was preparing for a dinner party at her house, where she intended to serve stew. She noticed that she was out of plastic spoons, so she bought a new package of spoons. Later, her husband also bought a package of 5 new spoons and gave them to Julia. While Julia was making the stew, she used three of the spoons to sample her stew. Later, when she went to set the table, she had a total of 12 spoons. How many spoons were in the package that Julia bought?

The answer is 10. The student has already guessed 7, then 9, and the conversation stops on this turn:

She used 3 of the 5 spoons her husband gave her, so there are 2 spoons left.

Each model writes the next teacher turn from there, in at most two sentences. The judge scores that turn against what the human teacher actually said, which was "She used 3 from 5 snd at the end at 12 left , can you work this out".

DeepSeek V4 Flash 0731\nTeaching score 0.64\n “Exactly! She used 3 of the 5 from her husband's package, so those 3 are gone. Now let's put the whole story into one equation. If Julia's package has x spoons, then before sampling she has x + 5 spoons total. After using 3 spoons, she has 12 left. Can you write that as an equation and solve for x?”\n Kimi K3\nTeaching score 0.55\n

“If she has 12 spoons now and 2 left from her husband's package, how many spoons were in Julia's package?”\n GLM-5.2\nTeaching score 0.84\n “Good observation that 5 - 3 = 2! But let's step back for a moment. The problem says Julia used 3 spoons to sample her stew, but it doesn't specify which spoons she used. Let's think about it this way. Julia starts with the spoons from her package, let's call that x, then her husband adds 5 more, and then 3 get used. If she ends up with 12 spoons, can you write that as an equation?”

Scores on this item come from an exploratory 30-dialog pilot at high reasoning effort rather than the aggregate below. GLM-5.2 corrects the assumption before adding structure, DeepSeek V4 Flash reaches the right equation after affirming part of the mistaken reasoning, and Kimi K3 gives the shortest response but leaves the misconception in place.

Results

Teaching quality and token counts are at high reasoning effort. Time to first visible token is with reasoning disabled.

| Model | Teaching quality | Median completion tokens | First visible token |\n| --- | --- | --- | --- |\n| GLM-5.2 | 0.680 | 156 | 0.94s |\n| DeepSeek V4 Flash | 0.653 | 365 | 0.57s |\n| Kimi K3 | 0.589 | 48 | 1.18s |\n No model leads on all three measurements.

Teaching quality on MathTutorBench

Mean reward-model margin over the human teacher response, reasoning effort high on every model

327 dialogs per model, 3 trials each. Shaded bands are 95% confidence intervals

GLM-5.2

DeepSeek V4 Flash

Kimi K3

| --- | --- | --- | --- |\n| 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | Teaching quality |\n| 0.680 | 0.653 | 0.589 |\n Completion tokens include hidden reasoning tokens, 293 of DeepSeek V4 Flash's 365 and 101 of GLM-5.2's 156, while Kimi K3 reported no reasoning tokens on most requests even at high.

Output tokens per response at high reasoning effort

Median completion tokens, split into reasoning and visible response

| Model | Completion tokens | Reasoning | Visible response |\n| --- | --- | --- | --- |\n| DeepSeek V4 Flash | 365 | - | - |\n| GLM-5.2 | 156 | - | - |\n| Kimi K3 | 48 | - | - |\n Reasoning effort matters most for Kimi K3, whose score fell 0.073 with reasoning disabled, against 0.023 for GLM-5.2 and 0.014 for DeepSeek V4 Flash. GLM-5.2 with reasoning disabled scored 0.657, above DeepSeek V4 Flash with reasoning on at 0.653. Turning reasoning on also delays the first visible token, moving DeepSeek V4 Flash from 0.57 seconds to 2.15 and GLM-5.2 from 0.94 to 2.66.

Time to first visible token with reasoning disabled

Median seconds before the user sees any output, reasoning effort set to none on every model

| Model | Seconds to first visible token |\n| --- | --- |\n| DeepSeek V4 Flash | 0.57s |\n| GLM-5.2 | 0.94s |\n| Kimi K3 | 1.18s |\n Timings come from single-request probes against one gateway on one day, so treat them as a floor for what someone waits rather than a forecast under production load. If your tasks look like parts of this set, start with GLM-5.2 for quality, DeepSeek V4 Flash for speed, or Kimi K3 for token cost, then confirm on your own data.

How to try Kimi K3 and DeepSeek V4

Both models are also available from code. Point the OpenAI or Anthropic SDK at the Braintrust gateway, set the model to kimi-k3 or deepseek-v4-flash-0731, and start sending requests.

const client = new OpenAI({
  baseURL: "https://gateway.braintrust.dev",
  apiKey: process.env.BRAINTRUST_API_KEY,
});

const response = await client.responses.create({
  model: "kimi-k3",
  input: [{ role: "user", content: "Say hello!" }],
});

Kimi K3 and DeepSeek V4 Flash are available now. Try them in Braintrust.