What Is Braintrust? How the AI Eval Platform Works in 2026

---

Braintrust is the platform a team reaches for when it needs to know whether an LLM agent got better or worse after a change. It captures what the agent did, scores it, and keeps every run for comparison. What it does not hold is the business meaning behind the data an agent read. Atlan is the Context Layer for AI, and this guide shows where each one fits next to the other AI agent evaluation platforms.

What it is	A commercial platform for evaluating and observing LLM applications and agents, built by Braintrust Data, Inc.
Four jobs	Tracing, scoring, experimenting, and discovering recurring failures
Scorer types	Prebuilt autoevals, custom code, LLM-as-a-judge, and human review
How it ships	SDKs in six languages, OpenTelemetry ingestion, a GitHub Action, and the bt eval CLI
Deployment	SaaS on Starter, Pro, and Enterprise; bring-your-own-cloud and self-hosted on Enterprise
What it leaves to you	The record of whether the context an agent read was certified, current, and permitted

---

What is Braintrust?

Braintrust is a commercial platform for evaluating and observing LLM applications and AI agents, built by Braintrust Data, Inc. and sold at braintrust.dev. Its homepage calls it the active observability platform for agents.

A naming note: this Braintrust has no connection to Braintrust at usebraintrust.com, which describes itself as an AI-powered talent network and carries a BTRST token section in its site footer. Searches for “Braintrust” return both, so check the domain.

The product covers four jobs:

• Tracing captures prompts, model responses, tool calls, retrieval steps, latency, token usage, and cost across development and production, using auto-instrumentation for supported providers.
• Scoring grades output with custom code, LLM-as-a-judge prompts, or human reviewers.
• Experimenting saves each eval run as an experiment, a permanent record that a team compares against a baseline.
• Discovering uses Loop, Braintrust’s AI agent, along with Topics and Patterns, to surface recurring problems across production traces.

---

What is Braintrust used for in AI development?

Braintrust covers the lifecycle from the first prompt draft to an agent serving production traffic. The table maps each stage to the job Braintrust does there.

Stage	What teams use Braintrust for	Features involved
Prototyping	Iterating on prompts and swapping models before committing to a full eval	Playgrounds, prompt versions, Braintrust Gateway
Pre-release testing	Checking what changed in scores when a prompt, model, or tool changes	Datasets, scorers, experiment comparison, CI
Production	Monitoring live quality, cost, and latency	Online scoring, alerts
Continuous improvement	Turning real failures into new tests	Topics, Patterns, human review, datasets populated from production

Common applications

Braintrust’s cookbook shows the range of systems teams evaluate with it:

• RAG pipelines: a Ragas recipe scores retrieval with ContextRecall and answers with AnswerCorrectness. The autoevals library also bundles context precision, context recall, and faithfulness scorers.
• Tool-using agents: Zapier’s recipe uses assertions to check whether a chatbot calls the right tool. The recipe reports a pass rate that moved from 50% to 75% after a prompt change.
• Text-to-SQL: the Text-to-SQL recipe runs the generated and expected queries against a DuckDB database and compares the results.
• Voice agents: the voice agent recipe scores a language classifier on synthetic calls.
• Safety checks: the prompt injection recipe moves a detector from 90% to 96.67% accuracy on 30 labeled examples.

Text-to-SQL is also where eval results depend most on the definitions an agent reads. In Atlan’s AI Labs benchmark, adding governed context improved text-to-SQL accuracy by 38%. Pairing SQL evals with a context layer helps a team see whether a failing query came from a wrong join or a wrong definition, and the guide to text-to-SQL for the enterprise walks through that split.

---

How does Braintrust work?

Braintrust describes a five-stage loop: instrument, observe, annotate, evaluate, and deploy. New production logs feed back into the observe stage. The mechanics come down to three building blocks.

Logs and traces

Braintrust publishes SDKs for TypeScript, Python, Go, Java, Ruby, and C#, and a Rust SDK in alpha that covers tracing and evals. Teams that already emit OpenTelemetry spans can send them to Braintrust, and a LangSmith integration traces LangSmith-instrumented code. Each request becomes a trace of nested spans.

• LLM spans: auto-instrumentation logs inputs, outputs, latency, token usage, and costs for every call to a supported provider.
• Application spans: retrieval steps, tool calls, preprocessing, and business logic can each be traced as spans.
• Framework integrations: the integrations page lists LangGraph, the OpenAI Agents SDK, the Claude Agent SDK, CrewAI, and the Vercel AI SDK, among others.
• Online scoring: online scorers run asynchronously in the background, so they add no latency to the application.

For a wider view of what a production team should trace, see AI agent observability and the comparison of context, data, and LLM observability.

Data, Task, and Scores

Braintrust’s evaluation quickstart defines three components. Data is a dataset of test cases with inputs and expected outputs. Task is the AI function under test, from a single prompt to a multi-step agent. Scores are the functions that measure output quality.

Scores come in four families, and most teams mix them:

• Autoevals is an MIT-licensed library with factuality, exact match, Levenshtein distance, JSON validation, and RAG metrics.
• Custom code covers rules a team can express deterministically, such as exact matching, pattern matching, and business logic.
• LLM-as-a-judge grades output against a natural-language rubric at span or trace scope, with an optional chain-of-thought setting and a pass threshold between 0 and 1.
• Human review lets reviewers score traces and dataset rows, which the documentation calls the source of ground truth and a check on automated scorers.

Teams that want to evaluate the trajectory of a multi-step agent can score at trace level, and the guide to AI agent trajectory evaluation covers what to measure. The wider question of metrics beyond accuracy has its own guide on agent evaluation benchmarks and metrics.

Experiments and CI

Each eval run creates an experiment that captures inputs, outputs, scores, and metadata. Set a baseline and Braintrust aligns test cases across experiments, adds score deltas to every row, and grades the experiment as an improvement, regression, tradeoff, or tie.

Braintrust documents three ways to run evals in CI:

• GitHub Action: braintrustdata/eval-action runs evals and posts a comment with results on the pull request.
• CLI for other CI systems: bt eval runs headless with JSON output, and the --first N or --sample N flags run a smoke test on pull requests while the full run waits for merges.
• Custom pass and fail logic: the CLI exits non-zero when an eval throws an exception. A team that wants a score drop to fail the build defines a Reporter whose reportRun function returns true or false.

---

Who uses Braintrust, and where does it fit in the stack?

Braintrust’s homepage names Cloudflare, Pylon, Notion, Box, Replit, Graphite, and Retool as customers. The documentation and cookbook point to five recurring jobs:

1. Regression testing: rerun a fixed dataset whenever a prompt, model, or tool changes, then read the experiment comparison.
2. Model swaps: playgrounds let a team swap models and compare outputs side by side before committing.
3. Production monitoring: score live traces online and route alerts to Slack or a webhook.
4. Dataset building: populate datasets from production so real failures become versioned test cases.
5. Agent trajectory review: custom code scorers assess agent trajectories and tool usage at trace level.

Where Braintrust sits next to a context layer

One way to picture an enterprise agent stack is four layers. Braintrust works in the evaluation and observability layer. Atlan works in the context layer beneath it, which holds the governed definitions, lineage, ownership, and policy an agent reads.

Layer	What it does	Example tools
Agent execution	Runs models, tools, and orchestration logic	LangGraph, OpenAI Agents SDK, Claude Agent SDK, CrewAI
Evaluation and observability	Traces, scores, and compares agent runs	Braintrust
Context layer	Builds, versions, and delivers definitions, lineage, ownership, and policy context	Atlan
Data estate	Stores the records agents query	Warehouses, lakehouses, BI tools, SaaS apps

The enterprise context layer guide describes that third row in detail, and the semantic layer for AI agents explains where metric definitions fit inside it.

---

What does an eval score tell you, and what does it leave open?

An eval score measures output against a rubric, on a specific dataset, at a specific moment. It shows that quality moved and roughly where, by test case, scorer, and span. It does not say why.

Seven failure points behind one bad score

A 2024 paper by Barnett and colleagues catalogued seven failure points of retrieval-augmented systems, drawn from three case studies in research, education, and biomedicine. Any of them can produce the same low correctness or faithfulness score, and each calls for a different fix.

Failure point	What happens	Where to look first
FP1: Missing content	The question cannot be answered from the indexed documents, and the system should decline instead of answering	Context supply
FP2: Missed top-ranked documents	The answer exists in a document that did not rank high enough to be returned	Retrieval
FP3: Not in context	Documents with the answer were retrieved but did not make it into the context for generation	Context assembly
FP4: Not extracted	The answer is in the context, but the model failed to extract it	Grounding or model
FP5: Wrong format	The model ignored an instruction to return a table or list	Model or prompt
FP6: Incorrect specificity	The answer is too broad or too narrow for the need	Prompt or model
FP7: Incomplete	The answer is correct but misses information that was in the context	Model

The last column is this guide’s mapping, not the paper’s. For a fuller treatment of the retrieval side, see how to evaluate RAG systems and the guide to agent context layers versus RAG.

LLM judges carry their own bias

Many scorers are themselves LLMs, which adds a second source of error. The MT-Bench study found that strong judges such as GPT-4 reached over 80% agreement with human preferences, the same level of agreement as between humans. The same paper names position, verbosity, and self-enhancement biases. A separate study found that reordering candidate answers let a weaker model beat ChatGPT in 66 of 80 test queries when ChatGPT served as the evaluator.

Braintrust’s human review is the documented check here: its documentation lists assessing the efficacy of automated scoring methods as a use case. Teams that want a vendor-neutral view of the scoring tools can read the LLM evaluation frameworks comparison.

The trace shows the failure, and the metadata shows its origin

A Braintrust trace records what the agent retrieved, what it sent to the model, and what came back. Whether the retrieved definition was the certified one, whether its upstream table changed last Tuesday, or whether a policy should have blocked it is metadata that lives in the data estate. Braintrust can attach custom metadata and tags to spans, so a team can log it, but something else has to supply it.

Without that record, a team is left with three competing hypotheses:

• Retrieval: the agent fetched the wrong asset or document.
• Grounding: the agent fetched the right asset, and its content was stale, uncertified, or contradictory.
• Model: the agent had correct, current context and still reasoned badly.

The first two feed the AI agent hallucination problem that teams try to catch with hallucination detection, and neither detector can tell which hypothesis is true without the record described next.

---

Where context lineage picks up: retrieval, grounding, or model

Context lineage is the record of where each piece of agent context came from: the source asset, the definition version, the owner who certified it, and the upstream changes that touched it. Paired with an eval trace, it turns three hypotheses into a short elimination. Atlan builds this record into its context layer, and the diagnostic logic below works with any eval platform.

Failure class	Question to ask	Evidence in the trace	Evidence in context lineage	Where the fix lives
Retrieval	Did the agent fetch the right asset?	Retrieved chunks or tables differ from the expected source	The certified asset existed and was current at run time	Index, chunking, ranking, query rewriting
Grounding	Was the fetched content correct and current?	The expected asset was retrieved	The definition was stale, deprecated, uncertified, in conflict, or changed upstream since the last passing run	Definitions, lineage, ownership, policy context
Model	Did the model misuse correct context?	Correct, current context appears in the prompt	No upstream change, and the definition was certified	Prompt, model choice, decoding, scorer rubric

Data lineage for AI explains why the upstream half of that table matters, and context observability for AI agents covers how to watch it in production.

---

How does Atlan work alongside an eval platform?

Atlan sits beneath an eval platform such as Braintrust and leaves scoring to it. The platform supplies the governed context agents read and to keep a versioned record of that context, so every eval result can be read against it. Four parts of the platform do that work:

• Context Engineering Studio builds context from existing assets such as dashboards and SQL, versions it in Context Repos, and tests it with eval suites built from your own dashboards and queries. The guides to context engineering for AI agents and the context repository describe the method.
• Context Agents generate descriptions, glossary terms, and metric definitions from SQL history, pipeline code, and BI semantics. Sage, the metric arbiter, surfaces two teams defining the same metric differently and routes the conflict to a steward.
• Lineage traces which upstream assets feed the ones an agent reads, so a team can check what changed before a score dropped.
• Context Lakehouse stores metadata in open Iceberg formats with time travel, which lets a team review what a definition said at an earlier point.

The Enterprise Data Graph ties these together, and business context for AI explains the gap it closes. The video below shows the build, test, and ship loop applied to context.

Connecting the two in practice

Braintrust’s integrations page lists AI providers, SDKs, agent frameworks, developer tools, and runtimes, and it names no data catalog or lineage integration. A team can still pair the two with a few working practices:

• Log the context version: write the Context Repo version or definition IDs into span metadata, so every score ties to the context in use.
• Rerun across context changes: run the same dataset before and after a context update to isolate its effect on scores. The guide to context evals versus data quality testing explains why that comparison is its own kind of test.
• Route grounding failures to owners: send traces judged wrong on certified definitions to the owning steward and not to the prompt backlog. Context testing for AI agents and context quality testing cover that handoff.
• Bring context into investigations: Loop can connect to external MCP servers, and Atlan exposes context through its MCP server. Braintrust labels that connection a public preview that is available on Braintrust-hosted deployments only, so test the pairing in your own environment. MCP is the protocol underneath.

---

What should you weigh before adopting Braintrust?

A handful of decisions shape how well Braintrust works in an enterprise:

• Deployment model: Braintrust documents SaaS on Starter, Pro, and Enterprise plans, while bring-your-own-cloud and self-hosted deployments require Enterprise. Braintrust always operates the control plane.
• Open-source scope: autoevals is MIT-licensed, and the platform itself is commercial.
• Scorer trust: validate every LLM judge against human labels before it gates a merge, and recheck whenever the judge model changes.
• CI semantics: decide up front what blocks a merge. The CI documentation fails the CLI on thrown exceptions, so score-based gates need a custom Reporter.
• Sensitive data: Braintrust offers masking functions and ingestion-time PII redaction. The redaction scans only incoming project logs and excludes attachments, and it sends selected fields to a detection model hosted on Baseten, so read the page before relying on it.
• Feature maturity: Patterns and Loop’s external MCP connections are public previews, and Topics on self-hosted deployments needs data plane v2.x and access from the account team.
• Context pairing: plan from day one how failing traces will be checked against certified definitions and lineage, or grounding errors will keep landing in the prompt backlog. The AI agent governance and guardrails checklist guides cover the controls around that.

For a feature and pricing comparison with LangSmith, Arize, Langfuse, and others, read the guide to Braintrust alternatives. Teams moving from proof of concept to production may also recognize the pattern described in AI agents stuck in proof-of-concept testing.

---

Moving forward with Braintrust

Braintrust gives AI teams a disciplined way to trace agents, score output with code, judges, and people, and compare every change against a baseline. Those scores describe the output. The cause of a drop often sits upstream, in the definitions, lineage, and policy context an agent reads.

Atlan supplies that governed context through Context Engineering Studio, Context Agents, lineage, and the Context Lakehouse. Read side by side, the eval result and the context record show that an agent failed and where the fix belongs. Teams choosing a context layer can start from the context layer evaluation criteria.

---

FAQs about Braintrust

1. What does Braintrust do?

Braintrust is a commercial platform for evaluating and observing LLM applications and AI agents. Teams instrument their code with Braintrust SDKs, which capture traces of prompts, tool calls, latency, token usage, and cost. Code, LLM judges, or human reviewers score those traces. Each run is saved as an experiment that a team can compare against a baseline, monitor in production, and run in CI before a change ships.

2. What is an eval in Braintrust, and how do scorers, datasets, and experiments fit together?

An eval has three components. Data is a dataset of test cases with inputs and expected outputs, Task is the AI function under test, and Scores are the functions that measure output quality. Running an eval creates an experiment, a permanent record of inputs, outputs, scores, and metadata. Comparing experiments against a baseline shows which cases improved or regressed.

3. Is Braintrust the same company as the Braintrust talent network or the BTRST token?

No. Braintrust at braintrust.dev, run by Braintrust Data, Inc., builds AI evaluation and observability software. Braintrust at usebraintrust.com describes itself as an AI-powered talent network, and its site carries a BTRST token section. The two products share a name and nothing else.

4. Is Braintrust open source, and can it be self-hosted?

The platform is commercial software, while the autoevals scorer library is open source under the MIT License. Braintrust documents SaaS on its Starter, Pro, and Enterprise plans. Bring-your-own-cloud and self-hosted deployments require an Enterprise plan, and Braintrust always operates the control plane.

5. How does Braintrust relate to observability tools like Arize, Langfuse, and LangSmith?

These tools all sit in the LLM tracing and evaluation category, so their feature sets overlap. Braintrust documents an integration that traces LangSmith-instrumented applications, so a team can send those spans to Braintrust. The differences between the tools are mostly deployment model, pricing, and instrumentation approach, and a separate Braintrust alternatives guide compares them.

6. Can Braintrust tell you why an agent failed, or only that it failed?

Braintrust shows where in a trace a failure happened, and its Loop agent investigates logs, experiments, and datasets in a project. A trace holds what the agent retrieved, so facts such as whether a definition was certified or changed upstream appear only if a team logs them as metadata. Separating retrieval, grounding, and model errors usually needs that record from the data estate.

7. What do you still need alongside an eval platform to run agents in production?

An eval platform measures behavior. Running agents reliably also depends on the inputs they read, such as governed definitions, lineage, policy context, and a history of what each definition said at a given time. A context layer supplies those inputs and keeps the record.

---

Sources

1. Braintrust: The active observability platform for agents, Braintrust. https://www.braintrust.dev/
2. Braintrust: AI-Powered Talent Network for Enterprise, Braintrust (usebraintrust.com). https://www.usebraintrust.com/
3. Braintrust workflow, Braintrust Docs. https://www.braintrust.dev/docs/workflow
4. Trace LLM calls, Braintrust Docs. https://www.braintrust.dev/docs/instrument/trace-llm-calls
5. Trace application logic, Braintrust Docs. https://www.braintrust.dev/docs/instrument/trace-application-logic
6. Custom tracing and metadata, Braintrust Docs. https://www.braintrust.dev/docs/instrument/custom-tracing
7. SDKs, Braintrust Docs. https://www.braintrust.dev/docs/reference/sdks
8. Integrations, Braintrust Docs. https://www.braintrust.dev/docs/integrations
9. LangSmith integration, Braintrust Docs. https://www.braintrust.dev/docs/integrations/sdk-integrations/langsmith
10. Score production traces, Braintrust Docs. https://www.braintrust.dev/docs/evaluate/score-online
11. Evaluation quickstart, Braintrust Docs. https://www.braintrust.dev/docs/evaluation-quickstart
12. autoevals, Braintrust on GitHub. https://github.com/braintrustdata/autoevals
13. Custom code evaluators, Braintrust Docs. https://www.braintrust.dev/docs/evaluate/custom-code
14. LLM-as-a-judge evaluators, Braintrust Docs. https://www.braintrust.dev/docs/evaluate/llm-as-a-judge
15. Human review, Braintrust Docs. https://www.braintrust.dev/docs/annotate/human-review/index
16. Datasets, Braintrust Docs. https://www.braintrust.dev/docs/annotate/datasets
17. Compare experiments, Braintrust Docs. https://www.braintrust.dev/docs/evaluate/compare-experiments
18. Run experiments in CI/CD, Braintrust Docs. https://www.braintrust.dev/docs/evaluate/run-in-ci
19. Playgrounds, Braintrust Docs. https://www.braintrust.dev/docs/evaluate/playgrounds
20. Prompt versions, Braintrust Docs. https://www.braintrust.dev/docs/evaluate/prompts/versions
21. Braintrust Gateway, Braintrust Docs. https://www.braintrust.dev/docs/deploy/gateway
22. Alerts, Braintrust Docs. https://www.braintrust.dev/docs/observe/alerts
23. Loop, Braintrust Docs. https://www.braintrust.dev/docs/loop
24. Connect external MCP servers to Loop, Braintrust Docs. https://www.braintrust.dev/docs/loop/mcp
25. Topics, Braintrust Docs. https://www.braintrust.dev/docs/observe/topics/index
26. Patterns, Braintrust Docs. https://www.braintrust.dev/docs/observe/patterns/index
27. Protect sensitive data, Braintrust Docs. https://www.braintrust.dev/docs/admin/data-management/protect-sensitive-data
28. Deployment options, Braintrust Docs. https://www.braintrust.dev/docs/admin/deployment/index
29. Cookbook, Braintrust Docs. https://www.braintrust.dev/docs/cookbook
30. Optimizing Ragas to evaluate a RAG pipeline, Braintrust Docs. https://www.braintrust.dev/docs/cookbook/recipes/SimpleRagas
31. How Zapier uses assertions to evaluate tool usage in chatbots, Braintrust Docs. https://www.braintrust.dev/docs/cookbook/recipes/Assertions
32. Text-to-SQL, Braintrust Docs. https://www.braintrust.dev/docs/cookbook/recipes/Text2SQL
33. Evaluating a voice agent, Braintrust Docs. https://www.braintrust.dev/docs/cookbook/recipes/VoiceAgent
34. Detecting prompt injections, Braintrust Docs. https://www.braintrust.dev/docs/cookbook/recipes/PromptInjectionDetector
35. Seven Failure Points When Engineering a Retrieval Augmented Generation System, Barnett et al., arXiv, 2024. https://arxiv.org/abs/2401.05856
36. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng et al., arXiv, 2023. https://arxiv.org/abs/2306.05685
37. Large Language Models are not Fair Evaluators, Wang et al., arXiv, 2023. https://arxiv.org/abs/2305.17926