How We Proved Metadata Delivers 38% Better AI Accuracy

Imagine everyone in the business – regardless of technical expertise – asking complex data questions in plain English and getting instant, accurate answers without waiting. Well, that technology is here. LLMs can now generate syntactically correct SQL from natural language.

But there’s a catch. What’s syntactically correct is often semantically wrong.

Countless failed production rollouts prove that generating a query isn’t the challenge. It’s generating the right query – one that understands business logic, unwritten rules, and specific database conventions.

We tested 522 queries to find a solution. Adding rich semantic metadata delivers a 38% relative improvement in AI-generated SQL accuracy, with a p-value less than 0.0001. Translation: there’s less than a 2 in 10,000 chance this is random luck.

It’s guesswork. It’s reliable data you can trust with your biggest decisions.

Why does AI get it wrong? Context.

An LLM is like a brilliant generalist with zero institutional memory. The AI can see schemas — table names, column names, etc. — but has no idea what they mean in business context.

It’s a context problem, not an intelligence problem.

Here’s how it breaks in production, using a Formula One dataset as an example. A user asks: “Which drivers were eliminated in the first round?” The AI sees the schema, looks for a column called “eliminated,” and generates plausible SQL: SELECT driver_name FROM results WHERE position IS NULL.

The result? Completely wrong. Why? The AI treated “eliminated” like missing data, not “slowest qualifying times in a racing context.” Without metadata explaining the domain rule, it makes reasonable-but-wrong assumptions.

With enhanced metadata, the AI understands that “eliminated” means slowest times. It generates the correct query: ORDER BY Q1_time DESC LIMIT 5.

That one piece of domain knowledge — that single piece of metadata — is the difference between giving a CFO totally false numbers and giving them the truth.

The common failure patterns show up everywhere:

Ambiguous terminology: Does “expired” mean contract ended or license lapsed? Does “qualified” mean certification or race qualification?
Column confusion: Multiple tables with similar names — nationality vs. country, three different columns all labeled “date”
Missing relationships: Wrong joins leading to Cartesian products or missing records

The solution is context. But how much difference does it actually make?

The 38% swing from AI failure to success

Our research tested 174 unique queries three times each — 522 total evaluations — using a Formula One dataset with 13 tables and 94 columns that mirrors enterprise data warehouse complexity.

We tested two conditions:

Baseline context: Bare minimum schema information (think schema, column type, etc., ~29 lines of metadata)
Enhanced context: Schema plus business glossaries, SQL usage patterns, and domain hints (~64 lines)

The results for simple, medium and complex queries:

Baseline: 16.1% win rate (roughly 1 in 6 queries answered correctly)
Enhanced: 22.2% win rate
Improvement: +38% relative improvement, p < 0.0001

The outcome was clear: better metadata directly improves AI performance. That’s the difference between a failed AI project and a successful one. With 95% of AI pilots failing in production, that swing makes adding a context layer to data infrastructure a no-brainer.

And the cost? Only $4.02 per 1,000 queries. Each improved query only needs to be worth two cents to justify the investment. A correctly answered self-service query — just factoring in user satisfaction and avoided support tickets — is conservatively worth 50 cents.

At that conservative value, the enhanced metadata delivers an ROI of 2,662%. But think about it in terms of preventing one bad business decision. If fixing one query stops the CFO from reporting the wrong earnings, that two-cent investment just saved millions.

The value is exponential – and that reframes the whole cost-benefit analysis.

The complexity sweet spot

Not all queries benefit equally. The research broke queries into three complexity categories, revealing exactly where metadata investment delivers maximum impact.

Simple queries (~70% of dataset) like “What is the average age of drivers?” saw a 1.15x improvement. LLMs can usually handle these with basic schema alone.

Complex queries (~4% of dataset) with multiple subqueries and intricate conditional logic saw a modest improvement (1.00x). These require more advanced techniques like chain-of-thought prompting.

Medium complexity queries (~27% of dataset) are the goldmine. These involve table relationships, aggregations, and domain-specific rules. They saw 2.15x improvement with enhanced metadata.

TL;DR: The medium complexity queries went from being mostly failures to often successes.

These are the “workhorse questions” business analysts ask every day. Investments in semantic metadata shouldn’t be aimed at the easiest questions or the nearly impossible ones. It should be focused squarely on medium complexity queries — that’s where you unlock self-service data for most users.

This is where Atlan helps maximize ROI, by providing a content layer that enriches medium-complexity queries with semantic metadata. In fact, customers in Atlan AI Labs workshops saw a 5x improvement in query accuracy just by adding metadata.

Format matters: The “just enough” principle

Here’s where it gets counterintuitive. Researchers tested two versions of enhanced metadata:

Optimized version (64 lines): 22.2% win rate
Verbose version (176 lines): 13.8% worse performance, 52% more expensive to run

Same information, different format, dramatically different results.

Why? LLMs suffer from attention decay with verbose content because it’s seen as noise. Comprehensive catalog descriptions perfect for humans become noise for code generation. Critical information gets diluted in paragraphs of explanatory text.

In other words, dumping your data catalog into the LLM prompt won’t work. You have to actively transform that raw data into AI-effective metadata.

The research identified four key principles for transformation:

Show how, not what. LLMs are code generation tools — they respond to runnable SQL patterns, not prose descriptions. Instead of “The milliseconds column contains lap time measurements,” provide: # Fastest lap: SELECT milliseconds FROM lap_times ORDER BY milliseconds ASC LIMIT 1
Prevent errors, don’t explain. Get ahead of known failure patterns. State rules upfront: “Eliminated means ORDER BY time DESC, not IS NULL.” Stop bad queries before the AI considers them.
Disambiguate similar columns. List confusable columns side-by-side with distinct usage rules. Example: drivers.nationality (driver’s home country) vs. circuits.country (race location).
Production-ready syntax. Always use actual production syntax the LLM needs to generate. Fully qualified names: formula_one_db.schema.drivers. Exact naming conventions your database requires.

These principles transform business glossaries, data model relationships, and column-level metadata into context an AI agent can actually use.

Optimizing the context layer for AI agents

This metadata-AI relationship extends far beyond query generation. Anywhere an AI agent needs to reason about your data — data quality validation, automated governance, intelligent discovery — optimized context is the multiplier.

The LLM is the engine, but the context layer is the high-octane fuel. Atlan captures, enriches, and delivers that fuel at scale.

AI models will always need context. The quality of that context directly determines outcomes. 38% improvement. 2.15x multiplier for critical queries. 2,662% ROI. These aren’t projections — they’re proven results.

Your data catalog likely has comprehensive documentation optimized for human search and browsing. But if you’re serious about building AI on top of enterprise data, where are you actually investing—models, or meaning?

Because what this research makes clear is that the metadata layer isn’t documentation. It’s infrastructure. It’s the foundation that determines whether an AI system reasons correctly or confidently gives you the wrong answer.

If you’re working on NL2SQL — or any “talk to data” use case — and you’re spending all your energy tuning prompts, swapping models, or adding guardrails, but not actively optimizing the context you give the model, you’re leaving most of the value on the table. The biggest gains don’t come from smarter models. They come from better context.

And that context doesn’t magically appear. It has to be designed. Curated. Optimized for machine actionability, not just human readability.

So ask yourself this: Is your metadata layer built for humans to browse… or for AI agents to reason with?

Because the future of AI-powered data isn’t model-first. It’s metadata-first. And the teams that understand that will be the ones whose AI actually works in production.

See the complete methodology, results, implementation guidelines, and ROI benchmarks here.

Share this article

Atlan is the next-generation platform for data and AI governance. It is a control plane that stitches together a business's disparate data infrastructure, cataloging and enriching data with business context and security.

Book a Demo Start Tour