Structured, unstructured, and semi-structured data each create distinct failure modes for AI agents. According to IBM (2025), unstructured data makes up 90% of all enterprise-generated data and is growing three times faster than structured data. Atlan’s context graph links structured metrics, unstructured documents, and semi-structured events to the same certified business concepts, so your agents work from shared meaning instead of raw storage.
Quick facts
Permalink to “Quick facts”| Data type | What it gives AI | Where it breaks | Context fix |
|---|---|---|---|
| Structured | Metrics, tables, entities, SQL-ready facts | Same field name can mean different business concepts | Glossary links, ownership, lineage, certification |
| Unstructured | Documents, tickets, policies, recordings, human context | No freshness, policy, owner, or source trust signal | Metadata enrichment before chunking and embedding |
| Semi-structured | JSON, XML, logs, API payloads, events | Schema-on-read shifts hide breaking field changes | Versioning, aliases, deprecation rules, schema lineage |
| Unified context | One semantic map across all three types | Requires context management above storage and retrieval layers | Context graph, live context, access policy, MCP delivery |
Where does your data estate break down for AI?
Run the Context Gap Calculator to see how structured, unstructured, and semi-structured sources score against agent-readiness.
Calculate Your GapHow does AI read across all three data types?
Permalink to “How does AI read across all three data types?”AI agents read across types in one task, the same challenge how enterprises use AI agents in production keeps surfacing. A customer health agent checks CRM fields, support tickets, product events, and policy documents before recommending outreach. Databricks recommends storing contextual metadata with chunks and embeddings because metadata improves filtering and retrieval. That principle applies beyond RAG: every data type needs context carried with it. The storage taxonomy matters less than the meaning taxonomy. Your context layer decides whether agents can reconcile signals from different formats.
Why does structured data fail on meaning?
Permalink to “Why does structured data fail on meaning?”Structured data looks safe because tables expose columns, types, primary keys, and SQL-friendly values. That helps analytics. It does not tell an agent which definition is canonical. In a customer success workflow, arr_current, contract_arr, and finance_arr can all be valid. The agent needs the certified term, owner, lineage, and approved use case before it reasons over revenue, the same gap that leaves agents worse than humans at data discovery when no one has certified which value is canonical. A feature store hits the identical wall one layer down the pipeline: feature stores for ML exist precisely because one certified metric definition has to survive from raw table to model input.
Why does unstructured data fail on trust?
Permalink to “Why does unstructured data fail on trust?”Unstructured data carries the richest business explanation. Tickets describe customer friction. PDFs explain policies. Call notes expose intent. The failure is trust. A document without freshness, owner, access policy, and source lineage becomes context poison. This is the AI hallucination pattern that context enrichment is designed to prevent. The same access-policy gap is why handling PII in AI pipelines has to happen before a document ever reaches an agent, not after.
Why does semi-structured data fail on drift?
Permalink to “Why does semi-structured data fail on drift?”Semi-structured data survives change by design. JSON payloads, API responses, and logs accept nested fields, optional fields, and evolving names. That flexibility helps integration, but agents need version history when customer_id becomes customerId, then cid_v3. Schema flexibility protected the application. It breaks the agent when no versioned semantic alias travels with the change. Data contracts for AI exist to enforce that versioned alias before a breaking field change ever reaches production.
How do data conflicts appear in production?
Permalink to “How do data conflicts appear in production?”A customer success agent asks whether an account is healthy. Structured CRM data says ARR is stable. Unstructured support tickets show repeated complaints about onboarding. Semi-structured product logs show a fall in daily active users after a feature flag changed. Each signal is true. The agent fails when it treats one type as the whole answer. The practical fix is not format conversion, your team needs a shared semantic map across all types. Resolving which system holds the canonical answer is the same problem systems of record and systems of knowledge are built to separate before an agent has to guess.
How does ARR split across systems?
Permalink to “How does ARR split across systems?”The structured CRM table uses arr_booked. Finance uses arr_recognized. Product analytics joins arr_current to usage cohorts. All three columns are valid. The agent needs disambiguation rules: which value answers renewal risk, which answers revenue recognition, and which answers usage-based expansion. The same three-system split shows up whenever real-time data reaches AI agents: a metric can be correct and stale at the same time.
How do stale documents poison retrieval?
Permalink to “How do stale documents poison retrieval?”A support playbook says a feature ships next quarter. The feature shipped nine months ago. A vector search index retrieves the paragraph because it is semantically close to the question. Without freshness metadata, the agent treats stale guidance as current policy. The issue is retrieval quality, but the root cause is a missing freshness signal on the document itself, the same data quality gap that undermines structured pipelines shows up here as staleness instead of bad values. Catching that gap before an agent acts on it is exactly what data observability for AI pipelines is built to do.
How does JSON drift break event streams?
Permalink to “How does JSON drift break event streams?”A product event stream changes trial_status to account_stage. Dashboards survive because the analytics team patches transformations downstream. The agent still queries the older field through a tool contract and receives null values. Schema flexibility protected the application. It broke the agent context because no versioned semantic alias traveled with the change. Preventing that failure starts earlier than the event stream itself, with the same groundwork covered in how to prepare enterprise data for AI agents.
How can you unify meaning across data types?
Permalink to “How can you unify meaning across data types?”Production AI needs semantic integration above the storage layer, the same argument behind why AI agents need an enterprise context layer in the first place. Apache Arrow standardizes columnar memory structures for data transport. AWS Lake Formation helps build, catalog, and secure data lakes. These layers help data move and control access. Agents still need business concepts mapped across all formats.
What is semantic normalization?
Permalink to “What is semantic normalization?”Semantic normalization maps field names, document concepts, and event labels to shared business terms. customer_id, account_id, and client_ref become related identifiers with clear scope. ARR links to one certified glossary definition, accepted variants, owners, and disambiguation rules. This is where context engineering matters more than raw data preparation, the same discipline semantic layers for AI agents are built to enforce. The same normalization has to hold even when the storage layer underneath changes, which is why a data lakehouse built for AI treats certified definitions as a layer above Iceberg, Delta Lake, or Snowflake, not a property of any one of them.
What's the ROI of unifying context across data types?
Run the Context Layer ROI Calculator to size the cost of agents reasoning over disconnected tables, documents, and events.
Calculate the ROIWhy must metadata travel with embeddings?
Permalink to “Why must metadata travel with embeddings?”Metadata should travel before and after embedding. Before embedding, it improves chunk selection, filtering, and policy checks. After embedding, it lets an agent explain why a source was retrieved, whether it is current, and which user can access it, the same role a metadata layer for AI plays across every retrieval path, not just embeddings. According to Atlan AI Labs research (2026), attaching metadata context to embeddings improves retrieval accuracy from 33% to 55%. That travel matters even more once embeddings span formats, the same cross-format challenge multimodal data for AI agents has to solve for images, audio, and video alongside text.
How do schema versions protect AI agents?
Permalink to “How do schema versions protect AI agents?”Schema versions should include aliases, deprecation status, owner, event source, last validation time, and downstream consumers. A schema registry alone describes fields. Agents also need semantic versioning that tells them whether an old field is safe, deprecated, or mapped to a newer concept. Decision traces link field versions to agent outputs, making every inference auditable, the same audit trail data lineage for AI builds for structured tables extended to schema versions on events.
How Atlan approaches structured vs. unstructured data for AI
Permalink to “How Atlan approaches structured vs. unstructured data for AI”The challenge
Permalink to “The challenge”Your agent sees tables, documents, and event streams as separate retrieval surfaces. The same business concept appears in different formats with different owners, freshness signals, access rules, and lineage coverage, the fragmentation metadata management for AI exists to close. The organizational cold start problem is most acute here: agents have no mechanism to resolve cross-format conflicts without a governed context layer.
The approach
Permalink to “The approach”Atlan acts as the governed context layer across all three data types. It links structured metrics, unstructured documents, and semi-structured events into one context graph. It adds owners, glossary terms, lineage, freshness, sensitivity tags, and policies, then delivers that context to agents through MCP.
The outcome
Permalink to “The outcome”Agents query governed context instead of isolated files, tables, or payloads. When a customer term differs across CRM, support tickets, and usage logs, the context layer exposes the relationship, the conflict, and the certified rule for resolving it, the same mechanism decision-support agents depend on to answer a business question instead of a query. Context Engineering Studio bootstraps the initial context map from SQL history, BI usage, and lineage records. Context Agents keep that map current as schemas evolve and documents age.
Real stories from real customers: unifying context across data types
Permalink to “Real stories from real customers: unifying context across data types”"AI initiatives require more context than ever. Atlan's metadata lakehouse is configurable, intuitive, and able to scale to hundreds of millions of assets."
Andrew Reiskind, Chief Data Officer, Mastercard
"Atlan is much more than a catalog of catalogs. It's more of a context operating system. Atlan enabled us to easily activate metadata for everything from discovery in the marketplace to AI governance to data quality to an MCP server delivering context to AI models."
Sridher Arumugham, Chief Data & Analytics Officer, DigiKey
Mastercard runs AI initiatives across a data estate spanning structured, unstructured, and semi-structured sources at a scale of hundreds of millions of assets, interpreting transaction data at the speed of transaction. DigiKey unified discovery, data quality, and AI model delivery across structured product data, unstructured documentation, and semi-structured catalog events into a single context operating system, rather than treating each data type as its own system to maintain. Both stories make the same point data mesh and AI makes at the architecture level: unifying context across systems beats governing each one separately.
How ready is your data estate for AI agents?
Run the AI Agent Context Readiness Checklist to score your structured, unstructured, and semi-structured sources against agent-readiness criteria.
Check Your ReadinessWhy unified context beats type conversion
Permalink to “Why unified context beats type conversion”Converting everything into one format feels tidy and fails the real problem. Tables lose document nuance. Documents lose metric authority. JSON events lose stable semantics when payloads drift. AI agents need all three views, with shared context that explains what each view means, who owns it, whether it is current, and how conflicts resolve.
The stronger architecture is structured truth plus unstructured explanation plus semi-structured signals, unified by shared context. Agents become safer when meaning is resolved before retrieval, not repaired after hallucination. Multi-agent memory silos compound this problem at scale: each agent builds isolated knowledge about different data types, producing conflicting answers from the same enterprise data. According to Zylos Research (2026), nearly 65% of enterprise AI failures in 2025 were attributable to context drift or memory loss during multi-step reasoning, not raw context-window exhaustion, the same failure mode this kind of format-level context drift produces at the data layer. A unified context layer prevents that from the first query.
FAQs about structured vs. unstructured data for AI
Permalink to “FAQs about structured vs. unstructured data for AI”1. What is the main difference between structured and unstructured data for AI?
Permalink to “1. What is the main difference between structured and unstructured data for AI?”Structured data gives agents organized values in tables, columns, and rows. Unstructured data gives agents meaning-rich content in documents, PDFs, recordings, images, and messages. AI agents need both authority and explanation. Structured data anchors the answer. Unstructured data adds the surrounding evidence.
2. How does semi-structured data fit into AI pipelines?
Permalink to “2. How does semi-structured data fit into AI pipelines?”Semi-structured data bridges systems through JSON, XML, logs, API responses, and event streams. It gives agents useful real-time signals, but the structure changes more often than table schemas. AI pipelines need schema versioning, aliases, and deprecation rules so agents understand when fields change meaning.
3. Can your team convert unstructured data into structured data instead?
Permalink to “3. Can your team convert unstructured data into structured data instead?”Conversion helps search and analysis, but it does not preserve all meaning. A support ticket converted into a table can lose tone, timing, and narrative context. The better pattern is extraction plus context: keep the original source, attach metadata, and map extracted concepts to certified business terms.
4. Why do AI agents hallucinate across data types?
Permalink to “4. Why do AI agents hallucinate across data types?”Agents hallucinate across data types when each source answers the same question through a different definition. A CRM table, support ticket, and product log can all describe customer status differently. Without a shared semantic layer, the agent selects one interpretation and sounds confident while missing the conflict.
5. What metadata should unstructured data include for AI?
Permalink to “5. What metadata should unstructured data include for AI?”Unstructured data should include owner, source system, creation date, update date, sensitivity level, policy scope, lineage, language, topic, and freshness status. This metadata helps agents decide whether a document is relevant, current, authorized, and trustworthy before it becomes part of an answer.
6. Is schema mapping enough for structured and semi-structured data?
Permalink to “6. Is schema mapping enough for structured and semi-structured data?”Schema mapping aligns fields and tables. It does not align meaning. Two fields can share a name and represent different concepts, while two different fields can represent the same concept. AI agents need semantic mapping, glossary terms, lineage, and certified definitions alongside schema mappings.
7. What does unified context mean for AI agents?
Permalink to “7. What does unified context mean for AI agents?”A unified context layer links structured, unstructured, and semi-structured data to shared business concepts, owners, policies, lineage, quality signals, and freshness rules. It gives AI agents one map for deciding which source to trust, how to reconcile conflicts, and which data a user can access.
Sources
Permalink to “Sources”-
IBM Think. (2025). Structured vs. Unstructured Data: 90% of enterprise data is unstructured. https://www.ibm.com/think/topics/structured-vs-unstructured-data
-
Apache Arrow. (2026). Arrow columnar format specification. https://arrow.apache.org/docs/format/Columnar.html
-
MongoDB. (2026). JSON and BSON. https://www.mongodb.com/resources/basics/json-and-bson
-
AWS Lake Formation. (2026). Tutorials for building, ingesting, sharing, and securing data lakes. https://docs.aws.amazon.com/lake-formation/latest/dg/getting-started-tutorials.html
-
Zylos Research. (2026). AI Agent Context Compression: Strategies for Long-Running Sessions. https://zylos.ai/research/2026-02-28-ai-agent-context-compression-strategies/
