---
title: "How an Unstructured Data Catalog Works for PDFs and Documents"
url: "https://atlan.com/know/data-catalog/how-does-an-unstructured-data-catalog-work-for-pdfs-and-documents/"
description: "An unstructured data catalog crawls PDFs and documents, extracts text and metadata, classifies content, and makes files searchable and governed for AI."
author: "Emily Winks"
author_role: "Data Governance Expert"
published: "2026-10-21"
updated: "2026-10-21"
---

> Atlan is hosting Context Conference, bringing together the leaders and builders at the frontier of giving AI the context it needs to understand their business. It runs online on October 28, 2026, from 11:00 AM to 2:00 PM ET. Atlan co-founder Prukalpa Sankar opens and closes the day. Leaders from AstraZeneca, BNY and Verizon share why they invest in context and what they get from it. Registrants get early access to The AI Context Gap, a new study from MIT Technology Review Insights. Register: https://atlan.com/context-conference/

## How does an unstructured data catalog work for PDFs and documents?

An unstructured data catalog works for PDFs and documents in four steps: it crawls the storage where files live, extracts text and metadata from each file, classifies the content against business terms and sensitivity labels, and publishes the result as a searchable, governed catalog entry. The file stays where it is. The catalog holds a record of what the file is, what it says, who owns it, and who may read it.

## Key takeaways

- The catalog has to build a description from three layers of metadata, and each layer comes from a different source.
- Without that step, a scanned contract enters the catalog as a file name and a size.
- The allowed answers come from the business glossary.
- A RAG system that retrieves the superseded version gives a confident, wrong answer.

---

An unstructured data catalog works for PDFs and documents in four steps: it crawls the storage where files live, extracts text and metadata from each file, classifies the content against business terms and sensitivity labels, and publishes the result as a searchable, governed catalog entry. The file stays where it is. The [catalog](https://atlan.com/what-is-a-data-catalog/) holds a record of what the file is, what it says, who owns it, and who may read it.

---

## What does an unstructured data catalog record about a PDF?

A table has a schema, so a catalog can describe it from column names alone. A PDF has no schema. The catalog has to build a description from three [layers of metadata](https://atlan.com/know/metadata-layer-for-ai/), and each layer comes from a different source.

| Layer | Example fields | Where it comes from |
|---|---|---|
| File metadata | name, path, size, file type, last modified, owner | the storage system, read during the crawl |
| Content metadata | title, author, summary, topics, entities, page count | parsing the file and, often, a language model |
| Governance metadata | business term, sensitivity label, steward, retention policy | classification rules and human review |

The file layer is cheap. On Databricks, reading a folder of PDFs in `binaryFile` format returns each file's `path`, `modificationTime` and `length`, plus its raw bytes ([Azure Databricks docs](https://learn.microsoft.com/en-us/azure/databricks/query/formats/binary)). That tells you a file exists. It tells you nothing about what the file says.

The content layer is where the work is. On Google Cloud, a model can read the raw PDF and write back a document title, author, version, summary and a confidence score from 0.0 to 1.0, as one cloud provider's [codelab on "dark data" PDFs](https://codelabs.developers.google.com/metadata-as-code-dark-data?hl=en) shows.

The governance layer turns a description into a control. The same build attaches a second record with a stewardship owner, a retention policy and compliance classifications such as FCC Part 15 ([codelab](https://codelabs.developers.google.com/metadata-as-code-dark-data?hl=en)).

The scale is why this matters. An August 2023 IDC white paper, sponsored by Box, puts [unstructured content](https://atlan.com/know/ai-agent/data-for-ai/structured-vs-unstructured-data-for-ai/) at 90% of enterprise data ([Box, citing IDC](https://blog.box.com/90-your-data-unstructured-and-its-full-untapped-value)).

---

## How does an unstructured data catalog extract text from PDFs and scanned documents?

Extraction starts with a crawl. The catalog connects to object storage, a file share or a document library, lists the files, and reads each one it has permission to read. One catalog's documentation [lists 12 formats](https://docs.ataccama.com/ataccama-one-agentic/latest/data-catalog/unstructured-data.html), including PDF, DOCX, PPTX, XLSX, HTML, Markdown and plain text.

Then the parser runs. Two kinds of PDF need two kinds of parser.

A born-digital PDF, exported from Word or a reporting tool, already holds a text layer. A parser reads it directly. [Apache Tika](https://tika.apache.org/), an open-source toolkit used for [search indexing](https://atlan.com/know/ai-agent/data-for-ai/enterprise-search-with-ai/) and content analysis, "detects and extracts metadata and text from over a thousand different file types."

A scanned PDF is a picture of a page. It needs [optical character recognition](https://atlan.com/know/ai-agent/data-for-ai/document-intelligence-for-enterprise/) (OCR). [Amazon Textract](https://docs.aws.amazon.com/textract/latest/dg/what-is.html), for example, detects typed and handwritten text and extracts forms and tables. Without that step, a scanned contract enters the catalog as a file name and a size.

Good parsers also keep structure. One [document partitioning](https://atlan.com/know/data-for-ai/document-parsing-for-ai-agents/) library splits a PDF into elements, with element types such as table, and tags each element with its file name, file type, page number, coordinates, language and a `parent_id` that records where the element sits in the document hierarchy ([partitioning walkthrough](https://unstructured.io/blog/optimizing-unstructured-data-retrieval)). That structure lets a later step point to the page a clause sits on, instead of only saying the file mentions it.

The output is text plus element metadata, ready for classification.

  WTF Is the Context Layer?
  A plain-language ebook on how data definitions, owners and policies become context that AI agents can use.
  Get the Ebook

---

## How does the catalog classify and tag documents after extraction?

Classification answers two questions about every file: what kind of document is this, and how sensitive is it?

Rules can do part of this: a regular expression for a card number, a keyword list for "contract." A pipeline can also send the extracted text to a [language model](https://atlan.com/know/what-is-a-large-language-model/) and ask it to choose from a fixed list of answers. On AWS, a pipeline can run the full loop, as a [public reference build](https://aws.amazon.com/blogs/big-data/improve-the-discoverability-of-your-unstructured-data-in-amazon-sagemaker-catalog-using-generative-ai/) from April 1, 2026 shows:

1. Textract extracts text from every page of the PDFs and images in an S3 bucket.
2. A model on Amazon Bedrock writes a summary and maps the text to business terms.
3. A [PII check](https://atlan.com/know/ai-agent/data-for-ai/how-to-handle-pii-in-ai-pipelines/) returns true or false, and the file gets a "sensitive" or "non-sensitive" term from a confidentiality glossary.
4. The catalog entry is updated and republished, so other teams can find it in search.

The allowed answers come from the [business glossary](https://atlan.com/know/ai-agent/how-to-manage-kpi-definitions-and-a-business-glossary-in-a-data-catalog/). A catalog that asks a model "what is this document?" with no list can get inconsistent labels. A catalog that asks "which of these document categories fits?" gets labels people can filter on. One commercial catalog defines both document categories and extraction entities, such as PII or product codes, as glossary terms, so every tag links to an agreed definition ([vendor documentation](https://docs.ataccama.com/ataccama-one-agentic/latest/data-catalog/unstructured-data.html)).

Model output is a draft. One cloud provider's [codelab](https://codelabs.developers.google.com/metadata-as-code-dark-data?hl=en) describes a production pattern in which any file scored below an example threshold of 0.85 confidence gets the status PENDING_STEWARD_REVIEW, so stewards can find and review the low-confidence files.

---

## How does an unstructured data catalog support RAG and AI agents?

[Retrieval-augmented generation (RAG)](https://atlan.com/know/what-is-rag/) feeds document passages to a model at question time, and it runs on the metadata the catalog builds.

[Chunks](https://atlan.com/know/chunking-strategies-rag/) inherit document metadata. When a pipeline splits a PDF into passages for a [vector index](https://atlan.com/know/what-is-a-vector-database/), each passage should keep its file name, page number, element type and sensitivity label. A retrieval query can then [filter before it ranks](https://atlan.com/know/advanced-rag-techniques/). A [public walkthrough on optimizing document retrieval](https://unstructured.io/blog/optimizing-unstructured-data-retrieval) filters on `{'Element Type': 'Table'}` to return only table passages.

[Access control](https://atlan.com/know/ai-agent-access-control/) has to travel with the text. If a PDF sits in a governed storage area, its chunks in a vector index should carry the same restriction. On Databricks, files sit in volumes and access is granted with SQL privileges such as `READ VOLUME` and `WRITE VOLUME` ([Azure Databricks docs](https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-privileges/privileges)).

[Natural-language search](https://atlan.com/know/conversational-search-enterprise-data/) runs over the summaries and tags the catalog wrote, so a search for "vendor contracts with auto-renewal" can match a file whose name says nothing. On AWS, the [reference build](https://aws.amazon.com/blogs/big-data/improve-the-discoverability-of-your-unstructured-data-in-amazon-sagemaker-catalog-using-generative-ai/) makes an asset findable in catalog search once its summary is published. [Hybrid search](https://atlan.com/know/hybrid-rag/) adds [keyword matching to meaning](https://atlan.com/know/semantic-search-vs-keyword-search/): a public [retrieval walkthrough](https://unstructured.io/blog/optimizing-unstructured-data-retrieval) pairs a BM25 keyword encoder with [dense embeddings](https://atlan.com/know/what-are-embeddings-ai-search/), so exact terms like a contract number still match.

The business case sits here. Gartner predicts that through 2026, organizations will abandon 60% of AI projects that are not supported by [AI-ready data](https://atlan.com/know/ai-readiness/ai-ready-data/) ([Gartner](https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk)). For document-heavy use cases, "AI-ready" means the catalog knows [which version of a file is current](https://atlan.com/know/ai-agent/context-versioning-for-ai-agents/), who owns it, and who may see it.

  Context Gap Calculator
  Estimate how much of your data estate is missing the context an AI agent needs to answer correctly.
  Calculate Your Context Gap

---

## What keeps a document catalog accurate over time?

Shared drives gain and lose files all day, so a document catalog decays fast. Five habits keep it honest.

[Re-crawl when files change](https://atlan.com/know/ai-agent/context-freshness/). Read each file's last-modified time and re-parse only changed files, so OCR spend stays low and approved tags stay put.

Find duplicates and [stale files](https://atlan.com/know/llm-knowledge-base-staleness/). The same contract often exists as `final.pdf`, `final_v2.pdf` and an email attachment. One unstructured data management guide recommends content checksums to detect exact duplicates, as one step in clearing redundant, obsolete or trivial (ROT) files ([unstructured data management guide](https://pentaho.com/insights/blogs/unstructured-data-management-strategies-tools-and-enterprise-solutions/)). A [RAG system](https://atlan.com/know/rag-accuracy-problems/) that retrieves the superseded version gives a confident, wrong answer.

Attach retention and ownership. Every governed file needs a named steward and a retention rule. Without an owner, nobody fixes a wrong tag. Without a retention rule, the catalog keeps indexing documents legal wanted deleted years ago.

Expect index lag. On Google Cloud, catalog search can lag updates by up to about three minutes, so one [codelab](https://codelabs.developers.google.com/metadata-as-code-dark-data?hl=en) builds in retries.

Track [lineage from file to agent](https://atlan.com/know/ai-agent/data-for-ai/data-lineage-for-ai/). Record which vector index or chunk set each file feeds, and which agents read from that index. When a policy PDF is replaced, the record shows which index to rebuild and which agents answered from the old text. One [lakehouse guide](https://unstructured.io/blog/your-lakehouse-handles-structured-data-brilliantly-unstructured-is-next) describes this as a single auditable path from document to agent.

Where to start: pick one AI use case and the one storage location that feeds it. Write the document categories and sensitivity labels into the glossary first, run extraction on that folder, and review every low-confidence tag before widening the crawl.

Related reading: how a catalog relates to [metadata management](https://atlan.com/data-catalog-vs-metadata-management/), to a [data warehouse](https://atlan.com/data-catalog-vs-data-warehouse/), and to a [semantic layer](https://atlan.com/know/ai-agent/semantic-layer/semantic-layer-vs-data-catalog/).

  Atlan in Action: Live Context Layer Demos
  Watch live sessions on how Atlan builds, reviews and governs context for AI agents.
  Watch a Live Demo

---

## How does Atlan catalog unstructured data?

Atlan treats documents as part of the same [context graph](https://atlan.com/know/what-is-a-context-graph/) as tables, so an agent can move from a revenue figure to the policy PDF that defines it. Atlan's position, set out in its [guide to unstructured data for AI](https://atlan.com/know/data-for-ai/unstructured-data-for-ai/), is to start with the files that feed deployed agents rather than crawl every share at once.

The file layer comes from storage connectors. Atlan's Amazon S3 crawler records buckets, folders and objects, with object properties that include key, size, last-modified time, content type, owner, version ID, Object Lock status and tags ([Atlan docs](https://docs.atlan.com/apps/connectors/storage/amazon-s3/references/what-does-atlan-crawl-from-s3)).

The classification layer can come from partners. On March 9, 2026, Atlan and BigID announced an integration in which BigID discovers and classifies structured and unstructured data, and its classification tags and policy alerts flow into Atlan's [catalog](https://atlan.com/know/data-catalog-for-ai/) and lineage ([how the joint solution traces unstructured data](https://atlan.com/know/unstructured-data-ai-lineage/); [PR Newswire](https://www.prnewswire.com/news-releases/bigid--atlan-introduce-the-first-unified-structured--unstructured-data-catalog-for-ai-governance-302708291.html)). A sensitivity label found in a document then follows that data downstream.

The review loop is the part with a measured result. Mastercard ran a two-week pilot in which [AI drafted descriptions](https://atlan.com/know/ai-agents-for-data-catalog/) and subject matter experts approved them before publishing. The team enriched about 30,000 assets and saved an estimated 6,200 steward hours, according to Zhenni Hu, Manager of AI and Data Governance at Mastercard ([Atlan recording](https://atlan.com/context-in-practice-mastercard-recording/)). Those assets sat across roughly 700 tables and views, not PDFs. The pattern is the same one a document catalog needs: the model drafts, a person approves, and the catalog records who approved it.

  Book a Demo

---

## FAQs about unstructured data catalogs

### What is an unstructured data catalog?

An unstructured data catalog is an inventory of files such as PDFs, Word documents, slides and scans. For each file it records storage details, extracted content such as a summary and topics, and governance details such as an owner and a sensitivity label. Teams use it to find documents, control access, and supply approved content to AI systems.

### Does an unstructured data catalog store the documents themselves?

No. The catalog stores metadata and a pointer to the file. The PDF stays in its original storage, whether that is an object store, a file share or a document library. Some pipelines also write extracted text or embeddings to a separate index, and that copy needs the same access rules as the source file.

### How does a catalog handle scanned PDFs?

A scanned PDF is an image of a page, so the catalog needs optical character recognition to read it. OCR services such as Amazon Textract detect typed and handwritten text and extract forms and tables. Without OCR, a scanned document enters the catalog with only its file name, size and dates, and search cannot find what it says.

### How accurate is AI tagging of documents?

Accuracy depends on how narrow the question is. A model choosing from a fixed glossary of document types gives labels people can filter on, and one catalog defines categories as glossary terms. Pipelines can attach a confidence score to each extraction. One cloud provider's codelab sends files below an example threshold of 0.85 to a steward.

### What is the difference between an unstructured data catalog and a vector database?

A vector database stores embeddings of text passages so a model can retrieve similar passages. A catalog records what each file is, who owns it, how sensitive it is, and which version is current. RAG systems work best with both: the catalog decides which files are eligible, and the vector index finds the right passage inside them.

---

## Sources

1. [Read binary files, Azure Databricks documentation (Microsoft Learn)](https://learn.microsoft.com/en-us/azure/databricks/query/formats/binary)
2. [Govern unstructured dark data with Gemini and Knowledge Catalog, Google Codelabs](https://codelabs.developers.google.com/metadata-as-code-dark-data?hl=en)
3. [90% of your data is unstructured, Box Blog (citing IDC, August 2023)](https://blog.box.com/90-your-data-unstructured-and-its-full-untapped-value)
4. [Unstructured data governance, catalog vendor documentation](https://docs.ataccama.com/ataccama-one-agentic/latest/data-catalog/unstructured-data.html)
5. [Apache Tika, Apache Software Foundation](https://tika.apache.org/)
6. [What is Amazon Textract?, AWS documentation](https://docs.aws.amazon.com/textract/latest/dg/what-is.html)
7. [Hybrid search with metadata for RAG systems, document-processing vendor blog](https://unstructured.io/blog/optimizing-unstructured-data-retrieval)
8. [Improve the discoverability of your unstructured data using generative AI, AWS Big Data Blog (April 1, 2026)](https://aws.amazon.com/blogs/big-data/improve-the-discoverability-of-your-unstructured-data-in-amazon-sagemaker-catalog-using-generative-ai/)
9. [Unity Catalog privileges reference, Azure Databricks documentation (Microsoft Learn)](https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-privileges/privileges)
10. [Lack of AI-ready data puts AI projects at risk, Gartner (February 26, 2025)](https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk)
11. [Unstructured data management: strategies, tools and enterprise solutions, data management vendor blog](https://pentaho.com/insights/blogs/unstructured-data-management-strategies-tools-and-enterprise-solutions/)
12. [Your lakehouse handles structured data. Unstructured is next, document-processing vendor blog](https://unstructured.io/blog/your-lakehouse-handles-structured-data-brilliantly-unstructured-is-next)
13. [BigID and Atlan introduce the first unified structured and unstructured data catalog for AI governance, PR Newswire (March 9, 2026)](https://www.prnewswire.com/news-releases/bigid--atlan-introduce-the-first-unified-structured--unstructured-data-catalog-for-ai-governance-302708291.html)