When to Use a Vector Database for Agentic Memory

---

A vector database lets an agent recall what it has seen by meaning instead of exact wording, which makes it a sensible store for memories that mostly accumulate: conversation logs, past actions, support history. It is a poor fit for facts that change, exact identifiers, and anything that must be ranked by recency. Atlan is the Context Layer for AI, and it covers the part a vector store cannot: the governed enterprise context, such as definitions, lineage, and ownership, that agents reach through Atlan’s MCP server while their own session memories stay in whatever store you choose.

Fits well	Additive memories such as logs, conversations, and agent activity, where an approximate recall is acceptable
Falls short	Facts that change, exact IDs, recency and importance ranking, relationships, and sensitive data
What it needs beside it	Metadata filters, exact or lexical search, re-ranking on age, and sometimes a graph
What every memory carries	Owner, scope, source, memory type, and timestamp
Not the same as	Agentic memory as a whole; extraction, consolidation, and deletion sit outside the vector index

---

Why use a vector database for agentic memory?

A large language model reasons from what it learned in training. Researchers behind the original retrieval-augmented generation paper describe that internal store as parametric memory, and note that providing provenance for a model’s decisions and updating its world knowledge remain open research problems. Retraining or fine-tuning to add one new fact is slow and expensive, so teams add non-parametric memory outside the model. The same paper pairs a pre-trained model with a dense vector index as its non-parametric memory.

A vector database is the most common form of that external memory because it does three things well:

• It updates by writing a row, with no retraining.
• It returns specific chunks of stored text, so the agent can point to where a memory came from.
• It grounds an agent in organizational knowledge, skills, and instructions.

Any lookup counts as non-parametric memory, whether it is keyword search, a database table, or a web request. Vector search adds the ability to match on meaning. Teams use vector databases heavily for retrieval-augmented generation, and using one alone as an agent’s memory is a different and harder job. The rest of this guide covers when it works, where it breaks, and how to build it so it holds up.

---

What is the difference between RAG and agentic memory?

Retrieval-augmented generation covers any retrieval an agent does while working: an API or tool call, a vector store, a vector and lexical hybrid, or a vector and graph hybrid. Memory is retrieved too. What changes is how the content gets there.

A RAG index is fed from source systems. The source stays authoritative, and a conflict gets resolved at the source. Agent memory is written by the agent, from conversations and actions. Nothing guarantees a new memory agrees with an old one. Memories repeat, drift, and contradict each other, so they need consolidation and cleanup that a RAG index mostly avoids.

That is why managed memory systems document extraction as a step of its own. Amazon Bedrock AgentCore Memory ships built-in strategies for semantic, user preference, summary, and episodic memory, and each strategy runs extraction, consolidation, or reflection steps driven by a system prompt. LangGraph’s memory documentation describes three kinds of long-term memory: semantic memory for facts, episodic memory for past experiences, and procedural memory for rules. Our guides cover types of agent memory, semantic versus procedural memory, and episodic memory in depth.

You can write extracted memories to Markdown files in object storage or a Git repository. Neither gives you fast retrieval by meaning. A vector index does, which is the case for putting it underneath.

---

How do vector databases handle recall?

Before a memory is written, it passes through an embedding model, after parsing and chunking if it is long. The resulting vector goes into the index, and that is what enables semantic search.

When the agent needs to remember something, it embeds its query and searches. Only the closest memories enter the context window, which saves space for the task. People and agents phrase the same thing differently, so matching on meaning recovers memories that keyword search would miss.

Two conditions mark a good fit. First, the memories mostly accumulate: logs, agent activity, conversations, with little chance that an old entry contradicts a new one. Second, an approximate answer is acceptable. If the agent recalls a close cousin of the right memory and nothing breaks, a vector store is fine.

---

Why doesn’t similarity search work for every agent memory?

Similarity search breaks down when a use case needs any of the following:

• Facts that change. A person’s job title is a fact with a lifespan. An embedding of the old title and one of the new title both stay in the index, and similarity cannot tell which one is current.
• Scoring on more than relevance. The Generative Agents paper retrieves memories by a weighted sum of recency, importance, and relevance, with each scaled to the range 0 to 1. Recency decays exponentially, importance is a score from 1 to 10 that a language model assigns, and relevance is cosine similarity. Similarity search supplies only the third.
• Exact lookups. Identifiers and keywords need exact matching. The test in the build section below shows a vector-only search ranking ticket #48231 above ticket #48213 when asked for #48213.
• Relationships. Embeddings capture how similar two items are, not how they connect. Questions like “which dashboards depend on this table” need a knowledge graph.
• Sensitive data. Storing PII or health data as memories is risky on its own, and embeddings are not a safe hiding place. A 2023 study of embedding inversion recovered 92% of 32-token text inputs exactly, and recovered full names from a dataset of clinical notes.

Most of these gaps close by pairing the vector index with the right second mechanism: a relational table for changing facts, a re-ranker for recency, exact or keyword search for identifiers, a graph for relationships, and redaction before anything is stored. Our comparisons of vector databases and knowledge graphs and of vector stores and graph databases go into the graph side, and context layer versus vector database covers what a vector store cannot hold at all. A vector database versus a metadata layer is the related question for enterprise data.

Atlan’s own walkthrough, What Is a Vector Database? Why RAG Still Returns Wrong Answers, covers the retrieval failures that carry over from RAG into agent memory.

---

How do the popular vector stores handle memory requirements?

Every store embeds and searches. The differences that matter for memory are how each one scopes a user’s memories, filters on metadata, and mixes in exact or keyword matching.

Store	Scoping a user’s or agent’s memories	Metadata filtering	Lexical or hybrid search
pgvector (Postgres)	A memory is a row; scope with a WHERE clause on ordinary columns, or a partial index	Standard SQL; the README warns that filtering applies after an approximate index scan, and offers iterative scans	Combine with Postgres full-text search; the README points to Reciprocal Rank Fusion for merging
Pinecone	One namespace per tenant; Pinecone describes it as physical isolation	Operators such as $eq, $gte, $in, $and; 40 KB of metadata per record	Dense and sparse vectors in one index, or separate indexes merged client-side
Qdrant	Recommends one collection with a tenant field in the payload and a keyword index flagged is_tenant	must, should, range, and datetime range conditions	Prefetch with RRF or DBSF fusion, plus formula queries with decay functions
Weaviate	Multi-tenancy puts each tenant on a separate shard; off by default	Filters apply to hybrid queries	BM25F keyword search fused with vector search; relative score fusion is the default since v1.24
Chroma	Filter on a metadata field such as user_id	where operators including $and, $gte, and $in	where_document with $contains filters on document text
Milvus	Four strategies: database, collection, partition, or partition key level	Filter expressions on scalar fields	Hybrid search across dense and sparse vector fields, merged with RRF or weighted reranking

Two rows deserve a closer read. Pinecone’s documentation says a query against one tenant’s namespace costs far less than the same query on a shared namespace, because a metadata filter on a shared namespace still scans all data. If every user is a tenant, namespaces matter more than filters. In pgvector, the README states that filtering is applied after the index scan, so a selective WHERE user_id = ... on an approximate index can return fewer rows than you asked for; its iterative scan setting exists for that case. Both are scoping details that decide whether one user can ever see another’s memories, which makes them governance questions as much as performance ones.

You can also use Amazon S3 Vectors, which adds native vector storage to S3 with metadata filtering. Top vector databases for enterprise AI compares the products on infrastructure and cost.

---

How do you build an agentic memory layer on a vector index?

Three requirements apply to every store:

1. Keep provenance, scope, and ownership with every memory. Who wrote it, for which user or agent, from which source.
2. Record the memory type. Semantic, episodic, or procedural, so the agent can ask for the right kind.
3. Write the time. Without a timestamp you cannot filter by age, decay by age, or retire old memories.

In Postgres, a memory is a row, and the embedding sits in a vector column next to those fields. This schema follows the syntax in the pgvector README; set the vector dimension to match your embedding model.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE memories (
    id          bigserial PRIMARY KEY,
    user_id     text NOT NULL,
    agent_id    text NOT NULL,
    memory_type text NOT NULL,   -- semantic | episodic | procedural
    source      text NOT NULL,   -- session, ticket, or document the memory came from
    content     text NOT NULL,
    created_at  timestamptz NOT NULL DEFAULT now(),
    embedding   vector(384)
);

CREATE INDEX ON memories USING hnsw (embedding vector_cosine_ops);

-- Scoped recall: filter by owner and type, order by cosine distance
SELECT content, source, created_at
FROM memories
WHERE user_id = 'u1' AND memory_type = 'episodic'
ORDER BY embedding <=> $1
LIMIT 10;

The same pattern in Python, with Chroma, runs in memory with no server. It uses Chroma’s where filter for scope and type, a cosine collection set through the documented configuration parameter, and a recency re-rank on top. Install it with pip install chromadb. Chroma downloads its default embedding model the first time you run it.

import time
import uuid

import chromadb

client = chromadb.Client()  # in-memory; use chromadb.PersistentClient(path="./memory-db") to keep data
memories = client.get_or_create_collection(
    "agent_memories", configuration={"hnsw": {"space": "cosine"}}
)


def remember(text, *, user_id, agent_id, memory_type, source):
    """Write one memory with the fields recall depends on later."""
    memories.add(
        ids=[str(uuid.uuid4())],
        documents=[text],
        metadatas=[{
            "user_id": user_id,          # scope and ownership
            "agent_id": agent_id,
            "memory_type": memory_type,  # semantic | episodic | procedural
            "source": source,            # provenance: session, ticket, document
            "created_at": int(time.time()),
        }],
    )


def recall(query, *, user_id, memory_type=None, k=3, half_life_days=30, pool=20):
    """Filter by scope, search by meaning, then re-rank with recency."""
    where = {"user_id": user_id}
    if memory_type:
        where = {"$and": [where, {"memory_type": memory_type}]}
    hits = memories.query(query_texts=[query], n_results=pool, where=where)
    now = time.time()
    scored = []
    for doc, meta, dist in zip(
        hits["documents"][0], hits["metadatas"][0], hits["distances"][0]
    ):
        age_days = (now - meta["created_at"]) / 86400
        recency = 0.5 ** (age_days / half_life_days)  # exponential decay
        relevance = 1 - dist                          # cosine distance -> similarity
        scored.append((relevance + recency, doc, meta["source"]))
    return sorted(scored, reverse=True)[:k]


remember("Priya approved Looker access for the finance team",
         user_id="u1", agent_id="support", memory_type="episodic", source="ticket-48231")
remember("Priya fixed the Okta login timeout issue",
         user_id="u1", agent_id="support", memory_type="episodic", source="ticket-48213")
remember("Always escalate SSO outages to the identity team",
         user_id="u1", agent_id="support", memory_type="procedural", source="runbook-12")
remember("Other tenant: reset the Okta timeout",
         user_id="u2", agent_id="support", memory_type="episodic", source="ticket-1")

for score, doc, source in recall("what did we do about the Okta timeout?", user_id="u1"):
    print(f"{score:.3f}  {doc}  [{source}]")

We ran this script against Chroma 1.5.9. It returned the Okta fix first, the escalation rule second, and the Looker approval last, and it never returned the other tenant’s memory, because the user_id filter runs before the ranking:

1.667  Priya fixed the Okta login timeout issue  [ticket-48213]
1.270  Always escalate SSO outages to the identity team  [runbook-12]
1.111  Priya approved Looker access for the finance team  [ticket-48231]

Recency has no built-in switch in this setup, so the script retrieves a pool of 20, scores each memory, and keeps the top three. The decay follows the idea from the Generative Agents paper, using a 30-day half-life that you should tune to how fast your memories go stale. Qdrant can do the equivalent inside the query: its formula query supports exp_decay on a datetime field, available from v1.14.0.

Where exact identifiers still break

Vector search is weakest on near-identical identifiers. We indexed three short memories and searched for ticket #48213:

import chromadb

client = chromadb.Client()
tickets = client.get_or_create_collection(
    "tickets", configuration={"hnsw": {"space": "cosine"}}
)
tickets.add(
    ids=["a", "b", "c"],
    documents=[
        "Ticket #48231: approved Looker access for the finance team",
        "Ticket #48213: fixed the Okta login timeout issue",
        "Shared the churn report in #finance on Slack",
    ],
)

query = "ticket #48213"
r = tickets.query(query_texts=[query], n_results=3)
r_exact = tickets.query(
    query_texts=[query], n_results=3, where_document={"$contains": "48213"}
)

Vector-only search put ticket #48231 first, at a cosine distance of 0.364, and ticket #48213, the one we asked for, second at 0.619. Adding the where_document filter, which keeps only documents whose text contains the string 48213, returned ticket #48213 alone. A substring filter is not full hybrid search. A store with BM25 or sparse-vector support, such as the Weaviate, Qdrant, Pinecone, and Milvus options in the table above, scores keyword and vector matches together. For identifiers, though, even a plain exact-match step beats pure similarity.

Beyond retrieval, the hard part of agent memory is upkeep: keeping memories consistent, updating the ones that change, and retiring the ones that contradict newer facts. None of that comes from the vector index. Our guides on the memory ingestion pipeline, how memory systems work, building a memory layer, and implementing long-term memory walk through that side.

---

Should you build on a vector database or use a managed memory service?

A managed service takes over extraction, consolidation, and recall, and usually stores the result in a vector database underneath. Mem0’s documentation describes three stores behind each memory: a SQL database as the source of truth, a vector database for embeddings, and an entity store that boosts memories sharing entities with the query. Its open-source overview lists a local Qdrant as the default vector store when you run it as a library. LangGraph’s store takes a different approach: memories are JSON documents under a namespace and a key, and you configure an embedding model on the store to search them by meaning. Amazon Bedrock AgentCore Memory, covered above, runs the extraction itself.

Building on a vector database yourself gives you control of the schema, the scoping rules, and what deletion means. You also write the extraction prompts and the conflict handling. Compare the options in best AI agent memory frameworks, LangGraph memory versus Mem0, and how to choose a memory architecture. A managed service still has no view into your enterprise systems, so it remembers what the agent saw and nothing else.

---

How does Atlan bring enterprise context to agentic memory?

An agent’s memory holds what it learned in its own sessions. It does not hold what the business already knows: which definition of revenue is certified, where a column comes from, who owns a table. You can wire each agent to each source through its own API and maintain that pipeline, or connect once to a context layer.

The Atlan MCP server is that connection. Its documentation says agents can search enterprise context, traverse end-to-end lineage, read governed definitions and glossaries, and run SQL at https://mcp.atlan.com/mcp. With OAuth, each person signs in with their own account, so tools run under that person’s permissions. Behind the server, the context layer organizes that knowledge in an Enterprise Data Graph and an Active Ontology, and teams assemble Context Repos in Context Engineering Studio, which the documentation describes as a workspace for assembling, testing, and deploying the business context agents need.

The two layers split cleanly. The vector store keeps session memories with an owner, a type, and a timestamp. The context layer supplies the facts that should come from a source of record, so they are looked up when needed and never stored as a memory that can drift. Memory layer versus context layer and active metadata for agent memory explain the split, and the context layer as enterprise memory shows it end to end.

---

Moving forward with vector memory

A vector database works as agent memory when the memories pile up rather than contradict each other, and when a near miss costs little. It strains when memories hold facts that change, need recency or importance scoring, depend on exact identifiers, or contain sensitive data. Build with metadata filters from the first commit, add exact or hybrid search before you need it, and treat extraction and cleanup as the real work.

Where the memories are about the business itself, the better question is whether they should be memories at all. Do your agents need to remember a definition, or look it up? How much of what sits in your memory store today would be wrong if its source changed last week?

---

FAQs about vector databases for agentic memory

1. Is a vector database the same thing as agentic memory?

No. A vector database is a storage and retrieval engine. Agentic memory is the whole system around it: deciding what to remember, extracting it from sessions, consolidating duplicates and contradictions, and retiring stale entries. A vector database can serve as the storage layer, and agent memory can also live in other stores.

2. Which vector database should you use for agent memory?

Pick on the requirements the memory has, not on a brand. If you already run Postgres and want memories next to relational data, pgvector keeps everything in one place. If you want per-user isolation, Pinecone namespaces, Weaviate tenants, and Qdrant payload partitioning each handle it differently. If you need lexical and semantic search together, check that the store has hybrid search, as Weaviate, Qdrant, Pinecone, and Milvus document.

3. Why does similarity search fail for some agent memories?

Similarity search returns what is closest in meaning, which is only one signal. It does not know that a newer fact replaced an older one, it can rank a near-identical identifier above the exact one, and it carries no importance or recency score. You add metadata filters, exact-match or lexical search, re-ranking, or a graph to cover those gaps.

4. How do you handle recency in a vector memory store?

Store a timestamp with every memory, retrieve a larger candidate pool by similarity, then re-rank with a decay on age. The Generative Agents paper combines recency, importance, and relevance, each scaled to the range 0 to 1, with an exponential decay for recency. Some stores, such as Qdrant, can apply a decay function inside the query itself.

5. Should you use a managed memory service instead of building on a vector database?

A managed service such as Amazon Bedrock AgentCore Memory or Mem0 takes over memory extraction and consolidation, usually on top of a vector database. Building it yourself gives you control over the schema, scoping, and deletion rules, at the cost of writing and maintaining the extraction and conflict-handling logic.

6. How does Atlan fit with a vector memory layer?

Atlan does not replace the vector store that holds what an agent learned in its own sessions. It supplies the governed enterprise context the agent needs to act: definitions, lineage, and ownership, which agents reach through the Atlan MCP server under the permissions set in Atlan.

---

Sources

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al.), arXiv. https://arxiv.org/abs/2005.11401
2. Built-in strategies, Amazon Bedrock AgentCore Developer Guide, AWS. https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/built-in-strategies.html
3. Memory overview, LangChain Docs. https://docs.langchain.com/oss/python/concepts/memory
4. Add memory (long-term memory with semantic search), LangChain Docs. https://docs.langchain.com/oss/python/langgraph/add-memory
5. Generative Agents: Interactive Simulacra of Human Behavior (Park et al.), arXiv. https://arxiv.org/abs/2304.03442
6. Text Embeddings Reveal (Almost) As Much As Text (Morris et al.), arXiv. https://arxiv.org/abs/2310.06816
7. pgvector: Open-source vector similarity search for Postgres, GitHub. https://github.com/pgvector/pgvector
8. Implement multitenancy, Pinecone Docs. https://docs.pinecone.io/guides/index-data/implement-multitenancy
9. Filter by metadata, Pinecone Docs. https://docs.pinecone.io/guides/search/filter-by-metadata
10. Hybrid search, Pinecone Docs. https://docs.pinecone.io/guides/search/hybrid-search
11. Hybrid search, Weaviate Documentation. https://docs.weaviate.io/weaviate/search/hybrid
12. Multi-tenancy, Weaviate Documentation. https://docs.weaviate.io/weaviate/manage-collections/multi-tenancy
13. Hybrid queries, Qdrant Documentation. https://qdrant.tech/documentation/concepts/hybrid-queries/
14. Filtering, Qdrant Documentation. https://qdrant.tech/documentation/concepts/filtering/
15. Multitenancy, Qdrant Documentation. https://qdrant.tech/documentation/guides/multiple-partitions/
16. Hybrid search with multiple vector fields, Milvus Documentation. https://milvus.io/docs/multi-vector-search.md
17. Multi-tenancy, Milvus Documentation. https://milvus.io/docs/multi_tenancy.md
18. Metadata filtering, Chroma Docs. https://docs.trychroma.com/docs/querying-collections/metadata-filtering
19. Configure a collection, Chroma Docs. https://docs.trychroma.com/docs/collections/configure
20. Query and get data, Chroma Docs. https://docs.trychroma.com/docs/querying-collections/query-and-get
21. Amazon S3 Vectors, AWS. https://aws.amazon.com/s3/features/vectors/
22. Memory types and storage, Mem0 Documentation. https://docs.mem0.ai/core-concepts/memory-types
23. Open source overview, Mem0 Documentation. https://docs.mem0.ai/open-source/overview
24. Set up Atlan MCP, Atlan Documentation. https://docs.atlan.com/product/capabilities/atlan-ai/how-tos/remote-mcp-overview
25. What is Context Engineering Studio, Atlan Documentation. https://docs.atlan.com/platform/context-engineering-studio/what-is-context-engineering-studio
26. What Is a Vector Database? Why RAG Still Returns Wrong Answers, Atlan (atlanHQ) on YouTube. https://www.youtube.com/watch?v=awPqwuZw9NI
27. Boolean expression rules, Milvus Documentation. https://milvus.io/docs/boolean.md