Skip to main content

How to use analyst reports to shortlist a data catalog (and the mistakes to avoid)

Emily Winks, Data Governance Expert, Atlan
Data Governance Expert
Updated:
|
Published:
17 min read

Key takeaways

  • A tool missing from a chart may simply have missed the inclusion bar.
  • The evaluation cut-off, not the cover date, tells you which product version was scored.
  • Write your own must-haves and knockouts before you reopen the chart.
  • Treat a vendor's press release or badge as that vendor's selection, not the whole finding.

How do you use analyst reports to shortlist a data catalog?

Use an analyst report to build a long list, then let your own weights and a hands-on test build the short list. Read the method before the chart, check who met the inclusion bar, check the evaluation date, swap in your own weights, and test the survivors on your own data. Atlan, an AI Context Platform, publishes two evaluation skills, plain-text checklists a person or AI assistant can follow, to apply the same tests to every candidate.

The 5 steps:

  • Read the method before the chart
  • Check who got on the chart and who was gated out
  • Check the evaluation cut-off, not just the cover date
  • Replace the weights with your own must-haves
  • Test the short list on your own data

See a context layer working on live metadata


An analyst chart is a long-list tool. Check its method, inclusion bar and evaluation cut-off, put your own weights on the criteria, and let a proof of concept on your data pick the short list. BARC’s 2026 Score compares 10 vendors; Forrester’s Data Governance Wave evaluated 13 on 28 criteria. Atlan, an AI Context Platform, publishes two plain-text evaluation skills for the hands-on step.

Build your shortlist matrix


Give it the tools on your long list and the systems they must cover. It returns a matrix on connector coverage, where lineage stops, who maintains the metadata and what happens if you leave, with untestable items marked requires trial. Read the skill.

Paste into a new chat

Use the skill at https://atlan.com/skills/catalog-tool-shortlist.md to build an evaluation matrix for my data catalog long list. Ask me for whatever it needs.

Run once in a terminal

curl -fsSL --create-dirs \
  -o ~/.agents/skills/catalog-tool-shortlist/SKILL.md \
  https://atlan.com/skills/catalog-tool-shortlist.md

For an agent

curl -fsSL https://atlan.com/skills/catalog-tool-shortlist.md

You lead data governance or data architecture, and someone has forwarded an analyst chart as the case for a new catalog. Read as a ranking, that chart can put a year of budget and migration work behind a tool nobody has tested on your systems. The analyst picks who qualifies, which criteria count and how much each weighs, before any vendor gets a score. New to the category? Start with what a data catalog does and how an AI data catalog differs. Then work through the five steps and check your plan against the eight common mistakes.


What does an analyst placement measure, and what does it leave out?

An analyst placement is an opinion about a defined slice of a market, scored on axes the firm chose, so it answers a narrower question than “which catalog is right for us”.

According to Forrester’s Wave methodology (Forrester, 2026), a Wave “is a guide for buyers considering their purchasing options in a technology marketplace and is based on our analysis and opinion”. IDC’s MarketScape page (IDC, 2026) says a MarketScape “assesses a specific offering from a particular vendor based on an inclusion criteria”.

BARC draws the line at ownership. According to BARC’s 2026 release (BARC, 2026), “Only vendor-owned functionality is evaluated, not OEM or partner solutions.”

How the firms describe their methods


Firms are listed alphabetically: BARC, Forrester, Gartner, IDC. The order says nothing about quality. The table covers BARC, Forrester and IDC; for Gartner, read the firm’s own method page on gartner.com.

Firm Axes or bands What the firm publishes about its method Where to read it
BARC Two axes, Portfolio Capabilities and Market Execution; five segments: Dominators, Market Leaders, Challengers, Specialists, Entrants Inclusion criteria, use cases and evaluation basis for the 2026 Data Intelligence Platforms Score; the Score itself is sold on request BARC Scores and the Score’s product page
Forrester Three bands, Leader, Strong Performer and Contender, for Waves kicked off on or after July 1, 2024; customer feedback replaced market presence on the graphic A public methodology that Forrester says it applies “consistently across all participating vendors” The Forrester Wave methodology
IDC Two axes, Capabilities (short term) and Strategy (a 3 to 5 year timeframe); four segments: Leaders, Major Players, Contenders, Participants; bubble size shows market share How the chart is built, on its public MarketScape page, and how vendors may describe results, in its promotional guidelines IDC MarketScape and the promotional guidelines

The axes differ by firm, so a top-right spot on one chart makes a different claim from a top-right spot on another. The steps below apply to any firm’s report, Gartner’s included.

For one firm’s chart read closely, see what the BARC Score for data intelligence platforms measures. The data catalog RFP template shows one way to fold several reports into a weighted scoring rule.

A chart answers who qualified, what counted and how much it weighed for the analyst’s market. Your shortlist needs those three answers for your team.


How to use analyst reports to shortlist a data catalog, in five steps

The five steps run from reading the report to testing on your own data.

  1. Read the method before the chart.
  2. Check who got on the chart.
  3. Check the date, then the cut-off behind the date.
  4. Replace the analyst’s weights with yours.
  5. Test the short list on your own data.

Before you start


Have the full report or the firm’s public method pages, a list of the systems the catalog must cover, and the names of the people who will run the trial.

Step 1: Read the method before the chart


Open the firm’s method page before any vendor position. According to Forrester (2026), “The analyst determines the inclusion criteria, the evaluation criteria, and the scoring framework”. Its criteria are meant to be “differentiating, rather than exhaustive”, so a capability you need may not be scored.

Each firm publishes different detail:

  • Forrester publishes its Wave method: how vendors take part, which inputs it uses (a questionnaire, a strategy and product demo session, and reference customers) and that the analyst weights criteria by importance.
  • BARC published the inclusion gates and the evaluation basis for its 2026 Score. That basis was “a detailed questionnaire, analyst briefings, and independent market research”, per BARC’s 2026 release (BARC, 2026).
  • IDC explains the chart, its axes and its segments on the public MarketScape page.

Note what the analyst measured, what it skipped, and whose product scope it counted. Done when: you can say in one sentence what the chart scores and what it leaves out.

Step 2: Check who got on the chart


Absence from a chart is not a low placement. Inclusion criteria decide who appears, and BARC’s, for example, combine scope gates with size gates.

According to BARC’s product page for the 2026 Score (BARC, 2026), platforms had to offer “data cataloging, a business glossary, support the development of context for AI, promote collaboration”. Vendors also needed:

  • a European branch and at least one other global location
  • at least 100 people
  • at least EUR 10 million in annual license and maintenance revenue from the evaluated suite
  • at least 75 customers in the data intelligence sector

BARC also excluded several platform-native catalogs because they “are not standalone platform offerings”, and one vendor that met the criteria asked to sit out the 2026 edition. Forrester works the other way on participation: it “reserves the right to evaluate products and services, regardless of vendor participation.”

Read the inclusion and participation notes before inferring why a vendor is missing. Then start the long list from the chart and add tools the gates kept out if they fit your stack. Agent context layer tools compared and the AI context platform guide cover candidates a catalog chart may not hold. If a platform-native option is on the table, weigh a full-stack AI platform against a best-of-breed context layer. Done when: every tool on your long list either met the gates or has a written reason to stay.

Step 3: Check the date, then the cut-off behind the date


The cover date tells you when a report came out. The evaluation cut-off tells you the latest product it could describe. According to Forrester (2026), “We evaluate products that are generally available no later than the time of the Wave questionnaire due date.” Forrester also says “We do not reevaluate companies based on changes announced during the evaluation period”, and a Wave runs “approximately 18 weeks from kickoff to publication”.

In a blog dated July 23, 2025, Forrester analyst Raluca Alexandru wrote that “Over the past six months, we evaluated 13 of the most significant data governance solution providers” on “28 criteria”, per the Forrester Wave for data governance solutions blog (Forrester, 2025). Read in November 2026, that evaluation window opened more than 20 months earlier. Look for the questionnaire or data-collection date in the report’s method notes, and ask each vendor which release was scored.

The market inside a report also moves between editions. BARC’s February 2025 data intelligence platforms release named 14 vendors, per BARC’s 2025 release (BARC, 2025); the May 2026 Score compares 10.

Atlan’s guidance, not an analyst statement: write down the cover date and the cut-off, then re-check every product claim you rely on against the vendor’s current documentation. Apply the same date to any build vs buy criteria for metadata tooling and to the three-year cost of build, buy or bundle. Done when: each report you rely on has two dates written next to it.

Step 4: Replace the analyst’s weights with yours


Every chart carries weights set for a whole market, and your team’s weights will differ. IDC’s MarketScape page says “Divisions between chart segments (Leaders, Major Players, Contenders, and Participants) change based on weightings between the two axes, Capabilities and Strategies.” Forrester’s analyst weights criteria “according to importance”, and the Wave offers “an interactive experience that allows clients to customize Forrester’s evaluation findings”. BARC’s Portfolio Capabilities axis is built from “a series of weighted sub-criteria”, per BARC Scores (BARC, 2026).

BARC states the consequence plainly in its 2026 release: “Putting governance and trustworthiness front and center leads to a different shortlist than wanting to make existing data assets available faster.”

Write five to eight requirements before you reopen the chart, each marked knockout or weighted preference. A knockout removes a tool on its own. Screen budget, deployment model, security review and metadata export early, so no proof of concept is spent on a tool you cannot buy or leave. An illustrative worksheet:

Requirement Knockout or preference Weight Evidence you will accept
Column-level lineage from your transformations to your BI fields Knockout n/a Your own pipeline, traced in a trial
Metadata export in a format another system can read Knockout n/a An export file you open yourself
Glossary terms linked to columns in two systems Preference 40 A live search on your data
An agent reads the metadata at runtime through an open interface Preference, or a knockout if AI readiness drove the purchase 60 A test call from your own agent

Starting points for the list:

Weights for preferences add up to 100, and the numbers above are examples only. Score each preference 0 to 3 on what your team saw, multiply by its weight, and keep the two or three highest totals that clear every knockout. Where a report lets clients customize findings, enter your weights there and see whether the order changes. Done when: every tool on the long list has a total, and the knockouts have removed what they should.

Step 5: Test the short list on your own data


BARC’s 2026 release recommends the last step: “A structured proof-of-concept phase significantly reduces the error rate in selection.” A report cannot see your systems, so this step settles what only your systems can show.

Atlan’s catalog-tool-shortlist skill, a plain-text method a person or AI assistant can follow, asks four questions a chart cannot answer:

  1. How many of your systems does each tool cover natively, and does “covered” mean schema only or schema plus lineage plus usage?
  2. Where does lineage stop?
  3. Who maintains the metadata after launch, and what makes them?
  4. What happens to the metadata if you leave?

The skill marks “anything not verifiable without a trial” as “requires trial” rather than guessing. Lineage is a common example, as a lineage proof of value and the column-level lineage test for Power BI show.

The governance-platform-evaluation-checklist skill scores only what your own team saw, 0 to 3: “A demo tops out at 2. Only their own data earns a 3.” A report’s findings, however well sourced, describe what someone else saw, so treat them as claims until your team sees the behavior itself.

Run the test on real jobs: self-service analytics, whether a tool can sit on top of your existing catalogs, and how far you can extend your current catalog for AI readiness.

Atlan’s rule: reports for the long list and the market’s vocabulary; your own weights and the proof of concept for the short list. When they disagree, investigate first. Give every vendor the same setup time and data, and record where vendor engineers stepped in. For behavior tested that way, the test outweighs the chart; support, viability and contract terms still need their own checks. Done when: every requirement for every shortlisted tool carries a score your own team saw, or a “requires trial” mark.


What mistakes do buyers make when they use analyst reports?

Eight mistakes are worth watching for, each a gap between what a chart shows and what the analyst’s own words say. The same checks apply to head-to-head reading, such as alternatives to an open-source catalog, a comparison of two catalogs, what each of two catalogs governs or catalogs compared for one platform.

Mistakes are ordered by the step of the shortlist where they happen: reading, inclusion, date, weighting, testing.

Mistakes before you weight anything


# Mistake What the analyst says What to do instead
1 Reading a placement as a rank order IDC tells vendors not to call a MarketScape a ranking: “It is not.” It also asks them to avoid superlatives such as “best” Read segments as groups, not positions
2 Relying on a report you have not read BARC’s 2026 Score is sold on request; Forrester’s methodology is public Read the full report through a subscription, a purchase or a vendor-supplied reprint you check for edition and date; treat a press release or badge as that vendor’s selection
3 Forgetting the chart only holds vendors that met the inclusion bar BARC gates on size and scope and excludes tools that are “not standalone platform offerings”; Forrester’s inclusion criteria “determine which vendors to include” Add tools that missed the gates but fit your stack
4 Comparing editions as if the market held still BARC’s 2025 release named 14 vendors; the 2026 Score compares 10 Before reading a vendor’s move between editions as a product change, check whether the vendor set changed
5 Reading an old report as today’s product Forrester scores products generally available by its questionnaire due date and does not re-evaluate later changes Re-check claims against current docs

Mistakes while weighting and testing


# Mistake What the analyst says What to do instead
6 Treating a top-right spot as a fit for your use case BARC: governance-first priorities lead “to a different shortlist”; its Specialists “win over customers with selected features and functionality” Re-score the top-right vendors against your priority; a Specialist may fit better
7 Using the chart’s weights instead of yours IDC: segment divisions “change based on weightings”; Forrester weights “according to importance” Score against your own must-haves
8 Skipping the hands-on test BARC: a structured proof of concept “significantly reduces the error rate in selection” Test the short list on your own data

Seven of the rows can be fixed at a desk. The last one needs your own data.


Why choose a catalog on how you will run it, not where it sits on a chart?

The AI barriers executives name sit in how data is governed and understood. A placement scores those capabilities across a market; it cannot show how they hold up on your systems, with your owners and your access rules.

MIT Technology Review Insights, in association with Atlan, polled 250 executives; according to the report’s landing page (MIT Technology Review Insights and Atlan, 2026), “68% of organizations rank governance, business understanding, or trustworthy outputs as their biggest AI barrier.”

The survey covers AI barriers, not catalog selection. When AI is a selection priority, it points at what to test: who owns the metadata, how it stays current, and who can read it. For AI readiness, the catalog-tool-shortlist skill adds one criterion: whether an agent can read the metadata at runtime through an open interface.

Test that interface in the trial with your own agent and access rules. The difference between a data catalog and a context layer, the idea of a context catalog, and what AI agents need from a data catalog help frame the test.


How does Atlan help you run the shortlist?

Atlan, an AI Context Platform, publishes the two skills used in step 5 so a team can run the same tests on every candidate. Each is a vendor-authored, text-only method that a person or an AI assistant can follow, and you can change its criteria to fit your stack.

  • catalog-tool-shortlist builds an evaluation matrix from your own candidates and your own stack. In its own words, it “does not supply a vendor ranking”, and that holds for Atlan too.
  • governance-platform-evaluation-checklist runs 22 behavior tests, each scored 0 to 3 on the evidence your team saw, with three to five knockouts agreed before the first demo. “Any knockout below 2 removes the platform.” For a knockout only a trial can show, such as lineage on your own pipelines, hold out for a 3.

Run them on Atlan the same way you run them on every other candidate.

If AI use is in scope, the trial may reach past the catalog to a context layer; whether you keep, extend or add to your catalog depends on the gaps it finds. Check whether a tool acts on metadata like an agentic data catalog, whether agents reach it as an MCP-connected data catalog, and how it keeps KPI definitions and a business glossary. Test how it works with your semantic layer and any data marketplace.

For how other teams use a catalog once it is chosen, see data catalog examples across industries.


What to do after the shortlist

A short list that survived a proof of concept is ready for the full cost and migration plan. Price it with the open source vs commercial total cost of ownership worksheet, check whether the winner can sit on top of the catalogs you already run, and compare build, buy and bundle costs over three years.

For a longer worksheet, get the 5-step guide to evaluating a data catalog. It runs from the problem a catalog solves through evaluation keys to choosing a partner.

One question no chart shows: who on your team will keep the metadata current a year after launch?


FAQs about using analyst reports to shortlist a data catalog

1. Are analyst reports enough to choose a data catalog?


No. BARC describes its Score as a foundation for a long or short list, and offers selection advisory alongside it. Connector coverage against your own systems, where lineage stops on your pipelines and who keeps the metadata current on your team still need a test on your own data.

2. How old is too old for an analyst report?


Judge age by the evaluation cut-off, not the cover date. Atlan’s guidance is to note two dates, the cover date and the evaluation cut-off, and to re-check every product claim you rely on against the vendor’s current documentation. Forrester, for example, evaluates only products generally available by its questionnaire due date, and a Wave takes about 18 weeks from kickoff to publication.

3. Does a Leader placement mean a tool is the best one?


No. IDC tells vendors not to describe a MarketScape as a ranking, because it is not one. IDC also says the lines between its Leaders, Major Players, Contenders and Participants move with the weighting between its two axes. A Leader placement reflects the analyst’s weights, which may not match yours.

4. Can a vendor missing from a report still be the right choice?


Yes. Inclusion criteria decide who appears. BARC’s 2026 Score required, among other gates, at least 100 people, EUR 10 million in annual license and maintenance revenue from the evaluated suite and 75 customers in the data intelligence sector, and it excluded tools it judged not to be standalone platform offerings. Absence alone does not show weaker capability: check whether the cause was a scope gate, a size gate, a standalone-offering exclusion or a vendor opting out, and weigh each cause yourself.

5. Which analyst reports cover data catalogs?


Firms file catalogs under different category names. BARC covers them in its Score for data intelligence platforms; the 2026 edition, published in May 2026, compares 10 vendors. Forrester’s closest recent example is a data governance Wave from 2025 that evaluated 13 providers on 28 criteria. Check each firm’s research index for its current category name and edition date.

6. How do I turn an analyst report into an RFP?


Borrow the report’s criteria as a vocabulary and set the weights yourself. List five to eight must-haves and knockouts from your own systems and use cases, and send vendors the tests you will run in the proof of concept. Score what you saw, not what the vendor or the report claimed.


Sources

  1. The Forrester Wave Methodology, Forrester
  2. The Forrester Wave: Data Governance Solutions, Q3 2025 (blog), Forrester
  3. BARC Scores, BARC
  4. BARC Score Data Intelligence Platforms, BARC
  5. From Data Catalog to Context Provider for AI Agents, BARC
  6. BARC Score: AI and Automation Propel Data Intelligence Platforms Forward, BARC
  7. IDC MarketScape, IDC
  8. IDC MarketScape Promotional Guidelines, IDC
  9. What’s really holding enterprise AI back?, MIT Technology Review Insights and Atlan
  10. catalog-tool-shortlist, Atlan
  11. governance-platform-evaluation-checklist skill, published on Data Governance Platform Evaluation Criteria: 22 Tests to Run, Atlan

Share this article

signoff-panel-logo

Atlan is the Context Layer for AI. It translates business knowledge, including data definitions, working procedures, and governance policies, into context AI can actually use. This knowledge lives in a single Enterprise Data Graph that every team and AI agent can reach.

In Atlan's research report, rich semantic metadata improved the accuracy of AI-generated SQL by 38%.

Atlan is recognized as a Leader across multiple Gartner reports and Forrester Waves, and is deployed at enterprises including Mastercard, Workday, General Motors, CME Group, HubSpot, FOX, Virgin Media O2, and Elastic.

Bridge the context gap.
Ship AI that works.