Metadata Lakehouse: real-time AI context on Apache Iceberg

Presenter

0:00 I think all of us remember the famous adage, garbage in, garbage out. But with AI it's worse. Garbage in, and confident garbage out. And we are really excited because of that to activate this context across all agentic applications, and... But let's talk about how we are gonna make it happen. Let's, let's take a step back first. Every software in this world that was built was always built on these three layers. Data at the bottom, logic in the middle, and experience on the top. But with AI, all three layers have changed. Data isn't just rows and columns anymore. It's documentation, images, conversations. Logic is no longer just if and else.

0:45 We've been trained to write logic like that. Now it's reasoning, learning, thinking, and we are outsourcing it to these general purpose agents. Experience isn't just point and click. It's conversations. That means we had to reimagine the whole infrastructure as well. And this is what our learnings were. Metadata is no longer just one-way traffic. It's not about just us consuming context, AI, AI consuming context. It's also about generating new context. There is so much interaction that humans are going to do with AI. There's so much context that'll be generated across the agentic life cycle. And I'm sure by now in forty-six minutes you've realized metadata is no longer just small data. It's huge.

1:30 It'll be generated in millions. So we had to rethink and learn from the world of data. So what, what did we care about then? We cared about scalability. Can it s-scale with exploding workloads? We had to care about versioning because there needs to be right audit how the context has changed over a period of time, so we know how the outcomes are changing. We should be able to do real-time querying. AI agents should be able to easily plug them in. And at the end of the day, it needs to be open and interoperable because the agents could be anywhere, could be built in any framework, in any runtime.

2:08 So just like in the BI era, there were data warehouses. In the AI era, we are really proud to introduce context stores powered by lakehouse architecture. So we are building this from the ground up with an open table format like Iceberg, which makes it versioned, open by design, and interoperable. You can bring your favorite compute to query, run analytics, run your AI workloads. You can bring your own model on top of this. It's... And it's not just a store. It has out-of-box capabilities with things like semantic and hybrid search. And like we always do, we keep it open and extensible with your frameworks, APIs, and apps that you can build on top of this.

2:52 And we are also really proud to partner with amazing partners in the ecosystem, just like with all the pillars to help accelerate this for our customers. Tom, do you want to take this and make it real for everybody here?

Tom

3:05 I would love to. Let me take the screen. Boom. All right. AI is only as powerful as the metadata that fuels it. And until now, metadata has been locked away in catalogs, hard to access and even harder to put to work. Atlan changes that. We've built a metadata lakehouse powered by Apache Iceberg and made it queryable both in Snowflake and Databricks. That means your metadata is no longer just documentation. It's data you can analyze, visualize, and act on. Let me show you. Here in Snowflake, you'll see the metadata lakehouse exposed just like any other database.

3:50 I can explore the tables, and if I wanted, I can even run a query. For example, I'm going to check the description quality on tables using Snowflake Cortex. Metadata isn't abstract anymore. It's insanely queryable. Now I'll hop over to Databricks, because you know what? Same metadata, same tables, same query, but with Databricks syntax available natively here too. That's the power of using an open table format like Iceberg. No lock-in, no silos, just one consistent foundation for all your metadata. And because metadata is now data, you can analyze it anywhere, like in a Sigma dashboard built directly on top of the metadata lakehouse.

4:39 This is one designed for a chief data officer who wants to know, "How ready are my domains for text-to-SQL use cases? Which domains are well-documented and governed? Which need investment?" This is just one example. The truth is, you can build any custom dashboard or any metric you can imagine. Operational efficiency, governance, adoption, AI readiness. The use cases are endless. That's the vision of the metadata lakehouse. Your metadata stored in an open format, queryable in Snowflake and Databricks, powering whatever use case your business needs. As a matter of fact, Ritik, would you mind taking over the screen and showing what our launch partner Sigma built on top of the metadata lakehouse?

David Lyons

5:26 Hello, everyone. I'm David Lyons with Sigma Computing. I'm a senior solution engineer here, and Sigma's thrilled to be a launch partner of Atlan's metadata lakehouse built on Iceberg. Today, we're going to show a pre-built Sigma application that offers a few representative examples around popular use cases such as compliance and storage utilization. Now, let's go ahead and get started. So in this particular view, I'm actually looking at compliance, where we're wanting to manage our tables and views and manage those that do not have tags. In this case, I actually see thirty-one percent actually don't have tags, which is a large number. In addition, the Sigma platform allows us to utilize a graphical visualization.

6:11 Here, we're wanting to look at our storage utilization and identify those large tables that may not be utilized. In this case, I can actually see our dim company has a popularity score of zero, but it's a fairly significantly large table. Other types of views we may want to look at is a historical analysis, where we're looking at tables without a description. Here we can see that over time and kind of track this information. Maybe some training for our data stewards to make sure that they are always adding a description when they add new objects in.

6:46 So here, very quickly, I just showed a few representative examples of the power of Sigma on top of your metadata lakehouse built on Iceberg, being able to not only visualize data, but also take action on that. Thank you very much.

Presenter

7:07 Thank you to our partner, Sigma. And I also want to talk about, we are launching this with our launch partners Snowflake, Sigma, and Databricks. You will keep hearing updates on how we take our context store to every person building agentic applications across the world.