The year our connectors stopped depending on who was online

Every Connector Had Its Experts .

Until this year, every connector we ran was tied to the specialists who knew it. A change to a connector waited for one of those engineers to be free, and every new connector we shipped made the fleet bigger without making anyone freer, which kept us on a hamster wheel.

Coding agents had not changed that. Our engineers already had them open on their laptops, each getting faster at the connectors they knew, but what any one of them learnt stayed where they learnt it, and a faster engineer is still only one engineer. Our bottleneck was where our knowledge lived: in a few people’s heads, on their laptops, and on dashboards that only they watched.

So at the start of the year we committed to working agentically across every part of how we build software, from the first line of a specification to the release, as part of Atlan’s wider effort to become a frontier company in the open. The first plan, in March, was deliberately cautious: run the agentic flow end to end on a few connectors before deciding how to organise everything else around it.

Getting this right mattered well beyond our own team, because of what connectors feed. Every AI agent an enterprise runs is only as good as the context it reads, and every platform builds that context for its own ecosystem, leaving a business with islands instead of one layer it can trust. Atlan’s bet is the neutral one: a single context layer across every system a business runs, where metadata, semantics, lineage and business knowledge come together for people and agents alike.

That layer does not come from a prompt. It comes from a pipeline, and the pipeline is made of connectors.

Every connector runs as a workflow that reaches into a source, extracts its metadata and publishes it into the customer’s context layer. Each of those runs is a crawl. A connector that feeds a context layer carries far more than table names. Moving data only needs the rows; a context layer needs to know what they mean and where they came from. We run them across warehouses, databases, BI, ETL, streaming and ERP, for some of the largest banks, retailers, healthcare systems and manufacturers in the world.

When a connector fails, the context layer becomes confidently wrong, and a person or an agent acts on metadata that is no longer true. So the hard problem is keeping every connector true, on sources that change every week. That is the Connectivity team’s job, for every connector it builds.

Looking back, our own team had the same shape as the problem we were feeding. We had islands of knowledge, each trusted by the person who held it, and no single layer that anyone else, person or agent, could read. This year we started building that layer for ourselves: first by creating one picture of the fleet, then by setting rules for every connector, and lastly by building agents to create those connectors.

Changing the Engine Without Stopping the Car .

The first large job we handed to agents touched every connector at once. For years our connector workflows ran on Argo, which our architecture now lists as the legacy path. A workflow engine is what schedules each crawl, retries a step that fails and keeps track of where every run got to. Moving to Temporal had been planned for a while, because Argo ran every step of a workflow as its own container, while Temporal lets us break a workflow into activities, units of work whose boundaries we choose and which are retried and resumed on their own. That gave us control over how a workflow is designed, not just how it runs.

Agents did the rewriting under one absolute rule: every capability a connector had on the old engine had to exist in the rewrite. The rewritten connectors run in production today, and we saw runtimes improve across the fleet. On comparable runs on both engines, Power BI ran nearly five times faster and Tableau more than twice as fast. The gains came from both the engine swap and the finer control it gave us over how each workflow is designed.

This move was ultimately about more reliable workflows and shorter runtimes. Fresher context followed, because a crawl that fails less often and finishes sooner catches up with its source sooner.

The migration also made our islands impossible to ignore. With every connector moving engines in stages, and both engines live at once, the answer to “which build is running where?” was spread across the engineers who owned each connector, and nobody could see it in one place.

One Place to See the Fleet .

That question is how knowledge first started to leave individual heads. Connector Pulse had been our team dashboard, built entirely by AI, where we tracked workflow success, time to resolve support tickets and how much of the work AI was doing. To answer which build is running where, we added a view that showed every deployment across the fleet, connector by connector, and Pulse started growing into something bigger than we had planned.

That single picture mattered beyond the migration. Each limit we hit on the way forced the next evolution of Pulse.

Fleet health followed: runs, successes and failures across every connector and tenant (each customer’s own Atlan instance) in one place. It showed us that a fleet-wide success rate, on its own, cannot tell us what to do.

Sliceable health answered that: health per customer, per connection, per failure pattern, so a number always came with who it was landing on.

Views on demand went further. A slice answers a question we already knew to ask. When a question came up that no existing view covered, the view itself could be created the same afternoon, because no person had to hand-build it.

Take the migration itself: proving the new engine was faster meant timing both engines for every connector, down to each activity in each step. Nobody had planned that view. We built it with agents in two days.

Agent access over MCP came next. MCP, the Model Context Protocol, is an open standard that lets an AI agent read from a tool directly. With it, the agents working the fleet and the people reading it pull from one source. This is the context-layer argument at the scale of one team: one picture of the fleet, read by everyone who works on it, where before each engineer had their own.

A board of team metrics became the surface the fleet is run from.

A Fix Is Spent, a Rule Keeps Paying .

Once there was one picture, the failures in it started to look familiar. For the first time, we could see the same failure twice. Before Pulse covered the whole fleet, we had separate team dashboards, each watched by the engineer who knew that connector, so a shape repeating across two connectors was never in one place to be seen. The unit of work was whatever sat in front of us: one failure, one expert, one fix, and the understanding ending inside that person.

When we could see the whole fleet, the natural unit became the class. We stopped asking why this crawl failed and started asking how many customers this shape of failure was landing on, and what they had in common. Each fix became a candidate for a rule, and the understanding behind it had somewhere to go other than one engineer’s memory.

That is where it starts compounding. Every connector is built on the Connector SDK, our internal toolkit for building applications on the Atlan platform. A fix is spent the moment it is applied. A rule written into the SDK keeps paying: on every connector it touches, on every tenant it has not reached, and on connectors nobody has built yet.

Even turning fixes into rules is no longer down to us alone. Periodically, an agent reviews the fixes merged across all our connectors and, when a pattern could affect the rest of the fleet, opens a pull request that writes it into the Connector SDK as a conformance rule. An AI reviewer checks it first, then an engineer decides whether it should exist at all. We keep that step human on purpose: a wrong rule would push wrong fixes into every connector it touches.

Once a rule merges, every connector is checked against it, including the ones we haven’t built yet, and agents fix the violations they find, most of them merging on their own.

Since what we know now lives in the SDK, a connector nobody has built yet will be able to start with all of it.

From a One-Line Ticket to a Connector .

Rebuilding connectors that already worked was the first test. A source nobody had connected was a harder one, and the clearest check on how much of what we know now lives somewhere an agent can read it.

In June, we filed a ticket with a title and nothing else: Build SAP Datasphere connector. The agent it was assigned to broke the work into thirteen steps, wrote a specification with 108 traceable requirements, then tested it against a live instance and found 22 of its own assumptions wrong, from field names to which endpoints exist. It stopped three times for an engineer to make a call, and by the next afternoon the connector was built, crawled, tested and waiting for review, work that used to take us weeks. It is now in public preview, and a model a business user works from traces back through Datasphere to the S/4HANA table the data came from.

What the agent had to get right was how an enterprise source really behaves. It also had to fit the source into the asset model every other connector already speaks, the shared vocabulary for what a connector finds, such as tables, columns and dashboards. That fit is what lets the new connector’s lineage join up with the systems around it. The agent did not start from a blank page: it built on the Connector SDK and asset models refined across every connector we have built, which is years of what our engineers knew, written down where an agent could use it.

Its three stops show how we split the work. The agent settled anything it could check against the documentation, the live source or a test. It stopped for the calls only a person should make: what the connector is for, which objects matter, and how its assets should be modelled. The shared layer holds what we have learnt, and deciding what a connector should be stays with people.

This was not a one-off. In September, three new capabilities on our Fabric connector (semantic-model measures, Dataflow lineage and Spark lineage) went from request to release in about a week, built by engineers working with agents, in time to be announced at Microsoft’s FabCon Europe.

Building was one half of the job. The other half was noticing what breaks, and that had always started with someone else telling us.

We Stopped Waiting for the Customer Ticket .

Before this year, failure triage started with a customer ticket, which meant the first person to know a connector had broken was usually the customer. Now, while the team sleeps, the daily sweep agent reads every failure the fleet produced, often before a customer notices one.

Pulse changed where the work starts. Every failure across the fleet now lands in one snapshot, and a growing number of agents, built on LangGraph, work from it:

· One that writes up the day’s failures each morning, with one ticket per distinct fault

· One that confirms whether a new release really caused the failures an alert reports

· An agent that takes a support ticket to a root cause and a fix

· One that turns a customer request for a new capability into tested changes

· An agent that watches run times and finds which step is slowing a connector down

These agentic loops also let us get ahead of some breaks. Sources usually announce a new API, a deprecated one or a change to how authentication works in advance, so we actively watch for those notices and fix a connector before it breaks.

We Misread Our Own System .

More than once this year, we misread our system. The times that cost us most were both about what we had given the agents to work from: once a premise nobody had checked, and once far more context than they needed.

We thought two agents agreeing meant the answer was right. A support ticket arrived with a lead in it, pointing at the wrong code. The investigating agent started from that lead; so did the second agent we run to check the first one blind. They agreed, the reviewer agent passed it, and the fix was written for the old engine, which that connector had already left. An engineer caught it, and it took no special knowledge: which engine a connector runs on is one of the first things Pulse shows. Two agents reading the same wrong premise had given us one opinion twice. By the next morning we were stripping leads like that from tickets, and the support agent now checks the engine before anything else.

We spent a month’s budget in a week, almost all of it on re-sending context. By the first week of August, our agents had spent more than in all of July. We first blamed a model switch, but about 94% of the spend was re-sending context: every turn sent back everything the agent running the job had done so far. On one ticket that reached half a million tokens, and a crash meant paying to rebuild it just to find its place.

So we took the running of the job away from the model, and that is why every agent graph we run now works the same way. An investigation is a series of steps, where code decides which step runs next. Some steps are plain scripts with no model at all, others get a small model or the strongest one, and each has its own budget of turns and spend. A fresh agent sees only what its step needs.

That week, we also turned Pulse onto the agents themselves. Pulse now has a section that tracks what each agent worked on, how its runs ended (including when it handed a ticket to a person) and what it cost, by graph and by ticket. Any agent the team runs is meant to be tracked there, in the same place as the fleet it works on.

What We’re Still Working On .

What we’ve built is still far from finished. What is left is mostly about what it still cannot see, and what happens when it is wrong.

Giving every issue an identity. Whether an issue comes from the daily sweep or from a customer, the graph first pins down what actually happened: the connector, the connection, the run and the failure, each traced to the system of record. That matters most for a customer’s ticket, which describes what they saw, and that isn’t always what happened. When a field is missing, the graph stops rather than guesses, which is the standard we hold the context layer to. It works for workflow failures. Lineage is harder, because a wrong edge fails nothing, so there is no failure to start from, only an agent downstream that trusts it.

From a right diagnosis to a right fix. A graph can get the diagnosis right and still get the fix wrong. In one case, a graph correctly worked out that a process had been killed for running out of memory, then proposed a fix that changed nothing about memory. Graphs fail most on connectors that do not follow the Connector SDK’s conventions, or whose details the graph was given wrong. So when a fix misses, we close the gap in the graph itself rather than in that one ticket. The graphs are starting to heal themselves: a second agent re-investigates every finished run blind and scores it. A run that scores badly becomes a change to the graph. It is early, and so far each miss is making the next diagnosis on that kind of failure more reliable.

Guardrails for a loop that feeds itself. This is the open problem we think about most. Agents write fixes and check every merged fix to decide which ones become rules, so the loop makes its own next task. The shared layer that lets one good rule reach every connector would carry a wrong one just as far into everything built on it, including connectors we have not built yet. It is the problem we started with, a context layer that is confidently wrong, turned on our own system. Before agents run more of that loop on their own, we are building the checks that catch a wrong rule first.

Where the Knowledge Lives Now .

From where we started, the Connectivity team has reinvented itself more this year than in the several before.

Two key things have moved: knowledge and judgement. What an engineer knew about a connector used to live in their head. Now it is built into the Connector SDK, into Pulse, and into specialised agents that work across every connector, and a new source can go from an empty ticket to review in about a day. Judgement also moved earlier: people now decide what a connector should be, and agents do the building.

The layer we built for ourselves this year is a small version of the one our customers are building across their whole business, and it taught us the same lessons up close. Knowledge compounds once it lives somewhere everyone can read it, and a shared layer earns trust again every day it is used. Our next step is a loop that can carry more of the work on its own, once the checks are in place that let us trust it to.