Runtime AI Governance vs Reviewing Models Before Deployment

Emily Winks

Data Governance Expert

Updated:Nov 1, 2026

Published:Nov 1, 2026

11 min read

Watch Context Layer Live Get the AI Context Stack

Key takeaways

Read together, the three frameworks treat a pre-release review as necessary and insufficient.
A read from a customer table followed by an outbound email can be harmless step by step and a data leak as a sequence.
The check runs outside the model, so a manipulated prompt cannot switch it off.
Runtime governance enforces limits, and the review is where those limits are set.

What is runtime AI governance and how does it differ from reviewing models before deployment?

Runtime AI governance is the set of policies, checks and logs that apply to an AI system while it runs in production: every prompt, output, tool call and data access is evaluated as it happens. Reviewing models before deployment is a one-time gate that tests a model and approves it for release. The difference is timing and object: review judges a model at a point in time, and runtime governance judges each live action for the life of the system.

Is your AI agent context production-ready?

Audit Your Runtime Controls

Describe your production agent and the controls you run today. It checks what the agent may do, what data it may reach, who reviews, and what you can reconstruct, then ranks the gaps. Read the skill.

Paste into a new chat

Use the skill at https://atlan.com/skills/agent-guardrails-audit.md to audit the runtime controls around our production AI agent. Ask me for whatever it needs.

Run once in a terminal

curl -fsSL --create-dirs \
  -o ~/.agents/skills/agent-guardrails-audit/SKILL.md \
  https://atlan.com/skills/agent-guardrails-audit.md

For an agent

curl -fsSL https://atlan.com/skills/agent-guardrails-audit.md

How does runtime AI governance differ from reviewing models before deployment?

The two answer different questions. A pre-deployment review asks “is this model fit to release?” Runtime governance asks “is this action, by this system, on this data, allowed right now?”

	Pre-deployment model review	Runtime AI governance
When it runs	Once, before release, and again after a major change	Continuously, on every live request
What it inspects	The model: accuracy, bias, robustness, documentation	The behavior: prompts, outputs, tool calls, data accessed
Typical evidence	Test reports, model cards, red-team results, sign-off	Per-request logs, policy decisions, alerts, incident records
Who acts	A review board or model risk team	Automated policy checks, with humans for escalations
What it catches	Design flaws, known weaknesses, poor training data	Prompt injection, drift, misuse, unsafe action sequences
Failure mode	Approves a model that later behaves differently	Blocks or flags too late if logs are not watched

Regulation already treats these as two stages. Under the EU AI Act, a high-risk system passes a conformity assessment before it reaches the market, and it needs a new one after any substantial modification. The same law then requires providers to run a post-market monitoring system that collects and analyzes performance data for the whole lifetime of the system. The first is a review. The second is runtime governance.

Review sets the boundary. Runtime governance holds it under real traffic, with real users, data and tools attached.

Why can’t a review before deployment cover what happens in production?

Three things change after release, and a static review sees none of them.

First, inputs change. A test suite samples prompts that the reviewers thought of. Live users and attackers send prompts nobody thought of, and data pulled from documents or web pages can carry instructions of its own. The OWASP Top 10 for LLM Applications names “excessive agency” as a risk class that lets damaging actions follow from “unexpected, ambiguous or manipulated outputs,” and traces it to three root causes: excessive functionality, excessive permissions and excessive autonomy. All three show up only when the system is connected to real tools.

Second, agents choose their own path. A classic model returns one prediction. An agent plans a sequence of steps, calls tools and reads data along the way. In a March 2026 paper, Kaptein, Khan and Podstavnychy argue that the execution path is the main object to govern for agents, and that static access rules cover only the paths a fixed rule happens to match. A read from a customer table followed by an outbound email can be harmless step by step and a data leak as a sequence.

Third, the system itself drifts. Model versions are swapped, prompts are edited, new connectors are added and the data underneath changes. For high-risk systems that keep learning after release, the EU AI Act exempts changes that a provider pre-determined at the initial assessment from a new conformity assessment. That leaves post-market monitoring as the main check on how those planned changes behave in real use.

What do the EU AI Act, NIST AI RMF and ISO/IEC 42001 require after deployment?

All three frameworks describe governance as a lifecycle activity, and each one names controls that only run in production.

The EU AI Act (Regulation (EU) 2024/1689) sets several runtime duties for high-risk systems:

Article 9 defines risk management as “a continuous iterative process” run “throughout the entire lifecycle,” fed by data from post-market monitoring.
Article 12 requires systems to allow “automatic recording of events (logs) over the lifetime of the system.”
Article 26 tells deployers to monitor operation, assign human oversight to people with the authority to act, keep automatically generated logs for at least six months, and suspend use and inform the provider when they see a risk.

The NIST AI Risk Management Framework places these controls in its Manage function. The NIST AI RMF Playbook lists MANAGE 4.1, which calls for post-deployment monitoring plans that capture and evaluate input from users, with appeal, override, incident response and change management. MANAGE 2.4 calls for mechanisms and named owners to supersede, disengage or deactivate a system that does not perform as intended.

ISO/IEC 42001, published in December 2023, specifies requirements for establishing, implementing, maintaining and continually improving an AI management system. Certification against it covers the management system, not a single model, so an organization is audited on whether its oversight keeps running after release.

Read together, the three frameworks treat a pre-release review as necessary and insufficient. Each expects evidence from production: logs, monitoring results, incidents and the decisions taken on them.

What controls make up runtime AI governance?

Runtime governance is a small set of controls placed in the execution path of an AI system. Each one draws on the EU AI Act, the NIST AI RMF, ISO/IEC 42001 or the OWASP guidance on excessive agency.

A policy check on each action. Every proposed output, tool call or data read gets a decision: allow, block, redact or escalate. The check runs outside the model, so a manipulated prompt cannot switch it off.
Least privilege per task. The agent holds only the permissions the current task needs, which matches the minimize-extensions and minimize-permissions mitigations that OWASP lists for excessive agency, alongside user approval for high-impact actions.
Logs that explain decisions. Each record holds who asked, which data and tools were used, which policy fired and what the system did. The EU AI Act requires high-risk systems to record events automatically (Article 12) and deployers to keep those logs (Article 26).
Human oversight with clear thresholds. People review escalations above a set risk or value, and named owners can stop the system, as NIST MANAGE 2.4 describes.
Drift and performance monitoring. Alerts fire when accuracy, input distribution or behavior moves past a threshold, and the alert opens a review.

Gartner predicts that “guardian agent” technologies, which review, monitor or block the actions of other AI agents, will take at least 10 to 15% of agentic AI markets by 2030.

If your agents run on Databricks, Unity AI Gateway adds spend limits at the gateway, with guardrails and payload logging in beta. Those controls still need to know what the data is, who owns it and which policy applies to it.

Do you still need to review models before deployment?

Yes. Runtime governance enforces limits, and the review is where those limits are set. Skipping the review leaves the runtime checks with nothing specific to enforce.

A review before deployment does work that cannot happen later. It confirms the training data is fit for use, documents intended use and known limits in a model card, sets the risk tier, and decides which actions need human approval. Under the EU AI Act, the conformity assessment for a high-risk system happens at this stage, and a substantial modification sends the system back through it.

The two stages feed each other. The review produces the policy and the thresholds. Runtime governance applies them on every request and produces the logs, alerts and incidents. Those results then flow back into the next review, which is the loop that Article 9 of the EU AI Act describes when it ties risk management to post-market monitoring data.

Three practices keep the two stages connected:

They keep one registry of every model, version and AI application, with an owner for each.
They link each model to the data it was trained on and the data it reads in production, so a data problem is traced to the models it affects.
They define review triggers in advance: a retrain, a new tool, a drift alert or a policy breach reopens the review.

For a full view of the lifecycle stages, see AI model governance and the model lifecycle. For who owns each decision, see the AI governance operating model.

How does Atlan support runtime AI governance?

Runtime checks are only as good as the context they read: what a model is, which data it touches, who owns it and which policy applies. Atlan holds that context.

According to Atlan’s documentation, AI governance in Atlan works as a control plane for AI assets, with visibility and discovery, lifecycle management, risk assessment and policy enforcement. Teams register AI models and their versions through a guided three-step wizard and register AI applications alongside them, then track each asset from creation to retirement. The AI governance overview page also describes visibility into what agents can access, policy enforcement at the asset level, and lineage that makes each decision auditable. That registry and lineage are what a runtime check needs to answer “is this allowed?” for a specific model and dataset. They also give the next pre-deployment review its evidence: which model versions changed since the last review, and which upstream data moved underneath them.

General Motors works with Atlan on registering its AI models and applications. Sherri Adame, Data Governance Lead at General Motors, said on Atlan’s customer page: “Atlan has been a really good partner in helping us figure out how to register AI models and applications, and what metadata to put in place to meet the transparency requirements that [AI Governance] legislation asks for.” In a Re:Govern session, GM described automating about 20 essential data quality checks across all of its AI and ML datasets to meet AI legislation.

FAQs about runtime AI governance

What is runtime AI governance in one sentence?

Runtime AI governance is the continuous enforcement and recording of policy on a live AI system. It checks each prompt, output, tool call and data access as it happens, logs the decision, and escalates to a person when a risk threshold is crossed. It runs for as long as the system is in production.

Is runtime AI governance the same as AI monitoring?

No. Monitoring observes and alerts, for example when accuracy drops or inputs drift. Runtime governance also decides: it can block, redact or escalate an action before it completes. Most programs use both, with monitoring results feeding the policies that the runtime checks enforce, and with logs from both kept as audit evidence.

Does the EU AI Act require runtime governance?

For high-risk systems, yes in substance. The Act requires a lifecycle risk management process, automatic event logging over the system’s lifetime, post-market monitoring by providers, and monitoring plus human oversight by deployers. Deployers must keep automatically generated logs for at least six months unless other law sets a different period.

Can a model pass review and still fail at runtime?

Yes. A review tests the model against the cases reviewers chose. In production, the model meets new inputs, injected instructions, new tools and changed data. An agent can also combine individually allowed steps into a harmful sequence, which no single-step test reveals. Runtime checks exist to catch exactly these cases.

Who owns runtime AI governance in an organization?

Ownership is usually split. A central AI governance or risk function sets the policies and thresholds. The AI platform team runs the enforcement and logging. Named business owners review escalations and decide whether to suspend a system. Data governance teams own the metadata and lineage that the checks rely on.

Sources

Share this article

Atlan is the Context Layer for AI. It translates business knowledge, including data definitions, working procedures, and governance policies, into context AI can actually use. This knowledge lives in a single Enterprise Data Graph that every team and AI agent can reach.

In Atlan's AI Labs benchmark, adding this context improved AI's text-to-SQL accuracy by 38%.

Atlan is recognized as a Leader across multiple Gartner reports and Forrester Waves, and is trusted by over 400 enterprises representing $10T+ in market cap, including Mastercard, Workday, General Motors, CME Group, HubSpot, FOX, Virgin Media O2, and Elastic.

Get a Custom Demo