--- title: "Runtime AI Governance vs Reviewing Models Before Deployment" url: "https://atlan.com/know/data-governance/runtime-ai-governance-vs-reviewing-models-before-deployment/" description: "Runtime AI governance checks what a live AI system does on every request. Pre-deployment model review checks it once before release. See how they differ." author: "Emily Winks" author_role: "Data Governance Expert" published: "2026-11-01" updated: "2026-11-01T00:00:00.000Z" --- ## What is runtime AI governance and how does it differ from reviewing models before deployment? Runtime AI governance is the set of policies, checks and logs that apply to an AI system while it runs in production: every prompt, output, tool call and data access is evaluated as it happens. Reviewing models before deployment is a one-time gate that tests a model and approves it for release. The difference is timing and object: review judges a model at a point in time, and runtime governance judges each live action for the life of the system. ## Key takeaways - Read together, the three frameworks treat a pre-release review as necessary and insufficient. - A read from a customer table followed by an outbound email can be harmless step by step and a data leak as a sequence. - The check runs outside the model, so a manipulated prompt cannot switch it off. - Runtime governance enforces limits, and the review is where those limits are set. --- Runtime AI governance is the set of policies, checks and logs that apply to an AI system while it runs in production: every prompt, output, tool call and [data access](https://atlan.com/know/ai-agent/how-to-give-ai-agents-access-to-enterprise-data/) is evaluated as it happens. Reviewing models before deployment is a one-time gate that tests a model and approves it for release. The difference is timing and object: review judges a model at a point in time, and runtime governance judges each live action for the life of the system. ### Audit Your Runtime Controls Describe your production agent and the controls you run today. It checks what the agent may do, what data it may reach, who reviews, and what you can reconstruct, then ranks the gaps. [Read the skill](/skills/agent-guardrails-audit.md). *Paste into a new chat* ``` Use the skill at https://atlan.com/skills/agent-guardrails-audit.md to audit the runtime controls around our production AI agent. Ask me for whatever it needs. ``` *Run once in a terminal* ``` curl -fsSL --create-dirs \ -o ~/.agents/skills/agent-guardrails-audit/SKILL.md \ https://atlan.com/skills/agent-guardrails-audit.md ``` *For an agent* ``` curl -fsSL https://atlan.com/skills/agent-guardrails-audit.md ``` Human Agent ClaudeChatGPTGeminiCursorOther Copy --- ## How does runtime AI governance differ from reviewing models before deployment? The two answer different questions. A pre-deployment review asks "is this model fit to release?" Runtime governance asks "is this action, by this system, on this data, allowed right now?" | | Pre-deployment model review | Runtime AI governance | |---|---|---| | When it runs | Once, before release, and again after a major change | Continuously, on every live request | | What it inspects | The model: accuracy, bias, robustness, documentation | The behavior: prompts, outputs, tool calls, data accessed | | Typical evidence | Test reports, model cards, red-team results, sign-off | Per-request logs, policy decisions, alerts, incident records | | Who acts | A review board or model risk team | Automated policy checks, with humans for escalations | | What it catches | Design flaws, known weaknesses, poor training data | Prompt injection, drift, misuse, unsafe action sequences | | Failure mode | Approves a model that later behaves differently | Blocks or flags too late if logs are not watched | Regulation already treats these as two stages. Under the [EU AI Act](https://atlan.com/know/ai-readiness/the-eu-ai-act-summary/), a high-risk system passes a [conformity assessment](https://artificialintelligenceact.eu/article/43/) [before it reaches the market](https://artificialintelligenceact.eu/article/16/), and it needs a new one after any substantial modification. The same law then requires providers to run a [post-market monitoring system](https://artificialintelligenceact.eu/article/72/) that collects and analyzes performance data for the whole lifetime of the system. The first is a review. The second is runtime governance. Review sets the boundary. Runtime governance holds it under real traffic, with real users, data and tools attached. --- ## Why can't a review before deployment cover what happens in production? Three things change after release, and a static review sees none of them. First, inputs change. A test suite samples prompts that the reviewers thought of. Live users and attackers send prompts nobody thought of, and data pulled from documents or web pages can carry instructions of its own. The [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) names "excessive agency" as a risk class that lets damaging actions follow from "unexpected, ambiguous or manipulated outputs," and traces it to three root causes: excessive functionality, excessive permissions and excessive autonomy. All three show up only when the system is connected to real tools. Second, agents choose their own path. A classic model returns one prediction. An agent [plans a sequence of steps](https://atlan.com/know/ai-agent/ai-agent-planning/), [calls tools](https://atlan.com/know/ai-agent/ai-agent-tool-use/) and reads data along the way. In a March 2026 paper, [Kaptein, Khan and Podstavnychy](https://arxiv.org/abs/2603.16586) argue that the [execution path](https://atlan.com/know/ai-agent/ai-agent-trajectory-evaluation/) is the main object to govern for agents, and that [static access rules](https://atlan.com/know/ai-agent/context-layer/context-layer-role-based-access-control/) cover only the paths a fixed rule happens to match. A read from a customer table followed by an outbound email can be harmless step by step and a data leak as a sequence. Third, [the system itself drifts](https://atlan.com/know/context-drift-ai-agents/). [Model versions are swapped](https://atlan.com/know/ai-model-versioning-best-practices/), [prompts are edited](https://atlan.com/know/ai-agent/centralized-prompt-management-for-enterprise/), new connectors are added and [the data underneath changes](https://atlan.com/know/ai-agent/data-for-ai/data-observability-for-ai-pipelines/). For high-risk systems that keep learning after release, the EU AI Act exempts changes that a provider [pre-determined at the initial assessment](https://artificialintelligenceact.eu/article/43/) from a new conformity assessment. That leaves post-market monitoring as the main check on how those planned changes behave in real use. --- ## What do the EU AI Act, NIST AI RMF and ISO/IEC 42001 require after deployment? All three frameworks describe governance as a lifecycle activity, and each one names controls that only run in production. The **EU AI Act** ([Regulation (EU) 2024/1689](https://eur-lex.europa.eu/eli/reg/2024/1689/oj)) sets several runtime duties for high-risk systems: - [Article 9](https://artificialintelligenceact.eu/article/9/) defines [risk management](https://atlan.com/know/ai-readiness/ai-risk-management/) as "a continuous iterative process" run "throughout the entire lifecycle," fed by data from post-market monitoring. - [Article 12](https://artificialintelligenceact.eu/article/12/) requires systems to allow "automatic recording of events (logs) over the lifetime of the system." - [Article 26](https://artificialintelligenceact.eu/article/26/) tells deployers to [monitor operation](https://atlan.com/know/ai-agent-monitoring/), assign human oversight to people with the authority to act, keep automatically generated logs for at least six months, and suspend use and inform the provider when they see a risk. The **NIST AI Risk Management Framework** places these controls in its Manage function. The [NIST AI RMF Playbook](https://airc.nist.gov/airmf-resources/playbook/manage/) lists MANAGE 4.1, which calls for post-deployment monitoring plans that capture and evaluate input from users, with appeal, override, incident response and change management. MANAGE 2.4 calls for mechanisms and named owners to supersede, disengage or deactivate a system that does not perform as intended. **ISO/IEC 42001**, published in December 2023, specifies requirements for [establishing, implementing, maintaining and continually improving an AI management system](https://www.iso.org/standard/42001). Certification against it covers the management system, not a single model, so an organization is audited on whether its oversight keeps running after release. Read together, the three frameworks treat a pre-release review as necessary and insufficient. Each expects evidence from production: logs, [monitoring results](https://atlan.com/know/ai-agent-observability/), incidents and the decisions taken on them. --- ## What controls make up runtime AI governance? Runtime governance is a small set of [controls placed in the execution path](https://atlan.com/know/ai-agent/enterprise-ai-agent-guardrails-checklist/) of an AI system. Each one draws on the EU AI Act, the NIST AI RMF, ISO/IEC 42001 or the OWASP guidance on excessive agency. - **A policy check on each action.** Every proposed output, tool call or data read gets a decision: allow, block, redact or escalate. [The check runs outside the model](https://atlan.com/know/harness-engineering/), so a [manipulated prompt](https://atlan.com/know/prompt-injection-attacks-ai-agents/) cannot switch it off. - **Least privilege per task.** The agent holds [only the permissions the current task needs](https://atlan.com/know/ai-agent-access-control/), which matches the minimize-extensions and minimize-permissions mitigations that [OWASP lists](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) for excessive agency, alongside user approval for high-impact actions. - **Logs that explain decisions.** Each record holds [who asked](https://atlan.com/know/ai-agent/ai-agent-identity/), which data and tools were used, which policy fired and what the system did. The EU AI Act requires high-risk systems to record events automatically ([Article 12](https://artificialintelligenceact.eu/article/12/)) and deployers to keep those logs ([Article 26](https://artificialintelligenceact.eu/article/26/)). - **Human oversight with clear thresholds.** People review escalations above a set risk or value, and [named owners](https://atlan.com/know/ai-agent/who-owns-ai-governance-caio-vs-cdo-vs-cto/) can stop the system, as [NIST MANAGE 2.4](https://airc.nist.gov/airmf-resources/playbook/manage/) describes. - **Drift and performance monitoring.** Alerts fire when [accuracy](https://atlan.com/know/ai-agent/ai-agent-accuracy/), [input distribution](https://atlan.com/know/context-drift-detection/) or behavior moves past a threshold, and the alert opens a review. Gartner [predicts](https://www.gartner.com/en/newsroom/press-releases/2025-06-11-gartner-predicts-that-guardian-agents-will-capture-10-15-percent-of-the-agentic-ai-market-by-2030) that "guardian agent" technologies, which review, monitor or block the actions of other AI agents, will take at least 10 to 15% of [agentic AI](https://atlan.com/know/what-is-agentic-ai/) markets by 2030. If your agents run on Databricks, [Unity AI Gateway](https://atlan.com/know/ai-agent/databricks/unity-ai-gateway/) adds spend limits at [the gateway](https://atlan.com/know/what-is-ai-gateway-llm-gateway/), with [guardrails](https://atlan.com/know/ai-agent/ai-guardrails-tools-compared/) and payload logging in beta. Those controls still need to know what the data is, who owns it and [which policy applies to it](https://atlan.com/know/data-governance/why-should-data-governance-rules-be-readable-by-ai-agents/). --- ## Do you still need to review models before deployment? Yes. Runtime governance enforces limits, and the review is where those limits are set. Skipping the review leaves the runtime checks with nothing specific to enforce. A review before deployment does work that cannot happen later. It confirms the [training data is fit for use](https://atlan.com/know/how-to-ensure-llm-training-data-quality/), documents intended use and known limits in a model card, sets the risk tier, and decides which actions need human approval. Under the EU AI Act, the [conformity assessment](https://artificialintelligenceact.eu/article/43/) for a high-risk system happens at this stage, and a substantial modification sends the system back through it. The two stages feed each other. The review produces the policy and the thresholds. Runtime governance applies them on every request and produces the logs, alerts and incidents. Those results then flow back into the next review, which is the loop that [Article 9](https://artificialintelligenceact.eu/article/9/) of the EU AI Act describes when it ties risk management to post-market monitoring data. Three practices keep the two stages connected: 1. They keep [one registry of every model, version and AI application](https://atlan.com/know/what-is-ai-registry/), with an owner for each. 2. They link each model to [the data it was trained on](https://atlan.com/know/training-data-lineage-for-llms/) and the data it reads in production, so [a data problem is traced to the models it affects](https://atlan.com/know/ai-agent/data-for-ai/data-lineage-for-ai/). 3. They define review triggers in advance: a retrain, a new tool, a drift alert or a policy breach reopens the review. For a full view of the lifecycle stages, see [AI model governance and the model lifecycle](https://atlan.com/know/ai-readiness/ai-model-governance/). For who owns each decision, see the [AI governance operating model](https://atlan.com/know/ai-governance-operating-model/). --- ## How does Atlan support runtime AI governance? Runtime checks are only as good as [the context they read](https://atlan.com/know/what-is-the-enterprise-context-layer/): what a model is, which data it touches, who owns it and which policy applies. Atlan holds that context. According to [Atlan's documentation](https://docs.atlan.com/product/capabilities/governance/ai-governance), AI governance in Atlan works as a [control plane](https://atlan.com/know/ai-control-plane/) for AI assets, with visibility and discovery, lifecycle management, risk assessment and policy enforcement. Teams [register AI models and their versions](https://docs.atlan.com/product/capabilities/governance/ai-governance/get-started/create-ai-model) through a guided three-step wizard and register AI applications alongside them, then track each asset from creation to retirement. The AI governance overview page also describes visibility into what agents can access, policy enforcement at the asset level, and lineage that makes each decision auditable. That registry and lineage are what a runtime check needs to answer "is this allowed?" for a specific model and dataset. They also give the next pre-deployment review its evidence: which model versions changed since the last review, and which upstream data moved underneath them. General Motors works with Atlan on registering its AI models and applications. Sherri Adame, Data Governance Lead at General Motors, said on [Atlan's customer page](https://atlan.com/customers/): "Atlan has been a really good partner in helping us figure out how to register AI models and applications, and what metadata to put in place to meet the transparency requirements that [AI Governance] legislation asks for." In a [Re:Govern session](https://atlan.com/regovern-watch-center/gm-proactive-governance/), GM described automating about 20 essential [data quality checks](https://atlan.com/know/data-for-ai/data-quality-for-ai-agent/) across all of its AI and ML datasets to meet AI legislation. --- ## FAQs about runtime AI governance ### What is runtime AI governance in one sentence? Runtime AI governance is the continuous enforcement and recording of policy on a live AI system. It checks each prompt, output, tool call and data access as it happens, logs the decision, and escalates to a person when a risk threshold is crossed. It runs for as long as the system is in production. ### Is runtime AI governance the same as AI monitoring? No. Monitoring observes and alerts, for example when accuracy drops or inputs drift. Runtime governance also decides: it can block, redact or escalate an action before it completes. Most programs use both, with monitoring results feeding the policies that the runtime checks enforce, and with logs from both kept as audit evidence. ### Does the EU AI Act require runtime governance? For high-risk systems, yes in substance. The Act requires a lifecycle risk management process, automatic event logging over the system's lifetime, post-market monitoring by providers, and monitoring plus human oversight by deployers. Deployers must keep automatically generated logs for at least six months unless other law sets a different period. ### Can a model pass review and still fail at runtime? Yes. A review tests the model against the cases reviewers chose. In production, the model meets new inputs, injected instructions, new tools and changed data. An agent can also combine individually allowed steps into a harmful sequence, which no single-step test reveals. Runtime checks exist to catch exactly these cases. ### Who owns runtime AI governance in an organization? Ownership is usually split. A central AI governance or risk function sets the policies and thresholds. The AI platform team runs the enforcement and logging. Named business owners review escalations and decide whether to suspend a system. Data governance teams own the metadata and lineage that the checks rely on. --- ## Sources 1. [Article 43: Conformity Assessment, EU Artificial Intelligence Act](https://artificialintelligenceact.eu/article/43/) 2. [Article 16: Obligations of Providers of High-Risk AI Systems, EU Artificial Intelligence Act](https://artificialintelligenceact.eu/article/16/) 3. [Article 72: Post-Market Monitoring by Providers and Post-Market Monitoring Plan for High-Risk AI Systems, EU Artificial Intelligence Act](https://artificialintelligenceact.eu/article/72/) 4. [LLM06:2025 Excessive Agency, OWASP Gen AI Security Project](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) 5. [Runtime Governance for AI Agents: Policies on Paths, arXiv](https://arxiv.org/abs/2603.16586) 6. [Regulation (EU) 2024/1689 (Artificial Intelligence Act), EUR-Lex](https://eur-lex.europa.eu/eli/reg/2024/1689/oj) 7. [Article 9: Risk Management System, EU Artificial Intelligence Act](https://artificialintelligenceact.eu/article/9/) 8. [Article 12: Record-Keeping, EU Artificial Intelligence Act](https://artificialintelligenceact.eu/article/12/) 9. [Article 26: Obligations of Deployers of High-Risk AI Systems, EU Artificial Intelligence Act](https://artificialintelligenceact.eu/article/26/) 10. [AI RMF Playbook: Manage, NIST AI Resource Center](https://airc.nist.gov/airmf-resources/playbook/manage/) 11. [ISO/IEC 42001:2023 Artificial Intelligence Management System, ISO](https://www.iso.org/standard/42001) 12. [Gartner Predicts That Guardian Agents Will Capture 10-15% of the Agentic AI Market by 2030, Gartner](https://www.gartner.com/en/newsroom/press-releases/2025-06-11-gartner-predicts-that-guardian-agents-will-capture-10-15-percent-of-the-agentic-ai-market-by-2030) 13. [AI governance, Atlan Documentation](https://docs.atlan.com/product/capabilities/governance/ai-governance) 14. [Create AI Model, Atlan Documentation](https://docs.atlan.com/product/capabilities/governance/ai-governance/get-started/create-ai-model)