[{"content":"From Model Risk to Agent Risk: A Practical Risk Management Approach for Organisations Rolling Out Agents Core idea: Agent risk management is not about proving a technical system is safe at a point in time. It is about governing delegated autonomy in a socio-technical system so that outcomes stay within policy, performance targets, and risk appetite over time. A socio-technical system with self correcting mechanisms\nSR 11-7 does not mention hallucination rates. It does not define retrieval precision. It has no section on prompt injection, corpus freshness, or agentic decision boundaries. It was issued in 2011, long before today’s enterprise rollout of large language models and agents.1\nAnd yet, if a financial institution is using GenAI or agentic systems in ways that could have material impact, SR 11-7 is still likely to be the lens an examiner brings into the room. Not because it is an AI-specific rulebook, but because its logic is technology-agnostic. The guidance defines model risk broadly as the potential for adverse consequences from decisions based on incorrect or misused model outputs, and it anchors validation around three enduring questions: is the approach conceptually sound, does it perform as intended over time, and do actual outcomes support continued use?2 The agencies have also made clear that this guidance is principles-based and does not require any specific model risk management framework or application.3\nBut OCC and SR 11-7 are US-specific references. The underlying supervisory logic is much broader.\nIn the UK, the PRA’s SS1/23 treats model risk as a risk discipline in its own right and sets five principles: model identification and risk classification, governance, development and use, independent validation, and mitigants. It also says those expectations are relevant to models using AI and machine learning to the extent those risks arise in models more generally.4 The FCA’s AI approach is similarly principles-based and outcome-focused: it says it does not plan extra AI-specific rules and will instead rely on existing frameworks, including Consumer Duty and SM\u0026amp;CR, to support safe and responsible adoption.5 The Bank of England and FCA are already seeing why this matters: in their 2024 survey, 75% of firms said they were already using AI, 17% of use cases involved foundation models, 55% had some degree of automated decision-making, and a third of use cases were third-party implementations.6\nIn Australia, the closest anchors are not AI-specific prudential rules, but APRA’s enterprise risk and operational resilience standards together with ASIC’s AI governance work. APRA’s CPS 220 requires a Board-approved risk appetite statement and a risk management framework that identifies, measures, evaluates, monitors, reports and controls material risks. CPS 230 and its practice guide add defined risk appetite supported by indicators, limits and tolerance levels, monitoring and testing of controls, scenario analysis, operational resilience, incident escalation, and management of service-provider risk. APRA can also require an independent review and remediation program where it identifies material weaknesses.78 ASIC’s REP 798, meanwhile, warns of a governance gap and asks firms who is accountable for AI use and outcomes, how consumer outcomes are measured, what human oversight is required, whether governance is leading or lagging AI use, and how third-party AI models are validated, monitored and reviewed.9 ASIC has also said it is not rushing to AI-specific regulation and will instead test how far its existing technology-neutral framework and powers can flex.10\nDifferent labels, same burden: define your standards, test against them, monitor real-world performance and harm, and show that accountability and remediation actually work. That is why the same conversation increasingly appears across the US, UK, and Australia, even when the underlying documents were not written for agents specifically.\nThat matters because many organisations are trying to solve the wrong problem, but we are starting to see similar logic being applied to Agents through things like AIUC-1 and Singapore\u0026rsquo;s new governance framework (more on that below).\nThe mistake: treating agents as just a better chatbot Most current Responsible AI programs were built for bounded systems. A model summarizes a document. It drafts a memo. It answers a question inside a defined workflow. In those environments, the main governance question is often: “Is the answer acceptable?”\nThat is why many firms already have familiar AI controls:\nreview processes prompt testing output sampling human sign-off red-teaming before launch Those controls made sense when systems produced outputs inside relatively narrow workflows and humans reviewed the result before anything important happened.\nAgents change that.\nSingapore’s 2026 Model AI Governance Framework for Agentic AI describes agents as systems that can plan across multiple steps to achieve objectives, with varying degrees of autonomy and action-taking. It also highlights two features that matter enormously for governance: action-space (what the agent is allowed to do) and autonomy (how much it can decide for itself when and how to act).11 Once an agent can use tools, write to systems, trigger transactions, coordinate with other agents, or operate over many steps, the governance problem is no longer mainly about the quality of a single answer.\nIt becomes a question of delegated authority, managing and minimising blast raduis while accepting that things will go wrong (like we do with Human\u0026rsquo;s in an organisation)\nThe real object of control is the socio-technical system The thing that creates risk is not just the model. It is not just the human. It is not just the workflow. It is the combined system: agents, people, tools, data, permissions, policies, vendors, operating environments, and the surrounding organisation.\nNIST’s AI Risk Management Framework treats AI risk as a socio-technical issue and emphasizes that effective risk management depends on accountability mechanisms, roles, responsibilities, culture, and incentives across the AI lifecycle. NIST also frames AI governance around a broad set of actors and lifecycle stages, with testing, evaluation, verification, and validation happening throughout rather than at a single gate.12\nThat is exactly the right frame for agents.\nWhy? Because agents move faster, touch more systems, and create more opportunities for failure before anyone sees the result. Their risks are not limited to “bad output.” Singapore’s agentic framework explicitly points to risks tied to planning, tool use, protocols, unauthorised actions, and multi-agent interactions, including cascading effects and orchestration drift.13\nIn practice, the control problem shifts from:\n“Did the model say something acceptable?”\nto\n“How do we know outcomes remain good enough over time? Are we taking actions to improve when things go wrong?”\nThese questions make up the foundations for a sociotechnical system that learns over time and converges to the right answer. Self correcting mechanisms.\nWhy point-in-time controls are not enough Agents are fundamentally difficult to govern with only deterministic, point-in-time safeguards.\nIn bounded AI use cases, you can often define a narrow task, test a representative set of prompts, set a review checkpoint, and be reasonably confident that the system will stay within a known envelope. With agents, the system is more open-ended. It interacts with external systems. It encounters changing data and policies. It may work through multiple reasoning steps. It may depend on upstream models and platforms that keep evolving. The environment changes while the agent is operating.\nSingapore’s guidance says this directly: because of the autonomous nature of agents and the changing environment, it is challenging to account for and test all possible outcomes before deployment, which is why gradual rollout and real-time monitoring after deployment are recommended.14\nASIC’s 2024 review points in the same direction. It found AI adoption accelerating, a shift toward more complex and opaque techniques such as generative AI, and cases where governance and risk management maturity did not keep pace with the scale and nature of AI use. It also found that some firms were still relying mainly on pre-deployment testing or trigger-based reviews, whereas better practice involved periodic review, root cause analysis, and assessment of consumer impact.9\nThat means the goal of risk management cannot be to prove that an agent is “safe” once and for all. For many material use cases, that is the wrong standard. The better standard is whether the organisation can keep the system within an acceptable operating envelope over time.\nWhat a supervisor is likely to care about This is where SR 11-7 becomes surprisingly useful, and where the cross-jurisdiction parallels become clearer.\nThe US guidance does not tell you how to measure hallucinations. It does not tell you what retrieval score is acceptable. It does not prescribe an agent evaluation harness. But it does give a practical supervisory logic. A validation framework should include:\nevaluation of conceptual soundness ongoing monitoring outcomes analysis15 The OCC’s examiner handbook extends that logic into governance: model risk should be managed like other types of risk, board and senior management should set a firm-wide framework, the framework should fit into broader risk management, model risk should be monitored in the aggregate, and the board should ensure it stays within tolerance or risk appetite.16\nThe same supervisory logic shows up elsewhere, just with different accents.\nUnited States: SR 11-7 and the OCC handbook ask about conceptual soundness, outcomes analysis, ongoing monitoring, effective challenge, and remediation.1516 United Kingdom: PRA SS1/23 asks for model definition, inventory, risk-based tiering, governance, board-approved model risk appetite, independent validation, performance thresholds, and mitigants when models underperform. The FCA adds a principles-based conduct and accountability layer through existing frameworks such as Consumer Duty and SM\u0026amp;CR.175 Australia: APRA asks for Board-approved appetite, indicators, limits and tolerance levels, monitoring and testing of controls, scenario analysis, operational resilience, service-provider risk management, and remediation. ASIC adds explicit questions on accountability, consumer outcomes, oversight, alignment of governance to use, and validation and monitoring of third-party AI models.789 In practice, the supervisory conversation often collapses into four questions:\nShow me your standards. What metrics, thresholds, and decision rules did you define, and why? Show me your tests. How did you evaluate the system against those standards before and after deployment? Show me your results. Did the system stay within the thresholds you set? Convince me it is reasonable. Would an informed third party view your approach as rigorous, proportionate, and defensible? That is the burden on the institution. You define the framework. You set the thresholds. You build the testing and monitoring infrastructure. Then you defend the whole approach to someone who may not care how elegant your RAG pipeline or orchestration layer is, but absolutely cares whether your governance is disciplined.\nA practical risk management approach for agents So what should organisations do?\nThey need a risk management approach designed at the organisational level, not just a collection of technical controls. The job is to supervise delegated autonomy in a living socio-technical environment.\nA practical approach has six parts.\n1. Define the operating envelope Start by defining what “good enough” means for the use case.\nThat includes:\npurpose and business objective relevant policies and obligations performance targets unacceptable outcomes affected stakeholders escalation triggers risk appetite stop conditions The key question is not whether the agent is perfect. It is whether the organisation has clearly defined the conditions under which the agent is acceptable to operate.\n2. Bound authority before you assess outputs For agents, authority matters as much as answer quality.\nMap the agent’s action-space:\nwhat it can read what it can write what it can approve what it can send what it can spend what it can trigger what it can change what it can delegate to other agents Then decide where humans must stay in the loop, where they only approve at significant checkpoints, and where monitoring plus after-the-fact review is enough. Singapore’s framework recommends defining significant checkpoints or action boundaries for human approval, especially for high-stakes or irreversible actions, and complementing that with automated monitoring and alert thresholds.18\n3. Test the system, not just the prompt For agents, testing needs to move beyond output quality alone.\nYou still need prompt testing, but that is not enough. You also need to test:\nend-to-end task execution policy compliance tool-use accuracy escalation behaviour permission boundaries failure recovery behaviour under degraded inputs interactions between agents behaviour when upstream systems or policies change That is much closer to system assurance than to classic model QA.\n4. Monitor continuously in production Monitoring is where agent governance becomes real.\nSR 11-7 already required ongoing monitoring because conditions, uses, and data sources change over time.19 For agents, that logic becomes central rather than secondary. Singapore recommends continuous post-deployment monitoring, logging, reporting, fail-safe mechanisms, alert thresholds, anomaly detection, and the ability to intervene in real time when agents behave unexpectedly.20\nThe UK and Australian analogues push the same way. PRA SS1/23 expects regular testing of data, model construct, assumptions, and outcomes; performance thresholds; back-testing; root cause analysis; and recalibration or redevelopment when thresholds are breached.17 APRA requires regular monitoring, review and testing of controls, board reporting, scenario analysis, and action where operational risk management falls below expectations.8 ASIC describes better practice as periodic post-deployment review, routine monitoring, root cause analysis, and investigation of possible consumer impact when unexpected outputs appear.9\nThis is the organisational muscle that matters most:\ntraces and logs tool-call monitoring thresholds for high-risk activities anomaly detection near-miss reporting drift detection override analysis pause / kill / revoke mechanisms trend analysis against expectations 5. Trust, but verify If the first line designs and runs the system, someone else needs to challenge whether it is actually working.\nIndependent review should not be ceremonial. It should test assumptions, challenge the chosen metrics, review incidents, look for control gaps across the aggregate system, and verify that outcomes still support deployment. SR 11-7 and the OCC handbook both emphasize effective challenge, validation, audit, documentation, and prompt remedial action when issues appear.21\nThat same expectation is explicit elsewhere. PRA SS1/23 makes independent model validation a standalone principle. APRA can require an independent review and a remediation program where it identifies material weaknesses in operational risk management. ASIC also asks whether non-technical functions such as compliance and internal audit have the skills and voice to engage with AI decisions and monitoring.1789\nThis is where many agent programs will succeed or fail. If the only evidence that controls work comes from the team that built the agent, the assurance story is weak.\n6. Correct quickly and continuously A good framework does not just identify issues. It drives correction.\nWhen outcomes move outside the accepted range, the response may include:\ntightening permissions changing routing adding human approval steps restricting tool access replacing a model or vendor redesigning the workflow retraining operators rolling back or decommissioning the agent SR 11-7 is explicit that when outcomes fall outside predetermined thresholds of acceptability, adjustment, recalibration, or redevelopment is warranted.22 PRA SS1/23 says much the same through thresholds, root cause analysis, recalibration and redevelopment.17 APRA likewise links material weaknesses to remediation, independent review, and potentially additional supervisory action.8\nThe shift in mindset The deepest change is philosophical.\nTraditional AI governance often assumes that risk management is mainly about approving a system. Agent governance is different. It is about supervising a dynamic operating system for delegated autonomy.\nThat means moving:\nfrom model risk to system risk from technical safeguards to organisational assurance from one-time approval to continuous supervision from answer quality to authority, outcomes, and intervention from static compliance to adaptive governance The goal is not to prove that the system will always be safe. The goal is to keep the socio-technical system converging toward outcomes that are good enough for your purposes, within policy, performance expectations, and risk appetite, even as agents, humans, workflows, and platforms evolve.\nThat is the risk management challenge organisations actually face when they roll out agents. And it is the standard a credible governance program should be built to meet.\nReferences Federal Reserve, SR 11-7: Guidance on Model Risk Management (2011).\nhttps://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nFederal Reserve, Supervisory Guidance on Model Risk Management (SR 11-7 attachment), including the definition of model risk and the three core validation elements.\nhttps://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nFederal Reserve, SR 21-8: Interagency Statement on Model Risk Management for Bank Systems Supporting BSA/AML Compliance (2021), noting the guidance is principles-based and does not require any specific model risk management framework or application.\nhttps://www.federalreserve.gov/supervisionreg/srletters/SR2108.htm\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nBank of England / PRA, SS1/23 – Model risk management principles for banks (published 2023, effective 2024), including five core principles, applicability across model types, and express relevance to AI and machine learning in modelling techniques.\nhttps://www.bankofengland.co.uk/prudential-regulation/publication/2023/may/model-risk-management-principles-for-banks-ss\nhttps://www.bankofengland.co.uk/-/media/boe/files/prudential-regulation/supervisory-statement/2023/ss123.pdf\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nFCA, AI and the FCA: our approach (2025/2026 updates) and Artificial Intelligence (AI) update – further to the Government’s response to the AI White Paper (2024), stating that the FCA’s approach is principles-based and outcome-focused, that it does not plan extra AI-specific regulations, and that existing frameworks such as Consumer Duty and SM\u0026amp;CR are relevant.\nhttps://www.fca.org.uk/firms/innovation/ai-approach\nhttps://www.fca.org.uk/publications/corporate-documents/artificial-intelligence-ai-update-further-governments-response-ai-white-paper\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nBank of England and FCA, Artificial intelligence in UK financial services – 2024, including figures on adoption, foundation models, automated decision-making, governance, accountability, and third-party implementations.\nhttps://www.bankofengland.co.uk/report/2024/artificial-intelligence-in-uk-financial-services-2024\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nAPRA, Prudential Standard CPS 220 Risk Management, including Board-approved risk appetite, institution-wide risk management framework, annual review, and management of material risks.\nhttps://www.apra.gov.au/sites/default/files/Prudential-Standard-CPS-220-Risk-Management-%28July-2017%29.pdf\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nAPRA, Prudential Standard CPS 230 Operational Risk Management and Prudential Practice Guide CPG 230 Operational Risk Management, including indicators, limits and tolerance levels, monitoring and testing of controls, scenario analysis, operational resilience, service-provider risk management, board oversight, incident escalation, independent review, and remediation.\nhttps://www.apra.gov.au/sites/default/files/2023-07/Prudential%20Standard%20CPS%20230%20Operational%20Risk%20Management%20-%20clean.pdf\nhttps://www.apra.gov.au/sites/default/files/2024-06/Prudential%20Practice%20Guide%20CPG%20230%20Operational%20Risk%20Management.pdf\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nASIC, REP 798: Beware the gap – Governance arrangements in the face of AI innovation (2024), including findings on governance gaps, consumer outcomes, accountability, oversight, lifecycle monitoring, and third-party model validation and review.\nhttps://download.asic.gov.au/media/mtllqjo0/rep-798-published-29-october-2024.pdf\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nASIC, AI: A blueprint for better banking? speech by ASIC Chair Joe Longo (2025), stating that ASIC is not rushing to more AI regulation and will use existing technology-neutral powers and framework in the meantime.\nhttps://www.asic.gov.au/about-asic/news-centre/speeches/ai-a-blueprint-for-better-banking/\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nIMDA Singapore, Model AI Governance Framework for Agentic AI (2026), sections on what agentic AI is, action-space, and autonomy.\nhttps://www.imda.gov.sg/-/media/imda/files/about/emerging-tech-and-research/artificial-intelligence/mgf-for-agentic-ai.pdf\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nNIST, AI Risk Management Framework (AI RMF 1.0), on socio-technical dimensions, accountability mechanisms, collective responsibility, and lifecycle risk management.\nhttps://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nIMDA Singapore, Model AI Governance Framework for Agentic AI (2026), discussion of risks from planning, tools, protocols, unauthorised actions, and multi-agent interactions.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nIMDA Singapore, Model AI Governance Framework for Agentic AI (2026), recommendation for gradual rollout and real-time monitoring because not all outcomes can be tested before deployment.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nFederal Reserve, Supervisory Guidance on Model Risk Management (SR 11-7 attachment), “Evaluation of conceptual soundness,” “Ongoing monitoring,” and “Outcomes analysis.”\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nOCC, Model Risk Management, Comptroller’s Handbook (updated handbook aligning with OCC Bulletin 2011-12), including that model risk should be managed like other types of risk and should fit into the broader risk management framework and risk appetite of the organisation.\nhttps://www.occ.treas.gov/publications-and-resources/publications/comptrollers-handbook/files/model-risk-management/pub-ch-model-risk.pdf\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nBank of England / PRA, SS1/23 – Model risk management principles for banks, including expectations on model inventory, risk-based tiering, model risk appetite, independent validation, regular testing, performance thresholds, root cause analysis, back-testing, and recalibration or redevelopment when thresholds are breached.\nhttps://www.bankofengland.co.uk/-/media/boe/files/prudential-regulation/supervisory-statement/2023/ss123.pdf\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nIMDA Singapore, Model AI Governance Framework for Agentic AI (2026), guidance on significant checkpoints, human approval, automated monitoring, and alert thresholds.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nFederal Reserve, Supervisory Guidance on Model Risk Management (SR 11-7 attachment), discussion of ongoing monitoring over time and regular analysis of data sources, conditions, and overrides.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nIMDA Singapore, Model AI Governance Framework for Agentic AI (2026), guidance on continuous monitoring, logging, reporting, fail-safe mechanisms, anomaly detection, and real-time intervention.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nFederal Reserve, Supervisory Guidance on Model Risk Management (SR 11-7 attachment); OCC, Model Risk Management, Comptroller’s Handbook.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nFederal Reserve, Supervisory Guidance on Model Risk Management (SR 11-7 attachment), on outcomes falling outside predetermined thresholds of acceptability and the need for adjustment, recalibration, or redevelopment.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://quintona.github.io/blog/posts/agent_risk_management_overview_blog-0.1/","summary":"\u003ch1 id=\"from-model-risk-to-agent-risk-a-practical-risk-management-approach-for-organisations-rolling-out-agents\"\u003eFrom Model Risk to Agent Risk: A Practical Risk Management Approach for Organisations Rolling Out Agents\u003c/h1\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eCore idea:\u003c/strong\u003e Agent risk management is not about proving a technical system is safe at a point in time. It is about governing \u003cstrong\u003edelegated autonomy\u003c/strong\u003e in a \u003cstrong\u003esocio-technical system\u003c/strong\u003e so that outcomes stay within policy, performance targets, and risk appetite over time. A socio-technical system with self correcting mechanisms\u003c/p\u003e\u003c/blockquote\u003e\n\u003cp\u003eSR 11-7 does not mention hallucination rates. It does not define retrieval precision. It has no section on prompt injection, corpus freshness, or agentic decision boundaries. It was issued in 2011, long before today’s enterprise rollout of large language models and agents.\u003csup id=\"fnref:1\"\u003e\u003ca href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\"\u003e1\u003c/a\u003e\u003c/sup\u003e\u003c/p\u003e","title":""}]