AI in financial services: good practice and areas for improvement

""

Good practice

Choosing where AI should support, automate or abstain. Firms considered which tasks AI can support or automate, which should remain staff-led, and where the AI tool should abstain, redirect or escalate to a person. This means building proportionate technical checks and continuous monitoring into the AI system, rather than only relying on a person to review, approve, challenge or override its outputs or decisions. Firms should be able to show that the controls they choose work in practice.

Managing bias risks at the design stage. Firms tested for unfair or unjustified differences across relevant consumer groups and scenarios. Depending on the use case, they considered factors such as the data and variables used, and compared outcomes and error rates. They addressed any issues by rebalancing the data, adjusting thresholds or adding controls. They retested to make sure those changes improved consumer outcomes without creating new gaps and then kept monitoring for bias.

Defining the consumer need before developing or deploying a product. Firms started with a clear consumer problem, target market, and intended outcome before deciding whether AI was the right tool and in what form, eg agentic and generative.

Testing outcomes across different consumer characteristics, including vulnerability and digital or financial capability, to support inclusive design.

""

Areas for improvement

Building the tool without establishing the consumer problem, target market or foreseeable harms. This raised concerns that the product would not meet the needs of the consumers likely to use it.

Guardrails that address one risk but create another. Some firms focused on technical model guardrails with limited consideration of how to support good consumer outcomes.

Excluding consumers in vulnerable circumstances by default. To reduce risk, some firms automatically excluded consumers with vulnerability indicators from AI journeys. This may be appropriate in some scenarios but could unfairly exclude those consumers if the firm doesn't provide another way for them to get support or benefits that meet their needs.

""

Smaller firms focus

It may not always be proportionate or feasible to test at scale. Smaller firms need to judge what is proportionate for them. We saw smaller firms use sample model outputs, add vulnerability markers to test cases, and compare outputs across scenarios. These approaches helped them spot potential bias or unfair variation without needing the same volume of testing as larger firms. 

Proportionately filtered and quality assured inputs can be useful where resources are limited. Good-quality sample data is vital when testing whether consumer-facing AI tools are producing appropriate outputs. One smaller firm, for example, used carefully filtered high-quality datasets that had been reviewed by people. This helped them provide known good outputs for testing their AI systems.

""

Good practice

Explaining the role and limits of AI at the right level for the consumer. Firms made it clear to consumers that they were using an AI-enabled tool, and explained:

• what it could and could not do
• when the consumer should look for help from another source

Using AI to make consumer communications clearer, with human review. Firms used AI to assess consumer communications and propose clearer wording. Staff reviewed all the changes before approving the final wording.

Using approved source material and guardrails to reduce hallucinations or inappropriate responses. Firms limited chatbots and generative AI assistants to approved source material, with checks and escalation routes. This meant issues could be identified and passed to a person where needed.

Maintaining auditable records. Firms kept records of consumer prompts, outputs, model versions, applied filters and escalation decisions. This helped monitor outcomes, complaints handling, explainability and investigation where a consumer challenged an outcome. We've seen positive examples of firms assessing their model’s performance against human-led processes to see how effective they are, and whether they need to add more checks.

""

Areas for improvement

Assuming AI outputs are clear without testing them. Some firms assumed their AI outputs were clear and helpful without testing whether different consumer groups understood them. This included consumers with lower financial capability, accessibility needs or vulnerability characteristics.

Giving consumers technical information without explaining what it means for them. Firms provided technical information about an AI model without explaining what it meant for the consumer, why it produced a certain outcome, what the consumer could do next, or whether the information was right for the consumer.

Overstating what an AI system can do. Firms overstated the accuracy, reliability or role of AI systems, or were unclear about whether AI or a person performed a particular function. This could mislead consumers and undermine informed decision-making and trust.

Taking a purely technical approach to explaining their model’s actions. Firms focused only on how a model works, rather than whether they could show good outcomes, spot poor outcomes, understand the likely causes and support challenge or redress.

""

Good practice

Identifying potential needs and responding appropriately. Firms used AI tools to spot potential vulnerability indicators, such as key words in consumer responses, spending patterns, or signs a consumer is struggling with a digital journey.

Giving consumers choice and continuity. Where appropriate, firms allowed consumers to choose between AI-enabled or non-AI support for straightforward tasks. Consumers could switch channels without having to repeat information. This helped maintain continuity and consumer control, and allowed firms to answer questions more efficiently. 

Learning from unexpected scenarios. Firms combined AI monitoring with sample reviews and structured governance so they could use unexpected scenarios, such as a consumer mentioning using a food bank or financial distress, to refine prompts, controls or escalation criteria. We have seen examples where firms used outliers to refine their models.

""

Areas for improvement

Weak monitoring and escalation. Firms left long gaps between reviews of live outcomes, or defined their escalation triggers too vaguely. Although there is no set timescale for firms' reviews and they will need to determine what is most appropriate for their use case, running them more frequently can help spot harm faster. This can be particularly effective when the firm has clear responsibilities and processes to address issues.

Assuming an LLM can reliably identify vulnerability without sufficient testing or safeguards. Some firms placed too much confidence in an LLM's ability to identify text-only indicators of vulnerability, particularly temporary ones. Firms should test their systems thoroughly to make sure they can identify gaps in approaches and make changes, eg add vulnerability indicators or questions the LLM could ask.

Creating inflexible automated journeys. Some automated interfaces made it harder for consumers to get support. Problems included unclear responses, not addressing the consumer's issue and not allowing them to move to another channel. 

""

Good practice

Structured challenge, testing and sign-off before deployment. Firms took a risk-based approach to challenging their plans for new deployments. They involved staff from areas across the business in testing and sign-off. Formal evaluation, clear accountability and executive approval helped identify and mitigate risks before launching new AI systems.

Central AI governance bodies. Firms, particularly larger ones, established central AI bodies, such as data ethics committees or AI boards, with members from different areas of the business. These bodies helped firms adopt AI safely and consistently; they share common learnings, set principles and risk tolerances, and inform senior oversight while avoiding unnecessary bottlenecks.

Scaling oversight through existing governance. Firms, including smaller firms, incorporated AI oversight into their existing governance frameworks. They used simpler risk tiers to increase testing, guardrails and monitoring for higher-risk use cases, overseen by their existing risk and monitoring committees.

Clear governance of AI throughout the consumer journey. Whether creating new oversight functions, or adapting existing approaches, firms planned governance frameworks that mapped their AI design, deployment and monitoring across the whole consumer journey, rather than treating models as standalone tools.

Escalating ethical issues for senior consideration. Firms set up committees to address ethical dilemmas emerging from AI systems. The committees considered the evidence and made strategic decisions for the business in line with their regulatory requirements. For example, if a tool spotted spending patterns that could suggest consumer vulnerability, an ethics committee could decide how to use this insight fairly.

Building governance layers for agentic AI. Before deploying more autonomous agentic systems, firms designed a suitable governance structure. This included escalation routes and compliance controls that maintained accountability, human oversight and risk management across complex workflows.

""

Areas for improvement

Informal approval processes or generic governance. Some firms didn't have formal processes to test, challenge and approve AI use cases. Their approach was inconsistent across different use cases, including the evidence they used to justify decisions. Control functions weren't involved enough, they didn't consider AI-specific risks properly, and they didn't adapt their governance processes proportionately to the potential impact.

Retrospective governance. Firms designed and deployed AI use cases without first considering how to govern them. They tried to fit them into rigid existing frameworks after they'd already made key design decisions. This made it harder to show who was accountable for decisions, manage AI-specific risks or tailor oversight to the use case.

Governance operating in isolation. Firms' governance procedures for AI were less effective if they didn't clearly link them to their existing governance structures. Some only had governance for specific AI lifecycle stages, eg post-launch, or governed AI decisions in isolation. This left gaps in the process. Some firms didn't have a structured approach to testing and governance, leading to gaps in auditability, documentation and risk management.

Unclear responsibility for AI systems. Responsibilities for AI systems in some firms were unclear, fragmented or overly diffused, particularly across technology and business teams. Firms that didn't coordinate fully across regions or business lines often ended up with ineffective and inconsistent controls.

""

Smaller firms focus

Scaled governance frameworks. We have seen examples of smaller firms scaling and tailoring their governance frameworks to their size, capability, and risk profile rather than copying large firm models. This allows them to deploy AI safely by focusing their in-house expertise where oversight and challenge are most needed. Smaller firms have also used phased and iterative AI introduction, simpler risk tiering, clear accountability and proportionate review processes, escalating only higher risks.

""

Good practice

Forward-looking monitoring. Firms adopted monitoring approaches that tracked real-world outcomes and impacts, and adapted to evolving risks, such as model drift.

Using structured evaluation frameworks to assess generative AI outputs. Firms assessed their outputs against clear criteria.

Ongoing training and capability reviews. Firms gave staff ongoing training and carried out capability reviews of their human-in-the-loop and oversight functions. This helped them maintain effective oversight and make decisions once they had deployed. Some training was role-specific:

• Senior stakeholders: to understand AI-related regulatory expectations, governance responsibilities, risk appetite, accountability and escalation routes.
• Technical teams: to understand model risk, data quality, ethics, guardrails, controls, testing and monitoring requirements.
• Human-in-the-loop: to understand a system’s purpose and limitations, how to interpret and challenge outputs, ways to manage automation bias, how to judge issues independently, and know when to escalate or override.

Periodic review of governance arrangements. Firms periodically reviewed their governance arrangements to make sure they remained appropriate, taking account of technological change and any new regulations.

""

Areas for improvement

Narrow monitoring focus. Where firms monitored technical metrics rather than outcomes, they had less chance of meaningful oversight. Some firms treated governance as a point-in-time exercise and didn't follow up with sufficient monitoring or review after deployment.

Reactive or static monitoring. Some firms didn't adapt their monitoring for AI systems, despite evolving and probabilistic outputs. Some had reactive approaches, so they could only identify harm after issues arose rather than anticipating it.

Oversight functions lacking AI-specific capability. Some firms didn't have enough expertise in their second- and third-line oversight functions to identify or challenge emerging risks. Their training was generic or limited to awareness, without supporting effective oversight. In these circumstances, firms didn't have effective escalation triggers or intervention mechanisms.

""

Smaller firms focus

Some small firms ensure effective oversight by focusing their more limited specialist expertise where they need it most, and giving baseline training to relevant teams. 

This helps build enough understanding to identify risks, challenge outputs and escalate issues.

""

Good practice

Assigning clear SMF ownership for AI risks. Firms mapped responsibilities for AI across senior management functions (SMF), establishing clear accountability, governance and oversight for AI risks. Some firms highlighted the importance of an ownership structure and tailored roles to each AI use case, rather than assigning a single owner to each model. Roles could include technical, research and business owners, with a senior person accountable for overall oversight.

Accountability assigned at the beginning. Senior managers were accountable and owned risks from the design stage onwards for each AI use case.

Informed, documented and auditable SMF oversight. Firms made sure senior managers received appropriate information and oversight through documented governance processes, so they could understand key risks and challenges, and make informed decisions. They had auditable records of decisions, including rationale, challenge, and escalation.

Clear escalation routes. Firms had clear, evidenced escalation routes, including formal threshold-based escalation to governance bodies.

Targeted SMF training. Firms gave senior managers targeted training so they could understand AI risks and discharge their responsibilities effectively.

""

Areas for improvement

Poor documentation. Some firms' decisions on AI tools were poorly documented with limited audit trails or supporting rationale. This meant that senior managers couldn't show the steps they'd taken to assess and mitigate risks.

Delegating responsibility without sufficient challenge. Senior managers delegated responsibility for AI development entirely to technical teams without sufficient understanding or effective challenge. This was particularly concerning for higher-risk AI use cases, including those affecting consumers.

Gaps between AI risk assessment and SMF accountability. Some firms had no clear link between risk identification and accountability. This made it harder to ensure risks were owned, escalated and addressed appropriately. This could limit senior managers' ability to demonstrate that they understood the risk, challenged it appropriately and took reasonable steps to manage it.

""

Smaller firms focus

Smaller firms may have simpler governance structures, but they still need to have clear responsibilities, understand the risks, and make informed decisions. They are accountable for how they use AI in their business, even where third-party providers are involved. 

Good practice included named accountable individuals, with clear decision ownership and sign-off. One firm mapped every material AI use case to a named senior manager, with documented risk appetite and escalation routes before deployment. 

Another small firm emphasised the need to record whether staff needed to review AI outputs, whether this happened, and whether they approved the outputs, overrode them or escalated issues. This created an audit trail showing how oversight operated in practice, rather than relying on an assumed control. 

Due diligence and oversight processes do not need to be overly complex, but they should be proportionate to the risk. For higher-impact AI uses, stronger examples included: documented vendor assessments, named internal owners, defined service and escalation expectations, periodic performance reviews and clear fallback arrangements if the tool does not operate as expected.

""

Good practice

Human-in-the-loop as a risk-based control. Firms did clear risk assessments to see where they needed human oversight, and what it should look like. We've also seen firms making sure an appropriately qualified person is involved, with the expertise to spot key risks, and the authority to escalate them when needed.

Using AI to support human judgement. Firms used AI systems to support rather than replace people’s judgement. Staff were accountable and had control at key points in the workflow, particularly where decisions affected consumers or regulatory outcomes.

Embedding escalation routes into AI processes. Firms included human-in-the-loop controls at key design and operation stages, with staff empowered to intervene with clear authority, expertise and remit. They set clear thresholds for when human intervention and escalation were needed, and defined how AI outputs could be challenged or overridden.

""

Areas for improvement

Excessive reliance on human-in-the-loop controls. Firms relied excessively on staff to oversee AI outputs, without addressing underlying model risk or governance responsibilities. Human review may mask weaknesses in the model, while unclear governance responsibilities can leave gaps in accountability, escalation and oversight.

Lack of meaningful oversight. There was a risk of human review of AI outputs being a 'box-ticking' exercise, rather than meaningful oversight.

""

Good practice

Board ownership of AI strategy and oversight. Effective boards treated AI as a strategic business issue. They considered how it supported business objectives and affected consumers and markets, and whether their governance, accountability and operational resilience arrangements were appropriate. Regular management information helped boards oversee adoption, challenge decisions and remain within their risk appetite.

Using risk appetite to guide AI development and oversight. Boards actively set the firm’s risk appetite for AI use cases. This informed areas where the firm wanted to develop its capabilities, and areas where risks were too great. We also saw firms applying risk-based approaches, using risk scoring or tiering, to calibrate their governance requirements.

Adapting risk frameworks for AI. Firms assessed whether their existing risk frameworks, eg model risk, were appropriate for AI and adapted them where needed. This ensured their existing controls remained fit for purpose, while addressing AI-specific risks that they may not capture.

Clear ownership of AI agents. Firms developing agentic use cases registered their agents in a central inventory so they were easier to oversee. This can help firms maintain a clear view of which agents are live, who owns them, what systems and data they can access, and what controls apply. This supports accountability, monitoring and change management, and helps firms intervene when risks emerge.

""

Areas for improvement

Undefined risk appetite and thresholds. Some firms had no defined risk appetite or thresholds, which limits their ability to judge acceptable outcomes. Some also default to avoiding risks that are close to regulatory boundaries, rather than identifying ways to manage and mitigate them, leading to unintended poor outcomes.

""

Smaller firms focus

Some small firms started with narrower, domain-specific use cases. They avoided large, upfront transformation or broad use case adoption, taking an incremental, risk-based approach. This allowed them to test governance, understand risks and build capability before scaling AI adoption. An incremental approach also reduced the risk of widespread harm from untested systems.

""

Good practice

Assessing hosting choices for AI resilience. Firms made considered decisions on whether to host their AI models on company-owned architecture or a third-party cloud graphics processing unit (GPU). They considered a range of factors, including differences in capability, security, and risk management.

Using backup infrastructure to support continuity. Firms ensured that their third-party suppliers maintained appropriate backup servers in different locations to ensure that AI models would continue to function if there was, for example, an unanticipated spike in demand.

Piloting lower-risk use cases before full deployment. When exploring how they could deploy AI solutions, firms ran lower-risk use cases to identify and mitigate any issues early on. This helped ensure those issues wouldn’t have a major operational impact.

Stress-testing AI tools before deployment. This enabled firms to identify potential vulnerabilities and refine guardrails so that the product was resilient before deployment.

Mapping AI dependencies into resilience planning. Firms considered their dependence on AI models, including those hosted externally, when assessing how severe but plausible scenarios could affect their business services. This is a requirement for firms in scope of SYSC 15A[9].

""

Areas for improvement

Failure to consider AI when reviewing operational resilience. Some firms did not appropriately consider their use of AI, or their dependence on AI models, in their operational resilience plans (where applicable). As firms moved from low-risk use cases into more material ones, they did not appropriately consider how it would affect their business.

""

Good practice

Reducing single-vendor dependence. Firms designed AI architecture that could operate with different providers, to reduce the risk of overreliance on, or inability to switch from, a single vendor. They also brought representatives of different functions into procurement processes before selecting vendors, including legal, compliance and risk.

Keeping clear records of AI use cases and providers. Firms clearly documented third-party arrangements. This included maintaining clear inventories of AI use cases and the models they were built on, as well as information sharing requirements. Some firms made a distinction in their inventories between experimental tools and production tools.

Maintaining visibility of vendor model changes. Firms may not be able to control changes made by large technology providers, but they can reduce disruption by understanding upcoming updates and planning for their impact. Some firms established vendor-oversight processes before deployment. They used contractual requirements to obtain notice of, and further information about, significant model changes. This helped them test and integrate updates without disrupting services.

Planning for disruption in third-party AI arrangements. Firms developed and tested contingency plans in case of unexpected changes to their AI provider arrangements. This can help firms understand the practical impact of provider changes, identify viable fallback options and reduce disruption if a model, service or contractual arrangement changes unexpectedly.

Assessing suppliers' AI use during procurement. Firms identified and assessed prospective suppliers’ use of AI as a routine part of procurement exercises and included contractual obligations for suppliers to use AI securely. Others introduced periodic questionnaires, vendor reassessments, or contractual provisions to help them identify new or modified AI components in third-party services.

""

Areas for improvement

Overreliance on untested vendor assurances. Some firms relied on vendors’ assurances without challenging or testing them. Firms remain accountable for outsourced AI services. If they don't test assurances, they may not understand model limitations, resilience dependencies or whether fallback arrangements would work in practice.

Overreliance on hard-to-replace AI providers. Firms didn’t do enough to prevent the risk of overreliance on individual AI providers. This can create operational resilience risks if failures or unexpected changes occur with the provider.

Single-platform dependence without exit options. Firms didn’t consider alternative providers, or develop plans in case the chosen model became unavailable or unsuitable. This may lead to firms being locked in, or dependent on a single provider to support AI use cases, reducing their ability to adapt to changing circumstances.