Advancing the Analytical Engine

08 September 2017

Analytical models and machines can require lengthy crafting and re-crafting. But the insights they offer are invaluable.

Speaker: Stefan Hunt, Head of Behavioural Economics and Data Science

Location: Edinburgh Credit Scoring and Credit Control Conference, 1 September 2017

The Analytical Engine of the ‘father of the computer’, Charles Babbage, is a sight to behold. Containing a mill, similar to a modern computer’s central processing unit, and a printing mechanism, you can go and see the unfinished model for yourself in the Science Museum in London. However, its journey from initial idea to prototype was not a simple one. 

Back in the early 1820s, Babbage first began work on his ‘Difference Engine’, a six-wheeled machine to perform certain mathematical calculations. But while work on creating the engine was delayed, Babbage kept thinking.

He re-evaluated his previous work and moved onto inventing the much-more revolutionary Analytical Engine, a general-purpose computer and one of the most celebrated icons in the history of computing. He continued working on his engines over several decades in the 19th Century, up to his death in 1871.

In the subsequent 146 years, much has obviously changed. Not least our methods, the scale of data we have and the speed that we can process information.

But the fundamental process of analytics still involves a data scientist, statistician or economist using a machine to better understand the world around us.

And like Babbage – albeit on a much more modest scale - the FCA has gone through a process of iteratively crafting, honing and redesigning its various analytical machines and models since the organisation took over the regulation of consumer credit in 2014.

Today, I’m going to discuss the FCA’s development of its own analytical designs, models and techniques for consumer credit. 

  • I will first provide some context for how our analytics support effective consumer credit regulation: why our models are designed the way they are and how they help us identify harm.
  • I will then discuss examples from our own 'analytical engine'. I will explain the analysis that allowed us to make decisions on setting a price cap on payday loans. And I will then describe the analysis we conducted more recently for high cost credit markets, in general, and for overdrafts, in particular. Together, these examples demonstrate the range of techniques that help us assess the probability of harm, develop evidence, create remedies and evaluate policy outcomes.
  • I will summarise the three main lessons we have learnt so far: the importance of credit reference agency (CRA) data; the importance of data direct from firms on consumer behaviour and outcomes; and the need to use a wide variety of methods, from advanced econometrics to qualitative research.
  • I will finally discuss some techniques, such as machine learning, which we are developing now and you will hear more about in the future.

But before I launch into the story since 2014, I want to briefly outline the FCA’s decision-making framework and how the organisation applies it to consumer credit. 

Consumer credit and the FCA decision-making framework

Consumer credit performs a variety of critical functions for UK consumers. It enables people to manage temporary cash-flow shortfalls that arise from receiving income at a later point than expenditures, such as rent. And it enables people to meet their longer-term needs and smooth the repayment of larger, indivisible purchases: for example, buying a car or household appliances or financing special events, such as weddings. These two functions are especially important for households who do not own property and so have no large asset to secure debt against.

You can see the importance of consumer credit in borrowing statistics: many people hold debt with products such as credit cards, motor finance or payday loans. As of November 2016 an estimated 27.4 million people, over half the UK adult population, had outstanding consumer credit debt.

One of the metrics that lenders care most about is ‘PD’, the probability of default on a credit agreement. PD is a key input for assessing the profitability of lending. PD comes in very different forms, as there is not a single definition of default. But the metric has some nice features. Default is a single observable, quantitative outcome. We can predict the likelihood of it and, after lending, assess how our prediction compares to subsequent realised default rates. This provides a valuable feedback loop of knowledge to improve the accuracy of our predictions by iteratively refining our methods and data. Better prediction enables firms to make more informed decisions over how to optimise their lending decisions, primarily for profit maximization. But prediction could also be used for assessing affordability and for treating customers fairly.

Unfortunately harm is not always a nice single, observable, quantifiable outcome.

The FCA’s recent Mission document set out its strategy for how it decides when and how to intervene in markets in order to deliver the greatest public value. The FCA has a slightly different target than industry, which is the reduction of harm to consumers or making markets work as well as they could. In the context of credit, I’m going to call this ‘PH’, the probability of harm to consumers. In credit, harm can come in many forms, including prices being too high, people borrowing too much or people taking out products when they would be better off not doing so.

Unfortunately harm is not always a nice single, observable, quantifiable outcome. When you bang your knee it will almost always obviously bruise, but you sometimes sustain other internal damage to the tissue or ligaments that may not be so immediate. With credit as with the human physiology, harm can be hard to predict, assess and evaluate.

The FCA uses a range of data and other information to predict where harm is likely to occur. Assessing harm can require large information gathering exercises and may need the inclusion of range of measures. While much assessment is quantitative, difficult judgements may be needed to interpret the evidence. Over time regulators then evaluate whether our interventions have been successful.

Economic theory lies at the heart of the FCA’s Mission. It provides a framework for considering when consumer harm may be caused by 'market failures', which could be corrected. Analytics provides the basis for testing whether markets are failing and harm is occurring.

Like all markets, credit markets can have issues, leading to harm for consumers. These 'market failures' can be traditional or 'neoclassical' failures, such as information asymmetry, where some market participants have more information than others. As you are all well aware, reducing information asymmetry is why we have credit reference agencies, to help improve the functioning of credit markets and foster their growth. There can also be behavioural market failures, where we see consumers struggling to act in their own best interest. This is the realm of behavioural economics, the intersection of psychology and economics.

A famous example that illustrates the difficulties people can have is where someone puts their credit card on ice, literally putting it in a tray of water and then into the freezer, to make sure that they can control their spending. This difficulty with self-control is something I am sure we can all relate to in some form. Whether it spending less on credit cards, struggling to avoid eating cake or just not being able to rouse yourself to get out of bed and go to the gym in the morning, we all wrestle with conflicting desires. It always seems to be something we’ll plan to do from tomorrow. But when tomorrow comes it is always the day after tomorrow we’ll start changing. Tomorrow never actually comes. With these kinds of behavioural market failures, assessing harm can be difficult, requiring many types of evidence and some judgement.

I will now talk about the judgements the FCA has made and analysis it has executed over the last three and a half years. So, I should first go back to 2014, when the FCA was developing the payday lending price cap…

Setting the payday lending price cap: the use of advanced analytics 

High-cost short-term credit lending – known colloquially as payday lending and I will stick with that term – became increasingly a national issue in the UK from 2010 onwards, on several occasions appearing on the front page of newspapers. There were a number of factors that drove these headlines, including an increasing number of people with payday debt approaching consumer organisations.

In early December 2013 Parliament created primary legislation mandating the FCA to impose a price cap on payday products offered by consumer credit firms and have it in operation in 13 months’ time, by 2 January 2015. The FCA had to decide on the structure and the level of the price cap.

To give some context, the FCA’s analysis subsequently found that in 2012 and 2013 the average payday loan was £270 for 17 days. The average cost was about 1.2% per day, when a variety of additional fees and charges were included. This amounted to £55 on average - a considerable amount for a small, short-term loan. As many as 4.6 million people, or around 10% of the adult population, had applied for a payday loan during these two years. And, for a customer’s first loan, firms were willing to lend to people with over a 50% chance of not paying back the loan – in some cases a fair bit over – because future loans to non-defaulted customers were so profitable. The FCA is worried about the potentially significant consequences for these high risk borrowers.

There were tricky trade-offs between protecting consumers, making sure that they had access to credit and ensuring effective competition in that market. The analysis needed to focus on three main questions:

  1. First, what happens to firms and firms’ lending decisions as a result of a price cap?
  2. Second, what options are there for consumers who no longer have access to payday loans?
  3. And, third, are these consumers better or worse off as a result of not getting payday loans?

The FCA wanted to approach the analysis in as rigorous a way as possible to allow the organisation to make decisions on these difficult trade-offs. But there was not a solid evidence base in the UK in order to take these decisions and the evidence from the US market was mixed.

Getting started: estimating the payday price cap’s impact on firms

Figure 1: Estimating the payday price cap’s impact on firms

To do this, the FCA created a model of how payday lenders make decisions. The lifetime profitability of new customers – based on their first loan and all future loans – was expected to broadly to increase with credit score, with the score for almost all firms based on proprietary credit models. In Figure 1 above, you can see the profitability is always increasing with increasing credit score (it is monotonic and linear), but the model did not require this.  Applicants below some certain level, a, are unprofitable and so the payday firm will fix a as its cut-off point in the lending decision-making process. All applicants with scores to the left of a would not get a loan.

With the introduction of a price cap, revenues decrease, reducing firms’ profitability, 'effect 1'. And some firms may exit given their fixed costs. Some people, those with credit scores between a and b, will no longer get loans, 'effect 2', and obviously we need to think about whether there is any harm done to these consumers. Lastly we have 'effect 3': that those people who still get loans get them more cheaply.

The 'supply-side' model helped answer the first of the three questions: what happens to firms and firms’ lending decisions as a result of a price cap? Just as in the model shown, it estimated expected customer lifetime profitability for different credit scores. And it allowed the FCA to model what would happen for a range of different cap structures and levels.

To do this, data was needed to create models. The organisation obtained details of the loans granted from the top 37 lenders in the market, covering 99% of the market or around 20 million loans. For 11 larger firms, covering approximately 90% of the market, details were provided on not only all loans, but all applications, accepted and denied, including details of the lender’s credit scoring process and the credit score assigned to each application. The organisation received fully-disaggregated revenues and costs at the loan level, so to examine the profitability of each loan. Individual loan applicants across all the firms were matched to six years of credit reference agency data, providing a wide picture of people’s borrowing behaviour. Overall, there were 4.6 million individuals in the dataset, including 1.5 million people who applied for payday loans for the first-time. For some firms there weren’t get good historical credit scores and the FCA had to create our own scores using the credit reference agency data and the firm’s behavioural data. I will not go into the details, as the analysis was pretty conventional.

Data analysis was performed to generate credit scores, estimate customer profitability, regenerate lending decisions and ultimately generate the models.

Figure 2: Estimating the impact of a cap on firms

Impact of cap on firms graph

These models worked for 8 different firms. This allowed the FCA to estimate what would happen to the market for a different cap. Figure 2 above illustrates one of our models. The model shows, for any given price cap, how firm revenue, the number of loans and many other metrics would change. And, by incorporating fixed costs, the FCA could model how many firms would be expected to stay in the industry. (At least in steady state with this partial equilibrium analysis that did not allow for firm to adjust different elements of their business model, e.g. the length of loans offered, to compensate for the new environment. We incorporated these adjustments in other analysis.)

This analysis provided a detailed estimate of the relevant different costs and benefits of the price cap, which allowed the FCA to take a decision on the level of the cap.

Shifting models: estimating the payday price cap’s impact on consumers 

But two questions remain what options are there for consumers who no longer have access to payday loans? And, are these consumers better or worse off as a result of not getting payday loans?

In many ways, these are much harder questions, because they concern harm to consumers. When we worry about people who have high credit risks taking out payday loans, we are concerned that, at the prevailing interest rates, these people are harming themselves by taking out payday loans. 

Approximately one in six people with consumer credit debt suffer moderate to severe ‘financial distress’

Earlier I discussed how, in a similar way, we may be harming our health substantially by not resisting bad food or not going to the gym. In this context too, the influences of our behavioural biases are incredibly important: lending choices can have far-ranging consequences, including harm to consumers. For instance, approximately one in six people with consumer credit debt suffer moderate to severe ‘financial distress’, experiencing financial difficulties or other issues such as mental health problems from the strain of repaying their debts.

In order to understand this, and other, issues properly and develop effective, evidence-based policy as a result, it is important that we not only understand where harm is being caused, but how and why. We need to assess the evidence carefully, including that from behavioural science.

An assessment was made on whether payday loans cause harm to high risk people close to the boundary of just being able to receive payday loans – and many different elements of consumer harm, including a range of different measures of financial distress, also needed to be considered.

Pinning down causation is difficult, but it turns out that for payday loans there was what economists call a ‘natural experiment’ – when treatment and control conditions are determined naturally, but the process resembles random assignment – that we could use to estimate the impact of payday loans. The FCA used a technique called regression discontinuity design.

Applying analytics: regression discontinuity design

Figure 3: Regression discontinuity design 

regression discontinuity design graph

Let me explain how this technique works. If you look at the left-hand side of figure 3 above, you can see the x-axis shows the proprietary credit score from a particular payday lending firm. The y-axis on the left hand side shows the likelihood of the customer getting a loan or not from any firm in the whole market, not just the firm that the consumer applied to. This is because consumers rejected by one firm could go to another one. There is a jump in the probability of the customer getting a loan at precisely the proprietary credit score that the firm uses, in this example at a credit score of 500.

Now turn and look at the chart on the right hand side of figure 3. Here you can see the same x-axis, credit score, but we have a new y-axis, in this case the probability of missing a payment on loans other than the payday loan. In this example the probability of missing a payment jumps at the credit score of 500. We can causally attribute the jump to the probability of getting a payday loan.

Another way to see this is to imagine people who have a credit score of 499 and a credit score of 501. Essentially these people are identical in all ways (and the FCA’s analysis verified this), apart from the group with the slightly higher credit score has a much higher likelihood of having a payday loan. This is just like a randomised controlled trial with the 501 credit score group being the treatment group and the 499 group being the control group. And we can see that the treatment effect of having a payday loan is negative. In this example, it is a 5.9% increase in missing non-payday payments.

The data that I described earlier was precisely configured to allow us to empirically estimate this model. You could see the credit scores of applications and whether the person got a loan anywhere in the whole market. And from the credit reference agency files, the FCA could observe a very rich array of observable credit outcomes, like the one shown in the right-hand of figure 3. This allowed the organisation to infer benefits or harms from payday use. 

Figure 4: Estimating the causal impact of payday loan use

Casual impact of payday loans

 

The financial situations of consumers at the margin of getting payday loans were worsened by receiving loans.

My team calculated the impact of payday loans in the whole market by aggregating the treatment effects for individual lenders, and a set of clear results was found. The most natural interpretation was that the financial situations of consumers at the margin of getting payday loans were worsened by receiving loans.

For example, figure 4 above shows our estimate of the impact of payday loans on the use of unarranged overdrafts. In the 12 months leading up to taking a payday loan there is no treatment effect: the group who received payday loans (just) had exactly the same overdraft usage as the group who did not receive payday loans (just). This is a falsification test: it tests whether the treatment group and control group can be considered comparable - as the comparison is made before the application for a payday loan it passes - it is exactly what should be seen.

In the month of receiving a payday loan the unarranged overdraft usage went down 1 to 2 percentage points, off of a base of around 25%. In others words, a quarter of this group of people were using an unarranged overdraft in any given month. This is again what we would imagine, as the people with payday loans had just borrowed £270 pounds or so.

But three months after applying for a loan the people who received loans are using unarranged overdrafts by 3 to 4 percentage points more, each month. And that increase persists for at least 12 months after the original loan application. This, therefore, looks like a short-term benefit followed by a significantly larger, more persistent and on-going cost. This is exactly what might be predicted if the borrowing behaviour was driven by present bias, or other similar behavioural effect, just like the 'not going to the gym' example I mentioned earlier.

In fact when the FCA looked at a variety of other measures and saw a financial deterioration: for example, a marked increase in delinquency and default on non-payday loan products, a 20 point decrease in credit score and a variety of other effects.

Broadening our scope: using surveys

Figure 5: Surveying consumers on their borrowing

If no payday loan... most do not borrow graph

Now, of course, credit files only say so much about the welfare outcomes of individual borrowers. To get a much fuller picture and so assess harm more completely a 2,000 person survey was also commissioned to understand other measures. It had a large array of questions: on financial distress, on subjective well-being, on borrowing from friends and family and a carefully sculpted question on the use of loan sharks (i.e. illegal money lending). The survey design mimicked that of the regression discontinuity design and focused on those individuals with credit scores close to the lenders’ credit score cut-offs.

As you can see from figure 5 above, we found that the majority of borrowers, close to half, went without any form of borrowing, and must have adjusted in some other way. The next most likely outcome was to borrow from friends and family. Importantly there was no increase in loan shark usage, no impact on subjective well-being and no impact on financial distress. Together with the results from the CRA files, the FCA concluded that those people at the margin of being able to borrow from payday lenders, who would no longer be able to borrow after the price cap was implemented, would mostly go without borrowing and in fact be better off from not using payday loans as they were being harmed by using the loans. The second and third questions were answered.

Together the evidence created provided the backbone for taking the tough decisions about the trade-offs and choosing the right structure and level for the price cap. The FCA set the cap at a rate of 0.8% per day, with a maximum cost of 100% of the amount borrowed, plus £15 pounds maximum for default fees.

Payday price cap: an update in 2017

We are now over two and a half years on from the price cap being implemented. And in July this year, as part of our broader on-going review of high-cost credit markets, the FCA published its first review of how the payday lending market is functioning.

There was broad agreement between industry, regulator and consumer bodies that the cap had worked well and did not need changing. The FCA found that the payday loan market now is significantly smaller: the number of loans issued fell by over 65%, from 10.3 million in 2013 to 3.6 million in 2016. And despite the fall in lending volumes there remain a healthy number of firms still active in the market (over 140 with permissions to lend and at least 30 issuing new loans as of December 2016). Consumer default rates have more than halved from 2014 to 2016 and debt charities are dealing with far fewer payday lending cases. Consumers without access went without, as predicted, and there wasn’t evidence that these consumers were increasingly turning to illegal lenders, as some feared.

The payday lending example covers off several of the data sources and analytical techniques that the FCA has been using. For my own team, there were three significant lessons:

  • first, that credit reference agency data was an incredibly important resource, in particular for looking across products on a consistent basis, understanding consumer behaviour and for taking a broad view of harm
  • second, individual data collected from firms was crucial for understanding consumer behaviour, in this case whether consumers had applied for loans and passed credit score checks at a particular firm and if they had been denied a loan whether they got one from another firm
  • third, that in order to get a full picture you sometimes need a mix of methods, including the use of survey data

I will now outline two more recent examples of how the FCA is applying its 'analytical engine' in different contexts.

Advanced analytics for consumer credit regulation: high cost credit

My next example outlines how the FCA used CRA data to undertake market-wide analysis for the FCA’s recent publication on high-cost credit in July. CRA data is especially insightful as we can look across different products on a consistent basis and see them in the context of a consumer’s portfolio of debt.

There are a wide variety of products potentially considered high-cost credit. Some of them you have heard of, such as payday loans. Others you may not have heard of, such as rent-to-own loans. These loans are typically referred to as being hire purchase agreements, whereby the borrower takes out credit to purchase a good – such as furniture or TVs - but does not gain ownership until the last payment has been made.

A key question for us as regulators is how similar or different are these markets?   

Figure 6: Measuring default rates over time

Default rates over time graph

The FCA’s analysis found there to be large differences in the size of these different markets. Arrears and default rates across these markets also differ considerably. Some had undergone substantial changes since FCA regulation began. The most notable in the case of home-collected credit where nearly 15% of loans issued in 2013 entered default, by 2016 this had decreased to under 5%.

Figure 7: Credit scores of high-cost users 

Credit scores of high cost use

The FCA examined the credit score (risk profile) of borrowers taking out products potentially considered to be high-cost in 2016 – but please note figure 7 above is indexed. There are three things that really stand out:

  • Firstly, the customer bases of these products are noticeably concentrated at the sub-prime end of the market.
  • Secondly, for most of these markets the distributions of the customer bases by credit score are remarkably similar.
  • Thirdly, the exception to these trends is catalogue credit. This is a revolving credit product attached to particular retailers, especially online fashion, and has a very different customer base that is typified by noticeably higher credit scores.

Figure 8: Breaking down different types of consumer debt 

Different consumer debt types

In figure 8 above, you can see quite large differences in the mix of debts held by borrowers using different high-cost products. Rent-to-own stands out as being a particularly concerning case. The median consumer has outstanding debt on eight products and over a third of their debt was on these very high-cost rent-to-own agreements. This is important:  these borrowers often have very low incomes and the costs of borrowing are often multiples of the retail value of the good. Guarantor loans, on the other hand, have lower costs and are typically taken out by consumers with relatively higher estimated incomes.

When we combine these insights with other findings from CRA data on market sizes, measures of consumer vulnerability and metrics of consumer harm we start to build up a detailed picture of how these markets work and where there may be a case for regulatory intervention. But I need to flag the range of our current 'analytical engine': we have only really scratched the surface so far of how such data can be used to inform our understanding of the behaviours of consumers and lenders.

Advanced analytics applied to overdrafts 

I will now talk through the example of our on-going analysis of consumer use of overdrafts. It illustrates how individual level data on consumer behaviour is instrumental for regulatory analysis.

Concern over the costs of using overdrafts has been a perennial issue. Who are the consumers who incur the charges and how much do they pay?

As with the research on payday lending, the FCA used its statutory powers to gather a big dataset to help understand consumer use of overdrafts and assess harm. This included anonymised data of the full transaction history of 250,000 customers for two years from the top six current account providers - this showed exactly what happened to each customer. Individual level data is particularly useful for regulators, because aggregate data can be misleading, especially if there are individuals or groups with vastly different behaviours and risks of harm.

Data science, in particular unsupervised machine learning, is especially helpful. It supports the understanding of different types of consumers and their respective patterns of usage. Such understanding of different consumer types helps us to better understand the nature of consumer harm and design remedies to target this.

Figure 9: Heatmap comparing the profiles of arranged and unarranged overdraft users 

Unarranged OD users younger and with lower incomes than arranged OD users

This is an early example of my team’s work. We look at the proportion of people using unarranged overdrafts by their age and monthly income. Consumers incurring many charges are typically younger 18-30 and there is little relationship with the amount of money flowing into their current account, a proxy for income.

Now if you compare this to those using arranged overdrafts, you find a very different consumer base. We estimate that consumers incurring arranged charges have higher incomes and tend to be older than unarranged user. So it certainly seems as though the distribution of consumers using arranged overdrafts differs somewhat from those using unarranged overdrafts.

Let’s dig into this a bit more. When thinking about harm, one issue is whether the same consumers are repeatedly incurring costs from using overdrafts. This could be a sign that they are not learning from past mistakes or are struggling in a debt trap they are unable to get out of.

Overdrafts clustering

Figure 10: Clustering arranged and unarranged overdraft users

Few persistent unarranged OD users compared to arranged OD users

My team clustered our dataset according to unarranged overdrafting patterns. These patterns included the frequency and length of overdraft episodes, as well as the time between consecutive overdrafting episodes. The clustering throws up four key groups of consumers, as you can see in the figure above.

To assess this look at when consumers incur charges for using overdrafts on a daily basis over a one year period. Each tiny row in the red portion of figure 10 above represents a different consumer and these are ordered by how frequently they use unarranged overdrafts. This only shows those who used unarranged overdraft at least once over a one year period.

This visual really helps us pick out different segments of consumers. Only a small proportion of consumers use unarranged overdrafts consistently. Half of those who incur charges do so only once or twice.

My team did the same for consumers incurring charges on arranged overdrafts. It is striking how different the right-hand chart of figure 10 looks. We can see a thick almost solid blue box at the top of the chart – this group is in their arranged overdrafts for almost every day over a one year period.

The band below it shows a bit more white space – these consumers have prolonged spells of arranged overdraft use but do dip in and out. Then below this we have people who rarely use their arranged overdraft facility.

Compared to consumers using unarranged overdrafts, a much larger proportion of consumers using arranged overdrafts are repeatedly incurring charges. This group at the top are especially important.

Analytics techniques: better together

I’ve hopefully now demonstrated the power of both CRA and individual firm level data. The final lesson I have learnt over the past few years is that different analytics methods often work better in conjunction than in isolation. I will continue to refer to overdrafts as an example as I illustrate this.

The FCA is using design thinking – psychology and marketing combined with qualitative research – in order to design alerts.

Alongside our data science work, we are combining our domain-specific knowledge on overdraft use with insights from behavioural economics about when people may be better off by not borrowing. We are applying a suite of analytical techniques to understand these tricky issues and assess consumer harm.

One strand of our work on overdrafts is assessing the effectiveness of alerts. The idea here is that an alert – such as a text message – notifies consumers that they are close to incurring an overdraft charge and enables them to act before incurring such charges.

The FCA is using design thinking – psychology and marketing combined with qualitative research – in order to design alerts. Consumers have given feedback on designs and ideas in order to give them the best chance of reducing harm (and minimising annoyance!) in the real world. These are techniques commonly used in industry to design effective marketing campaigns and are increasingly important in the design of regulations.

Big data analysis can help us understand the optimal groups of consumers these alerts should be sent to and under what circumstances. To complement this, the FCA is also working with leading academics to develop innovative econometric techniques that assess whether the times consumers use overdrafts are likely to be mistakes.

Last, but by no means least, are field experiments, also known as randomised controlled trials. There are quite a few firms who regularly trial changes before launching them. This is great! Such testing really helps us as regulators to understand what does and does not work. It can also throw up unexpected findings. There are helpful to know in advance of considering whether or not to make rules affecting a whole market.

My team are carrying out trials both in the ‘field’ with real consumers making real financial decisions and in hypothetical situations, either studied in the 'lab' or online. These have pros and cons. Lab tests are often faster to run than field trials and the former also often enables us to test a larger set of potential interventions. In some cases, such as price comparison websites, lab tests can provide the ideal environment for such testing.

But the FCA does not do lab or field trials for all of its policies. This is simply neither practical nor necessary. The organisation is planning to publish a paper shortly discussing how and when it decides to conduct trials. But, in brief, it tends to conduct trials in cases where a proposed intervention is likely to have a significant impact on the market - and there is sufficient uncertainty over the intervention’s impact to warrant testing. The work on alerts is one example.

Analytics at the FCA: piecing the components together 

There are many similarities between the analytical work the FCA does and that done by industry. But there are also a number of differences, in use of technique, but also, of course, in objective.

As a public body the FCA’s aim is simple: to serve the public interest by improving the way financial markets work and how firms conduct their business. But in order to form judgements and make decisions that prevent harm to consumers, it relies on accurate measures of consumer and societal welfare. Over the past few years of FCA regulation of consumer credit, there are three key lessons that I have learnt.

First, whilst there is a difference in the focus for analytics for the FCA and firms, credit reference agency data is a key source of information for both. For the regulator, credit reference agency data is an essential resource for revealing market trends, consumer behaviour and emerging risks, as I’ve mentioned when considering the FCAs review of high cost credit.

Second, data from firms on real consumer behaviour at the individual level, rather than the aggregate, can be instrumental for regulatory analysis. I have spoken about how this was used to help set the level of the payday loan cap, as well as how we’ve used it in investigating overdraft alerts.

Finally, just as an engine requires a combination of various cogs, wheels and bolts to function effectively, an array of different analytical techniques need to be used together in order to achieve optimum results within our regulatory context. These range from:

  • econometric studies that use natural experimental techniques such as regression discontinuity
  • to surveys of consumers and other qualitative methods: that can be used to understand how consumers think about particular settings
  • to experimental techniques, including both field and lab experiments

Individually each of these techniques may only shift a few gears or move some cogs. These may have practical limitations – for instance, field trials take a long time to carry out and are not always practical. Behavioural economics shows us that people’s preferences might not always be reflected in their actions. It, therefore, also requires careful analysis to interpret survey results that rely on consumers to introspect and judge their own actions. Although we have to be judicious and proportionate in how we choose what level of detail to go into in our analysis, understanding consumer welfare and the potential for harm requires a full suite of analytical tools. 

Optimising analytics: past, present and future

Babbage’s story of design and invention in the 19th century, teaches us the lasting value of continual improvement, innovation and evaluation. From the past few years of consumer regulation too, I have taken my own lessons on how to use analytics to inform an approach to problems and the design of interventions.

Whilst you can travel to see a portion of the Babbage’s 'Analytical Engine' for yourselves in South Kensington, hopefully today I’ve also given you more than a peek under the bonnet at the analytical techniques currently used by the FCA.

It is committed to making sure that it is an evidence-based regulator - the research I’ve discussed today is part of that commitment. Over the next 12 months you can also expect to see how research from a variety of the projects mentioned today will inform future policy.

Get Insight in your inbox