It sounds obvious. Before you introduce a policy you should have grounds to think it will actually work.

With this in mind, policy makers have put a lot of resource over the years into evidence based policy. But there remains a difficult question. Does this offer genuine assurance that a policy will be effective?

Take striking a match, striking by itself is not enough to light the match, as anyone who has tried to light a barbecue on a windy day will testify. It will only produce a flame if oxygen is present, the match and the box are dry, it is not windy, and so on.

The basic point here, as philosophers of science Nancy Cartwright and Jeremy Hardie have argued, is that having evidence of one cause (striking the match) isn’t enough to be confident that the effect you want (the match lights) will materialize. You still need evidence for other ‘support’ factors too – the calm, dry conditions and the oxygen.

This has implications for policy making.  If you want to be confident that your policy will work, you must know what other support factors are required.

Covering your ski injuries

In finance, this is not always straightforward. If, say, a financial regulator found a firm selling a travel insurance product to skiers without notifying them that it didn’t cover winter sports, one obvious remedy might be mandatory disclosure. In other words, instructing firms to let customers know if there policy includes coverage for ski injuries.

Will this work? Possibly. But as in the match example, regulators need to know what else needs to hold to be confident the disclosure will be effective.

Will the customer notice the disclosure? Will they read it? Will they understand it? Will they be able to weigh up the inclusion or exclusion of winter sports against other factors they care about? Like striking a match on windy day, disclosure alone may not guarantee people make informed choices.

If the regulator has sufficient time and resource, they could gather evidence for each of these support factors.

They might design the disclosure to be obvious and clear (and perhaps test this in a trial) so they are sure consumers notice it and read it. They might carry out a survey. They might well look at literature on financial capability to assess how likely consumers are to understand the disclosure in the context of insurance products.

They might also look to the growing body of research on behavioural biases to understand how the human mind reacts to disclosure.

After all that, they might trial the final disclosure to get evidence that it works in something approaching a real life purchasing situation. At this point, the regulator might be pretty confident the disclosure will work.

You need theory too

There is, however, an important layer of additional complexity. Regulators don’t just need evidence, they also need theories to build an accurate story (or ‘view’) of how things work together. 

Most importantly, theories are needed to say how different parts of the causal system act - and interact - to bring about their effects and also to set out the causal factors that are and aren’t relevant.

In the hypothetical disclosure example, the ‘theory’ is how our skiers get, digest and use information to make purchasing decisions. This, in turn, helps you identify what support factors are needed for the disclosure to work. It also allows you to see where else you might need to get evidence - directing your search in fruitful ways.

Multi-disciplinarity

Financial regulators also need wide-ranging and diverse sources of theory for building a picture of the causal system underlying an issue of interest. 

To list a few possibilities:

• there is the economics of how the relevant markets and competition work
• the political, legal and institutional workings of financial services markets and their regulation
• there is the psychology of how consumers and financial services staff behave
• there is accounting to understand, map out and measure firms’ financial activities
• there are the structures and norms through which firms govern
• there are the cultural drivers of staff behaviour

That list could be extended easily. The point is that financial regulation problems tend to be deeply complex and cut across many specialisms. Meaning you often need to take a multi-disciplinary approach to problem solving in financial regulation – a point picked up in Peter Andrews’ recent speech 'Beyond economics'.

The practical consequence of this is that even an apparently ‘simple’ and intuitive intervention like disclosure, could require a lot time (and a great deal of evidence) before you can be confident it will work. 

Conclusion - the policy maker’s uncomfortable role

This leads to a slightly uncomfortable truth. Policy makers don’t have the resources and time to investigate everything that may matter to the problem at hand. Also other factors, like the need to try to solve urgent issues, means they have to impose limits on their investigations. So there are limits to the extent to which the causal system can be accurately and fully set out.

The benefit in this approach is that serious issues are dealt with in a more timely fashion and without expending too much resource. But it also means uncertainties (and sometimes ‘deep’ uncertainties) remain - so there is an implicit gamble that the causal understanding of the underlying drivers is ‘good’ enough. Of course, other factors will influence what is ‘good enough’ here – such as the need that policy-making processes and interventions meet requisite legal standards.

This, in turn, places significant importance on ex-post evaluation, where evidence is gathered on how previous policies have or have not worked. But this is a discussion for another note.

This article builds on a talk given by philosopher Prof Nancy Cartwright at the FCA in December 2016. Our thanks go to both Nancy Cartwight and economist Jeremy Hardie CBE.