During the time of British colonial rule in India, the government was concerned about the number of venomous snakes in Delhi, so it offered a bounty for every dead cobra. Initially this proved a successful strategy, resulting in many dead cobras.
However some of the more enterprising citizens realised they could breed cobras and derive an income from collecting bounties. When the government finally cottoned on, they scrapped the reward scheme and snake breeders proceeded to release the now-valueless cobras onto the streets. Delhi ended up with more snakes than before.
The British government’s spectacular misjudgement coined the term ‘cobra effect’ and illustrates an important point. Policies that seem entirely sensible can lead to unintended – and sometimes unpleasant – consequences.
This is not, of course, news. Few doubt the importance of investigating outcomes to assess the merit of interventions. It’s implicit in any commitment to 'evidence-based policy making'.
Curiously though, in the real world we see relatively little effort to test whether a past intervention has led to a ‘cobra-effect’. A surprise because, in theory at least, so-called ‘ex- post evaluations’ offer public policy makers a compelling list of benefits.
The most obvious is that testing your intervention can give you ways to improve (or remove) it, as well as to possibly introduce new or adapted rules that support your objectives.
On top of this, evaluations are handy in giving us a better general knowledge of what works and what doesn’t. Some types of rules may prove consistently fruitful, others may not have their intended impact, and some rules have entirely unpredictable consequences. Evaluations of policies already implemented can therefore provide policy makers with a treasure chest of evidence to feed in to future regulation design.
Last but not least, evaluations are an essential means for regulators to demonstrate their transparency, accountability and public value, an imperative given the privileged position they occupy as both ‘rule-makers’ and enforcers.
So, if evaluations are this important and so popular among economists, why is there such a dearth of them? Of the major regulators in the UK, just a handful of policy evaluations were published last year, prompting the National Audit Office to release a report calling for a ‘step change improvement in the maturity of performance measurement’.
Barriers to evaluation
I suspect one reason why we see a lack of evaluations is that public policy makers have to overcome at least three big incentive hurdles.
The first is reputational risk. Shining a spotlight on existing problems (or things which could have gone better) takes guts; it could generate bad headlines. And even if no problems are uncovered, evaluation may look like an expensive exercise in self-congratulation. Problem or no problem, a regulator could appear in a negative light.
The second hurdle is saliency – a common behavioural bias in decision-making is to opt for what is prominent, novel and in your attention. In this context, evaluations can look pretty unattractive if they are soaking up time, manpower and data that could be put to work on whatever the immediate public, business or political concern of the day might be. In other words, day-to-day priorities and problems that need fixing can crowd out evaluation of previous interventions, especially if those interventions don’t obviously appear harmful.
The third hurdle is fear of litigation. Regulators, unlike government departments, have no legal compulsion to perform an evaluation – so they risk creating a rod for their own back if litigious agents try to exploit their honesty.
A benign environment, which accepts that a regulator can get things wrong, is part of a healthy legal system, and a regulator needs other market participants not to penalise it for trying to find and address issues.
Beside this, there are a number of other considerations. Evaluations are not a panacea for public policy: they are inevitably imperfect and, sometimes, may have a limited capacity for delivering better regulation (eg the regulator could lack the political discretion to change the policy). If lots of interventions happen at once, or they happen at the same time as other events, they can be impossible to disentangle and to ascribe causal impacts, so an evaluation is unlikely to be enlightening.
But this does not really challenge the argument that, where it’s appropriate to do so, evaluations ought to be undertaken more often and more rigorously.
The interesting question is how you address the incentives that act as hurdles. Below I suggest a change of culture, a change of process, and a change of focus.
Firstly, policy makers could adopt a more mature approach to self-criticism. This might involve questioning received wisdom and accepting that regulators are just as liable to ‘groupthink’ as any firm is. It could also include more candid communication of findings. To that end, as is good practice in ex ante evaluations (see FCA OP 23), ex post evaluations should be published.
To achieve this, it might be helpful if there were an external party to provide disinterested challenge and advice – this could be from an independent academic, consultancy or another public agency. If done properly, this could boost a regulator’s reputation - especially in the longer term.
A second important step is to embed evaluation into the policy-making process. This is vital for tackling a lot of the data problems which often weigh on ex post evaluations.
If ex post evaluation becomes part of the policy cycle, it would ensure that essential data is gathered that can support future evaluation – for example, an assessment of what the market looks like before any intervention.
It should hardly need saying, but lacking a robust baseline and counterfactual seriously undermines any analysis. By carefully planning for ex post evaluations before a policy is put in place, causal inferences may be drawn out far more clearly (eg by staggering the policy’s implementation).
Finally, any evaluations should be centred on outcomes – not mere ‘activity’. Imagine if the British government in India had based its success metrics only on the number of cobras killed or bounties paid out. Hardly worthwhile measures of whether its policy was working – more useful would be the number of injuries/fatalities from cobra bites.
This is a point the NAO alluded to last year by making the point that regulators tend to measure and report on activities and deliverables, not whether any intended outcomes are being achieved.
Why does this all matter? Ultimately evaluations should make for better policy, which should result in better outcomes. The challenge is to move from theoretical affirmation to practical application.
Addressing conflicting incentives is a tentative step to finding our cobras. Once we’ve found them, we can think about how best to deal with them.