In the last six years, power posing, the act of adopting confident body language in order to increase performance, has taken off amongst science-savvy business people, helped along by a highly popular TED talk and significant media exposure. The scientific evidence for this comes mainly from a 2010 paper in Psychological Science by Dana Carney, Amy Cuddy and Andy Yap, which found that participants who adopted a high power pose, such as leaning forward across a table, showed increased testosterone, feelings of power and tolerance for risk and decreased cortisol compared with those adopting a low power pose.
However, in 2015, a group of six researchers replicated the experiment with a much larger sample and were only able to replicate one of the main findings – that power poses increased feelings of power. Contrary to Carney, Cuddy and Yap, they found no evidence that power posing affected levels of testosterone or cortisol, nor risk taking behaviour. This has led to questions over the reliability of the original research, particularly given the small sample.
But what does controversy over power posing have to do with research for financial regulation?
The answer lies in the way science works: incrementally. It has long been accepted that the pursuit of knowledge is advanced through original studies followed by replications, confirmations, disconfirmations and gradual refinements, eventually allowing us to hone in on true effects. This is the same whether we are investigating power posing or personal loans. But various factors may disrupt this process; the most pernicious of which is, arguably, publication bias.
The perils of publication bias
Publication bias is the selective publication of 'interesting' findings, i.e. those which show statistically significant effects, or that are favourable to a researcher’s prior work or stated hypotheses. Meanwhile null results - findings that a treatment did not work - may be rejected by journal editors or simply filed away in a desk drawer by researchers. This is a big problem in social sciences research; a 2014 paper found that results where all or most hypotheses were supported by the statistical tests were 40 percentage points more likely to be published than null results in a known population of 221 trials.
Publication bias threatens science by distorting the body of evidence, lending credence to published theories and effects which may not be real, either because they occurred by chance, or because researchers (often unwittingly) used small samples or techniques such as p-hacking, which increase the chances of finding a significant result.
Whilst publication bias and related research practices may sound like academic issues (in both senses of the word!), they have real and serious consequences. For example, relying on flawed evidence can cause health services to dispense ineffective or even harmful medicines or it can waste governments’ time and money on keep-out-of-jail schemes which actually result in more teenagers being sent to jail.
The effect on financial services
We see similar stories in research on financial behaviour. For example, numerous organisations worldwide expend considerable energy and resources on financial education schemes, despite numerous experiments and a widely-cited meta-analysis which find that this has little long term effect on financial behaviour.
As regulators and policymakers, we need to be able to rely on good quality evidence in order to understand the markets we regulate and the consumers we protect, and this task may be made more difficult by publication bias. Fortunately, a range of organisations have committed to improving this problem, including AllTrials, which is calling for all past and present clinical trials to be registered and their full methods and summary results reported, Cochrane, which produces systematic reviews of primary research in medicine and, the Abdul Latif Jameel Poverty Action Lab, which maintains a searchable database of 774 randomised evaluations in 69 countries including many on-going experiments.
As researchers at the UK’s Financial Conduct Authority, we recently published a paper rounding up all of our previously unpublished experimental research. The eight experiments try to answer practical questions, such as How can we design disclosure about annuities to help people get a better deal? How can firms improve customers’ engagement with their mortgages? What messages encourage customers to claim compensation? We also test behavioural insights in a novel setting, which have shown promising effects in academia or elsewhere; improving compliance and engagement amongst regulated firms using communications.
While some experiments corroborate existing research or find interesting effects, others did not find any statistically significant effects. As well as putting the trials through independent review, we have published all of these results, rather than just those within a particular study which reach statistical significance, to support good research and good regulation: improving evidence, being transparent about our methods and combatting publication bias. We also share some of the practical lessons we have learned, in the hope that others may benefit from them.
