Background: There appears to be considerable variation between different national jurisdictions and between different sectors of public policy in the use of evidence and particularly the use of randomised controlled trials (RCTs) to evaluate non-healthcare sector programmes.
Methods: As part of a wider study attempting to identify RCTs of public policy sector programmes and the reasons for variation between countries and sectors in their use, we carried out a pilot study which interviewed 10 policy makers and researchers in six countries to elicit views on barriers to and facilitators of the use of RCTs for social programmes.
Results: While in common with earlier studies, those interviewed expressed a need for unambiguous findings, timely results and significant effect sizes, users could, in fact, be ambivalent about robust methods and robust answers about what works, does not work or makes no difference, particularly where investment or a policy announcement was planned. Different national and policy sector cultures varied in their use of and support for RCTs.
Conclusions: In order to maximise the use of robust evaluations of public programmes across the world it would be useful to examine, systematically, cross-national and cross-sectoral variations in the use of different methods including RCTs and barriers to and facilitators of their use. Sound research methods, whatever their scientific value, are no guarantee that findings will be useful or used. 'Stories' have been shown to influence policy; those advocating the use of RCTs may need to provide convincing narratives to avoid repetition about their value.