Who’s afraid of policy experiments?

Public policies have uncertain effects. As such, prospective and retrospective policy evaluation is of great importance in the quest for evidence-based public policy. In this context, randomized policy experiments provide a particularly important tool. The idea is to implement a policy for a randomly chosen subset of units only (e.g., a random subset of individuals, regions or firms), and estimate the policy’s causal effect by comparing the treatment and the control groups of units. 

Notwithstanding the benefit of this tool, policy experiments are not commonly used in practice. Why not? A recent study estimates the extent to which this reluctance may be due to voters’ perceptions regarding the value of experiments as well as policy makers’ perceptions of the voters’ perceptions.

The study is based on a unique representative survey of Dutch voters (N= 2,114). The survey text first sketches out what a randomized experiment could look like and proposes a series of six specific policies: (1) spending more money to combat tax evasion by large companies; (2) a program to help the unemployed find vacancies; (3) anonymising applications for job vacancies at a municipality to reduce hiring discrimination; (4) spending more money on detecting benefit fraud; (5) a new policing approach (‘Very Irritating Police’) to reduce nuisance by loitering youth and asylum seekers; (6) softer job application requirements for welfare recipients.

The survey recipients were asked to rate the following options: (1) implementing the policy in question; (2) running an experiment before implementation; (3) sticking to the status quo (i.e., no implementation). 

The table below shows the results regarding the appreciation for these alternatives.

	Experiment	Implement	Not impl.
Combat tax evasion	54	63	10
Help unemployed	64	45	14
Anonymize job applications	54	47	26
Combat benefit fraud	57	52	16
Policing to reduce loitering	58	49	17
Soften welfare requirements	47	19	51

As can be seen, the data suggests that voters appreciate policy experiments. For example, for four out of the six policies, experimentation gets higher popular support than both implementation and no implementation. It can also be shown that voters’ support for experimentation is strongest when they have no strong opinion about the policy,

A majority of the electorate (60%) rate it as ‘good’ or ‘very good’ if one of their favourite political parties at the next elections would be in favour of running policy experiments much more often. Only 15% find this ‘bad’ or ‘very bad’. The modal voter also says that she is more likely to vote for the party when it is in favour of much more policy experimentation.

The authors then extend their data-gathering effort by surveying politicians as well. 126 out of 725 invited politicians replied. Though this is not a representative sample, it provides some interesting results. The authors randomly divided the 126 respondents into two equal-sized groups. One group was informed about the results from the voters’ survey ex-ante, whereas the group was only informed ex-post. In this manner, the authors are able to estimate the extent to which politicians’ opinions are affected by voters’ opinions.

Politicians are slightly more positive about policy experiments than voters are. There is also a strong positive correlation between a politician’s belief about voters’ appreciation for policy experiments and his/her own appreciation for policy experiments. Specifically, politicians in the treatment group who become informed that voters appreciate policy experiments more than they initially thought, express more appreciation for policy experiments as compared to control politicians with the same prior beliefs. Reassuringly, politicians who already held correct beliefs about voters’ appreciation do not respond to the treatment. 

Robert Dur, Arjan Non, Paul Prottung and Benedetta Ricci

Who’s afraid of policy experiments?

The Economic Journal, Vol 135 (2024), pp. 538–555