Harnessing the potential of natural experiments for policy making

The poster boy of this approach is the excellent ‘randomized controlled trial’ (RCT), which is now being widely used to test the effectiveness of any intervention. Essentially, it consists of two time periods (one before the intervention and the other after) and two sets of subjects (one to which the intervention is applied, or the ‘treatment group’) and the other to which it does not (the ‘control’). Group’ ‘). Now, by ‘randomizing’ the subjects—i.e. they are almost identical except for their random assignment to the ‘intervention’ or ‘no intervention’ group—such a test can determine the effect of the intervention by looking at the desired outcome of the group compared to the other. assigned intervention. In general, if the observed change in outcome is significantly larger in the ‘intervention’ subject group than in the other, the RCT would suggest that this difference is due to the intervention, thereby pointing to its efficacy.

However, RCTs have their drawbacks. For example, Nobel laureate Angus Deaton has argued that randomness does not actually result in creating identical subjects, so differences in results should not be attributed to interference alone. Furthermore, just because an interference works in place A does not guarantee that it will work in place B (the problem of ‘external validity’). What is more relevant for policy purposes is that sometimes it is not ethically or politically feasible to choose one subject over another for intervention (imagine choosing one district for vaccination and the other for the common reason). leaving what you want to use). In contexts where RCTs are not feasible, other methods should be explored. This is where natural experiments from backstage come into the limelight.

Natural experiments occur when a particular intervention has been implemented, but the circumstances surrounding the implementation are not under the control of the researchers, unlike RCTs. Often such interventions occur as part of government programs or sudden unprecedented events (such as an economic crisis or pandemic). Such events provide unique opportunities to ‘naturally’ split a sample into treatment and control groups. If good quality data are available in the period before and after the intervention, stronger evidence with limited bias can be found. Often, with an interdisciplinary approach, natural experiments facilitate evidence-based policy making.

Researchers used the Indo-Gangetic plains to obtain evidence on the health impact of air pollution in India by geographic units (Saraswat and Bansal), epidemics (Fenske, Gupta and Yuan used changes in India’s female labor force participation). used to estimate influenza pandemic). rate), financial crises (Greenstone, Maas, and Nguyen used the Great Depression to understand the impact of debt-market shocks on employment in the US), and intervention (Jensen took advantage of the information reform in Kerala one-price law Tested the introduction of mobile phones, while Jensen and Oster studied how the introduction of cable TV improved Indian school enrollment and decreased domestic violence and son preference by increasing women’s autonomy within the home), among other things Apart from.

More specifically, natural experiments have at least three main strengths compared to RCTs. First, such experiments arising from situations when the policy is being implemented in some states or districts and not in others, or an exogenous event that affects some areas more than others, may address ethical or political viability concerns. can do These settings can be used to evaluate policy impact, as the treatment and control groups were created without tampering by an inquisitive investigator. Second, they provide evidence rooted in real-world situations, in contrast to RCTs that require controlled environments that are often considered to be a significant departure from reality. Therefore, one can be sure of the external validity of the results. Third, natural experiments can create more robust evidence at times. The bias brought about by the coincidental presence of other factors influencing the impact of the intervention on outcome and the researchers’ lack of control over policy roll-out, however, remain of concern when using natural experiments.

Natural experiments have in the past been marked as a potential goldmine for policy insights for India. The Economic Survey of 2016-17 referred to demonetisation in 2016 as an interesting natural experiment on replacement with other forms of cash. Administering the second dose of a particular COVID vaccine this May in Siddharthnagar district of Uttar Pradesh was another natural experiment to know the efficacy of a vaccine cocktail compared to both doses of a vaccine. How well India uses natural testing settings for public policy will depend on greater awareness of the concept at all levels of governance. A national level effort to crowd-source proposals to study natural experiments would be a leap in the right direction.

Yashaswini Saraswat and Anshuman Kamila are respectively Assistant Directors in the Office of Development Monitoring and Evaluation, NITI Aayog and Assistant Directors in the Department of Economics, Ministry of Finance. These are the personal views of the authors.