India’s official survey system has been battling censorship and apathy over the years, with a private agency coming to our rescue. The ‘Consumer Pyramid’ household survey by the Center for Monitoring Indian Economy (CMIE) helped us take a peek into the life of an average Indian household. During the first lockdown, it was thanks to CMIE that we came to know about the scale of urban unemployment and the problem of migration of urban workers to rural farms. Official surveys confirmed these events much later.
CMIE’s ability to publish data on an almost real-time basis and its independence from state mandates has earned it the praise and approval of a growing tribe of economists and policy victors. From evaluating poverty levels to estimating Covid deaths, CMIE data has become the first port of call for many.
Yet, as its database usage has grown, so has the number of queries about it. Economists Jean Dreze and Anmol Somanchi found that several demographic groups (very young, uneducated and wealth-poor) are under-represented in CMIE’s survey compared to the latest National Family Health Survey (NFHS, 2019-21). Adding to these concerns, World Bank economists Sutirth Sinha Roy and Roy van der Weid said that the very wealthy are also underrepresented in the CMIE survey.
Economists Rosa Abraham and Anand Srivastava found that CMIE reports fewer women in the workforce than the Periodic Labor Force Survey (PLFS). The female labor force participation rate estimated by CMIE is almost half of the ‘official rate’ estimated by PLFS. CMIE shows a higher share of respondents with post office savings, pension (or provident fund) schemes and insurance products (life and health) as compared to the All India Credit and Investment Survey (AIDIS) 2019, says Niyati Agarwal and colleagues at Doorstep Research I got it.
Many economists and statisticians suspect that these discrepancies are due to flaws in the CMIE’s survey methodology and design. Writing for the India Forum, statistician Salil Sanyal has raised questions about CMIE’s stratification strategy (over-sampling of urban areas), and its failure to adhere to statistical norms of probability sampling.
Sampling theory demands that households in a primary sampling unit (usually a village or urban ward) are selected at random from a list of all households in that unit. A convenient alternative is to choose the first house at random, but use a fixed interval to select subsequent houses. This sampling interval is calculated by dividing the total number of families by the desired sample size. So, if there are 300 households in a village and a sample of 30 is to be taken, the sampling interval is 10 (300/30). The surveyor first chooses a random number between 1 and 300. Let’s say that number becomes 25. Then the list would sample the 25th house first, followed by the 35th, 45th, and so on. Most of the official surveys in India use this technique as it saves time.
The problem with CMIE’s survey is that its field staff were unable to list homes in the primary sampling units due to practical difficulties (including safety concerns in some locations) at the initial stage. Hence CMIE used the ‘Jugaad’ method to select the houses. The field staff was asked to enter the main road of the village, check the number of houses on it, and choose a random number between 5 and 15 for the selected houses. Once the main road ends, the enumerators move on to the inner roads. The absence of a complete list, the absence of a random onset, and the use of ‘ad-hoc’ intervals (5–15) for selected families inject bias into the sampling process, wrote Sanyal.
Other statisticians argue that household selection of CMIE is problematic because housing arrangements in rural India are far from random. Rich families are often grouped on the main street, while poor families may be grouped in a settlement on the periphery of the main settlement. Economists Vikas Raval and Jessim Paes have also expressed similar concerns. So take Dreze and Somanchi.
CMIE takes these concerns seriously and is currently investigating whether its survey methods lead to any bias, its managing director Mahesh Vyas said. In every village, the CMIE team is checking whether the border is missing. The findings of this investigation will be published by October, and corrections to the sample (where necessary) will be effective from January to April 2023, Vyas said over email.
Vyas said the CMIE is also analyzing the discrepancies with the official dataset, a process that will take some time. Critics expressing concern about CMIE’s survey methods often use the NSS (National Sample Survey) as their reference. But the CMIE survey is not envisaged as a replica of the NSS, Vyas said. Still, CMIE is trying to create “some sort of mapping” between the NSS/NFHS definitions and systems and those used by CMIE, he said.
CMIE’s sincerity in addressing the concerns of data users is commendable. But given that these reviews and fixes will take time, it’s worth looking carefully at CMIE’s survey numbers for now. Rather than relying on its survey numbers to measure levels of various economic indicators related to employment and living conditions, it may make sense to use the data to track changes over time. Despite its flaws, the survey is likely to capture major changes in unemployment or living conditions, as was the case in 2020.
Prameet Bhattacharya is a Chennai based journalist. His Twitter handle is pramit_b. Is