"Household surveys in crisis"

This article summarizes some highly important work by Bruce Meyer and colleagues on the ongoing deterioration of the reliability of household survey data, and its implications for measuring important social and economic outcomes. Measurement is a dry and boring topic to many, but anyone who has relied on Census-type surveys to analyze social trends has to be quite concerned.

The Current Population Survey (CPS) is among the most important and widely used of these surveys. Conducted by the Census Bureau and the Bureau of Labor Statistics, it is the source of our monthly information on unemployment rates, as well as a standard source for income and poverty measures on an annual basis.

Our understanding of poverty as well as the effectiveness of anti-poverty policies rests critically on having good measures of the various sources of cash and non-cash income and assistance received by low-income Americans. As Meyer and Nikolas Mittag show in a new working paper, respondents to the CPS often fail to report receipt of various government transfers, and when they do they frequently under-report the amount. Hence the bias is strongly in the direction of over-estimating poverty, and under-estimating the poverty-reducing effect of transfer programs.

How can we tell what people are actually receiving? Meyer and Mittag link the CPS data for New York State to administrative records of the government agencies that are actually writing the checks. They find, for example, that for people reporting incomes at less than half the official poverty line, the missing transfers are worth a little more than the entire amount of their reported cash income. As a consequence of such measurement errors, "the poverty reducing effect of all programs together is nearly doubled" when transfers are fully accounted for.

The most poorly measured form of assistance in their New York data is housing subsidies. One reason for this may be that when these payments are made directly to the landlord, survey respondents may not be aware of the actual amount, and underestimate it. The CPS tries to impute rental assistance when it is not provided by respondents, but the imputation procedures are highly inaccurate, and downward-biased. Survey responses are incomplete and biased for other transfers as well, perhaps due to survey "fatigue", or stigma, or something else.

So there's bad news, but also some good news. The bad news is that these workhorse surveys are not nearly as reliable as we would have liked to believe, not through negligence or malfeasance on the part of the officials who run them, but because it is very difficult to elicit accurate responses. The good news is that administrative and other data sources may provide a more accurate picture. And better news still is the finding that thanks to government transfer programs, the social safety net is actually a lot more effective than we had thought.