From Doorstep to Dial Tone: How Survey Mode Reshapes Development Statistics

When people in rural Nigeria answer survey questions over the phone, they often describe their lives differently than when an interviewer sits with them face to face. That difference, new research shows, is not a small technical detail. It can significantly change how poverty, health, work, and well-being are measured, and how policies are designed.

A study by researchers from the World Bank's Development Data Group and ETH Zurich takes a close look at how survey mode itself shapes responses. Using a large experiment embedded in Nigeria's national household survey, the researchers show that phone surveys consistently produce different answers from in-person interviews, even when the same people are asked the same questions.

A Rare Experiment That Isolates the Truth

To understand whether phone surveys truly change answers, the researchers designed an unusually careful experiment. Nearly 1,000 rural households were interviewed twice within a few days: once in person and once by phone. Every household was given a mobile phone so that lack of access would not distort the results. The order of interviews was randomly assigned, and the questions were exactly the same.

This design allowed the researchers to isolate the survey mode, separating it from other factors, such as who owns a phone, when interviews take place, or whether people change their answers because they were asked earlier. Few studies in low- and middle-income countries have been able to do this so cleanly.

Big Differences Across Everyday Indicators

The results are striking. Across 20 commonly used indicators, phone survey responses differ from in-person responses by about 17 to 18 percent on average. These differences appear across health, food security, work, household businesses, exposure to shocks, and wellbeing.

In many cases, people report more when interviewed by phone. Illness, visits to health providers, health spending, COVID-19 vaccination, running a household business, and food consumption are all reported at higher rates over the phone. This suggests that phone interviews often encourage more "yes" answers or more frequent reporting.

But the pattern is not always one-sided. When asked about their wellbeing, people tend to rate their current lives worse over the phone than in person, even while sounding more optimistic about the future. Income adequacy and hours worked in farming are also reported lower by phone. The direction of bias depends on the topic.

Why the Averages Hide a Bigger Problem

Looking only at averages hides the most worrying finding. When the same individuals answered the same questions twice, more than half of their responses changed depending on whether the interview was by phone or in person. For questions with many answer options, such as ten-step wellbeing scales or expectations about weather risks, over 70 percent of respondents gave different answers.

Even when average differences are small, this often happens because positive and negative changes cancel each other out. In reality, many people are inconsistent across modes. This means that simply adjusting for survey mode in statistical analysis may fix the average but leaves large individual-level errors untouched.

The study also shows clear behavioral patterns. Over the phone, respondents are more likely to say "yes," agree with statements, choose extreme values, and round numbers like money spent or hours worked. These problems grow as questions become more complex.

What This Means for Data and Policy

Phone surveys are not going away. They are faster, cheaper, and essential during crises. The study does not argue against using them. Instead, it sends a warning: phone and in-person data are not directly interchangeable.

Survey designers need to simplify questions, limit long response scales, and test phone questionnaires more carefully. Comparing phone data with in-person benchmarks and monitoring response patterns in real time can help catch problems early. Education also matters, better-educated respondents are less affected by survey mode, suggesting that question design must account for cognitive burden.

The message is clear. As governments and donors rely more heavily on phone surveys, they must recognize that how questions are asked can strongly shape the answers. Ignoring survey mode risks building policy on numbers that change as soon as the interview moves from the doorstep to the phone line.