Human Science · Inquiry · Quantitative · Surveys

Human Science

Inquiry

Quantitative

Surveys

Fig. 1. Graph showing the deterioration of response rates by the Pew Research Center, February 27, 2019, fair use.^[1]

Fig. 2. It’s more with a six or seven percent response rate.^[2] Comic by Bob Thaves, October 14, 2022, via GoComics, fair use.

When the response rate, which should be ninety percent of a representative sample or more,^[3] is instead in the single digits,^[4] pollsters have a self-selecting sample that cannot represent non-respondents, particularly when, due to non-response, we have no idea, none whatsoever, how members of that self-selecting sample differ from all those non-respondents. This completely invalidates the methodology and I don’t care what rhetoric pollsters deploy to excuse themselves—the claim that polling works regardless^[5] is belied by all the instances in which it doesn’t^[6]—or what statistical magic they think they can employ to get around this problem—an absence of data remains an absence of data. This is a non-methodology that persists because we are desperate for an imaginary certainty of numbers,^[7] and because people have built entire careers around this non-methodology.

Steven Shepard, “Report: Phone polls aren’t dead yet,” Politico, May 15, 2017, https://www.politico.com/story/2017/05/15/pollsters-phone-polls-238409

Courtney Kennedy and Hannah Hartig, “Response rates in telephone surveys have resumed their decline,” Pew Research Center, February 27, 2019, https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/

David A. Graham, “The Polling Crisis Is a Catastrophe for American Democracy,” Atlantic, November 4, 2020, https://www.theatlantic.com/ideas/archive/2020/11/polling-catastrophe/616986/

Mona Chalabi, “The pollsters were wrong – again. Here’s what we know so far,” Guardian, November 4, 2020, https://www.theguardian.com/us-news/commentisfree/2020/nov/04/the-pollsters-were-wrong-again-heres-what-we-know-so-far

Steven Shepard, “Dem pollsters acknowledge ‘major errors’ in 2020 polling,” Politico, April 13, 2021, https://www.politico.com/news/2021/04/13/dems-polling-failure-481044

Dan Balz, “2020 presidential polls suffered worst performance in decades, report says,” Washington Post, July 18, 2021, https://www.washingtonpost.com/politics/2020-poll-errors/2021/07/18/8d6a9838-e7df-11eb-ba5d-55d3b5ffcaf1_story.html

David Byler, “Polling is broken. No one knows how to fix it,” Washington Post, July 22, 2021, https://www.washingtonpost.com/opinions/2021/07/22/polling-is-broken-no-one-knows-how-fix-it/

Joshua Yaffa, “Why Do So Many Russians Say They Support the War in Ukraine?” New Yorker, March 29, 2022, https://www.newyorker.com/news/news-desk/why-do-so-many-russians-say-they-support-the-war-in-ukraine

Nate Cohn, “Yes, the Polling Warning Signs Are Flashing Again,” New York Times, September 12, 2022, https://www.nytimes.com/2022/09/12/upshot/polling-midterms-warning.html

Perry Bacon, Jr., “Polls are useful. They just can’t predict elections in swing states,” Washington Post, September 27, 2022, https://www.washingtonpost.com/opinions/2022/09/27/polls-accuracy-midterm-elections-perry-bacon/

Adam Taylor, “Why Bolsonaro and the global right-wing love to hate on election polls,” Washington Post, October 3, 2022, https://www.washingtonpost.com//world/2022/10/03/bolsonaro-brazil-polls-trump/

David Byler, “The (early) verdict: The polls were okay!” Washington Post, November 9, 2022, https://www.washingtonpost.com/opinions/2022/11/09/polls-early-midterms-verdict-improvement/

Richard Hertz, “Too many predictions, not enough data,” Santa Rosa Press-Democrat, January 13, 2023, https://www.pressdemocrat.com/article/opinion/close-to-home-too-many-predictions-not-enough-data/

1. ^[1]Courtney Kennedy and Hannah Hartig, “Response rates in telephone surveys have resumed their decline,” Pew Research Center, February 27, 2019, https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/↩
2. ^[2]Courtney Kennedy and Hannah Hartig, “Response rates in telephone surveys have resumed their decline,” Pew Research Center, February 27, 2019, https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/↩
3. ^[3]This according to Valerie Sue, the professor in my first research methods class, at California State University, Hayward (now East Bay), Fall 2003.↩
4. ^[4]Steven Shepard, “Report: Phone polls aren’t dead yet,” Politico, May 15, 2017, https://www.politico.com/story/2017/05/15/pollsters-phone-polls-238409; Courtney Kennedy and Hannah Hartig, “Response rates in telephone surveys have resumed their decline,” Pew Research Center, February 27, 2019, https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/↩
5. ^[5]Steven Shepard, “Report: Phone polls aren’t dead yet,” Politico, May 15, 2017, https://www.politico.com/story/2017/05/15/pollsters-phone-polls-238409; Courtney Kennedy and Hannah Hartig, “Response rates in telephone surveys have resumed their decline,” Pew Research Center, February 27, 2019, https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/↩
6. ^[6]Dan Balz, “2020 presidential polls suffered worst performance in decades, report says,” Washington Post, July 18, 2021, https://www.washingtonpost.com/politics/2020-poll-errors/2021/07/18/8d6a9838-e7df-11eb-ba5d-55d3b5ffcaf1_story.html; David Byler, “Polling is broken. No one knows how to fix it,” Washington Post, July 22, 2021, https://www.washingtonpost.com/opinions/2021/07/22/polling-is-broken-no-one-knows-how-fix-it/; Mona Chalabi, “The pollsters were wrong – again. Here’s what we know so far,” Guardian, November 4, 2020, https://www.theguardian.com/us-news/commentisfree/2020/nov/04/the-pollsters-were-wrong-again-heres-what-we-know-so-far; David A. Graham, “The Polling Crisis Is a Catastrophe for American Democracy,” Atlantic, November 4, 2020, https://www.theatlantic.com/ideas/archive/2020/11/polling-catastrophe/616986/; Steven Shepard, “Dem pollsters acknowledge ‘major errors’ in 2020 polling,” Politico, April 13, 2021, https://www.politico.com/news/2021/04/13/dems-polling-failure-481044; Adam Taylor, “Why Bolsonaro and the global right-wing love to hate on election polls,” Washington Post, October 3, 2022, https://www.washingtonpost.com//world/2022/10/03/bolsonaro-brazil-polls-trump/↩
7. ^[7]Jacques Ellul, The Technological Society, John Wilkinson, trans. (New York: Vintage, 1964).; Neil Postman, Technopoly: The Surrender of Culture to Technology (New York: Vintage, 1993).↩