Human Science · Inquiry · Quantitative

Quantitative research relies on numbers to represent reality. There are many numbers I might use to describe you, including your age, your height, your weight, your bank balance, your credit score, your intelligence quotient (IQ). But do such numbers tell your story?

Quantitative research in Human Science or the social sciences is, at its very foundation, a superficial look at our reality, but is fully embraced by positivism as ‘objective,’ a ‘God’s eye’ view which humans in fact cannot possess.

We are subjective, our reality is subjective. In its disdain for subjectivity, positivism thus rejects our reality. And, in its preference for quantitative results, positivism chooses a superficial account over ‘rich narrative.’

Most quantitative research involves statistical methodology. Though I had a couple statistics classes along the way, I am not a statistician, and this can pose a challenge when, in a literature review, I draw upon research that uses arcane statistical methods which I am ill-prepared to evaluate. There are many such methods and in these cases, I am hoping that peer reviewers have done their job well. Unfortunately, that’s a problem.

One of my longstanding suspicions about quantitative research—I remember thinking about this while walking down a hallway on the east side of Meiklejohn Hall at California State University, East Bay—is that some of this stuff looks for all the world like teenage boys from an earlier era and their cars, tweaking their engines to try to get better performance. The specific accusation here is that researchers are tweaking their methods to obtain desired results.

I am not a statistician so I knew I would never be able to prove this sort of research misconduct.

But it turns out I was right. Research fraud can be difficult to detect or prove, but Gary Smith offers some hints.^[1] I talk about “big data” mining in my critique of artificial idiocy;^[2] that appears here as “HARKing,” “hypothesizing after results are known.”^[3]

In [Andrew] Gelman’s garden-of-forking-paths analogy, p-hacking occurs when a researcher seeks empirical support for a theory by trying several paths and reporting the path with the lowest p-value. Other times, a researcher might wander aimlessly through the garden and make up a theory after reaching a destination with a low p-value. This is hypothesizing after the results are known — HARKing.^[4]

The pressure scholars feel to publish, to maintain eligibility for tenure, and thus to escape the low wages and other abuse of adjuncts, yields perverse results:

Some are tempted by an even easier strategy — simply make up whatever data are needed to support the desired conclusion. When Diederik Stapel, a prominent social psychologist, was exposed in 2011 for having made up data, it led to his firing and the eventual retraction of 58 papers. His explanation: “I was not able to withstand the pressure to score points, to publish, to always have to be better.” He continued: “I wanted too much, too fast.”

It is just a short hop, skip, and jump from making up data to making up entire papers. In 2005, three MIT graduate students created a prank program they called SCIgen that used randomly selected words to generate bogus computer-science papers. Their goal was to “maximize amusement, rather than coherence” and, also, to demonstrate that some academic conferences will accept almost anything.^[5]

SCIgen is apparently still available and some are using it. And yes, the implication is dire for peer review:^[6]

Cyril Labbé, a computer scientist at Grenoble Alps University, wrote a program to detect hoax papers published in real journals. Working with Guillaume Cabanac, a computer scientist at the University of Toulouse, they found 243 bogus published papers written entirely or in part by SCIgen. A total of 19 publishers were involved, all reputable and all claiming that they publish only papers that pass rigorous peer review. One of the embarrassed publishers, Springer, subsequently announced that it was teaming with Labbé to develop a tool that would identify nonsense papers. The obvious question is why such a tool is needed. Is the peer-review system so broken that reviewers cannot recognize nonsense when they read it?^[7]

And of course, none of this stuff can be replicated, leading to scientific reversals.^[8] Some reversals are to be expected—scientific conclusions are always tentative. But we’re likely seeing more and these combine with political interference, as with COVID-19, to undermine public trust in science.^[9]

Statistical methods generally rely upon a representative sample to represent the entire population. Very strictly speaking, this is an ecological fallacy, using one unit of analysis, in this case, a subset of the population under study, to represent another, in this case the entire population, and this is one reason studies should be replicated. Even when properly conducted, incorrect results happen; when studies are replicated enough times, incorrect results can be weeded out.

A representative sample also relies heavily on participation of people selected in that sample. This has proven particularly problematic in survey research.

Experiments
- True Experiments
- Quasi-Experiments
- Nonexperiments
Surveys
Content Analysis
Artificial Idiocy