Statistics and Lies

I was recently having a discussion with a group of students, specifically about Marvin Harris’ discussion of the importance of statements of co-variance and his call for a more statistically oriented anthropology in The Rise of Anthropological Theory (affectionately – or disaffectionately – referred to as The RAT during my time as a master’s student at the University of Georgia).

One student objected that “Statistics are basically just lies.”

I was a bit taken aback by this.

Statistics can be used to mislead or distort things. For example, it’s fairly common to encounter figures on median income for U.S. households in the mainstream mass media. There’s no particular reason to doubt the accuracy of such figures in most cases, but one could begin to wonder why reportage of mean household income is much less common, much less why the two central tendency measures are so rarely seen together. But statistics per se aren’t lies.

Statistics involves a set of analytical tools and ways of thinking about sets of data. As with any other tool, statistics can be misused. But saying that statistics are lies because they can be used to lie strikes me a bit like saying that words are inherently lies because words are used to lie. (There are some who think that – but they’re lying.)

Still, there is a real and strong distrust of statistics among many cultural anthropologists and scholars in the humanities disciplines. This seems to me to derive from the now old (and tired) divide between “quantitative” and “qualitative” scholarship and the strong mutual distrust that has permeated that divide.

I’ve written before that this is a false divide. There is no non-quantitative research. All scholarship involves an awareness of quantity, whether in the binary mathematics of presence/absence; rough quantification along the lines of something being present in small or large amount, or happening frequently, continuously, or infrequently; or the highly enumerated quantification of precise counting. There is no non-qualitative research. All scholarship involves choice of what to pay attention to, count, etc.

Moreover, the emphasis on the qualitative/quantitative labels tends to obscure what all good scholarship shares in common, which is measurement and interpretation (see “Measurement and Interpretation”). If one moves past the qual/quant divide (the sort of attitude of “I’m not the sort of scholar who does statistics” or “I’m not the sort who pays attention to anything that can’t be quantified” [by which most mean enumeration, because again, there’s nothing that’s without quantity]) then a whole range of analytical tools and ways of thinking are opened up as possibilities, to be deployed as best fits the research question at hand rather than as best fits an ideological commitment to being “qualitative” or “quantitative.”

Measurement and Interpretation: Let Us Speak No More of Quantitative and Qualitative Research

I have long thought that the division between quantitative and qualitative research was a false divide. There is no pure quantitative nor pure qualitative research. There is always a qualitative element in quantitative work – you might be counting things, but the choice of what’s relevant to count is an inescapable and qualitative decision. Likewise, there’s always a quantitative element in any qualitative research, even if only in the rudimentary sense that it makes a difference whether there’s a lot of something or only a little, whether something is always occurring, occurs every day, once a year, or is a unique occurrence.

More and more, though, I begin to think that the use of the terms distracts from rather than facilitates scholarly communication and would be better replaced by an emphasis on measurement and interpretation. Many social scientists, when asked, pay lip service to the notion I outlined above that there is no pure quantitative or qualitative research, but then go on acting as if there were. This, I think, is done largely uncritically and at least partly (if not largely) out of mutual contempt for number-fetishizing quant types and muddle headed, fuzzy thinking qual types. If we chucked the qual and quant labels, perhaps we could better focus on things that all decent research has in common (whether everyone knows it or not): measurement and interpretation.

Measurement: There is no immeasurable

A lot of “qualitative” social scientists, including most cultural anthropologists (including myself much of the time), tend to be wary of “quantitative” research because they perceive it as ignoring things that are not easily counted and uncritically or simplistically counting things that seem easy to count. Frankly, a lot of “quantitative” work does do these things, though there’s also a lot that doesn’t. What could be better recognized by some quant types is the interpretive nature of choosing what to count, but what qual types could recognize is that we’re all engaged in measurement. There are phenomena that are not easy to count, but there are no observable phenomena that are not measurable.

There are different sorts of measurement. Some things can only be measured in fairly basic and imprecise terms – the binary measurement of the simple presence or absence of a phenomenon or trait, or rough measurement of quantity, e.g. something is absent, present in small quantity or frequency, or present in high quantity or frequency. Other things can be very precisely measured. So, highly “qualitative” ethnography involves measurement just as much as the most “quantitative” of quantitative sociological research. Once we recognize that we’re all involved in measuring, we all ought to measure things as precisely as possible – sometimes that might involve quantification and in other cases might involve simply notation of the presence or absence of something. There’s no reason to be wary of measurement, but good reason to be wary of measurement that is less precise than it reasonably could be or purports to be more precise than it can be or is.

Interpretation: What’s the Significance of Statistical Significance?

As with measurement, all research involves interpretation whether we realize it (or like it) or not. I alluded above to the interpretive quality of measurement – knowing what it makes sense to measure is an interpretive maneuver. Further, it is always necessary to interpret the results of measurement. (Just as quant researchers are often more aware than qual of the need to measure, qual researchers are often more aware of this fact than quant.) Measurements, and even basic analyses, alone never mean anything, e.g. an analysis indicating statistical significance of a set of data doesn’t indicate at all what the meaning of the data is, but simply that there's a low probability that the data are as they are by random chance. There’s no reason to be wary of interpretation, but good reason to be wary of uncritical interpretation not based on sound logical argumentation and good measurement or to be wary of interpretation by those not aware they are engaging in interpretation.