In research, don’t do things you don’t see the point of

When I started reading quantitative research reports, I hadn’t taken any methods or statistics classes, so small wonder that I didn’t understand why certain background variables on the participants were collected, why it was reported how many of them were women and how many of them were men, and what all those numbers in the results sections meant. However, I was willing to assume that these reports had been written by some fairly intelligent people and that, by the Gricean maxim of relevance, these bits and bobs must be relevant — why else report them?

It’s now fifteen years later, and I still haven’t taken any methods or statistics classes. But, as you can tell from a quick glance at the blog archive, I’ve come round to the view that researchers often take actions that don’t actually help them to address their research questions and that much information that is almost routinely reported in research papers is irrelevant to the nominal goal of that research paper (i.e., answering its research questions). Part of the reason that researchers do things that don’t make much sense is that they have misunderstood what some statistical tool does. But I suspect that another part of the reason is that beginning researchers don’t quite see the point of some procedures they run and of some snippets of information they provide but nonetheless assume that other researchers do understand why these are important. From my own experience and discussions with former students, I think that there’s a vicious circle at play:

1. Students read articles with lots of numbers and procedures they don’t really understand or see the point of.
   
2. They reasonably but often incorrectly assume that these ubiquitous numbers and procedures must be integral to the research report.
   
3. As students become researchers, they still haven’t quite understood whether or why all those numbers and procedures are relevant. But they assume that they are relevant. So they’d better also include them in their own reports, or they’d be betraying their own ignorance. Luckily, even if you don’t know what p-values, correlation coefficients and reliability coefficients actually express, computing them is a piece of cake.
4. During peer review, you’re more likely to be chastised for not including some piece of information than for including a couple of irrelevant numbers. So beginning researchers may rarely be forced to consider the added value of their go-to procedures and of the information they routinely provide.
5. A new cohort of students reads the published research, see 1).

It’s not that the beginning researchers in this scenario have misunderstood the tools they use — they have no conception of what these tools do, let alone a false one. All that is required for them to run superfluous procedures and include irrelevant information in their reports is that they think that other people see the relevance of what they’re doing — even if they themselves do not.

Now, it’s hard to stop using tools you’ve misunderstood the purpose of since you won’t know that you’ve misunderstood that purpose. But if you’re a young scholar and you want to run some analysis or report some numbers that are commonly run or reported in your line of work, first ask yourself and your colleagues how running this analysis or reporting these numbers would help you or readers of your work help answer your study’s research questions or make the answers easier to understand. Risk appearing ignorant and don’t cram your research reports with analyses and numbers you don’t see the added value of.

By the same token, if a young scholar asks you which statistical test they should use, first ask them why they think they need a test at all and what exactly it is they want to test. Similarly, if a novice asks you how they can run this or that analysis, ask them how they think running such an analysis would help them address their research question. Even if the added value of such an analysis is clear to you, it may not be clear to them.

Edit (February 21, 2022): Also see Daniël Lakens’ blog post The New Heuristics, where he proposes researchers should adhere to the adage justify everything.