On superficially similar research questions

Researchers often think they’ve clearly specified their research question when they haven’t. I encountered a case in point of this when I reviewed a manuscript a while back. Throughout this manuscript, the authors described the goal of their study in a few different ways. At first blush, these formulations seemed to express the same overarching objective, but on closer inspection, they corresponded to different statistical questions. The consequence was that the manuscript ended up reporting on a range of analyses without it being clear how they related to the study’s goal.

In this blog post, I’ll use a generically worded version of this study to illustrate how similar-sounding research questions may be substantially different statistical questions. Throughout this illustration, I make a few simplifying assumptions that strip away many of the complications that you would have to grapple with in a real project. This, I think, is a useful technique when clarifying research questions: before you start to tackle the statistical intricacies, first consider how you would have to answer your research question as currently stated in a simplified scenario. This may help you realise what it is that you actually want to accomplish. Once that is clear, the analytical approach can be tailored to answer that question rather than some approximation of it.

Setting

So as not to identify the study in question, I’ll only sketch the study in broad, generic strokes. The authors collected, for a set of units, two numerical predictors thought to reflect a property of these units. I’ll creatively call these predictors \(X\) and \(X'\). The variable \(X\) corresponds to a standard, well-established operationalisation of the property of interest; the variable \(X'\) corresponds to an alternative operationalisation suggested by the authors. For each unit we also have a numerical outcome \(Y\).

Objective

Throughout their article, the researchers formulated their objective in four different ways. I’ve paraphrased these in the same generic notation as above.

(F1) How well can we predict \(Y\) using \(X'\)? (At the start of the introduction.)

(F2) How much better can we predict \(Y\) if we also use \(X'\)? (At the end of the introduction.)

(F3) How large is the difference between the correlation between \(X'\) and \(Y\) and the one between \(X\) and \(Y\)? (At the end of the methods section.)

(F4) How much better can we predict \(Y\) if we use \(X'\) instead of \(X\)? (At the end of the methods section.)

At some level, these four questions are really just variations on the same theme: Given that everyone uses \(X\) to model \(Y\), is the newly proposed \(X'\) of any use?

But as we’ll see, formulations (F1) through (F4) are different in the sense that, given the same data, you may end up with a different answer depending on which exact question you’re asking.

Simplifying assumptions

I will show that formulations (F1) through (F4) correspond to different statistical questions even under ideal circumstances in which many real-world complications do not occur. To this end, I make two simplifying assumptions.

(A1) We have an infinite amount of perfectly measured data.

This assumption means that we don’t have to estimate anything or deal with sampling error: as long as we know what to compute, we can obtain the correct answer with infinite precision.

(A2) The predictors \(X\) and \(X'\) have a joint multivariate distribution, and the outcome \(Y\) is a weighted sum of these predictors with some Normal noise added to it. More specifically, the data-generating mechanism is as follows:

\[ \begin{align*} (X, X') &\sim \textrm{MultiNormal}_2(\boldsymbol 0, \boldsymbol \Sigma), \\ Y &= \beta_XX + \beta_{X'}X' + \varepsilon, \\ \varepsilon &\sim \textrm{Normal}(0, \sigma^2_{\varepsilon}), \end{align*} \] where \(\boldsymbol \Sigma\) is a covariance matrix and where \(\varepsilon\) and \((X, X')\) are independent.

Assumption (A2) implies that the joint distribution of \((X, X', Y)\) is multivariate Normal, which simplifies calculations considerably. Further, assumptions (A1) and (A2) jointly imply that, for whatever values of \(\boldsymbol \Sigma, \sigma^2_{\varepsilon}, \beta_X\) and \(\beta_{X'}\) we choose, the answers to (F1) through (F4) can be calculated precisely without any need for nuance in the form of sampling error, interactions, nonlinearities, etc.

The only thing standing between us and the answer to the research question is figuring out what the research question really is.

Under more realistic circumstances, additional complications will arise. But if the different formulations can already lead to different answers in this well-behaved multivariate Normal setting with perfect information, then they are clearly different for conceptual reasons rather than because of implementation details. At the end of the post, I’ll briefly discuss how more realistic assumptions can further exacerbate the differences between these formulations.

Identifying the relevant comparisons

Each formulation targets another comparison between two statistical quantities.

Formulation (F1)

Formulation (F1) asks how well we can predict \(Y\) given \(X'\). Under our assumptions, a natural way to formalise this question is by comparing the variability of \(Y\) (i.e., \(\textrm{Var}(Y)\)) to the variability of \(Y\) once you’ve accounted for \(X'\) (i.e., \(\textrm{Var}(Y | X')\)). The larger this reduction in variability, the better we can predict \(Y\) given \(X'\).

NoteFormulae

Both quantities can be computed using standard formulae.

First, the variance in \(Y\) is \[ \begin{align*} \textrm{Var}(Y) &= \textrm{Var}(\beta_XX + \beta_{X'}X' + \varepsilon) \\ &= \beta_X^2 \textrm{Var}(X) + \beta_{X'}^2 \textrm{Var}(X') + 2\beta_X\beta_{X'}\textrm{Cov}(X, X') + \textrm{Var}(\varepsilon), \end{align*} \] where \(\textrm{Var}(X), \textrm{Var}(X')\) and \(\textrm{Cov}(X, X')\) are entries in the covariance matrix \(\boldsymbol \Sigma\).

Second, the variance in \(Y\) given \(X'\) does not depend on the observed value of \(X'\) and can be computed using the conditioning formula for multivariate Normal distributions:

\[ \begin{align*} \textrm{Var}(Y|X') &= \textrm{Var}(Y) - \frac{\textrm{Cov}^2(X', Y)}{\textrm{Var}(X')}, \end{align*} \] where

\[ \begin{align*} \textrm{Cov}(X', Y) &= \textrm{Cov}(X', \beta_XX + \beta_{X'}X' + \varepsilon) \\ &= \beta_X\textrm{Cov}(X, X') + \beta_{X'}\textrm{Var}(X'). \end{align*} \]

Under our assumptions, the quantities \(\textrm{Var}(Y)\) and \(\textrm{Var}(Y | X')\) are equal if and only if \(\textrm{Cov}(X', Y) = 0\). This is the case in particular if both \(\beta_{X'} = 0\) and \(\textrm{Cov}(X, X') = 0\). The difference between these two quantities becomes large whenever \(X'\) contains a lot of information about \(Y\). This can be achieved, for instance, by picking highly correlated \(X\) and \(X'\) or by setting \(\beta_X\) to 0 and \(\beta_{X'}\) to a large absolute value.

If Formulation (F1) is what you’re interested in, a sensible first port of call would be to compare the mean squared error of a simple regression model with \(X'\) as its predictor to the mean squared error of an intercept-only model.

Formulation (F2)

Formulation (F2) asks how much better we can predict \(Y\) if we also use \(X'\). As a reviewer, I understood this to ask how much uncertainty in \(Y\) remains after accounting for both \(X\) and \(X'\) compared to after accounting only for \(X\). Under our assumptions, this amounts to comparing \(\textrm{Var}(Y | X)\) to \(\textrm{Var}(Y | X, X')\).

NoteFormulae

The variance in \(Y\) given both \(X\) and \(X'\) is easy to compute:

\[ \textrm{Var}(Y | X, X') = \textrm{Var}(\varepsilon) = \sigma^2_{\varepsilon}. \]

The variance in \(Y\) given only \(X\) can be computed like we did above:

\[ \begin{align*} \textrm{Var}(Y|X) &= \textrm{Var}(Y) - \frac{\textrm{Cov}^2(X, Y)}{\textrm{Var}(X)} \\ &= \beta_{X}^2 \textrm{Var}(X) + \beta_{X'}^2 \textrm{Var}(X') + 2\beta_X\beta_{X'}\textrm{Cov}(X, X') + \sigma^2_{\varepsilon} - \frac{\textrm{Cov}^2(X, Y)}{\textrm{Var}(X)}. \end{align*} \]

Some algebra reveals that the quantities \(\textrm{Var}(Y|X)\) and \(\textrm{Var}(Y | X, X')\) are equal if and only if \(\beta_{X'} = 0\) or if the absolute correlation between \(X\) and \(X'\) is maximal. The difference between these quantities can be made large by making \(X'\) but not \(X\) highly informative about \(Y\). This can be achieved, for instance, by setting \(\textrm{Cov}(X, X') = 0\), \(\beta_X = 0\) and choosing large values for \(\beta_{X'}\) and \(\textrm{Var}(X')\).

If Formulation (F2) is what you’re interested in, a sensible approach is to compare the mean squared error of a multiple linear regression model with both \(X\) and \(X'\) as predictors to the mean squared error of a simple linear regression model with only \(X\) as its predictor.

Formulation (F3)

Formulation (F3) differs from the others in that it explicitly presupposes a particular statistical tool: correlation coefficients. Specifically, it targets the difference between the correlations \(\textrm{Cor}(X, Y)\) and \(\textrm{Cor}(X', Y)\).

NoteFormulae

The correlation between \(X\) and \(Y\) is

\[\textrm{Cor}(X, Y) = \frac{\textrm{Cov}(X, Y)}{\sqrt{\textrm{Var}(X)\textrm{Var}(Y)}} = \frac{\beta_{X}\textrm{Var}(X) + \beta_{X'}\textrm{Cov}(X, X') }{\sqrt{\textrm{Var}(X)\textrm{Var}(Y)}};\] the one between \(X'\) and \(Y\) similarly is \[\textrm{Cor}(X', Y) = \frac{\beta_{X'}\textrm{Var}(X') + \beta_{X}\textrm{Cov}(X, X') }{\sqrt{\textrm{Var}(X')\textrm{Var}(Y)}}.\]

The parameters can be chosen in many different ways in order to make these two quantities identical, with the simplest case being \(X' = X\). If you want to make them as different from each other as possible, you could set \(\boldsymbol \Sigma = \begin{pmatrix}1 & 0 \\ 0 & 1\end{pmatrix}\) and pick \(\beta_X = -\beta_X' \neq 0\) for large absolute values of \(\beta_X\).

If Formulation (F3) is what you’re interested in, you need to compare two dependent correlations rather than run regression analyses.

Formulation (F4)

Formulation (F4) asks how much better \(Y\) can be predicted given only \(X'\) compared to when you’re only given \(X\). Under our assumptions, a sensible comparison involves the quantities \(\textrm{Var}(Y | X)\) and \(\textrm{Var}(Y | X')\) that we computed above.

It’s easy to make these quantities coincide: simply set \(X' = X\). The difference between them is large if \(X'\) contains considerably more information about \(Y\) than does \(X\). This can be achieved, for instance, by setting \(\textrm{Cov}(X, X') = 0\), \(\beta_X = 0\) and choosing large values for \(\beta_{X'}\) and \(\textrm{Var}(X')\).

If Formulation (F4) is what you’re interested in, a sensible first approach is to compare the mean squared errors of two simple regression models: one with \(X\) as its predictor and one with \(X'\) as its predictor.

Formulations (F1) through (F4) are different

The four formulations may superficially seem interchangeable. To show that they are not, I’ll construct situations where some formulations indicate a difference whereas others do not.

To show that Formulation (F1) differs from Formulations (F2) through (F4), consider the case where \(X'\) and \(X\) are the same variable and \(\beta_X = \beta_{X'} \neq 0\). In this case, the comparison for Formulation (F1) will show that \(X'\) is to some extent predictive of \(Y\); the extent of this predictive usefulness can be made arbitrarily large. However, the comparisons for Formulations (F2) through (F4) will show no difference whatsoever. All of these comparisons target some difference between \(X\) and \(X'\) in terms of how these predictors relate to \(Y\), whereas Formulation (F1) only pertains to how well \(X'\) predicts \(Y\).

Similarly, it can be shown that Formulation (F2) differs from Formulations (F3) and (F4). To this end, we can set, for instance,

\[\boldsymbol \Sigma = \begin{pmatrix}1 & 0 \\ 0 & 1\end{pmatrix}\] and \(\beta_X = \beta_{X'}\). In this case, the predictor \(X'\) contributes to the variance of \(Y\) independently of \(X\), which is precisely what the comparison implied by Formulation (F2) picks up on. However, Formulations (F3) and (F4) compare, in different ways, to what extent the contributions of \(X\) and \(X'\) to \(Y\) differ—and in this case, both predictors make exactly equal contributions.

Finally, we can show that Formulations (F3) and (F4) differ. To see this, again use \[\boldsymbol \Sigma = \begin{pmatrix}1 & 0 \\ 0 & 1\end{pmatrix}\] and set \(\beta_X = -\beta_{X'} \neq 0\). Then the correlation coefficient for \(X'\) and \(Y\) will differ in sign from that for \(X\) and \(Y\), which is what the comparison for Formulation (F3) will pick up on. But both \(X\) and \(X'\) will predict \(Y\) with equal accuracy, so the comparison for Formulation (F4) won’t reveal any differences.

All four formulations, then, are conceptually distinct.

Further complications

The setting above is quite unrealistic. But the idealisation makes it harder rather than easier to come up with situations in which the four different formulations of the research goal give conflicting answers.

For instance, while Formulations (F3) and (F4) are different, our multivariate Normal setting implies that any time the comparison for Formulation (F4) reveals a difference, then so will the comparison for Formulation (F3); just not the other way around. But once we consider settings where \((X, X', Y)\) does not have a multivariate Normal distribution, it’s easy to come up with examples where \(X\) and \(X'\) differ in their predictive usefulness but where the correlation between each predictor and the outcome is zero; see the blog post What data patterns can lie behind a correlation coefficient? for examples where one variable is highly predictive of another yet not correlated with it.

Further, you can open a whole can of worms by considering settings where \(X\) and \(X'\) interact in their effect on \(Y\), where these predictors do not have a multivariate Normal distribution, or where the noise does not follow the same Normal distribution for all units. In such settings, different measures of variability (e.g., the variance vs the mean absolute error) may yield different answers, the variability of \(Y\) given \(X\) needn’t be constant, and yet other complications may arise.

Additionally, in real-life settings, the answers to any of these formulations won’t be as clear cut as they are in our idealised setting with infinite data as each numerical answer will come with some uncertainty attached to it. But before addressing such complications, we first need to be clear-eyed about what we want to figure out in the first place.

So what’s the correct formulation?

Despite their lack of realism, I think there is some value in making simplifying assumptions and considering scenarios in which different answers will be given to different formulations of ostensibly the same research goal: stripping away some of the complications may help us better appreciate what it is we, or the researchers, are interested in.

So, out of the four formulations, what’s the correct one?

The boring answer is that it depends. But at the risk of committing the statisticians’ fallacy, I’m going to go ahead and say that either Formulation (F2) or Formulation (F4) may be best aligned with what is likely to be the researchers’ objective:

• Formulation (F2) concerns the added value of predictor \(X'\) and is suited if you’re interested in potentially recommending to other researchers that they should consider taking into account both \(X\) and \(X'\) when modelling \(Y\).
• Formulation (F4) is suited if you’re interested in potentially recommending to other researchers that they should forego collecting or computing \(X\) and use \(X'\) instead when modelling \(Y\).

Nothing prevents the researchers from trying to answer the research questions underpinning both of these formulations; it’s just that they should be aware that these are two different research questions with potentially different answers.

By contrast, Formulation (F1) seems premature. If it should turn out that the researchers are able to recommend that \(X\) should no longer be used and that \(X'\) is to be preferred, then it does become a sensible question to ask how good of a predictor of \(Y\) \(X'\) is in isolation. But this question cannot be answered using the same data set that is also used as the basis for this recommendation due to possible selection bias.

As for Formulation (F3), finally, I fail to see a use-case for it that isn’t better addressed by Formulations (F2) or (F4). Generally, I’m quite skeptical of research questions phrased in terms of correlation coefficients as these often poorly summarise the relation between two variables.

Conclusion

Statistical analyses serve to answer statistical questions. It is imperative that these questions align closely with the substantive research questions. But quite often, research questions are vague or formulated in several logically distinct ways. Considering idealised settings that strip away real-world complications is useful for clarifying which statistical quantities are actually of interest. The whole rigmarole of fitting and refitting models, accounting for interactions, nonlinearities, dependencies, and what-have-you is only going to be of any use once you know what you’re looking for.