Power simulations for comparing independent correlations
Every now and then, researchers want to compare the strength of a correlation between two samples or studies. Just establishing that one correlation is significant while the other isn’t doesn’t work – what needs to be established is whether the difference between the two correlations is significant. I wanted to know how much power a comparison between correlation coefficients has, so I wrote some simulation code to find out.
The contour plots below show the power of comparisons with sample sizes of 2×20, 2×40 and 2×80 observations for all combinations of population correlation coefficients. For instance, the first contour plot shows that you have about 90% power to find a significant difference between two correlation coefficients if the true population correlation in population A (x-axis) is 0.4 and -0.6 in population B (y-axis) and both sample contain 20 observations. If the correlation in population B is -0.2, however, you have less than 50% power. In blue is the contour line for 80% power for reference.
For unequal sample sizes, the contour plot might look like this:
Not really any new insights, but a good opportunity to stress once again that The difference between “significant” and “not significant” is not itself statistically significant.
And I got to play around with the
which are quite useful for avoiding for-loops in simulations (see below).
These simulations estimate the power for comparisons of independent correlations. Independent correlations are correlations computed for different samples (or different studies). An example of dependent correlations would be when you measure a variable, e.g. Italian proficiency, and correlate it to two other variables (e.g., French proficiency and Spanish proficiency) using the same participants. Since you used the same participants, there will exist some intercorrelation between French proficiency and Spanish proficiency, which needs to be taken into account when comparing the correlations between Italian and French proficiency on the one hand and Italian and Spanish proficiency on the other.
First load the
psych packages (run
install.packages(c("MASS", "psych")) if they aren’t installed yet).
mvrnorm function from the
we can generate samples drawn from a bivariate normal distribution with a specific population correlation coefficient (the numbers of the antidiagonal in the
Sigma parameter; in this example: 0.3).
cor we compute the sample correlation coefficients for these samples;
these will differ from sample to sample.
r.test function from the
we can compute the significance of the difference between two sample correlation coefficients.
In this case, the correlation coefficients were computed for independent samples,
r34 parameters are specified.
With that out of the way, we now write a new function,
that generates two samples of sizes
from bivariate normal distributions with population correlations of
Now we write another function,
compute.power, that takes
compute.p, runs it, say, 1000 times,
and returns how many p-values lie below 0.05 – i.e., the comparison’s estimated power.
Here’s where the R fun begins.
I want to compute the power not only for a single comparison,
but for nearly the whole
popr34 spectrum of possible comparisons:
-0.95 v. -0.90, -0.95 v. -0.85, …, 0.7 v. -0.3 etc.
All relevant correlations are stored in
outer function, I generate a grid featuring every possible combination of coefficients in
corrs and run
compute.power on each combination using
Here, I estimate the power for a comparison with two samples of 20 observations.
contour, the results matrix is then visualised:
This code could probably be optimised a bit; the power for the comparison between -0.5 and 0.3 is obviously identical to the power for the comparison between 0.5 and -0.3, for instance.