Walkthrough: A significance test for a two-group comparison

I wrote an R function that’s hopefully useful to teach students what significance tests do and how they can and can’t be interpreted.

Update (2023-08-06): By now, this function has long been integrated in the cannonball package. I’ve edited this blog post correspondingly. The installation instructions for the cannonball package are available from its GitHub page.

The basic use is simple. Just run walkthrough_p() and follow the on-screen instructions.

cannonball::walkthrough_p()

What does the function do?

Data are generated from a normal distribution with the requested standard deviation. Then, the data points are randomly assigned to two equal-sized groups. Data points in the intervention group receive a boost as specified by diff. Finally, a significance test is run on the data.

By default, the significance test is a two-sample Student’s t-test. Technically, the p-value from this test is the probability that a t-statistic larger than the one observed would’ve been observed if only chance were at play, but the walkthrough text says that is the probability that a mean difference larger than the one observed would’ve been observed if only chance were at play. That is, I use the t-test as an approximation to a permutation test. Switch on pedant mode if you want to run a permutation test.

Parameters

• n: the number of data points per group
• diff: The boost that participants in the intervention group receive.
• sd: The standard deviation of the normal distribution from which the data were generated.
• showdata: Do you want to output a dataframe containing the plotted data (TRUE) or not (FALSE, default)?
• pedant: Do you want to run the significance test in pedant mode (TRUE; performs a permutation test) or not (FALSE, default; performs Student’s t-test)?

Examples

cannonball::walkthrough_p(n = 12, diff = 0.2, sd = 1.3)

# Save data and double check results
dat <- cannonball::walkthrough_p(n = 10, diff = 0.2, 
                                 sd = 2, showdata = TRUE)
t.test(score ~ group, data = dat, var.equal = TRUE)

# Run in pedant mode (= permutation test)
dat <- cannonball::walkthrough_p(n = 13, diff = 1, sd = 4, 
                                 pedant = TRUE, showdata = TRUE)
t.test(score ~ group, data = dat, var.equal = TRUE)