# Walkthrough: A significance test for a two-group comparison

significance
R
teaching materials
Author

Jan Vanhove

Published

April 16, 2019

I wrote an R function that’s hopefully useful to teach students what significance tests do and how they can and can’t be interpreted.

Update (2023-08-06): By now, this function has long been integrated in the `cannonball` package. I’ve edited this blog post correspondingly. The installation instructions for the `cannonball` package are available from its GitHub page.

The basic use is simple. Just run `walkthrough_p()` and follow the on-screen instructions.

``cannonball::walkthrough_p()``

## What does the function do?

Data are generated from a normal distribution with the requested standard deviation. Then, the data points are randomly assigned to two equal-sized groups. Data points in the intervention group receive a boost as specified by `diff`. Finally, a significance test is run on the data.

By default, the significance test is a two-sample Student’s t-test. Technically, the p-value from this test is the probability that a t-statistic larger than the one observed would’ve been observed if only chance were at play, but the walkthrough text says that is the probability that a mean difference larger than the one observed would’ve been observed if only chance were at play. That is, I use the t-test as an approximation to a permutation test. Switch on pedant mode if you want to run a permutation test.

## Parameters

• `n`: the number of data points per group
• `diff`: The boost that participants in the intervention group receive.
• `sd`: The standard deviation of the normal distribution from which the data were generated.
• `showdata`: Do you want to output a dataframe containing the plotted data (TRUE) or not (FALSE, default)?
• `pedant`: Do you want to run the significance test in pedant mode (TRUE; performs a permutation test) or not (FALSE, default; performs Student’s t-test)?

## Examples

``````cannonball::walkthrough_p(n = 12, diff = 0.2, sd = 1.3)

# Save data and double check results
dat <- cannonball::walkthrough_p(n = 10, diff = 0.2,
sd = 2, showdata = TRUE)
t.test(score ~ group, data = dat, var.equal = TRUE)

# Run in pedant mode (= permutation test)
dat <- cannonball::walkthrough_p(n = 13, diff = 1, sd = 4,
pedant = TRUE, showdata = TRUE)
t.test(score ~ group, data = dat, var.equal = TRUE)``````