# Common-language effect sizes

simplicity
effect sizes
R
Author

Jan Vanhove

Published

November 16, 2016

The goal of this blog post is to share with you a simple `R` function that may help you to better communicate the extent to which two groups differ and overlap by computing common-language effect sizes.

## What is the ‘common-language effect size’?

In 1992, McGraw and Wong introduced the common-language effect size, which they defined as

the probability that a score sampled at random from one distribution will be greater than a score sampled from some other distribution.

For instance, if you have scores on an English reading comprehension task for both French- and German-speaking learners, you can compute the probability that a randomly chosen French-speaking learner will have a higher score than a randomly chosen German-speaking learner. This gives you an idea of how much the groups’ scores overlap, and the number can more easily be communicated to an audience that has no firm notion of what quantiles are or of what standardised effect sizes such as d = 0.3 mean.

## Computing common-language effect sizes in R

Below I first generate some data: 40 data points in a group creatively called `A` vs. 30 data points in group `B`.

``````# Generate data
set.seed(16-11-2016)
df <- data.frame(Outcome = c(rbeta(40, 4, 14),   # 40 observations in A
rbeta(30, 2, 4)),   # 30 in B
Group = c(rep("A", 40), rep("B", 30)))``````

A couple of boxplots to show the spread and central tendencies:

And the key summary statistics:

``````# A tibble: 2 × 3
Group  Mean Standard_Deviation
<chr> <dbl>              <dbl>
1 A     0.244             0.0841
2 B     0.301             0.172 ``````

On the basis of the group means and standard deviations, McGraw and Wong’s common-language effect size can be computed as follows:

``pnorm(0, 0.24 - 0.30, sqrt(0.084^2 + 0.172^2), lower.tail = FALSE)``
``[1] 0.38``

I.e., there’s a 38% chance that if you put an observation from Group A and one from Group B together at random, the one from Group A will be greater.

Strictly speaking, McGraw and Wong’s method assumes normally distributed, continuous data. While they point out that their measure is quite robust with respect to this assumption, you can use a brute-force method that doesn’t make this assumption to see if that yields different results.

Edit: On Twitter, Guillaume Rousselet suggested a quicker and mor exhaustive brute-force method for computing common-language effect sizes. I’ve updated the code and post to implement his suggestion.

On https://janhove.github.io/RCode/CommonLanguageEffectSizes.R, I provide a function, `cles.fnc()`, that pairs each observation from the first group to each observation from the second group and then checks how often the observation from the first group is larger than the one from the second group. Ties are also taken into account.

Here’s how the `cles.fnc()` function works:

``````# Read in the function
source("https://janhove.github.io/RCode/CommonLanguageEffectSizes.R")

# Set variable you want to compare between the groups,
# the group name,
# the baseline level,
# and the dataset:
cles <- cles.fnc(variable = "Outcome", group = "Group", baseline = "A", data = df)``````
``````Common-language effect size:

The probability that a random Outcome observation from group A
is higher/larger than a random Outcome observation from the other group(s):

Algebraic method:   0.38
Brute-force method: 0.4``````

The results for both methods aren’t identical (38% vs. 40%), but they’re in the same ballpark. This is more often the case than not.

You can turn off the output by setting the parameter `print` to `FALSE`:

``````## (not run)
# cles <- cles.fnc(variable = "Outcome", group = "Group", baseline = "A", data = df, print = FALSE)``````

You can also extract information from the `cles` object if you want to pass it on to other functions:

``cles\$algebraic # McGraw & Wong's method``
``[1] 0.38``
``cles\$brute     # brute-force method``
``[1] 0.4``

## An example with non-overlapping distributions

The code below generates a dataset with two non-overlapping groups.

``````# Generate data
set.seed(16-11-2016)
df <- data.frame(Outcome = c(sample(0:20, 10, replace = TRUE),
sample(21:25, 10, replace = TRUE)),
Group = c(rep("A", 10), rep("B", 10)))``````

Boxplots:

McGraw & Wong’s (1992) method suggests that there’s a 4% chance that a random observation in A will be higher than one in B. This may well be true at the population level, but it’s clearly not true at the sample level. The brute-force method pegs this probability at 0%, which may be wrong at the population level, but it’s clearly correct at the sample level.

``cles <- cles.fnc(variable = "Outcome", group = "Group", baseline = "A", data = df)``
``````Common-language effect size:

The probability that a random Outcome observation from group A
is higher/larger than a random Outcome observation from the other group(s):

Algebraic method:   0.04
Brute-force method: 0``````

## For use with more complex datasets

Let’s say you have data from a longitudinal study in which you collected data for Groups A and B at Times 1, 2 and 3, and you want to compare the groups at each time:

``````# Generate data
set.seed(16-11-2016)
df <- data.frame(Time = c(rep(1, 40), rep(2, 40), rep(3, 40)),
Group = rep(c(rep("A", 25), rep("B", 15)), 3),
Outcome = c(rnorm(25, 1.1, 1.0), rnorm(15, 1.2, 0.3),
rnorm(25, 1.2, 0.6), rnorm(15, 1.2, 0.8),
rnorm(25, 1.8, 0.4), rnorm(15, 1.7, 1.0)))``````

The boxplots:

Using the `by()` function, you can run `cles.fnc()` separately for each `Time`. For more complex datasets, you can include more variables in the `INDICES` list.

``````cles <- with(df, # select dataframe df
by(data = df, # group dataframe df
INDICES = list(Time = Time), # by Time
# and run cles.fnc() within each group
FUN = function(x) cles.fnc("Outcome", "Group", "A", data = x, print = FALSE)))
cles``````
``````Time: 1
\$algebraic
[1] 0.45

\$brute
[1] 0.55

------------------------------------------------------------
Time: 2
\$algebraic
[1] 0.46

\$brute
[1] 0.5

------------------------------------------------------------
Time: 3
\$algebraic
[1] 0.75

\$brute
[1] 0.77``````

## Software versions

``devtools::session_info()``
``````─ Session info ───────────────────────────────────────────────────────────────
setting  value
version  R version 4.3.1 (2023-06-16)
os       Ubuntu 22.04.3 LTS
system   x86_64, linux-gnu
ui       X11
language en_US
collate  en_US.UTF-8
ctype    en_US.UTF-8
tz       Europe/Zurich
date     2023-08-27
pandoc   3.1.1 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)

─ Packages ───────────────────────────────────────────────────────────────────
package     * version date (UTC) lib source
cachem        1.0.6   2021-08-19 [2] CRAN (R 4.2.0)
callr         3.7.3   2022-11-02 [1] CRAN (R 4.3.1)
cli           3.6.1   2023-03-23 [1] CRAN (R 4.3.0)
colorspace    2.1-0   2023-01-23 [1] CRAN (R 4.3.0)
crayon        1.5.2   2022-09-29 [1] CRAN (R 4.3.1)
devtools      2.4.5   2022-10-11 [1] CRAN (R 4.3.1)
digest        0.6.29  2021-12-01 [2] CRAN (R 4.2.0)
dplyr       * 1.1.2   2023-04-20 [1] CRAN (R 4.3.0)
ellipsis      0.3.2   2021-04-29 [2] CRAN (R 4.2.0)
evaluate      0.15    2022-02-18 [2] CRAN (R 4.2.0)
fansi         1.0.4   2023-01-22 [1] CRAN (R 4.3.1)
farver        2.1.1   2022-07-06 [1] CRAN (R 4.3.0)
fastmap       1.1.0   2021-01-25 [2] CRAN (R 4.2.0)
forcats     * 1.0.0   2023-01-29 [1] CRAN (R 4.3.0)
fs            1.5.2   2021-12-08 [2] CRAN (R 4.2.0)
generics      0.1.3   2022-07-05 [1] CRAN (R 4.3.0)
ggplot2     * 3.4.2   2023-04-03 [1] CRAN (R 4.3.0)
glue          1.6.2   2022-02-24 [2] CRAN (R 4.2.0)
gtable        0.3.3   2023-03-21 [1] CRAN (R 4.3.0)
hms           1.1.3   2023-03-21 [1] CRAN (R 4.3.0)
htmltools     0.5.5   2023-03-23 [1] CRAN (R 4.3.0)
htmlwidgets   1.6.2   2023-03-17 [1] CRAN (R 4.3.1)
httpuv        1.6.11  2023-05-11 [1] CRAN (R 4.3.1)
jsonlite      1.8.7   2023-06-29 [1] CRAN (R 4.3.1)
knitr         1.39    2022-04-26 [2] CRAN (R 4.2.0)
labeling      0.4.2   2020-10-20 [1] CRAN (R 4.3.0)
later         1.3.1   2023-05-02 [1] CRAN (R 4.3.1)
lifecycle     1.0.3   2022-10-07 [1] CRAN (R 4.3.0)
lubridate   * 1.9.2   2023-02-10 [1] CRAN (R 4.3.0)
magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.3.0)
memoise       2.0.1   2021-11-26 [2] CRAN (R 4.2.0)
mime          0.10    2021-02-13 [2] CRAN (R 4.0.2)
miniUI        0.1.1.1 2018-05-18 [1] CRAN (R 4.3.1)
munsell       0.5.0   2018-06-12 [1] CRAN (R 4.3.0)
pillar        1.9.0   2023-03-22 [1] CRAN (R 4.3.0)
pkgbuild      1.4.2   2023-06-26 [1] CRAN (R 4.3.1)
pkgconfig     2.0.3   2019-09-22 [2] CRAN (R 4.2.0)
pkgload       1.3.2.1 2023-07-08 [1] CRAN (R 4.3.1)
prettyunits   1.1.1   2020-01-24 [2] CRAN (R 4.2.0)
processx      3.8.2   2023-06-30 [1] CRAN (R 4.3.1)
profvis       0.3.8   2023-05-02 [1] CRAN (R 4.3.1)
promises      1.2.0.1 2021-02-11 [1] CRAN (R 4.3.1)
ps            1.7.5   2023-04-18 [1] CRAN (R 4.3.1)
purrr       * 1.0.1   2023-01-10 [1] CRAN (R 4.3.0)
R6            2.5.1   2021-08-19 [2] CRAN (R 4.2.0)
Rcpp          1.0.11  2023-07-06 [1] CRAN (R 4.3.1)
readr       * 2.1.4   2023-02-10 [1] CRAN (R 4.3.0)
remotes       2.4.2   2021-11-30 [2] CRAN (R 4.2.0)
rlang         1.1.1   2023-04-28 [1] CRAN (R 4.3.0)
rmarkdown     2.21    2023-03-26 [1] CRAN (R 4.3.0)
rstudioapi    0.14    2022-08-22 [1] CRAN (R 4.3.0)
scales        1.2.1   2022-08-20 [1] CRAN (R 4.3.0)
sessioninfo   1.2.2   2021-12-06 [2] CRAN (R 4.2.0)
shiny         1.7.4.1 2023-07-06 [1] CRAN (R 4.3.1)
stringi       1.7.12  2023-01-11 [1] CRAN (R 4.3.1)
stringr     * 1.5.0   2022-12-02 [1] CRAN (R 4.3.0)
tibble      * 3.2.1   2023-03-20 [1] CRAN (R 4.3.0)
tidyr       * 1.3.0   2023-01-24 [1] CRAN (R 4.3.0)
tidyselect    1.2.0   2022-10-10 [1] CRAN (R 4.3.0)
tidyverse   * 2.0.0   2023-02-22 [1] CRAN (R 4.3.1)
timechange    0.2.0   2023-01-11 [1] CRAN (R 4.3.0)
tzdb          0.4.0   2023-05-12 [1] CRAN (R 4.3.0)
urlchecker    1.0.1   2021-11-30 [1] CRAN (R 4.3.1)
usethis       2.2.2   2023-07-06 [1] CRAN (R 4.3.1)
utf8          1.2.3   2023-01-31 [1] CRAN (R 4.3.1)
vctrs         0.6.3   2023-06-14 [1] CRAN (R 4.3.0)
withr         2.5.0   2022-03-03 [2] CRAN (R 4.2.0)
xfun          0.39    2023-04-20 [1] CRAN (R 4.3.0)
xtable        1.8-4   2019-04-21 [1] CRAN (R 4.3.1)
yaml          2.3.5   2022-02-21 [2] CRAN (R 4.2.0)

[1] /home/jan/R/x86_64-pc-linux-gnu-library/4.3
[2] /usr/local/lib/R/site-library
[3] /usr/lib/R/site-library
[4] /usr/lib/R/library

──────────────────────────────────────────────────────────────────────────────``````