False-positive psychology

Author

Jan Vanhove

Published

July 28, 2026

Explanation

You can use the app below to reproduce some of the simulations from Simmons et al.’s (2011) False-positive psychology. The app simulates the following behaviour:

  • A researcher recruits a number of participants (default: 20) who are randomly assigned to one of two conditions.
  • Two outcome variables are measured per participant. These variables may be correlated with each other (default correlation: \(r = 0.5\)).
  • The researcher analyses both variables with separate significance tests to check if they differ significantly between the two conditions. S/he does this three times:
    • The first time s/he analyses the first variable separately.
    • Then s/he analyses the second variable separately.
    • Finally, s/he averages the first and second variable per participant and analyses these averages in a third significance test.
  • If none of these three significance tests return a significant result (i.e., if none returns \(p < 0.05\)), s/he recruits a couple of additional participants (default: 10) and analyses the data anew.
  • S/he keeps on recruiting and analysing data until at least one significance test yields a significant result or until s/he reaches the maximum number of participants s/he can pay (whichever comes first).
  • S/he writes up the results of his study, reporting only the lowest of the p-values s/he obtained.

Simulation

Please have some patience: This app simulates and analyses a large number of datasets. This takes a while.

#| '!! shinylive warning !!': |
#|   shinylive does not work in self-contained HTML documents.
#|   Please set `embed-resources: false` in your metadata.
#| standalone: true
#| viewerHeight: 850

library(shiny)
library(shinythemes)
library(ggplot2)
library(dplyr)
library(MASS)
library(gridExtra)

theme_jv <- function(font_size = 9) {
  theme_bw(base_size = font_size) %+replace%
    theme(
      panel.grid = element_blank(),
      axis.ticks = element_line(colour = "black"),
      axis.text = element_text(colour = "black")
    )
}
theme_set(theme_jv(14))

# p-value function
situation_AB.fnc <- function(
    min_n = 20, max_n = 30, add = 10, r = 0.50
) {
  group <- rep(c(-0.5, 0.5), times = max_n)
  
  outcomes <- MASS::mvrnorm(
    n = 2 * max_n,
    mu = c(0, 0),
    Sigma = matrix(c(1, r, r, 1), nrow = 2)
  )
  
  average <- rowMeans(outcomes)
  
  df <- data.frame(
    group = group,
    outcome1 = outcomes[, 1],
    outcome2 = outcomes[, 2],
    average = average
  )
  
  n <- min_n
  p_value <- Inf
  
  repeat {
    if (p_value < 0.05) break
    if (n > max_n) break
    
    tests <- summary(
      lm(cbind(outcome1, outcome2, average) ~ group,
         data = df[1:(2 * n), ])
      )
    
    p_1 <- tests[[1]]$coefficients[2, 4]
    p_2 <- tests[[2]]$coefficients[2, 4]
    p_3 <- tests[[3]]$coefficients[2, 4]
    
    p_value <- min(c(p_1, p_2, p_3))
    
    n <- n + add
    
    if (add <= 0) break
    if (max_n <= min_n) break
  }
  
  p_value
}

ui <- fluidPage(
  theme = shinytheme("united"),
  titlePanel("False-positive psychology"),
  
  sidebarLayout(
    sidebarPanel(
      sliderInput(
        "min_n",
        "Minimum number of participants in each group:",
        min = 5, max = 100, value = 20, step = 1
      ),
      sliderInput(
        "max_add",
        "Maximum number of additional participants in each group:",
        min = 0, max = 50, value = 10, step = 1
      ),
      sliderInput(
        "n_add",
        "After how many new participants per group should the data be analysed again?",
        min = 0, max = 50, value = 10, step = 1
      ),
      sliderInput(
        "r",
        "Correlation between the dependent variables:",
        min = -1, max = 1, step = 0.05, value = 0.5
      ),
      actionButton("go", "Simulate!")
    ),
    
    mainPanel(
      plotOutput("pValueDistribution", width = "450px", height = "750px")
    )
  )
)

server <- function(input, output) {
  generate_p_values <- eventReactive(input$go, {
    n_sims <- 1000
    p_values <- numeric(n_sims)
    
    withProgress(message = "Simulating experiments", value = 0, {
      for (i in seq_len(n_sims)) {
        p_values[i] <- situation_AB.fnc(
          min_n = input$min_n,
          max_n = input$min_n + input$max_add,
          add = input$n_add,
          r = input$r
        )
        incProgress(1 / n_sims)
      }
    })
    
    p_values
  })
  
  output$pValueDistribution <- renderPlot({
    p_values <- generate_p_values()
    df <- data.frame(p_values)
    false_positive_rate <- mean(p_values < 0.05)
    margin_of_error <-
      1.96 * sqrt(false_positive_rate * (1 - false_positive_rate) / length(p_values))
    
    p1 <- ggplot(df, aes(x = p_values, fill = factor(p_values < 0.05))) +
      geom_histogram(colour = "black", breaks = seq(0, 1, by = 0.05)) +
      scale_fill_manual(values = c("#2b83ba", "#d7191c")) +
      geom_hline(yintercept = nrow(df) * 0.05, linetype = "dashed") +
      xlab("p-value") +
      ylab("Number of simulations") +
      labs(
        title = paste("Distribution of lowest p-values in", nrow(df), "experiments"),
        subtitle = paste(
          "Type I error:", round(false_positive_rate, 2),
          "\u00B1", round(margin_of_error, 2)
        )
      ) +
      theme(legend.position = "none")
    
    significant_only <- df %>%
      filter(p_values < 0.05) %>%
      mutate(p_group = cut(p_values, breaks = seq(0, 0.05, 0.01))) %>%
      group_by(p_group) %>%
      summarise(n = n()) %>%
      ungroup()
    
    p2 <- ggplot(significant_only, aes(x = p_group, y = n)) +
      geom_path(group = 1) +
      geom_point() +
      xlab("p-value") +
      ylab("Number of simulations") +
      expand_limits(y = 0) +
      labs(
        title = paste("Distribution of the", sum(p_values < 0.05), "p-values below 0.05"),
        subtitle = ""
      )
    
    gridExtra::grid.arrange(p1, p2, ncol = 1)
  })
}

shinyApp(ui = ui, server = server)

Exercises

Before running the simulation to answer each question, first reason through what will happen.

  1. Increase the ‘maximum number of additional participants in each group’ to 30. Leave the other settings at their default values. How will the graphs change?
  2. Leaving all other settings as they currently are, what will happen if instead of analysing the data after 10 new participants per condition, they’re analysed after 5 new participants per condition? Or after just 2 new participants per condition?
  3. What will happen when the correlation between the two outcome variables becomes weaker (e.g., \(r = 0.1\) instead of \(r = 0.5\))? Why?
  4. What will happen when the correlation between the two outcome variables becomes stronger (e.g., \(r = 0.95\))? Why?
  5. For which combination of the different parameters will you obtain the highest Type-I error?
  6. For which combination of the different parameters will you find a Type-I error rate of about 5%? Are there any parameters that don’t play a role?

Reference

Simmons, Joseph P., Leif D. Nelson and Uri Simonsohn. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22(11). 1359–1366.