Draft: Replication success as predictive utility
In recent years, a couple of high-profile, multi-lab attempts to replicate previous findings have been conducted in psychology, but there wasn’t much consensus about when a replication attempt should be considered as confirming the original finding. As I started dabbling in predictive modelling a couple of months ago, I started thinking that it could be useful to view replication success in terms of how accurately the original finding predicts the replication: If taking the original finding at face value permits more accurate predictions about the patterns in the replication data than ignoring it does, the original finding can at least be said to contribute to our body of knowledge. This approach, I think, has some key advantages over the methods used to quantify replication success in the multi-lab replication attempts that I mentioned earlier.
I went in a bit head over heels and started writing R functions and an accompanying paper outlining my thoughts and how the R functions could be used, only to discover after having pretty much finished it that Verhagen and Wagenmakers (2014) had already put forward a suggestion bearing many similarities to mine a few years earlier. Their approach differs in some practical and philosophical aspects from mine (they’re not concerned with prediction, for instance), and I think my suggestion still has some additional value. That said, these similarities should have prompted a thorough revision of the accompanying paper, for which I haven’t had the time yet. Still, I thought I’d share the R functions and paper and solicit some comments in the meantime.
- Download the paper.
- Download the R functions.
- Download the dataset for the first worked out example.
- Download the dataset for the second worked out example.
Comments more than welcome!