Successful Reproducibility Hackathon Bern, May 5th 2023: First Findings
On May 5th 2023 the Bern node of the SwissRN organized a Reproducibility Hackathon. Thirteen people showed up, re-analysing the statistical analyses of pre-registered reports from the Reproducibility Project Cancer Biology. After eight hours of intense code hacking (with decent breaks for pizza and cake), over 60 confirmatory tests have been re-analysed. Participants got the original pre-registration protocol as well as the published data, while they did not see the code and the report of the study outcome published by the authors beforehand. The primary question we addressed with this Hackathon was, whether the pre-registration protocols provide an unambiguous plan how the data will be treated and analysed or whether they leave room for interpretation –so called “researchers’ degrees of freedom”. A first glance at the outcomes gives a mixed picture –with perfect replication in some cases, while in other cases the pathways of statistical analyses diverged, when authors and hackers had different views on removal of outliers, deviations from normality, data transformation, and adjustment for multiple testing. The immediate feedback from the participants of the Hackathon was rather positive, though several hackers found it surprisingly difficult to navigate through the jungle of provided data files and bring the data into shape for analysis. A more in-depth report of our findings and a summary of the experience will follow.
