Open Science

• What was the replication crisis?

  • Methodological crisis in which the results of studies were not reproducible when tested again
  • Affects many scientific disciplines
  • Bem (2011) published that showed evidence for 'Extrasensory Perception', lots of media coverage, provided evidence that the future could influence the present, used a standard psychological paradigm, published in a good journal, standardised stat methods
  • Science appeared broken
• What effect did the replication crisis have?

  • Pressure on academics to publish in top journals
  • Pressure to produce new and exciting results as this meant you had a higher chance of being in top journals
  • Led to lots of false positives
  • Needs to be less focus on end goals
• Open science

  Refers to a set of research practices designed to overcome limitations that previous scientific methodologies has presented to us:

  • Reproducibility
  • Replicability

  In order to asses this, we need research practices and data to be transparent and accessible
• Reproducibility

  If we have the same data and perform the same analysis, do we get the same results? (same data, re-do stats test)
• Replicability

  If we repeat the experiment with the same methods do we get the same results? (new data, new participants)
• Principles of open science:

  • Open access
  • Open data, Materials and Code
  • Reproducible Analyses
  • Preregistration
  • Replication Research
  • Teaching Open Science
• Open Access


  Refers to unrestricted public access to research. Typically used in reference to published journal articles but anything but anything can be made open access- student work, materials, code and data.

  Benefits:

  • Accumulation of knowledge
  • Increased citation of work
  • More media coverage
  • Support meta-research practice (meta-analysis)
• Open data, Materials and Code

  APA require researchers to be willing and able to make data available with editors for 5 years after publication. They share Data, Protocols and Code for Experiments and Analysis

  Benefits:

  • Verification- results can be checked in order to minimise errors and biases
  • Analytic reproducibility- checking which steps were taken to prepare and analyse the data
• Reproducible Analyses

  Need to provide materials to enable others to generate the exact same results as those reported, adequate documentation needs to be kept.

  Good Principles:

  • Provide clear annotations of what documents are
  • Store original data files separately
  • Record all steps of data processing
  • Use open source software where possible
• Preregistration

  Researchers are encouraged to submit plans for the specific research questions that they wish to address and the analyses they will conduct prior to data collection.

  • Confirmatory research- focuses on confirming hypotheses or research questions
  • Exploratory research- focuses on generating hypotheses or research questions

  Pre-registration promotes purely confirmatory research and reduces the risks of false positive results
• Replication Research

  The process of repeating research to verify findings.

  Enables confidence in results and helps to build theories.

  • Direct replication- reproduce the elements that produced the original findings (an attempt to reproduce a previous result with the same procedure) assess whether similar findings are produced in subsequent attempts
  • Conceptual replications- change at least one aspect f the original procedure (an attempt to reproduce a result using a different methodology) assess whether similar findings are produced under different conditions
• Teaching Open Science

  There is a need to teach the principles of open science

  • Promotes best practice
  • Increases the critical evaluation of research

  Increasingly being taught at undergraduate level and several teaching resources are made openly available to promote this process