Replication Crisis

• How the Replication Crisis became a Revolution Review of the History of Psychologists
• Early Philosophers


  • First big ideas about mind and science
• Enlightenment


  • Growing questions about mind, mechanism, empiricism
• Early psychologists


  • How to be experimentalists: study perception, consciousness, intelligence
• Psychoanalysts


  • The importance of the unconscious, inner conflict
• Behaviorists


  • No more "mind" silliness! Behavior Only!
• Cognitive Revolution


  • The mind is back in psychology, study as information processing
• Paradigm Shift


  Dominant schools of thought about how to study the mind scientifically have changed
• Paradigm shifts often occur during periods of upheaval, revolution
• The "replication crisis" in contemporary psychology is the focus of the final lecture
• Reproducibility


  The extent to which consistent results are observed when scientific studies are repeated

  
• Reproducibility is a major demarcation between science and pseudo-science
• Characteristics of Science


  • Systematic Observation
  • Ruthless peer review
  • Considers all evidence
  • Invites Criticism
  • Repeatable results
  • Limited claims
  • Specific terms, operational definitions
  • Engages community
  • Changes with new evidence
  • Follows evidence where it leads
• Characteristics of Pseudoscience


  • Anecdotal Evidence
  • No peer review
  • Considers only positive evidence
  • Dismisses criticism
  • Non-repeatable results
  • Grandiose claims
  • Vague terms and ideas. Science-y jargon
  • Isolated
  • Dogmatic and unyielding
  • Starts with a conclusion, works back to confirm
• The process of how psychological data is often collected is described as a "psychological fairy tale"
• Psychological fairy tale data collection


  1. Generate a hypothesis

  2. Collect some data

  3. Repeat step 3 until you have enough studies to publish
• Fairy tales are not real and actually pretty messed up
• In 2010, the Dutch social psychologist Diederik Stapel was investigated for fraud, with 25 published papers based on fabricated data
• Darryl Bem's ESP study raised concerns about reproducibility in psychology
• The Open Science Collaboration examined the reproducibility of 100 studies from 3 prominent journals, finding that only 36% of replications had significant effects in the same direction as the original studies
• Psychology has a serious reproducibility problem, threatening the reputation of psychology as a science
• Confirmation bias


  Tendency to seek out information that verifies your theory (validation), and not seek information that falsifies your theory (falsification)
• The replication crisis led to personal conflicts and fights between researchers
• Some solutions that have been recommended include: replicating studies, being aware of p-hacking, boosting statistical power, open data/materials/analysis, pre-registered confirmatory studies, incorporating open science practices in teaching, insisting on open science practices as reviewers, and rewarding open science practices
• The replication crisis in psychology should be viewed in historical context as part of the evolution of psychological science