Reliability and validity

• Reliability

  • Refers to the consistency of results
  • studies are considered reliable if they can be repeated with similar outcomes.
  • Quantitative methods are generally the most reliable.
  • Examples of quantitative methods include:
  • Experiments
  • Interviews and Questionnaires
  • Observations
• Laboratory Experiments

  • Considered reliable due to standardised procedures and controlled environments meaning each participant has the same experience
  • The controlled settings facilitate easy replication and repetition of experiments.
• Interviews and questionnaires

  • Interviews and questionnaires are tools to gather participants' thoughts and feelings.
  • Consistency in responses is expected from participants when the tools are used.
  • These methods can demonstrate high reliability.
  • The same researcher can ask identical questions in the same manner
  • Closed questions are more likely to produce reliable results as the choices are fixed
• Observations:

  • Consistency: The same observer should get the same results if they repeat the observations.
  • Inter-observer Reliability: If there are two observers, they should get the same scores on their behaviour category checklists.
  • This consistency between observations helps ensure reliability in the data collected.
• Qualitative methods

  • tend to be less reliable
  • Unstructured interviews and case studies are hard to repeat in the exact same way
  • Lack of Reliability: This makes them less reliable and lack consistency due to variability in results.
  • Example of Unstructured Interviews: Different interviewers might ask different follow-up questions, leading to different results
  • Interpretation in Case Studies: Data from case studies can be interpreted in various ways, adding to inconsistencies.
• Inter-rater Reliability

  • To check the reliability of results, involve more than one researcher in the study.
  • Agreement on Findings: If researchers agree on their findings and get similar results, this indicates reliability.
  • Inter-rater Reliability: The agreement between different researchers is referred to as inter-rater reliability
  • The results have inter-rater reliability
• Validity

  • Accuracy of Results: Refers to how accurate or true the results of a study are.
  • Real-Life Reflection: Also considers whether the study accurately reflects real-life situations or behaviors.
• Internal validity

  • Checks if the study measures exactly what the researcher intended

  External Validity

  • Determines if the study’s findings can be generalised to other situations, populations, or time periods.
• Sampling methods

  • Goal of Sampling: To create a participant group that represents the target population.
  • Opportunity Sampling: Has low representativeness (may not reflect the broader population well).
  • Stratified Sampling: Has high representativeness, better reflecting the target population.
• Experimental design

  • Repeated Measures: Can lead to order effects (participants may perform differently based on the order of tasks), which affects validity.
  • Independent Groups: Participant differences (variables) can impact validity, but random allocation helps reduce this issue.
  • Matched Pairs: Has the highest validity because participants are paired based on similar characteristics, reducing variability.
• Quantitative methods:

  • Laboratory Experiments:
  • High control over tasks and settings, but often very artificial.
  • The more artificial the setting or task, the lower the validity (less realistic reflection of everyday life).
  • However greater control enhances validity
  • Field Experiments:
  • More natural settings increase realism.
  • Tasks can still be artificial
  • P's may be aware they are being studied
  • Presence of extraneous variables (uncontrolled factors) can affect results, making it unclear if changes in the dependent variable (DV) are truly due to the independent variable (IV).
• Methods producing numerical data

  • Reduces complex behaviors to simple scores.
  • Lacks additional detail, meaning it doesn’t capture the full context or depth of behaviour.
  • This reduction limits understanding and thus reduces validity
• Qualitative methods

  • Purpose of Qualitative Methods: Aimed at gathering in-depth information that quantitative methods miss.
  • Case Studies: Provide high validity by offering deep insight into participants’ thoughts, behaviors, and perspectives.
  • Challenges with Qualitative Data:
  • Difficult to analyze due to its detailed and complex nature.
  • Often subjective, as researchers’ expectations can influence the interpretation.
  • This subjectivity can reduce the overall validity of the conclusions.
• Threats to validity

  • Demand Characteristics
  • Social desirability
  • Artificial setting
  • Participant variables
  • Order effects
  • Observer effect