Reliability and validity

• Reliability


  The consistency of a study or measuring device within a study, referring to whether a study produces similar results if replicated
• Test-retest reliability


  The same study/analysis/test is repeated several times under the same conditions at different times, and the results are compared
• Inter-observer/inter-rater reliability

  More than one researcher independently carries out the observation/analysis, and the results are compared
• Improving reliability in experiments


  1. Control the investigation as much as possible

  2. Operationalise variables

  3. Pilot test
• Improving reliability in questionnaires


  1. De-select or re-write complex/ambiguous questions

  2. Use more closed questions
• Improving reliability in interviews


  1. Use the same interviewer each time

  2. Train interviewers carefully to ensure questions are asked in the same way

  3. Use structured interviews
• Correlation coefficient


  +0.80 indicates high reliability
• Inter-observer/inter-rater reliability


  • More than one researcher independently carries out the observation/analysis
  • The results of the two analyses are compared
  • The correlation between the two results is calculated
  • A strong positive correlation between the scores indicates good reliability
• Observations/Content Analysis


  • Behavioural categories carefully operationalised and discussed so that observers/raters are clear about how to interpret behaviour
  • Record the data so it can be reviewed
  • Train the observers/raters
• Internal validity


  Concerns whether the test/measure/findings actually measures what it was intended to/claims to measure
• Construct validity


  Concerns the ability of the study to test the hypothesis it was designed to test
• External validity


  Concerns whether the findings of the study can be generalised beyond the study itself
• Ecological validity


  • The degree to which the findings of a study can be generalised to other situations, places and conditions
  • Usually, whether a measure of behaviour accurately reflects the way in which the behaviour would occur in normal circumstances
• Temporal validity


  Where the findings from research that took place at a certain point in time accurately reflect the way that behaviour would occur at a different point in time
• Population validity


  Where the findings from research can be generalised from the sample to other populations
• Face validity


  Where a behaviour appears on the face of it to represent what is being measured
• Concurrent validity


  • Where performance on one measure correlates highly with performance on another established measure of the same variable
  • The extent to which a psychological test/scale/measure relates to an established psychological test/scale/measure
  • Close agreement is indicated if the correlation between the two data sets exceeded +0.8
• Threats to validity


  • Extraneous variables
  • Investigator effects
  • Demand characteristics
  • Social desirability bias
  • Poor operationalisation of variables
  • Experimental design – order effects
  • Experimental design – participant differences
• Extraneous variables


  • Situational- All variables except the IV should be keep constant and use standardised procedures
  • Participant- The blind technique: participants do not know what group or condition they are in
• Investigator effects


  • Standardised procedures/script
  • Using the same investigator
  • Double-blind technique: neither the participants nor the experimenter know what each condition or group represents
• Demand characteristics


  • Blind procedures
  • Conduct field experiments
  • Covert observations
• Social desirability bias


  • Include a lie scale
  • Anonymity
• Poor operationalisation of variables


  • Clearly define all research variables or use measures and scales that have been validated in previous studies
  • Clearly define behavioural categories
• Experimental design – order effects

  Counterbalance to reduce the impact of order effects
• Experimental design – participant differences

  • Reduce participant variables by randomly allocating ppts to conditions- ppts have the same chance of being in either condition
  • Use a different design: e.g. matched pairs