Reliability

• Reliability = Refers to how consistent an observed effect is.
• Inter-Observer Reliability = The extent to which there is agreement between two or more observers involved in observations of a behaviour.
• Reliability = The consistency of measurements.
• Test-Retest Reliability = The same test or interview is given to the same participants on two occasions to see if the same results are obtained.
• Internal Reliability = A measure of the extent to which something is consistent within itself. It refers to what is happening inside the study.
• Internal reliability is assessed by the split-half method - the test is split in half and assessed. If the test is assessing the same thing, there should be a close correlation in the scores derived from both halves.
• External reliability = A measure of consistency over several different occasions. It refers to what is going on outside of the study.
• External reliability is assessed by the test-retest method and by inter-observer reliability.
• Improving reliability - Lab experiments:

  • Control extraneous variables
  • Clearly described and in depth method section
  • Have standardised instructions.
• Improving reliability - Interviews:

  • Use a pilot study.
• Reliability of Observations

  Assessing Reliability:

  • If the observations are reliable the second set of observations should be more or less the same as the first set.
  • A better way to assess accuracy is to have 2 or more observers making separate recordings and then comparing these records.
  • The extent to which the observers agree is called inter-observer reliability.
  • This can be calculated as a correlation coefficient for pairs of scores.
• Reliability of Observations

  Improving Reliability:

  • If the score for inter-rater reliability is low it can be improved by:
  • Operationalising behavioural categories clearly - Stops observers interpreting actions differently.
  • Some observers may need more practice using behavioural categories so they can respond more quickly.
• Reliability of Self-Report Techniques:

  Assessing Reliability - Inter-interviewer reliability:

  • A researcher could assess the reliability of one interviewer by comparing answers on one occasion with answers from the same person with the same interviewer a week later.
  • The researcher might want to assess the reliability of 2 interviewers using the same method as with 2 observers.
• Reliability of Self-Report Techniques:

  Assessing Reliability - Test-retest reliability:

  • Test-retest reliability is used to assess the reliability of a psychological test or other self-report measure.
  • The test/questionnaire designer gives the test to a group of people and then gives the same people the same test a second time. Usually there is a short interval between the tests so people don't remember their answers.
  • If the measure is reliable the outcome should be the same every time.
  • The scores for each person are compared using correlation.
• Reliability of Self-Report Techniques:

  Improving Reliability - Reducing Ambiguity:

  • Low reliability in a psychological test may be because some test items are ambiguous so people give different answers.
  • In such cases test items need to be re-examined and rewritten.
• Reliability of Experiments

  Improving Reliability - Standardisation:

  • It is important the procedures in an experiment are exactly the same each time because otherwise the performance of participants cannot be compared. Procedures are standardised.
  • If another researcher wishes to repeat the experiment, they also need to use exactly the same procedures.