Types of Reliability

• What is reliability?

  Refers to the degree of consistency of findings from an investigation or measuring device used
• What are the 2 ways of assessing reliability

  • Test-retest
  • Inter-observer/inter-rater
• What is test-retest reliability?

  • administering the same test or questionnaire to the same person on different occasions
• How can test-retest reliability be used?

  • Reliability will be shown through the same, or very similar, results being obtained. (consistency)
  • conducted with questionnaires and psychological tests but can be applied to interviews
  • must be sufficient time between the test and retest to ensure the participant cannot recall their answers but not so long that their attitudes, opinion or abilities may have changed.
• How many test-retest reliability indicate a high measure?

   answers must be correlated to be similar

  Significant correlation = good reliability of the measuring instrument
• What is inter-observer reliability?

  • The extent to which there is an agreement between 2 or more observers observing a behaviour
  • Measured by correlating the observations of the observers
• What is the number that indicates high inter-observer reliability?

  Total number of agreements / total number of observations must be higher or equal to +.80
• How can inter-observer reliability be used?

  • One researchers' interpretation of events can be widely different (subjectivity bias)
  • A pilot study can be conducted to ensure that the servers are applying behavioural categories in the same way to someone else’s
  • Observers watch the same event but record their data independently.
  •   Data collected should correlate to be reliable.
• How to improve reliability within questionnaires?

  • reliability of questionnaires over time should be measured using the test-retest metho
  • Comparing two sets of data should produce a correlation that exceeds +.80
  •  A questionnaire that produces low test-retest reliability may require rewriting
• How to improve reliability within interviews?

  • Use the same interviewer each time (ensures consistency and EV of individual differences)
  • If this is not possible, ensure all interviewers are properly trained
  • This is to reduce leading or ambiguous questions which may decrease reliability
• How to improve reliability within observation experiments?

  • making the behavioural categories operationalised – measurable
  • If categories are not operationalised well, different observers have to make their own judgements of what to record where and may well end up with differing inconsistent records
• How to improve reliability within lab experiments?

  • Researcher has strict control over variables to ensure IV is affecting DV
  • Replications of experiments can be conducted to ensure the measured effect is consistent and didn't occur by chance