Reliability

• Reliability


  The extent to which results are consistent
• Assessing and improving reliability

  1. Test-retest reliability

  2. Inter-observer reliability
• Inter-observer reliability


  The extent to which multiple observers agree in their tallied recordings
• High inter-observer reliability means that the observers recorded the same behaviours so ended up with consistent (reliable) findings
• Using only one observer could result in subjectivity. This means that if another observer was used, they may end up with different (unreliable) results
• Assessing inter-observer reliability


  1. Two observers watch the same people at the same time but tally their recordings independently

  2. The observers' recordings are then correlated on a scattergram to identify whether there is a positive correlation between them

  3. The strength of the correlation is then assessed by calculating a correlation coefficient

  4. A statistical test (Spearman's rho or Pearson's R) is used to assess whether the correlation is significant
• Ways to improve inter-observer reliability

  • Establish clear behavioural categories to avoid subjective interpretation
  • Train the observers with examples to ensure they understand the behavioural categories and how to tally them
  • Film the participants so that behaviours could be reviewed as often as required so that none are missed
• Test-retest reliability


  The extent to which the results of a study are consistent when the study is replicated
• The results of a study may be affected by an extraneous variable. This means that if the study was conducted again, the researcher may gather different (unreliable) findings
• Assessing test-retest reliability


  1. The same participants would complete the study on more than one occasion

  2. Each participants' scores are correlated on a scattergram to identify whether there is a positive correlation between them

  3. The strength of the correlation is then assessed by calculating a correlation coefficient

  4. A statistical test (Spearman's rho or Pearson's R) is used to assess whether the correlation is significant
• Ways to improve test-retest reliability

  • Experiments: Use lab experiments and standardisation
  • Observations: Use controlled observations, clear behavioural categories and use the same observer(s)
  • Questionnaires: Use closed questions, make sure anonymity is offered both times
  • Interviews: Use structured interviews, use the same interviewer(s), train all interviewers