reliability

• what does reliability mean?

  • Questions the extent to which the tests scales surveys observations or experiments that psychologist use are consistent in the sense that their measurements of behaviour are the same every single time they are used
  • Reliability is a measure of consistency
• there are two ways of assessing reliability. What are they?

  • Test – retest
  • Inter – observer reliability
• what is test – retest when assessing reliability?

  • This involves administering the same test or questionnaire to the same person on different occasions.
  • If the test or questionnaire is reliable, then the results obtained should be the same or similar each time they administered
  • there must be sufficient time between test and retest to ensure that the participant cannot recall their answers to the questions to a survey but not so long that their attitudes may have changed
• what is inter – observer reliability when assessing reliability?

  • when conducting an observation, a researcher interpretation of events may differ widely from someone else’s. this is called subjectivity
  • To overcome this observers should not observe alone but instead conduct their research teams of at least two
  • Inter - observer reliability must be established
  • This may involve a small scale trial run (pilot study) of the observations in order to check that observers are applying behaviour categories in the same way, or it may be reported at the end of the study to show that the data collected was reliable
  • Observers must watch the same event or sequence of events but record the data independently
  • as with the test - retest methods the data collected by the two observer should be correlated to assess its reliability
• how can we improve reliability in questionnaires?

  • A questionnaire that produces a low test – retest reliability (correlation that is below +0.80) may require some of the items to be removed or rewritten
  • If some questions are unclear and ambiguous or complex, they may be interpreted differently by the same person on different occasions
  • One solution would be to replace some of the open questions with fixed choice alternatives which may be less ambiguous
• how can we improve reliability in interviews?

  • Four interviews probably the best way of ensuring reliability is to use the same interviewer each time.
  • If this is not possible or practical or interviews must be probably trained, so, for example, one particular interviewer is not asking questions that are two leading or ambiguous
  • This is more easily avoided instructed interview interviews where the interviewers behaviour is more controlled by fixed questions
  • Interviews that are more instructed are likely to be less reliable
• how can we improve reliability in experiments?

  • Lab experiments are often described as being reliable because the researcher can exert strict control over many aspects of the procedure, such as the instructions that participants receive and the conditions within which they are tested
  • This is more about precise replication of a particular method though rather than demonstrating reliability of finding
  • That said, one thing that might affect the reliability of a finding is if the participants were tested under slightly different conditions each time they were tested
• how can we improve reliability in observations?

  • The reliability of observations can be improved by making sure that behavioural categories have been properly operationalised and that they are measurable and self evident
  • Categories should not overlap and all possible behaviour should be covered on the checklist
  • Categories are not operational or are overlapping different observers have to make their own judgement of water record and may well end up with different and inconsistent records