Reliability

• what is reliability?

  • a measure of consistency
  • this can refer to consistency over time, so that it is possible to obtain the same results on subsequent occasions when the same measure is used, or consistency between researchers when collecting their data
• what is the test-retest method?

  • the test or questionnaire is simply administered again on a second occasion, the results of the two tests are correlated to check for similarity
  • if a significant positive correlation is obtained (+0.8) then the reliability of the measuring instrument is assumed to be accepted
• what is inter-rater/observer reliability?

  • observers should conduct their observations in teams of at least two so that data on observed behaviours can be reliably collected, their results should then be correlated to check for reliability
  • a pilot study is used to get the most appropriate and accurate behavioural categories, using correlations which must be +0.8 and significant to be reliable
• when using a test-retest, what would make the test reliable?

  if on the second occasion, using the same conditions and people, the results are the same, or at least similar on both occasions
• what are some issues with test-retest method?

  there must be enough time between the two occasions so that the participants don't simply remember their answers from the first time, but also not so far apart that their answers will have completely changed from the first time (except on occasions where a change is expected e.g. after taking medication)
• how does test-retest method work with quantitative results?

  a correlation can be made to test how reliable the questionnaire (e.g.) is, if correlation is +0.8 and significant, then results are reliable
• when is inter-rater reliability mostly used?

  mostly relevant to observational methods which are open to interpretation
• what are some issues with inter-rater reliability?

  • different researchers may interpret the same situation in a different way to another researcher - subjectivity bias
• why must inter-rater reliability be established?

  to ensure that more than one researcher interprets a situation in the same way
• how can the reliability of questionnaires be improved?

  • low correlation using test-retest may require removal/rewriting of questions due to ambiguity - interpret differently on separate occasions, or may be too complex to guarantee same answer
  • open questions can be replaced with closed questions that only allow for fixed responses - are usually less ambiguous
• how can the reliability of interviews be improved?

  • unstructured - low reliability
  • structured - interviewers ask same questions to each participant, so they are not leading or ambiguous - higher reliability
  • different interviewers may ask questions in different styles, different accentuation on certain words, so is best to use same interviewer whenever possible
• how can you improve the reliability of experiments?

  • lab experiments with high control - high reliability, however conditions and instructions given must be exactly the same
  • instructions should be standardised to enforce this, and extraneous variables controlled as much as possible
  • field experiments - lower reliability
• how can you improve the reliability of observations?

  • behavioural categories could be misinterpreted, and must cover all possible behaviours - if some are missing/overlap the observers make own judgements on what to record and where (subjective)
  • categories must be operationalised to avoid this, must be measured and self-evident
  • using more than 1 (2 is best) observers means inter-observer reliability can be measured