Reliability

• What is reliability?

  consistency of results. If you repeat the same study/test, you should get the same outcome. If two observers are observing the same behaviour, they should record the same results
• What does a study also lack if it lacks reliability?

  validity - if a study gives different results each time it is carried out, then its validity is questionable as there must be something happening within the study which is affecting the DV
• What are the 2 types of reliability?

  internal and external
• What is internal reliability?

  consistent within itself
• What is external reliability?

  consistent over time
• What are the threats to reliability?

  - unstandardised procedures (if pps are asked questions differently each time they will answer differently)

  - ambiguous questions (pps may be unsure of how to answer)

  - confusing behavioural categories (observers will interpret the behaviour in their own way, which will reduce their inter-rater reliability

  - untrained observers (they may not know when to note things down or what constitutes a category)
• How do we asses reliability in observations?

  1) repeat an observation (watch same pps but tally independently)

  2) compare scores

  3) analyse using a stats test (Pearsons/ Spearmans) to look for a correlation - it is reliable if correlation is above 0.8 (0 = no correlation)
• How do we improve reliability in observations?

  - make behavioural categories clear and operationalised (not overlapping etc)

  - give observers practice time
• What is intra-observer reliability?

  within - one observer decides what to observe, records observation, observes video again and record data, looks for correlation, if above 0.8 say reliable
• What is inter-observer reliability?

  between - multiple observers agree on what to observe, observe same group at same time, independently record data, compare scores, look for correlation, if above 0.8 say reliable
• How do we assess reliability in self-report methods?

  - test retest -> give questionnaire to a group, wait and give it a second time

  - split half -> pps complete half a questionnaire/ interview, then the other half

  - inter interviewer -> interviewing the same person twice (if the outcome of both interviews is similar (correlation of above 0.8) say reliable)
• How do we improve reliability in self-report methods?

  - use a structured interview

  - carry out a pilot study

  - ensure questions are clear and not ambiguous

  - more closed questions (harder to misinterpret)

  - use the same interviewer every time

  - use properly trained interviewers
• How do we assess reliability in experiments?

  - check whether the method for measuring the DV is consistent

  - if using a scientific measure like CM/hrs/kgs, then look at how they are being measured
• How do we improve reliability in experiments?

  - if data is gathered by self-reports use the same procedure with all pps and make sure methods are consistent

  - standardised methods and procedures