reliability

• Reliability

  • Reliability is a measure of whether something stays the same, i.e is consistent
  • The results of psychological investigations are said to be reliable if they are similar each time, they are carried out using the same design, procedure, and measurements
  • Reliability can be split into two main branches: internal and external reliability
• Internal reliability

  • Internal reliability describes the internal consistency of a measure (i.e. consistency within itself), such as whether the different questions(known as ‘items’)in a questionnaire are all measuring the same thing
  • One way to assess this is by using the split half method, where data collected is split randomly in half and compared, to see if results taken from each part of the measure are similar
  • It therefore follows that reliability can be improved if items that produce similar results are used
• External reliability

  • This assesses consistency when different measures of the same thing are compared, i.e. does one measure match up against other measures?
  • Discrepancies will consequently lower inter-observer reliability, e.g. results could change if one researcher conducts an interview differently to another
  • Such reliability issues can be improved by standardising procedures (i.e. making sure that procedures are carried out the same way each time), for instance by implementing interviewer training, and/or practice through pilot studies
• What is one method of assessing reliability in research?

  Test-retest
• What does test-retest assess in a research tool?

  It assesses the external reliability
• How is the test-retest method conducted?

  By presenting the same participants with the same test on two separate occasions
• What correlation coefficient indicates that a test is reliable in the test-retest method?

  At least a 0.8 correlation
• What should happen to the results if the test or questionnaire is reliable?

  The results should be the same or very similar each time they are administered
• What is the ideal time gap between the initial test and retest?

  Enough time so participants cannot recall their answers, but not so long that their opinions change
• Why is it important to have a sufficient time gap between tests in the test-retest method?

  To prevent participants from recalling their answers
• What could happen if the time gap is too long in the test-retest method?

  The participant's opinions or attitudes may have changed
• Ways of assessing reliability : inter-observer reliability

  • It is very important to establish inter-observer reliability when conducting observational research

  •it refers to the extent to which two or more observers are observing and recording behaviour in the same way

  • This may involve a pilot study (small-scale trial run) of the observation to check that observers are applying behavioural categories in the same way
• What method is used to measure the reliability of questionnaires over time?

  Test-retest method
• How is the reliability of questionnaires assessed using the test-retest method?

  By correlating two sets of data to see if they match
• Which statistical test is mentioned for measuring correlation in questionnaires?

  Spearman's rho
• What must the correlation coefficient be for data to be considered reliable?

  +0.80 or above
• What action might be taken if a questionnaire produces low test-retest reliability?

  Edit the questionnaire
• What is one suggested solution to improve reliability in questionnaires with low test-retest reliability?

  Replace open questions with closed questions
• What are the steps involved in assessing the reliability of questionnaires using the test-retest method?

  • Administer the questionnaire twice to the same participants
  • Collect two sets of data
  • Correlate the data using a statistical test (e.g., Spearman's rho)
  • Calculate the correlation coefficient
  • Determine if the coefficient is +0.80 or above for reliability
• Improving reliability: interviews

  • For interview, the best way to ensure reliability is to use the same interviewer each time
  • If this is not possible, interviewers should be trained, so one interviewer is not asking more leading or ambiguous questions for example
  • This is more easily avoided in structured interviewed where the interview is more controlled through the use of fixed questions (as opposed to unstructured interviews which are more flexible and free-flowing)
• Improving reliability: experiments

  • Lab experiments are the method most often considered to be reliable due to the high degree of control over many aspects of the procedures, such as the instructions that participants are receive and conditions within which they are tested
  • However one thing that might affect the reliability is if participants were tested under slightly different conditions each time
• What is one way to improve reliability in observations?

  By properly operationalising behavioural categories
• What does operationalising an abstract concept involve?

  Turning it into measurable behaviours
• How can social anxiety be operationalised in observations?

  By measuring behavioural avoidance of crowded places
• Why should behavioural categories not overlap?

  To ensure clear and consistent observations
• What should be included in a checklist for observations?

  All possible behaviours
• What happens if categories are not operationalised well?

  Observers may make differing and inconsistent records
• What is the consequence of having overlapping or absent categories in observations?

  It leads to inconsistent judgments by different observers