Reliability and Validity

• Reliability- Whether something is consistent. In the case of a study, whether it is replicable
• Internal reliability assesses the consistency of results across items within a test.
• External reliability refers to the extent to which a measure varies from one use to another
• This can be assessed using:.

  • Inter-rater reliability
  • Test-retest reliability
• Inter-rater reliability- extent to which two or more observers are observing and recording behaviour in the same way. We can assess it by comparing the results of two researchers on the same thing and see how well they correlate. We can improve inter rater reliability with training, practice in a pilot study, better operationalisation, and ensuring that each researcher has the same ability to see the subjects.
• Test-retest reliability- Involves presenting the same participants with the same test or questionnaire on two separate occasions and seeing whether there is a positive correlation between the two. If correlation is poor then we should alter the task until it produces higher correlation.
• Validity- Whether something is true (measures what it sets out to measure).
• Internal validity- The extent to which a test is consistent within itself, such as whether the different questions (known as ‘items’) in a questionnaire are all measuring the same thing.:
• Factors that affect internal validity include (definitions found under Control of extraneous variables):

  • Participant variables (i.e: personality, demand characteristics
  • Lack of experimental control (this covers order and Investigator effects)
  • Situational variables (i.e: time of day and temperature)
  • Researcher bias (i.e: lack of objectivity)
• Types of internal validity include:

  • Content validity
  • Face validity
  • Split-half method
• Content validity- The extent to which the questions/measurements in the study measure what we think we are measuring rather than some other factor.
• Split-half method- Comparing two halves of a test, questionnaire, or interview. We can test this by having participants sit the two halves of the task separately and seeing if their scores correlate (i.e: each person scores similarly in both tasks). To improve this we can remove and/or change questions to improve correlation.
• Face validity- Simple way of assessing whether a test measures what it claims to measure which is concerned with face value – e.g. does an IQ test look like it tests intelligence.
• External validity- Whether it is possible to generalise the results beyond the experimental setting (i.e: onto the wider population).
• Types of external validity include:

  • Ecological validity
  • Mundane realism
  • Population validity
  • Temporal validity
  • Concurrent validity
• Ecological validity- The extent to which the findings of a research study are able to be generalized to real-life settings
• Mundane realism- is the task similar to those encountered in real life
• Population validity- Whether the research can be generalised to other people/populations.
• Temporal validity- Refers to how likely it is that the time period when a study was conducted has influenced the findings and whether they can be generalised to other periods in time
• Concurrent validity- Comparing a new test with another test of the same thing to see if they produce similar results. If they do then the new test has concurrent validity
• External validity is improved by replicating the study in new situations or groups, and via use of field studies and naturalistic observations.