Reliability and Validity

• Reliability

  Reliability is the extent to which results or procedures are consistent (you can replicate the experiment and gather similar results). It's more likely to be reliable if it has a standardised procedure
• Assessing reliability

  There are two main ways of assessing reliability;

  • inter-observer reliability
  • test re-test reliability
• Inter-observer reliability


  This is the extent to which scores are consistent between multiple observers/raters
• Assessing reliability using inter-observer reliability

  researchers must have the same training and be using the same behaviour categories

  • to asses reliability 2 psychologists, conduct the study (e.g observation/content analysis) seperatley/independently
  • they then check for agreement, if they find similar results it is reliable, if they do not then it is not reliable
  • this could be done by using a correlation, a strong positive correlation of +0.8 shows that the observers gave similar results, so it has reliability
• Improving inter-observer reliability:

  training;

  • give the observers' better training so they can easily identify the behaviour

  operationalisation;

  • make sure both observers are clear on what they are looking for
  • these could be checked using a pilot study

  view of behaviour;

  • video the behaviour and let them view the same video of the behaviour (may allow them to see it from a different perspective)
• Test retest reliability

  this is the extent to which a tool is consistent or stable over time. Giving the same results from the same test at a different occasion
• Assessing reliability using test-retest reliability

  to asses test-retest reliability a test would be given to a person on two different occasions (e.g a questionnaire/replication of experiment) and the scores are compared

  • if they find similar results, it is reliable, if they do not then it is not reliable
  • this could be done by using a correlation, a strong positive correlation of +0.8 or above shows that the test gave similar results on both occasions, so it has good test-retest reliability
• Improving test-retest reliability

  -standardisation - this means keeping things the same. This includes standardising the instructions, materials and procedure

  • identify and eliminate potential extraneous variables - using a lab experiment helps with this
• Validity

  validity is the extent to which a test measures what its supposed to be measuring (whether the IV is the only thing affecting the DV). If a study is valid and accurate then a cause and effect relationship can be established making the experiment more scientific
• Types of validity

  There are 2 main types of validity;

  • internal - inside the study
  • external - outside the study
• Internal validity

  the extent to which we can say that the test measures what it is supposed to be measuring and that the findings are due to the IV and nothing else. We can therefore establish a true cause and effect relationship
• Improving internal validity

  one factor that affects the internal validity is extraneous variables

  • to improve the internal validity extraneous variables must be minimised
  • careful planning and careful choice of research design should be used to improve internal validity (lab studies have a higher validity than field experiments)
• External validity

  this is the extent to which the findings can be generalised beyond the study itself. There are 3 types; ecological, population and temporal
• Ecological validity

  the extent to which behaviours observed and recorded in a study reflect the behaviours that actually occur in the real world
• Improving ecological validity

  -conduct experiments in more natural environments

  -use tasks that represent everyday life

  -use materials that represent everyday life
• Population validity

  this is the extent to which results can be generalised to the rest of the target population
• Improving population validity

  -conduct studies on a large sample size

  -use random/stratified sampling (or a combination of both) to ensure your sample accurately reflects the target population

  -conduct research cross-culturally and across different ages and genders
• Temporal validity

  this is the extent to which the results can be generalised to the modern day
• Improving temporal validity

  -do the test again in the modern day to see if the results are still accurate
• Assessing validity

  this can be measured in 2 ways;

  • concurrent validity
  • face validity
• Concurrent validity

  concurrent validity involves assessing how closely the scores on a test match a different measurement from the same participants

  • high concurrent validity is where there is close agreement between the data produced by the new test compared to the established test. Close agreement is indicated if the correlation between the two sets of data produced by the two tests exceeds +0.8
• Face validity

  face validity is less rigorous and involves looking at the test to see if they are genuinely measuring what it is supposed to be