Reliability & Validity

• what is the definition of Reliability?

  the measure of consistency
• What are the ways of assessing Reliability?

  • test-retest for experiments/questionnaires
  • inter-observer reliability for observations
• what is 'test-retest'?

  conducting the same test/questionnaire again on the same group or person.

  If the scores correlate (statistical test) then the reliability of the measuring instrument is good. If the correlation score is 0.8 then it has reliability
• what is 'inter-observer reliability'?

  • when two or more observers separately observe the behaviour in the observation and compare their results afterwards.
  • if the results have a 0.8 correlation then the results have inter-observer reliability
• In what ways can reliability be improved?

  • 1 - if the questionnaires produce low test-retest reliability then the questions on the test should be changed so they aren't ambiguous e.g. change open questions to closed questions
  • 2 - in interviews using a structured interview and the same interviewer that is trained ensures that they don't ask the participant leading questions or ambiguous questions
• In what ways can reliability be improved? Part 2

  • 3 - making sure that the behavioural categories are operationalized and don't overlap each other in an observation ensures higher reliability in the observation
  • 4 - by using standarised procedures in experiments
• define 'validity'

  the extent to which an observed effect is genuine and can be generalized beyond the research conducted e.g. to the public or a population
• what are the types of validity?

  • external validity
  • internal validity
  • ecological validity
  • temporal validity
  • population validity
  • mundane realism
• what is external validity?

  when the results can be generalized to the outside world/beyond the experiment
• what is internal validity?

  The extent to which a study accurately measures the relationship between variables without any confounding factors due to the manipulation of the independent variable
• what is ecological validity?

  The extent to which research findings can be generalized to real-world settings/every-day life
• what is Mundane Realism?

  extent to which the task in a research study mirrors the real-life experiences of participants
• what is temporal validity?

  whether the findings can hold true over time
• what is population validity?

  The extent to which findings from a study can be generalized to the larger population.
• Ways of assessing validity

  • Face validity - whether a test, scale or measure tested what it was intended to
  • concurrent validity - whether the test/scale closely matches the results of a well established test in that area of work (with a correlation of 0.8 or above)
• how do you improve the validity in experiments?

  • use a control group -> so the researcher can better asses the change in the dependent variable due to the independent variable
  • standardised procedures -> no investigator effects
  • using single-blind and double-blind procedures -> participants and researchers are unaware of the true aims and purposes, their behaviour is natural & no investigator effects
• what is a single-blind procedure?

  Participants are unaware of the true aims and purposes -> so their behaviour is natural and there are no demand characteristics
• what is a double-blind trial?

  when the researchers and the participants don't know the true aims or purposes of the study -> 3rd party is involved to conduct the investigation to avoid demand characteristics and investigator bias
• how do you improve validity in questionnaires?

  • adding a lie scale to the test -> to measure the participants consistency in answers & to control social desirability bias
  • having the participants names be anonymous -> replaced with identifier codes
• how do you improve the validity of observations?

  • using covert observations
  • making sure behaviour categories are operationalised
• what are examples of qualitative methods of research?

  • case studies
  • interviews (unstructured and open questions)
• what are strengths of qualitative methods?

  • often have more ecological validity
  • triangulation
• what is a weakness of qualitative methods?

  • researchers have to demonstrate interpretational validity to their conclusions
• what does triangulation mean in qualitative methods?

  the use multiple methods or data sources that develop a comprehensive understanding of phenomena