Validity

• Selecting an outcome measure:

  • All outcome measures should be accurate and feasible
  • What does accurate mean?
  • Valid
  • Reliable
• Validity:

  • means whether a study measures what it intends to measure
• Types of Validity:

  • face validity
  • construct validity
  • content validity
  • criterion validity
• Construct validity - What is a construct?

  • An artificial framework that is not directly observable
  • Abstract ideas that explain observable behaviours
  • E.G: depression, wellness, IQ ...
• Construct validity:

  • Do scores on the test accurately reflect the “construct” being measured?
  • Is the test a consistent reflection of the underlying theory of the “construct”
• Construct Validity:

  • Can be established by reviewing the literature of the theory that pertains to the topic being researched
  • Can also be established by examining the convergent and divergent validity of a measure
• Construct Validity:

  • Convergent Validity:
  • compares the target test with other measures believed to measure the same construct
  • The results should correlate highly if the same construct is reflected in both tests
  • Divergent Validity:
  • compares the target test with other measures believed to measure different characteristics or traits
  • A low correlation is expected in this case
• Construct Validity:

  • Convergent Validity:
  • compares the target test with other measures believed to measure the same construct
  • The results should correlate highly if the same construct is reflected in both tests
  • Divergent Validity:
  • compares the target test with other measures believed to measure different characteristics or traits
  • A low correlation is expected in this case

  
• Content Validity:

  • Does the test measure a range of behaviours based on all of the theoretical concepts of the construct?
  • Typically refers to questionnaires
• Content Validity:

  • A wide range of observable, quantifiable behaviours should be contained in the measure
  • The content should logically follow on from the literature reviewed for the construct validity stage as well as personal and professional experience
  • Can be established by using expert or user review, or by generating and testing hypotheses
• Content Validity:

  • Example: Assessing ability of people with COPD to do ADLs using a questionnaire
  • Conduct a literature review to establish the ADLs that are important to people with COPD
  • Develop the questionnaire based on the literature review
  • Ask people with COPD to review the questionnaire and decide if it covers to topics that are important to them and if there’s anything missing
• Criterion Validity:

  • Criterion validity is established by comparing the new measure with an accepted gold standard of measurement a.k.a. a criterion measure
• Definitions of Types of Validity:

  • Face Validity - Does the test look as though it is measuring what it is supposed to?
  • Construct Validity - Does the test measure the theory relating to the topic under investigation?
  • Content Validity - Does the test measure a full range of behaviours expected to emerge from the theory?
  • Criterion validity - Does the test show good levels of agreement with an accepted gold standard?
• Validity at the “study level”

  • the term “validity” is also often used in the critical appraisal of whole research studies
  • In this case it means something a little different
  • When we ask “Is this study valid?” we are asking 2 overall questions, regarding internal validity and external validity
• • internal validity:
  • CAN I TRUST THESE RESULTS?
  • ARE THE METHODS ROBUST?
  • WHAT IS THE RISK OF BIAS?
  • external validity:
  • ARE THE RESULTS GENERALIZABLE ?
• Internal validity

  • Refers to how well controlled a study is
  • Internal validity is high when:
  • The risk of bias is low
  • The risk of confounding is low
  • The methods are tightly controlled
• Threats to internal validity:

  • History/Maturation
  • Regression to the mean
  • Experimenter effect
  • Participant effects
  • Unreliable/invalid measurements
  • Selection/Assignment
  • Confounding
  • Attrition/dropout
• External Validity:

  • Refers to how far the results can be generalized to wider populations
  • External validity is high when:
  • Sampling is broad and representative
  • The study conditions mimic real world condition
• Threats to external validity:

  • Non random sampling (recruitment bias)
  • Restrictive inclusion/exclusion criteria
  • Tightly controlled experimental conditions
  • Study conducted in a highly unique environment
  • Small study size