Threats to Validity

• Two general types of research designs:

  • Internal Validity
  • External Validity
• Internal Validity - the extent to which the changes observed in the dependent variables are caused by the independent variables.
• Questions of internal validity - cannot be answered positively unless the design provides adequate control of extraneous variables.
• Internal validity is essentially a problem of control.
• Anything that contributes to the control of a research design contributes to its internal validity.
• External validity - refers to generalizability or representativeness of the findings.
• Questions addressed in external validity:

  • What groups, settings, experimental variables, and measurement variables can these findings be generalized?
• Threats to internal validity:

  • History
  • Maturation
  • Testing
  • Instrumentation
  • Regression
  • Differential selection of participants
  • Experimental mortality
• History - specific events or conditions, other than the treatment, may occur between the 1st and 2nd measurements of the participants to produce changes in the DV.
• Maturation - processes that operate within the participants simply as a function of the passage of time
• Testing - exposure to a pretest may affect participants' performance on a 2nd test, regardless of the independent variable
• Instrumentation - changes in the measuring instruments, in the scores, or in the observers used may produce changes in the obtained measures
• Regression - if groups are selected on the basis of extreme scores, statistical regression may operate to produce an effect that could be mistakenly interpreted as an experimental effect
• Differential selection of participants - important differences may exist between the groups before the independent variable is applied
• Experimental mortality - occurs when there is a differential loss of respondents from the comparison groups
• Ways to control threats to internal validity:

  • Random assignment
  • Randomized matching
  • Homogeneous selection
  • Building variables in the design
  • Analysis of covariance
  • Using participants as their own controls
• Randomized matching - match on as many variables as possible and then randomly assign one member of the pair to the independent variables, other goes to the control group
• Homogeneous selection - select samples that are as similar as possible on some extraneous variable.

  • Example: IQ, age
• Building variables in the design - include the extraneous variable as one of the independent variables examined
• Analysis of covariance - removing portion of performance that is systematically related to an extraneous variable
• Using participants as their own controls - participants are in each of the experimental conditions, one at a time.
• Types of external validity:

  • Population external validity
  • Ecological external validity
  • External validity of operations
• Population external validity - identifying the population to which results may be generalizable
• Ecological external validity - concerned with generalizing experimental effects to other environmental conditions

  • Settings
• External validity of operations - concerned with how well the operational definitions and the experimental procedures represent the constructs of interest

  • Would the same relationships be found if a different researcher used different operations in investigating the same question?
• Rothman's Causal Pies - attempt to account for the multifactorial nature of causation.