Week 7

• Defining features of an experiment

  Manipulation: the experimenter manipulates (creates different levels of) an independent variable (IV) … and examines the effect of this manipulation on a dependent variable (DV)

  Control: The experimenter seeks to control other variables. If this is not done, changes in the DV can be explained by something other than the manipulation of the IV.
• A research study has internal validity if...

  it produces a single, unambiguous explanation for the relationship between two variables.

  A threat to internal validity is any factor that allows for an alternative explanation.”  
• Threats to internal validity

  confounding from environmental factors: when a feature of the environment (e.g., time of day, location, surroundings, etc.) differs systematically between the conditions of the experiment.

  Confounding from time-related variables:

  • History
  • Maturation
  • Instrumentation
  • Regression towards the mean
  • Order effects, practice or fatigue (“progressive error”)
  • Order effects + carry-over effects
• Confounding from time-related variables: History

  Environmental events that affect scores in one treament differently than another.

  Can be something that happens inbetween pre-test and post-test. Example - Seasonal effects on depression scores in a study looking at a new treatment for a mood disorder.

  Cultural changes also a good example
• Confounding from time-related variables: Maturation

  Changes in pps occurring over time (but aren’t responses to specific events).

  Can include age, tiredness during an experiment, boredom etc.

  Usually of particular concern for experiments spanning long time periods, particularly for child development studies (changes in abilities occur quickly, particularly in vocabularly which occur in dramatic ‘bursts’). This is often why child development studies have very narrow age criteria (and why we count age in months for toddlers!)
• Confounding from time-related variables: Instrumentation

  issues with the instrument used to measure the DV.

  E.g. same behaviour can be judged differently at different times (e.g., in observational/behavioural research).

  Similarly changes in equipment accuracy over time (e.g., Eye-tracking, your hair absorbing saline gel in EEG, etc.).
• Confounding from time-related variables: Regression towards the mean

  any situation where you select data based on an extreme value on some measure.

  Because the measure has natural variation, it almost guarantees that on a later measurement, scores will be less extreme than the first one, purely by chance.
• Confounding from time-related variables: Order effects, practice or fatigue (“progressive error”) 

  memory of task content can inflate scores on subsequent tests. Tiredness during a long test can decrease performance.

  One way to counteract is to include an additional group of subjects at the time of final testing.
• Confounding from time-related variables: Order effects, carry-over effects 

  Initial treatment ‘carries over’ to influence response to the next test, even when the treatment has been withdrawn.

  Example – cross-over studies looking at a drug vs placebo treatment. If the drug cures the condition in those receiving the treatment first, then this can underestimate its effect when averaged across both time points.
• Concurrent Conditions with Multiple Randomly-Ordered Trials (CCwiMROT)

  do not experience all the stimuli in one condition, followed by all the stimuli in another condition.

  • trials from the two conditions were interspersed with each other (order is random, with a new random order set by the computer for each participant).
  • no systematic relationship between the order in which PPs encountered the trials from each condition
  • no progressive error (practice and fatigue): distributes order effects equally among the pps across all conditions
  • protect against history or instrumentation
• What design to choose

  
• Null hypothesis Statistical Testing (NHST)

  Backbone of historical scientific research, particularly medical, psychological, and biological

  NHST IS ESSENTIAL FOR ANYONE WANTING TO UNDERSTAND PREVIOUS RESEARCH

  get a clear picture of how to interpret the findings from NHST appropriately

  • we test the probability of the data assuming the NH is true
• Constructing the null hypothesis

  Hypothesis of interest: body worn cameras reduce the use of force

  THE NULL: Cameras do not reduce the use of force

  Data use: How likely is the data observed if the NH is true

  Interpretation: if the data is sufficiently unlikely under the NH, then we can reject the NH in favour of the alternative, otherwise we fail to reject the NH
• The process

  
• Fitting a model

  
• Neyman-Pearson approach


  • we reject the Ho when it is false (true +ve)
  • we can fail to reject the Ho when it is true (true -ve)
  • Reject Ho when it is aactually true (Type I error - false +ve)
  • Fail to reject Ho when it is actually false (Type II error—false -ve).
• Rational equations