C6-C8

• 1. p-value can only be affected by sample size, therefore large sample data may provide small, unimportant effects; small sample data may hide large, important effects
• Non-significant results only tell us that an effect is not big enough to be found given the sample size
• Significant test statistics are based on probabilistic reasoning
• Criterion significance levels


  Probability level; willing to accept as the likelihood that the results were due to sampling error
• All-or-nothing thinking – p<.05 is merely a rule of thumb and not a threshold to decide a 1-0 situation
• We counter all-or-nothing thinking by looking at confidence intervals
• The intentions of the scientist

  Affects the conclusions from NHST before data collection
• Success


  Defined by a scientist's results being significant
• Researcher degrees of freedom


  Scientists may selectively report their results to focus on significant findings and exclude non-significant ones
• 1. p-hacking


  Practices that lead to the selective reporting of significant p-value; reporting only the one that yields significant results
• HARKing


  Presenting a hypothesis that was made after data collection as though it were made before data collection
• EMBERS (Ways to counter the pitfalls)

  • Effect sizes
  • Meta-analysis
  • Bayesian estimation
  • Registration
  • Sense
• 1. p-values can indicate how incompatible the data are with a specified statistical model (i.e., Ho). The p-value can indicate how much the data can contradict the specific, expected statistical model
• A p-value suggesting compatibility with a hypothesis does not mean the hypothesis is the sole true explanation
• Open science


  Movement; makes the process, data, and outcomes of research freely available to everyone
• Pre-registration


  Practice; making all aspects of your research process publicly available before data collection
• Registered report


  Submission; academic journal; intended research protocol
• Peer Reviewers' Openness Initiative

  Scientists; commit to the principles of open science; acting as expert reviewers
• Effect size


  An objective, usually standardized measure of magnitude of observed effect; affected by sample size but not attached to a decision rule; affects how closely sample effect size matches the population e.f.s
• Standardized effect sizes


  • The ability to compare effect sizes across different studies
• Effect size guidelines


  • Cohen's d - .2 (small); .5 (medium); .8 (large)
  • Pearson's r - .10 (small); .30 (medium); .50 (large)
  • Odds Ratio – effect size for counts (frequency); categorical variables; 2x2 con. Table
• Meta-analysis


  Uses studies to get a definitive estimate of the effect in the population
• Weighted average in meta-analysis

  Each effect size is weighted by its precision
• Bayesian statistics


  Using the data you collect to update your beliefs
• Bayes' Theorem


  Conditional probability of two events = individual probabilities & inverse conditional probability; used to update prior distribution w/ data; used to update prior belief in a hypothesis based on the observed data
• Prior probability


  Belief in a hypothesis before considering the data
• Marginal likelihood/evidence


  Probability of the observed data
• Likelihood


  Probability; observed data could be produced given the hypothesis/model
• A posterior distribution can be used to obtain a point estimate
• Power


  • The ability to detect a significant effect, when it exists
  • Ability of a test to reject a Ho correctly
  • Having a power result of 0 means you cannot find a difference or relationship between variables/means
  • 0.1-0.3 are low power values; 0.8-0.9 are high power values
• Factors affecting power


  • Size of the effect expected to be found
  • Criterion significance level (value of the significance level at which you are prepared to accept that results are probably not due to sampling error)
  • No. of participants
  • Type of statistical test used
  • Between-participants/within-participants design
  • Hypothesis is 1-tailed/2-tailed
• Within-groups variance/within-participants variability


  Variation in experimental scores among identically treated individuals within the same group who experienced the same experimental conditions
• If you did not have enough power in a study, you wouldn't have been able to find an effect
• In the case of a study having an enormous amount of participants but the effect size still being small, there can truly be no effect at all
• The more power a test has, the narrower the confidence interval
• Confidence interval


  Statistically determined interval estimate of a population parameter
• Independent samples t-test


  Compare mean scores of two different groups of people
• Paired samples t-test


  Compare means scores for the same group of people on two different conditions
• Rationale for t-tests


  • 2 samples of data are collected and the sample means calculated
  • If the samples come from the same population, their means are expected to be roughly equal
  • The difference between collected sample means and the difference between sample means we expect to obtain if there were no effect are compared (means of two conditions are compared)
• 1. value


  The higher the t-value, the more likely it is that the difference between groups is not the result of sampling error. Likelihood of having obtained the observed differences between two groups by sampling error