STATS

• Reliability


  How consistently a method measures something
• When you apply the same method to the same sample under the same conditions, you should get the same results. If not, the method of measurement may be unreliable or bias may have crept into your research.
• Types of reliability


  • Test-retest
  • Interrater
  • Parallel forms
  • Internal consistency
• Test-retest reliability


  Measures the consistency of results when you repeat the same test on the same sample at a different point in time
• Many factors can influence your results at different points in time: for example, respondents might experience different moods, or external conditions might affect their ability to respond accurately.
• The smaller the difference between the two sets of results, the higher the test-retest reliability.
• Test-retest reliability


  • A test of color blindness for trainee pilot applicants
  • A questionnaire to measure the IQ of a group of participants
• Interrater reliability


  Measures the degree of agreement between different people observing or assessing the same thing
• People are subjective, so different observers' perceptions of situations and phenomena naturally differ. Reliable research aims to minimize subjectivity as much as possible so that a different researcher could replicate the same results.
• Interrater reliability


  • Observational study where a team of researchers collect data on classroom behavior
  • A team of researchers observe the progress of wound healing in patients
• Parallel forms reliability

  Measures the correlation between two equivalent versions of a test
• Parallel forms reliability


  • Different versions of a reading comprehension test
  • A set of questions to measure financial risk aversion
• Internal consistency


  Assesses the correlation between multiple items in a test that are intended to measure the same construct
• Internal consistency


  • Questionnaire to measure customer satisfaction with an online store
  • Set of statements designed to measure optimistic and pessimistic mindsets
• It's important to consider reliability when planning your research design, collecting and analyzing your data, and writing up your research.
• Methodology and relevant form of reliability

  • Measuring a property that you expect to stay the same over time - Test-retest
  • Multiple researchers making observations or ratings about the same topic - Interrater
  • Using two different tests to measure the same thing - Parallel forms
  • Using a multi-item test where all the items are intended to measure the same variable - Internal consistency
• Validity


  How accurately a method measures something
• If a method measures what it claims to measure, and the results closely correspond to real-world values, then it can be considered valid
• 4 main types of validity

  • Construct validity
  • Content validity
  • Face validity
  • Criterion validity
• Construct validity


  Does the test measure the concept that it's intended to measure?
• Content validity


  Is the test fully representative of what it aims to measure?
• Face validity


  Does the content of the test appear to be suitable to its aims?
• Criterion validity


  Do the results accurately measure the concrete outcome they are designed to measure?
• In quantitative research, you have to consider the reliability and validity of your methods and measurements
• Construct validity


  • Evaluates whether a measurement tool really represents the thing we are interested in measuring
  • It's central to establishing the overall validity of a method
  • Construct validity is about ensuring that the method of measurement matches the construct you want to measure
• Construct


  A concept or characteristic that can't be directly observed, but can be measured by observing other indicators that are associated with it
• To achieve construct validity, you have to ensure that your indicators and measurements are carefully developed based on relevant existing knowledge
• Content validity

  Assesses whether a test is representative of all aspects of the construct
• Face validity


  Considers how suitable the content of a test seems to be on the surface
• Criterion validity


  Evaluates how well a test can predict a concrete outcome, or how well the results of your test approximate the results of another test
• Criterion variable


  An established and effective measurement that is widely considered valid, sometimes referred to as a "gold standard" measurement
• Cronbach's Alpha


  A coefficient that measures the internal consistency, or reliability, of a set of survey items
• Cronbach's Alpha

  • Quantifies the level of agreement on a standardized 0 to 1 scale
  • Higher values indicate higher agreement between items
• High Cronbach's Alpha values


  Indicate that response values for each participant across a set of questions are consistent
• Cronbach's Alpha


  Helps analysts evaluate the quality of a survey or assessment instrument during the design phase before deploying it fully
• Surveys and assessment instruments frequently ask multiple questions about the same concept, characteristic, or construct</b>
• By including several items on the same aspect, the test can develop a more nuanced assessment of the phenomenon
• Correlation analysis


  A statistical technique used to study causal relationship among variables
• Regression analysis


  Used to determine the nature of relationships (impact, influence)
• In a two-variable linear regression or simple linear regression

  A positive relationship occurs when the 2 variables increase at the same time, while a negative relationship occurs when one variable increases and the other decreases or vice versa