Reliability

• General Types of Reliability

  • Test-Retest
  • Parallel-Forms
  • Internal Consistency
  • Inter-Rater
• Test-Retest Reliability


  Consistency of same test over time
• If measuring trait or characteristic that doesn't change over time, test should have high test-retest reliability
• Test-Retest Concerns


  • Practice Effects - Scores change due to increased familiarity with test, not change in ability
  • Time Between Tests - Practice effects may decrease if long time passes between tests
• Tests should be assessed at different time intervals
• Parallel-Forms Reliability


  Consistency of different versions of the same test
• Parallel-Forms Reliability


  1. Large pool of test items

  2. Split them into two separate tests

  3. Administer them one after the other
• High parallel-form reliability


  Different forms of test are very similar
• Low parallel-form reliability


  Different forms are not comparable
• Parallel-Forms Concerns


  • Need different but equivalent forms of same test
  • Item pool should be more than twice size of a single test form
• Internal-Consistency Reliability


  Consistency of individual items. Do they measure the same ability or trait
• Inter-Rater Reliability


  Consistency of different observers/raters
• Inter-rater reliability coefficients are typically lower than other types of reliability estimates. However, it is possible to obtain higher levels of inter-rater reliabilities if raters are appropriately trained
• Collecting Behavioral Data


  1. Clearly define behaviors before observation begins

  2. Establish inter-observer reliability