2. Reliability

• Reliability


  Dependability or consistency of the instrument or scores obtained by the same person when re-examined with the same or an equivalent instrument
• Error variance


  A test score attributable to sources other than the trait or ability measured
• Potential Sources of error variance

  • Assessors
  • Measuring Instruments
  • Random errors such as luck
• Classical Test Theory


  Each test-taker has a true score on a test that would be obtained but for the action of measurement error
• Assumption 6: Testing and Assessment can be conducted in a Fair and Unbiased Manner
• Assumption 7: Testing and Assessment Benefit Society
• Reliability


  Dependability or consistency of the instrument or scores obtained by the same person when re-examined with the same test on different occasions, or with different sets of equivalent items
• A test may be reliable in one context, but unreliable in another
• Reliability Coefficient


  Index of reliability, a proportion that indicates the ratio between the true score variance on a test and the total variance
• Classical Test Theory (True Score Theory)


  Score on an ability test is presumed to reflect not only the test-taker's true score on the ability being measured but also the error
• Error


  Component of the observed test score that does not have to do with the test-taker's ability
• Errors of measurement are random
• When you average all the observed scores obtained over a period of time, then the result would be closest to the true score
• The greater number of items, the higher the reliability
• Factors that contribute to consistency

  • Stable attributes
• Factors that contribute to inconsistency


  • Characteristics of the individual, test, or situation, which have nothing to do with the attribute being measured, but still affect the scores
• Goals of Reliability


  • Estimate errors
  • Devise techniques to improve testing and reduce errors
• Variance


  Useful in describing sources of test score variability
• True Variance


  Variance from true differences
• Error Variance


  Variance from irrelevant random sources
• Measurement Error


  All of the factors associated with the process of measuring some variable, other than the variable being measured
• Positive Measurement Error


  Can increase one's score
• Negative Measurement Error


  Decrease one's score
• Sources of Error Variance

  • Item Sampling/Content Sampling
  • Test Administration
  • Test Scoring and Interpretation
• Random Error


  Source of error in measuring a targeted variable caused by unpredictable fluctuations and inconsistencies of other variables in measurement process (e.g., noise, temperature, weather)
• Systematic Error


  Source of error in a measuring a variable that is typically constant or proportionate to what is presumed to be the true values of the variable being measured
• Reliability refers to the proportion of total variance attributed to true variance
• The greater the proportion of the total variance attributed to true variance, the more reliable the test
• Error variance may increase or decrease a test score by varying amounts, consistency of test score, and thus, the reliability can be affected
• Test-Retest Reliability


  An estimate of reliability obtained by correlating pairs of scores from the same people on two different administrations of the test
• Carryover Effects


  Happened when the test-retest interval is short, wherein the second test is influenced by the first test because they remember or practiced the previous test = inflated correlation/overestimation of reliability
• Practice Effect


  Scores on the second session are higher due to their experience of the first session of testing
• Test Sophistication


  Items are remembered by the test takers especially the difficult ones/items that we got highlight confused
• Test Wiseness


  Might inflate the abilities of test takers
• Mortality


  Problems in absences in second session (just remove the first tests of the absents)
• Intelligence, motor test, traits, reaction time (stable traits)
• Coefficient of Stability


  Statistical tool: Pearson R, Spearman Rho
• Parallel Forms Reliability


  Established when at least two different versions of the test yield almost the same scores
• Parallel Forms


  Each form of the test, the means, and the error variances, are EQUAL; same items, different positionings/numberings
• Alternate Forms


  Simply different version of a test that has been constructed so as to be parallel