Test Utility

• Test utility


  The usefulness or practical value of testing to improve efficiency
• Factors that affect a test's utility

  • Psychometric soundness
  • Costs
  • Benefits
• Psychometric soundness


  Refers to the reliability and validity of a test
• Reliability


  Tells us something about how consistently a test measures what it measures
• Validity


  Tells us something about the practical value of the information derived from scores on the test
• Costs (in the context of test utility)

  Disadvantages, losses, or expenses in both economic and non-economic terms
• Costs related to testing


  • Purchasing a particular test
  • Purchasing a supply of blank test protocols
  • Computerized test processing, scoring, and interpretation from the test publisher or some independent services
• Other testing costs


  • Payment to professional personnel and staff associated with test administration, scoring and interpretation
  • Facility rental, mortgage, and/or other charges related to the usage of the test facility
  • Insurance, legal, accounting, licensing, and other routine costs of doing business
• Non-economic costs


  Such as "loss of confidence" or costs in terms of loss
• Benefits


  Profits, gains, or advantages, both economic and non-economic
• Benefits of testing


  • Proper recruitment of personnel
  • Increase in quality performance of workers
  • Good work environment in admission programs
• Utility analysis


  A family of techniques that entail a cost-benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment
• Methods of utility analysis

  • Expectancy data
  • Taylor-Russel table
  • Naylor-Shine tables
  • Brogden-Cronbach-Gleser formula
  • Decision theory

  
• Expectancy data


  Provides an indication of the likelihood that a test-taker will score within some interval of scores on a criterion measure
• Taylor-Russel table


  Used to calculate the utility gain resulting from the use of a particular selection instrument under specified conditions

  
• Naylor-Shine tables


  Used to calculate the dollar amount of a utility gain resulting from the use of a particular selection instrument under specified conditions
• Utility gain


  An estimate of the benefit (monetary or otherwise) of using a particular test or selection method
• Productivity gain


  An estimated increase in work output
• Decision theory


  Provides guidelines for setting optimal cutoff scores
• Factors considered in decision theory


  • Pool of job applicants
  • Complexity of the job
  • Cut scores in use
• Cut score


  A reference point derived as a result of a judgement and used to divide a set of data into two or more classifications, with some action to be taken or some inference to be made on the basis of these classifications

  
• Relative cut score


  A reference point in a distribution of test scores used to divide a set of data into two or more classifications that is set based on norm-related considerations rather than on the relationship of test scores to a criterion

  
• Fixed cut score


  Typically set with reference to a judgement concerning a minimum level of proficiency required to be included in a particular classification
• Multiple cut scores


  The use of two or more cut scores with reference to one predictor for the purpose of categorizing test takers
• Multiple hurdle


  A cut score is in place for each predictor used, and the achievement of a particular cut score on one test is necessary in order to advance to the next stage of evaluation in the selection process
• Angoff method


  Judgements of experts are averaged to yield cut scores for the test
• Known groups method


  A cut score is set on the test that best discriminates the two groups' test performance
• IRT-based methods


  Cut scores are typically set based on test takers' performance across all the items on the tests, with each item associated with a particular level of difficulty

  
• Bookmark method


  Typically used in academic applications, where the bookmark serves as the cut score
• Method of predictive yield


  A technique for setting cut scores which took into account the number of positions to be filled, projections regarding the likelihood of offer acceptance, and the distribution of applicant scores
• Discriminant analysis


  Provides insight regarding the relationship between identified variables and two naturally occurring groups