4. Utility

• Utility


  Usefulness or practical value of testing to improve efficiency
• Utility can tell us something about the practical value of the information derived from scores on the test
• Utility helps us make better decisions
• Higher criterion-related validity


  Higher utility
• Cost


  Disadvantages, losses, or expenses both economic and noneconomic terms
• Benefit


  Profits, gains or advantages
• Utility Analysis


  Family of techniques that entail a cost-benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment
• Expectancy table


  Provide an indication that a test-taker will score within some interval of scores on a criterion measure – passing, acceptable, failing
• Expectancy table might indicate future behaviors, then if successful, the test is working as it should
• Cut Score


  Reference point derived as a result of a judgement and used to divide a set of data into two or more classifications
• Relative Cut Score


  Reference point based on norm-related considerations (norm-referenced), not fixed per se
• Fixed Cut Scores


  Set with reference to a judgement concerning minimum level of proficiency required; e.g., Board Exams
• Multiple Cut Scores


  Refers to the use of two or more cut scores with reference to one predictor for the purpose of categorization
• Multiple Hurdle


  Multi-stage selection process, a cut score is in place for each predictor
• Compensatory Model of Selection

  Assumption that high scores on one attribute can compensate for lower scores
• Angoff Method


  1. Setting fixed cut scores

  2. Requires expert judges to discuss the issues involved in determining a pass mark and to evaluate the examination by using a well-defined and rational procedure

  3. Low interrater reliability
• Known Groups Method


  1. Collection of data on the predictor of interest from group known to possess and not possess a trait of interest

  2. The determination of where to set cutoff score is inherently affected by the composition of contrasting groups
• IRT-Based Methods


  1. Item-Mapping Method: arrangement of items in histogram, with each column containing items with deemed to be equivalent value

  2. Bookmark Method: expert places "bookmark" between the two pages that are deemed to separate testtakers who have acquired the minimal knowledge, skills, and/or abilities from those who are not

  3. Cut scores are typically set based on testtaker's performance across all the items on the test
• Method of Predictive Yield

  Took into account the number of positions to be filled, projections regarding the likelihood of offer acceptance, and the distribution of applicant scores
• Discriminant Analysis


  1. Shed light on the relationship between identified variables and two naturally occurring groups

  2. Used by the researcher to analyze the research data when the criterion or the dependent variable is categorical and the predictor or the independent variable is interval in nature

  3. Enables the researcher to examine whether significant differences exist among the groups, in terms of the predictor variables

  4. Identify two groups of people who represent two distinct categories of some trait