CH7-PSM106

• Utility

  ● In everyday language, it refers to the usefulness of something or some process.

  ● In the language of psychometrics, utility or test utility means much the same thing; it refers to how useful a test is. More specifically, it refers to the practical value of using a test to aid in decision making.

  ● In the context of testing and assessment, it is the usefulness or practical value of testing to improve efficiency.

  ● It is also used to refer to the usefulness or practical value of a training program or intervention.
• Factors That Affect a Test’s Utility

  Psychometric Soundness

  Costs

  Benefits
• Index of reliability — can tell us something about how consistently a test measures what it measures.
• ● Index of validity — can tell us something about whether a test measures what it purports to measure.
• Index of utility — can tell us something about the practical value of the information derived from scores on the test
• Costs

  ● In considerations of test utility, factors variously referred to as economic, financial, or budget-related in nature must certainly be taken into account.

  ● One of the most basic elements in any utility analysis is the financial cost of the selection device (or training program or clinical intervention) under study.

  ● In the context of test utility, costs refers to disadvantages, losses, or expenses in both economic and noneconomic terms.

  ● The term costs can be interpreted in the traditional, economic sense; that is, relating to expenditures associated with testing or not testing.
• Benefits

  ● Refers to profits, gains, or advantages that can be viewed in both economic and noneconomic terms.

  ● From an economic perspective, the cost of administering tests can be minuscule (extremely small) when compared to the economic benefits—or financial returns in dollars and cents—that a successful testing program can yield.
• ● Examples of noneconomic benefits that also has economic benefits:

  ■ an increase in the quality of workers’ performance;

  ■ an increase in the quantity of workers’ performance;

  ■ a decrease in the time needed to train workers;

  ■ a reduction in the number of accidents;

  ■ a reduction in worker turnover
• Utility Analysis

  ● A family of techniques that entail a cost–benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment
• Expectancy data - An expectancy table can provide an indication of the likelihood that a test taker will score within some interval of scores on a criterion measure—an interval that may be categorized as “passing,” “acceptable,” or “failing.”
• Taylor-Russell tables - provide an estimate of the extent to which inclusion of a particular test in the selection system will improve selection
• Test’s validity — the computed validity coefficient.
• Selection ratio — a numerical value that reflects the relationship between the number of people to be hired and the number of people available to be hired.
• Base rate — the percentage of people hired under the existing system for a particular position.
• Limitations:

  ● The relationship between the predictor (the test) and the criterion (rating of performance on the job) must be linear.

  ● There is potential difficulty of identifying a criterion score that separates “successful” from “unsuccessful” employees
• Naylor-Shine tables - entails obtaining the difference between the means of the selected and unselected groups to derive an index of what the test (or some other tool of assessment) is adding to already established procedures.
• The Brogden-Cronbach-Gleser (BCG) Formula - used to calculate the dollar amount of a utility gain (an estimate of the benefit, monetary or otherwise, of using a particular test or selection method) resulting from the use of a particular selection instrument under specified conditions.
• Productivity gain refers to an estimated increase in work output
• ● Hit — A correct classification

  ● Miss — An incorrect classification; a mistake
• ● Hit rate — The proportion of people that an assessment tool accurately identifies as possessing or exhibiting a particular trait, ability, behavior, or attribute.

  ● Miss rate — The proportion of people that an assessment tool inaccurately identifies as possessing or exhibiting a particular trait, ability, behavior, or attribute.
• ● False positive — A specific type of miss whereby an assessment tool falsely indicates that the testtaker possesses or exhibits a particular trait, ability, behavior, or attribute.

  ● False negative — A specific type of miss whereby an assessment tool falsely indicates that the testtaker does not possess or exhibit a particular trait, ability, behavior, or attribute.
• Decision Theory and Test Utility The most oft-cited application of statistical decision theory to the field of psychological testing is Cronbach and Gleser’s Psychological Tests and Personnel Decisions. Decision theory provides guidelines for setting optimal cutoff scores.
• Cronbach and Gleser (1965) presented 1. a classification of decision problems; 2. various selection strategies ranging from single-stage processes to sequential analyses; 3. a quantitative analysis of the relationship between test utility, the selection ratio, cost of the testing program, and expected value of the outcome; and 4. a recommendation that in some instances job requirements be tailored to the applicant’s ability instead of the other way around (a concept they refer to as adaptive treatment).
• Practical Considerations when Conducting Utility Analyses

  1. The pool of job applicants
  2. The complexity of the job
  3. The cut score (cutoff score) in use
• The pool of job applicants.

  If you were to read a number of articles in the utility analysis literature on personnel selection, you might come to the conclusion that there exists, “out there,” what seems to be a limitless supply of potential employees just waiting to be evaluated and possibly selected for employment.
• The complexity of the job. In general, the same sorts of approaches to utility analysis are put to work for positions that vary greatly in terms of complexity.
• Cut score is a reference point (usually numerical) derived as a result of a judgment and used to divide a set of data into two or more classifications, with some action to be taken or some inference to be made on the basis of these classifications
• Relative cut score

  • May be defined as a reference point—in a distribution of test scores used to divide a set of data into two or more classifications—that is set based on norm-related considerations rather than on the relationship of test scores to a criterion.
  • Also referred to as a norm-referenced cut score because this type of cut score is set with reference to the performance of a group (or some target segment of a group).
• Fixed cut score

  • Reference point in a distribution of test scores used to divide a set of data into two or more classifications that is typically set with reference to a judgment concerning a minimum level of proficiency required to be included in a particular classification.
  • May also be referred to as absolute cut scores
• Multiple cut scores - refers to the use of two or more cut scores with reference to one predictor for the purpose of categorizing test takers
• Multistage (or multiple hurdle) selection process

  • At every stage a cut score is in place for each predictor used. The cut score used for each predictor will be designed to ensure that each applicant possesses some minimum level of a specific attribute or skill. In this context, multiple hurdles may be thought of as one collective element of a multistage decision-making process in which the achievement of a particular cut score on one test is necessary in order to advance to the next stage of evaluation in the selection process.
• Compensatory model of selection

  • An assumption is made that high scores on one attribute can, in fact, “balance out” or compensate for low scores on another attribute.
  • According to this model, a person strong in some areas and weak in others can perform as successfully in a position as a person with moderate abilities in all areas relevant to the position in question
• Methods for Setting Cut Scores

  The Angoff Method

  The Known Groups Method

  IRT Based Method (Item-mapping Method and Bookmark Method)

  Other Methods (Method of Predictive Yield and Discriminant Analysis)
• The Angoff Method

  Devised by William Angoff (1971), this method for setting fixed cut scores can be applied to personnel selection tasks as well as to questions regarding the presence or absence of a particular trait, attribute, or ability
• The Angoff Method

  A weak point of the Angoff method is when there is low inter-rater reliability and major disagreement regarding how certain populations of test takers should respond to items
• The Known Groups Method known as the method of contrasting groups.
• The Known Groups Method

  Entails collection of data on the predictor of interest from a)_groups known to possess, and b) not to possess, a trait, attribute, or ability of interest. Based on an analysis of this data, a cut score is set on the test that best discriminates the two groups’ test performance.
• In the item response theory (IRT) framework, each item is associated with a particular level of difficulty. In order to “pass” the test, the test taker must answer items that are deemed to be above some minimum level of difficulty, which is determined by experts and serves as the cut score
• Item-mapping Method - a technique that has found application in setting cut scores for licensing examinations. - It entails the arrangement of items in a histogram, with each column in the histogram containing items deemed to be of equivalent value
• Bookmark Method

  • more typically used in academic applications.
  • This method begins with the training of experts with regard to the minimal knowledge, skills, and/or abilities that test takers should possess in order to “pass.” Subsequent to this training, the experts are given a book of items, with one item printed per page, such that items are arranged in an ascending order of difficulty.