Psych assessment

• Utility


  The usefulness or practical value of testing to improve efficiency
• Utility


  The usefulness or practical value of a training program or intervention
• Judgments concerning the utility of a test are made on the basis of test reliability and validity data as well as on other data
• Psychometric Soundness


  • A test is said to be psychometrically sound for a particular purpose if reliability and validity coefficients are acceptably high
  • An index of reliability can tell us something about how consistently a test measures what it measures
  • An index of validity can tell us something about whether a test measures what it purports to measure
  • Index of utility can tell us something about the practical value of the information derived from scores on the test
• Costs


  Disadvantages, losses, or expenses in both economic and non-economic terms
• Costs of testing


  • Financial cost of the selection device (training program) under study
  • Payment to professional personnel and staff associated with the test
  • Rental, mortgage, other charges related to the test facility
  • Insurance, legal, accounting, licensing
• Benefits


  Profits, gains, or advantages
• Noneconomic benefits of testing in industrial settings

  • Increase in the quality of workers' performance
  • Increase in the quantity of workers' performance
  • Decrease in the time needed to train workers
  • Reduction in the number of accidents
  • Reduction in worker turnover
• Utility Analysis


  A cost–benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment
• Purposes of Utility Analysis for Tests

  • Preferability of one test over another for a specific purpose
  • Preferability of one assessment tool over another (e.g., test vs. behavioral observation)
  • Addition of tests or assessment tools to those already in use
  • Preferability of no testing or assessment at all
• Purposes of Utility Analysis for Training Programs or Interventions

  • Preferability of one training program over another
  • Preferability of one intervention method over another
  • Improvement of a training program by adding or subtracting elements
  • Improvement of an intervention method by adding or subtracting elements
  • Preferability of no training program at all
  • Preferability of no intervention at all
• Expectancy Data Approach


  Converts a scatterplot of test data into an expectancy table to indicate the likelihood that a test-taker will score within a certain range on a criterion measure
• Hits and Misses Table

  Hits: A correct classification

  Miss: An incorrect classification, a mistake

  Hit rate: Proportion of people that an assessment tool accurately identifies as possessing/exhibiting a particular trait

  Miss rate: Proportion of people that an assessment inaccurately identifies particular trait

  False positive: A specific type of miss whereby an assessment tool falsely indicates that the test-taker possesses or exhibits a particular trait

  False negative: A specific type of miss whereby an assessment tool falsely indicates that the test-taker does not possess or exhibit a particular trait
• Taylor-Russell Tables


  Provides an estimate of the extent to which inclusion of a particular test in the selection system will improve selection

  Computed validity coefficient - The value assigned for the test's validity

  Selection ratio - is a numerical value that reflects the relationship between the number of people to be hired and the number of people available to be hired

  Base rate - refers to the percentage of people hired under the existing system for a particular position
• Naylor-Shine tables


  An alternative table that is likely to have an average increase in criterion performance as a result of using a particular test or intervention
• Brogden-Cronbach-Gleser Formula


  Used to calculate the dollar amount of a utility gain resulting from the use of a particular selection instrument under specified conditions

  Utility gain - refers to an estimate of the benefit (monetary or otherwise) of using a particular test or selection method

  N represents the number of applicants selected per year

  T represents the average length of time in the position (or tenure)

  rxy represents the (criterion-related) validity coefficient for the given predictor and criterion

  SDy represents the standard deviation of performance (in dollars) of employees

  Zm represents the mean (standardized) score on the test for selected applicants

  The second part of the formula represents the cost of testing, which takes into consideration the number of applicants (N) multiplied by the cost of the test for each applicant (C)
• Decision theory and Test utility


  • Classification of Decision Problems: They categorized decision problems and proposed various selection strategies, including single-stage processes and sequential analyses

  Quantitative Analysis: They conducted a quantitative analysis to explore the relationship between test utility, selection ratio, cost of testing programs, and expected value of outcomes

  Adaptive Treatment: They recommended tailoring job requirements to the applicant's ability, rather than vice versa, which they termed adaptive treatment
• Practical Considerations in Utility Analysis

  • Base Rates: Extreme base rates, whether very low or high, can impact the accuracy of decisions based on tests

  Pool of Job Applicants: The assumption of a limitless pool of job applicants might not hold true for all positions

  Acceptance of Job Offers: Not all selected candidates may accept job offers, especially top performers who might have multiple offers

  The Complexity of Jobs: Jobs vary greatly in complexity, and utility analysis methods may need to be adapted accordingly

  The Cut Scores in use: Different types of cut scores, such as relative and fixed, are used in decision-making processes

  Compensatory Models of Selection: Compensatory selection models assume that high scores in one area can compensate for low scores in another
• Angoff Method


  Experts in the area provide estimates regarding how test takers who have at least minimal competence for the position should answer test items correctly
• Known Groups Method


  The method of contrasting groups, entails collection of data of interest from groups known to possess, and not to possess, a trait, or ability of interest
• IRT-Based Methods


  Cut scores are typically set based in test takers performance across all the items on the test

  Some portion of the total number of items on the test must be correct in order for the test takers to pass the test

  Item-mapping method: Entails the arrangement of items in a histogram, with each column in the histogram containing items deemed to be of equivalent value

  Bookmark method: Use of this method begins with the training of experts with regard to the minimal knowledge, skills, and/or abilities that test takers should possess in order to pass
• Method of Predictive Yield

  A technique for setting cut scores which took into account the number of positions to be filled, projections regarding the likelihood of offer acceptance, and the distribution of applicant scores
• Discriminant Analysis


  An approach to setting cut scores employs a family of statistical techniques to shed light on the relationship between identified variables (such as scores on a battery of tests) and two (and in some cases more) naturally occurring groups
• Test Development


  An umbrella term for all that goes into the process of creating a test
• Stages of Test Development

  • Test Conceptualization

  Test Construction

  Test Tryout

  Item Analysis

  Test Revision
• Test Construction


  Involves writing, revising, and formatting test items, setting scoring rules, and overall designing and building the test
• Test Tryout


  It is administered to a representative sample of test-takers under conditions that mimic the final test administration
• Item Analysis


  Used to determine which test items are effective, which need revision, and which should be discarded
• Test Revision


  Format for the purpose of improving the test's effectiveness as a tool of measurement
• Preliminary Questions for Test Development

  • Purpose and Measurement: What is the test designed to measure?

  Objective: What is the objective of the test?

  Need: How does it compare to existing tests in terms of reliability, validity, comprehensiveness, and administration time?

  User and Purpose: Who will use the test & what is the purpose?

  Test Taker: Who is the target audience of the test?

  Content: What content will the test cover and why?

  Administration: How will the test must be administered?

  Format: What is the ideal format?

  Alternate Forms: Should alternate or parallel forms of the test be developed based on a cost-benefit analysis?

  Training & Interpreting: What background and qualifications will a prospective user of data
• Norm-referenced


  Comparisons typically are insufficient and inappropriate when knowledge of mastery is what the test user requires
• Criterion-referenced testing


  Commonly employed in licensing contexts, be it a license to practice medicine

  Employed in educational contexts in which mastery of particular material must be demonstrated before the student moves on to advanced material
• Development of Criterion-referenced Instruments

  Derives form a conceptualization of the knowledge or skills to be mastered
• Development of criterion-referenced test

  Procedure may entail exploratory work with at least two groups of test takers: one group known to have mastered the material and another group known not to have mastered the material
• Norm-referenced comparisons


  Typically insufficient and inappropriate when knowledge of mastery is what the test user requires
• Development of criterion-referenced instruments

  1. Derives form a conceptualization of the knowledge or skills to be mastered

  2. Procedure may entail exploratory work with at least two groups of test takers: one group known to have mastered the knowledge or skill being measured and another group known not to have mastered such knowledge

  3. Pilot work to evaluate test items
• Test construction


  1. Scaling to assign numbers to responses so that a test score can be calculated

  2. Rating scale to record judgements

  3. Summative scale to obtain final test score by summing ratings

  4. Likert scale to scale attitudes

  5. Unidimensional vs multidimensional scales

  6. Comparative scaling, categorical scaling, Guttman scaling, scalogram analysis
• Item pool


  Reservoir or well from which items will or will not be drawn for the final version of the test
• Comprehensive sampling


  Provides a basis for content validity of the final version of the test
• Item formats


  • Selected response format
  • Constructed-response format
  • Multiple-choice
  • Matching
  • Binary choice
  • Completion
  • Essay