5.1 Validity

• Criterion-related validity is a judgment of how adequately a test score can be used to infer an individual’s most probable standing on some measure of interest.
• A criterion is (1) the standard against which the test is compared; (2) a direct and independent measure of what the test is designed to predict.
• An adequate criterion is relevant for the matter at hand, valid for the purpose for which it is being used, and uncontaminated, meaning it is not part of the predictor.
• Concurrent validity: an index of the degree to which a test score is related to some criterion measure obtained at the same time (concurrently).
• Predictive validity: an index of the degree to which a test score predicts some criterion, or outcome, measure in the future.
• Incremental validity: the degree to which an additional predictor explains something about the criterion measure that is not explained by predictors already in use.
• Construct validity: is a judgment about the appropriateness of inferences drawn from test scores regarding individual standings on a variable called a construct.
• A construct is an unobservable trait that is known to exist. It is a theoretical, intangible quality in which individuals differ.
• Experimental design validity: Using experimentation to show that a test measures a concept
• Factor analysis: Statistically examining the relationship between subscales and the larger construct
• Methods to establish construct validity:

  1. experimental design
  2. factor analysis
  3. convergence with other instruments
  4. discrimination with other measures.
• Convergent evidence - scores on the test undergoing construct validation tend to correlate highly in the predicted direction with scores on older, more established, tests designed to measure the same (or a similar) construct.
• Discriminant evidence - validity coefficient showing little relationship between test scores and other variables with which scores on the test should not theoretically be correlated
• Bias: a factor inherent in a test that systematically prevents accurate, impartial measurement.
• Rating error: a judgment resulting from the intentional or unintentional misuse of a rating scale.
• Leniency Error -raters' tendency is to rate all employees at the positive end of the scale
• Severity Error – a type of rating error in which the ratings are consistently overly negative, particularly about the performance or ability of the participants.
• Central Tendency Error - a rater place most items in the middle of a rating scale.
• Halo effect - a tendency to give a particular person a higher rating than he or she objectively deserves because of a favorable overall impression.
• Fairness: The extent to which a test is used in an impartial, just, and equitable way
• Test blueprint: A plan regarding the types of information to be covered by the items, the number of items tapping each area of coverage, the organization of the items in the test, etc.
• Content validity: is determined by the degree to which the questions, tasks or items on a test are representative of the universe of behavior the test is designed to sample.
• Face Validity: a judgment concerning how relevant the test items appear to be.
• Test users may validate a test with their own group of test takers – local validation
• Validation: the process of gathering and evaluating evidence about validity.
• The validity of a test is the extent to which it measures what it claims to measure. It defines the meaning of test scores