VALIDITY

• Validity


  A judgment or estimate of how well a test measures what it is supposed to measure
• Validity


  • Evidence about the appropriateness of inferences drawn from test scores
  • Degree to which the measurement procedure measures the variables to measure
  • Inferences - logical result or deduction
  • May diminish as the culture or times change
• Valid test


  • Predicts future performance
  • Measures appropriate domain
  • Measures appropriate characteristics
• Validation


  The process of gathering and evaluating evidence about validity
• Validation Studies


  Yield insights regarding a particular population of test takers as compared to the norming sample described in a test manual
• Internal Validity


  Degree of control among variables in the study (increased through random assignment)
• External Validity


  Generalizability of the research results (increased through random selection)
• Conceptual Validity


  • Focuses on individual with their unique histories and behaviors
  • Means of evaluating and integrating test data so that the clinician's conclusions make accurate statements about the examinee
• Face Validity


  A test appears to measure to the person being tested than to what the test actually measures
• Content Validity


  • Describes a judgement of how adequately a test samples behavior representative of the universe of behavior that the test was designed to sample
  • When the proportion of the material covered by the test approximates the proportion of material covered in the course
  • More logical than statistical
  • Concerned with the extent to which the test is representative of defined body of content consisting the topics and processes
  • Panel of experts can review the test items and rate them in terms of how closely they match the objective or domain specification
  • Examine if items are essential, useful and necessary
  • Construct underrepresentation: failure to capture important components of a construct
  • Construct-irrelevant variance: happens when scores are influenced by factors irrelevant to the construct
  • Lawshe: developed the formula of Content Validity Ratio
  • Zero CVR: exactly half of the experts rate the item as essential
• Criterion Validity


  • More statistical than logical
  • A judgement of how adequately a test score can be used to infer an individual's most probable standing on some measure of interest - the measure of interest being criterion
  • Criterion: standard on which a judgement or decision may be made
  • Characteristics: relevant, valid, uncontaminated
  • Criterion Contamination: occurs when the criterion measure includes aspects of performance that are not part of the job or when the measure is affected by "construct-irrelevant" (Messick, 1989) factors that are not part of the criterion construct
• Concurrent Validity


  If the test scores obtained at about the same time as the criterion measures are obtained; economically efficient
• Predictive Validity


  Measures of the relationship between test scores and a criterion measure obtained at a future time
• Incremental Validity


  The degree to which an additional predictor explains something about the criterion measure that is not explained by predictors already in use; used to improve the domain
• Construct Validity (Umbrella Validity)

  • Covers all types of validity
  • Logical and statistical
  • Judgement about the appropriateness of inferences drawn from test scores regarding individual standing on variable called construct
  • Construct: an informed, scientific idea developed or hypothesized to describe or explain behavior; unobservable, presupposed traits that may invoke to describe test behavior or criterion performance
  • One way a test developer can improve the homogeneity of a test containing dichotomous items is by eliminating items that do not show significant correlation coefficients with total test scores
  • If it is an academic test and high scorers on the entire test for some reason tended to get that particular item wrong while low scorers got it right, then the item is obviously not a good one
  • Some constructs lend themselves more readily than others to predictions of change over time
  • Method of Contrasted Groups: demonstrate that scores on the test vary in a predictable way as a function of membership in a group
  • If a test is a valid measure of a particular construct, then the scores from the group of people who does not have that construct would have different test scores than those who really possesses that construct
  • Convergent Evidence: if scores on the test undergoing construct validation tend to highly correlated with another established, validated test that measures the same construct
  • Discriminant Evidence: a validity coefficient showing little relationship between test scores and/or other variables with which scores on the test being construct-validated should not be correlated
  • Test is homogenous
  • Test score increases or decreases as a function of age, passage of time, or experimental manipulation
  • Pretest-posttest differences
  • Scores differ from groups
  • Scores correlated with scores on other test in accordance to what is predicted
• Factor Analysis


  • Designed to identify factors or specific variables that are typically attributes, characteristics, or dimensions on which people may differ
  • Developed by Charles Spearman
  • Employed as data reduction method
  • Used to study the interrelationships among set of variables
  • Identify the factor or factors in common between test scores on subscales within a particular test
  • Explanatory FA: estimating or extracting factors; deciding how many factors must be retained
  • Confirmatory FA: researchers test the degree to which a hypothetical model fits the actual data
  • Factor Loading: conveys info about the extent to which the factor determines the test score or scores
  • Can be used to obtain both convergent and discriminant validity
• Cross-Validation


  • Revalidation of the test to a criterion based on another group different from the original group form which the test was validated
  • Validity Shrinkage: decrease in validity after cross-validation
  • Co-Validation: validation of more than one test from the same group
  • Co-Norming: norming more than one test from the same group
• Bias


  • Factor inherent in a test that systematically prevents accurate, impartial measurement
  • Prejudice, preferential treatment
  • Prevention during test dev through a procedure called Estimated True Score Transformation
• Rating


  • Numerical or verbal judgement that places a person or an attribute along a continuum identified by a scale of numerical or word descriptors known as Rating Scale
  • Rating Error: intentional or unintentional misuse of the scale
  • Leniency Error: rater is lenient in scoring (Generosity Error)
  • Severity Error: rater is strict in scoring
  • Central Tendency Error: rater's rating would tend to cluster in the middle of the rating scale
  • One way to overcome rating errors is to use rankings
  • Halo Effect: tendency to give high score due to failure to discriminate among conceptually distinct and potentially independent aspects of a ratee's behavior
• Attempting to define the validity of the test will be futile if the test is NOT reliable