PA LESSON 4

• Test development


  An umbrella term for all that goes into the process of creating a test
• Test conceptualization


  1. Brainstorming of ideas about what kind of test a developer wants to publish

  2. Determining construct, goal, user, taker, administration, format, response, benefits, costs, interpretation

  3. Determining whether the test would be norm-referenced or criterion-referenced

  4. Pilot work/pilot study/pilot research - preliminary research surrounding the creation of a prototype of the test

  5. Attempts to determine how best to measure a targeted construct

  6. Entail lit reviews and experimentation, creation, revision, and deletion of preliminary items
• Test construction


  1. Writing test items, revisions, formatting, setting scoring rules

  2. It is not good to create an item that contains numerous ideas

  3. The test administered may be different for each testtaker, depending on the test performance on the items presented
• Item pool


  Reservoir or well from which the items will or will not be drawn for the final version of the test
• Item banks


  Relatively large and easily accessible collection of test questions
• Computerized adaptive testing


  Interactive, computer administered test-taking process wherein items presented to the testtaker are based on the testtaker's performance on previous items
• Floor effects


  There is some lower limit on a survey or questionnaire and a large percentage of respondents score near this lower limit (testtakers have low scores)
• Ceiling effects


  There is some upper limit on a survey or questionnaire and a large percentage of respondents score near this upper limit (testtakers have high scores)
• Item branching


  Ability of the computer to tailor the content and order of presentation of items on the basis of responses to previous items
• Item format


  Form, plan, structure, arrangement, and layout of individual test items
• Dichotomous format


  Offers 2 alternatives for each item
• Polychotomous format


  Each item has more than 2 alternatives
• Category format


  Format where respondents are asked to rate a construct
• Category format


  • Checklist
  • Guttman scale
• Selected response format


  Require testtakers to select response form a set of alternative responses
• Multiple choice


  Has three elements: stem (question), correct option, distractors (several incorrect alternatives also called foils)
• Effective distractors


  A distractor that was chosen equally by both high and low performing groups that enhances the consistency of test results
• Ineffective distractors


  May hurt the reliability of the test because they are time consuming to read and can limit the no. of good items
• Cute distractors


  Less likely to be chosen, may affect the reliability of the test because the testtakers may guess from the remaining options
• Matching item


  Test taker is presented with 2 columns: premises and responses
• Binary choice


  Usually takes the form of a sentence that requires the testtaker to indicate whether the statement is or is not a fact (50%)
• Constructed-response format


  Requires testtakers to supply or to create the correct answer, not merely selecting
• Constructed-response format


  • Completion item
  • Short-answer
  • Essay
• Scaling


  Process of setting rules for assigning numbers in measurement
• Nominal scale


  Involve classification or categorization based on one or more distinguishing characteristics, label and categorize observations but do not make any quantitative distinctions between observations, mode
• Ordinal scale


  Rank ordering on some characteristics is also permissible, median
• Ratio scale


  Contains equal intervals, has no absolute zero point (even negative values have interpretation to it)
• Interval scale


  Has true zero point (if the score is zero, it means none/null), easiest to manipulate
• Paired comparison


  Produces ordinal data by presenting with pairs of 2 stimuli which they are asked to compare, respondent is presented with 2 objects at a time and asked to select one object according to some criterion
• Rank order


  Respondents are presented with several items simultaneously and asked to rank them in order or priority
• Constant sum


  Respondents are asked to allocate a constant sum of units, such as points, among set of stimulus objects with respect to some criterion
• 1. Sort technique


  Sort object based on similarity with respect to some criterion
• Continuous rating


  Rate the objects by placing a mark at the appropriate position on a continuous line that runs from one extreme of the criterion variable to the other
• Itemized rating


  Having numbers or brief descriptions associated with each category
• Likert scale


  Indicate their own attitudes by checking how strongly they agree or disagree with carefully worded statements that range from very positive to very negative towards attitudinal object, principle of measuring attitudes by asking people to respond to a series of statements about a topic, in terms of the extent to which they agree with them
• Visual analogue scale


  A 100 – mm line that allows subjects to express the magnitude of an experience or belief
• Semantic differential scale


  Derive respondent's attitude towards the given object by asking him to select an appropriate position on a scale between 2 bipolar opposites
• Staple scale


  Developed to measure the direction and intensity of an attitude simultaneously
• Summative scale


  Final score is obtained by summing the ratings across all the items
• Thurstone scale


  Involves the collection of a variety of different statements about a phenomenon which are ranked by an expert panel in order to develop the questionnaire