29 Dec 2020
School
Department
Course
Professor
Week 8: Assessment
Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research
(Rodriguez, 2005)
• All reductions in number of options will result in significant changes in mean item difficulty
• Item discrimination: difference in proportion of correct responses for upper scoring vs. lower
scoring 27% of students; point-biserial correlation between item and given
format test score
o Nearly all reductions in number of options will result in significant changes
• Test score reliability: reliabilities reported for each set of items in given format
o Differences in reliability varied significantly across trials
• Test score validity: format effects on validity-related interpretations of test scores where validity
evidence reported in form of concurrent criterion-related validity coefficients
o Both students resulted in statistically negligible change in criterion-related validity
evidence when reducing number of options to 3
• Option deletion method: reduction in number of options deteriorates quality of items
o NO relation with change in item difficulty or discrimination
• Item-writing rule should be revised: 3 options optimal in most settings
• Has been suggested that we use as many plausible distractors as feasible
o Yes… But in most cases, only 3 are feasible
o Using more options does little to improve item and test score statistics, typically just
results in implausible distractors
• Less time needed to prepare 2 plausible distractors, more 3-option items can be administered per
unit of time (potentially improving content coverage)
o Inclusion of more high-quality items per unit of time should improve test score reliability,
providing additional validity-related evidence regarding consistency of scores and score
meaningfulness/usability
• Threat of guessing and having greater chance of correct guess with 3 items than 5 hasn’t
prevailed since examinees unlikely to engage in blind guessing
o Rather educated guessing for eliminating distractors
Review of multiple-choice item-writing guidelines for classroom assessment (Haladyna et al., 2002)
• Content concerns:
o Base each item on important content; avoid trivial
o Use novel material to test higher level learning
▪ Paraphrase textbook/lecture language to avoid testing for pure recall
o Keep content of each item independent form content of other items
o Avoid over-specific/general content
o Avoid opinion-based and trick items
▪ If minimum 48% picked same wrong answered, considered a trick
o Keep vocabulary simple
▪ Achievement construct being measured can be affected by reading demand
• Writing the stem:
o Ensure directions very clear
o Include central idea in stem instead of choices
o Avoid window dressing (excessive verbiage)
o Word stem positive, avoid negatives such as NOT or EXCEPT
▪ If used, use cautiously and always ensure they appear capitalized and bold