PSYC 400 Chapter 8: Assessment

1 views3 pages
29 Dec 2020
Week 8: Assessment
Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research
(Rodriguez, 2005)
All reductions in number of options will result in significant changes in mean item difficulty
Item discrimination: difference in proportion of correct responses for upper scoring vs. lower
scoring 27% of students; point-biserial correlation between item and given
format test score
o Nearly all reductions in number of options will result in significant changes
Test score reliability: reliabilities reported for each set of items in given format
o Differences in reliability varied significantly across trials
Test score validity: format effects on validity-related interpretations of test scores where validity
evidence reported in form of concurrent criterion-related validity coefficients
o Both students resulted in statistically negligible change in criterion-related validity
evidence when reducing number of options to 3
Option deletion method: reduction in number of options deteriorates quality of items
o NO relation with change in item difficulty or discrimination
Item-writing rule should be revised: 3 options optimal in most settings
Has been suggested that we use as many plausible distractors as feasible
o Yes… But in most cases, only 3 are feasible
o Using more options does little to improve item and test score statistics, typically just
results in implausible distractors
Less time needed to prepare 2 plausible distractors, more 3-option items can be administered per
unit of time (potentially improving content coverage)
o Inclusion of more high-quality items per unit of time should improve test score reliability,
providing additional validity-related evidence regarding consistency of scores and score
Threat of guessing and having greater chance of correct guess with 3 items than 5 hasn’t
prevailed since examinees unlikely to engage in blind guessing
o Rather educated guessing for eliminating distractors
Review of multiple-choice item-writing guidelines for classroom assessment (Haladyna et al., 2002)
Content concerns:
o Base each item on important content; avoid trivial
o Use novel material to test higher level learning
Paraphrase textbook/lecture language to avoid testing for pure recall
o Keep content of each item independent form content of other items
o Avoid over-specific/general content
o Avoid opinion-based and trick items
If minimum 48% picked same wrong answered, considered a trick
o Keep vocabulary simple
Achievement construct being measured can be affected by reading demand
Writing the stem:
o Ensure directions very clear
o Include central idea in stem instead of choices
o Avoid window dressing (excessive verbiage)
o Word stem positive, avoid negatives such as NOT or EXCEPT
If used, use cautiously and always ensure they appear capitalized and bold
Unlock document

This preview shows page 1 of the document.
Unlock all 3 pages and 3 million more documents.

Already have an account? Log in

Get OneClass Notes+

Unlimited access to class notes and textbook notes.

YearlyBest Value
75% OFF
$8 USD/m
$30 USD/m
You will be charged $96 USD upfront and auto renewed at the end of each cycle. You may cancel anytime under Payment Settings. For more information, see our Terms and Privacy.
Payments are encrypted using 256-bit SSL. Powered by Stripe.