January 29, 2014
Incremental validity: how much of the test actually gives you useful information.
Partial and semi-partial correlation: take into account that the variables overlap with each other.
Part of incremental validity. Can’t count a variable more than once if it is included within another
Construct validity: combination of content validity and relationships with other instruments. Need
to look at related fields to see if your area has anything to cover uniquely. Correlations between
scores on instrument and other measures of the construct of interest. Need to understand all
areas before you can specify. Questioning of the mental processes of the examinees in
Convergent/discriminant validity: showing that if you took one test and scored high on one test
… see if you score high on another one. Or low on the other.
Need to consider correlations between
The construct and the method used. If you can test a contrast in different ways and get a similar
result or correlation.
Item difficulty: the percentage of people that get it right.. low=small percentage got it right . low
is c.. high would be a/ useful questions would be . a=2/3 b=3/3 c=1/3. Everyone gets b right so it
tells us nothing.
A b c
1 1 1 1
2 0 1 0
3 1 1 0
February 3, 2014
Item analysis: a set of procedures used to assess the functioning of an assessment instrument.
At the item level and under a classical test theory measurement model ( observed score=true +
error- trait error- systematic error) which also has to do with if the test is assessing the right
thing. Observed score must be correlated with the true score and also with error (there is a
relationship between them- no correlation would be a perfect model but there is always some form of error). True score and error must be zero (they have to be two separate entities or else
the stats won’t work and there is a relationship going on).
Dependant v. = linear combination of IV = error
There are many individual item analyses that can and are done
- Item ranges
o Check that the items fit into their expected ranges
o Scores that are outside the valid range need to be dealt with
o Imputation (use the mean of the group or use a model), randomization (randomly