From Data Collection to Evaluation of Effectiveness: Maximizing Information from Rater-Based Assessments via Many-Facet Rasch Measurement

Description: To support the validity of score interpretation from rater-based assessments often used in certification assessment programs, analysis with the Many-Facet Rasch measurement (MFRM) model reduces the impact of rater severity and any differences in exam materials across candidates. However, to use MFRM effectively, the scoring criteria must be well defined, and the rating plan must ensure connectedness of the examinees, raters, and exam materials. Additionally, good rater training is essential to achieve quality outcomes. Use of the MFRM model provides extensive rating quality indices to evaluate case material, rater performance, and scoring criteria. In this session, learn from veteran psychometricians and board staff on how to prepare your rater-based assessment for MFRM analysis, evaluate assessment quality, and identify rater aberrations to continuously improve your exam program in successive administrations.

Citations and Literature References (if none, write n/a): Engelhard, G. (1997). Constructing rater and task banks for performance assessments. Journal of Outcome Measurement, 1(1),19–33.
Linacre J.M. (1994). Many-Facet Rasch Measurement, 2nd Ed. Chicago: MESA Press
Myford, C. M., & Wolfe, E. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of applied measurement, 4(4), 386-422.
Myford, C. M., & Wolfe, E. W. (2004). Detecting and measuring rater effects using many-facet Rasch measurement: Part II. Journal of applied measurement, 5(2), 189-227.
Schoepp, K., Danaher, M., & Kranov, A.A. (2018). An effective rubric norming process. Practical Assessment, Research & Evaluation, 23(11).