Modification indices for diagnostic classification models

Abstract

Diagnostic classification models (DCMs) have emerged as promising tools for the evaluation of a student’s mastery of the essential skills in a content domain based upon their responses to a set of test items. This dissertation presents a method for identifying specific areas of misfit for a given DCM, thereby providing information about how the model could be subsequently modified so as to improve its fit to empirical data. This method is analogous to the modification indices widely used in structural equation modeling, which are based on the score test. The order constraints imposed on the item parameters in DCMs necessitate the use of a one-sided score statistic with a null distribution that is a mixture of chi-sqaured distributions. In this dissertation, modification indices are developed for use in the detection of under-specification of the Q-matrix, which specifies the skills measured by each of the test items, and for use in the detection of under-specification of the model parameters of the particular DCM chosen for the analysis of the item response data. The results of a simulation study show that the Type I error rates of modification indices for DCMs are acceptably close to the nominal significance level when the appropriate mixture chi-sqaured reference distribution is used, but that a multiplicity correction is necessary in the presence of multiple testing. The simulation results indicate that modification indices are very powerful in the detection of an under-specified Q-matrix and have ample power to detect the omission of model parameters albeit only in large samples or when the items are highly discriminating between students that have mastered or have not mastered the skills measured by the item. An application of modification indices for DCMs to an analysis of response data from a large-scale administration of a diagnostic test designed to assess teachers’ conceptual understandings of fraction arithmetic demonstrates their use in practice.