Show simple item record

dc.contributor.authorJimenez, Albert Manuel
dc.description.abstractThe purpose of this study was to investigate the processes and procedures impacting the validation and the establishment of inter-rater reliability of observation instruments used in a teacher evaluation context. A newly-created observation instrument used in an extensive teacher evaluation program in a southeastern school district served as the object of investigation in this research. Inter-rater reliability coefficients of the instrument were assessed as part of the validation of the evaluation system. Additionally, two methods of inter-rater reliability that correct for chance agreement were examined to determine if the Gwet AC1 statistic, which is often used in a medical context but rarely in an education one, outperformed the typically provided kappa statistic. The inter-rater reliability coefficients for all videos and all items combined were in an acceptable range. This was also the case for most individual standards as well. Gwet’s AC1 statistic regularly outperformed the kappa statistic as a chance-corrected measure of inter-rater reliability. This finding held for all teachers combined, for the highest-rated teacher, and for the lowest-rated teacher, suggesting that Gwet’s AC1 statistic shows promise for future inter-rater reliability studies in a teacher evaluation context. While Gwet’s AC1 statistic outperformed kappa for the lowest-rated teacher, what was clear is the inter-rater reliability coefficients for the lowest-rated teacher suggests that consistently and accurately identifying poorly performing teachers is elusive. Additionally, this finding suggests the possibility that standards by which teachers are traditionally assessed enabling accurate identification for poor performing teachers may be underdeveloped. Further research in this area is warranted.
dc.subjectClassroom Observation Instruments
dc.subjectTeacher Evaluation
dc.subjectTeacher Evaluation System Creation
dc.subjectInter-rater Reliability
dc.subjectGwet’s AC1 Statistic & Teacher Evaluation
dc.titleAssessing the inter-rater reliability of a system-wide teacher evaluation observation instrument
dc.title.alternativemoving beyond the kappa paradox
dc.description.departmentLifelong Education, Administration, and Policy
dc.description.majorEducational Administration and Policy
dc.description.advisorSally Zepeda
dc.description.committeeSally Zepeda
dc.description.committeeNoel Gregg
dc.description.committeeAllan Cohen

Files in this item


There are no files associated with this item.

This item appears in the following Collection(s)

Show simple item record