Ratings of L2 oral performance in English

Abstract

Judgments of nonnative speaker (NNS) oral language performance are notoriously susceptible to rater biases. While acoustically measurable elements of pronunciation are indeed relevant to comprehensibility, variance in oral proficiency scores due to rater background and linguistic stereotypes constitute trait-irrelevant error. This study tested the supposition that raters’ background characteristics - including attitudes toward World Englishes - influence their rating of oral performances. In addition, a brief psychosocial intervention was undertaken in an attempt to mitigate the impact of potentially biasing rater characteristics. Seventy US undergraduate students rated the speaking and teaching proficiency of eleven international teaching assistants (ITAs). The ITA speech samples were acoustically analyzed on 12 measures of speech rate, pauses, stress, and intonation. Following the intial rating, one group of raters participated in a social-psychological intervention (training), in which undergraduates solved mystery puzzles with ITAs and interacted informally. Finally, 63 raters (including 29 trained) rated the same ITA speech samples a second time. All data collection from participant-raters was conducted online, including brief interviews as a qualitative supplement to the main results. Separate multiple regressions were employed to account for the variance in each dependent variable (rater judgment scores), based on linear combinations of independent variables (rater and speaker characteristics). Rasch modeling also yielded measures of rater stringency. Analysis results revealed that about 20-30 % of variance in proficiency ratings were attributable to rater background characteristics. Rater native English speaker status, amount of contact with NNSs, prior teaching experience, and negative past experience in ITA courses affected student judgments of ITAs’ accented speech. In contrast, 60-70% of the variance in ITAs’ oral performance ratings was attributable collectively to objectively measured prosodic pronunciation factors, especially acoustic fluency. The intercultural sensitization intervention mitigated the impact of rater biases on ratings of ITA instructional competence. Among recommendations for screening and training raters warranted by this study is the notion that students who feel that their class grades have been harmed by NNS instructors should be disqualified as raters in high stakes speech assessment. Conversely, NNSs who wish to improve their perceived oral proficiency should work on avoiding filled and irregularly placed pauses.