Differences in Rater Behavior

1063 Words3 Pages

Differences in rater behaviors are among the factors responsible for variability in the decision making process(DMP) during ratings. The interference of either the rater rating style or rater experience determines the validity and reliability of the rating score and the rater themselves. Factors related to rater inconsistencies identification and measurement in DMP is necessary to avoid factors underlying variability in decision making process . Several studies have identified rater proficiency level, rater experiences and tasks as its factors. The purpose of this paper is to critically review two articles that contribute to describe insights of rater behavior related to the factors studied. Barkaoui (2010) ‘Variability in ESL Essay Rating Processes: The Role of the Rating Scale and Rater Experience’ identifies effects of rating scales and experiences on raters behavior through think aloud protocol. While Baker (2012)’Individual Differences in Rater Decision-Making Style: An Exploratory Mixed-Methods Study” defines and addresses decision making style (DMS) as a factor related to decision making process in writing assessments rating. The following summary and critical evaluation of these articles explicitly provide what involves in raters decisions as well as the strength and weaknesses of these two articles. Summary of text 1 Barkaoui (2010) quantitative study of variability in raters assessments discusses the impact of rating scales and rater experiences in writing rating (p.54).12 ESL essays were holistically and analytically assessed by 25 raters, novice and experienced( Barkaoui,2010,p.56). Results shows that rating scales has greater impact on rating processes, especially in analytical ratings( Barkaoui,2010,p.56). ... ... middle of paper ... ... in rater training or rater profiling as well. As a final suggestion, the two studies discussed in this paper successfully identifies additional factors to rater decision making process and will be of great benefit for those who are interested in the area of rater assessments and rating consistencies. Works Cited Baker, B. A. (2012). Individual differences in rater decision-making style: An exploratory mixed- methods study. Language Assessment Quarterly, 9(3), 225-248. Retrieved from http://search.proquest.com.ezp.lib.unimelb.edu.au/docview/1364738673?accountid=12372 Barkaoui, K. (2010). Variability in ESL essay rating processes: The role of the rating scale and rater experience. Language Assessment Quarterly, 7(1), 54-74. Retrieved from http://search.proquest.com.ezp.lib.unimelb.edu.au/docview/744444316?accountid=12372

More about Differences in Rater Behavior

Open Document