A. D. SCHENCK, E. DALY
may independently contribute to the assignment of a holistic
writing score. All independent variables had tolerance levels
above 2 and variance inflation factors (VIF) below 5, suggest-
ing that one factor was not significantly related to another (Ap-
pendix C).
Although more study is needed to confirm and increase the
predictive validity of variables used within this study, the highly
significant results suggest that empirical methods of calculating
EFL writing quality may be both a valid and reliable tool for
education. The use of empirical methods has several advantages
over traditional rubrics. One distinct advantage is that it can re-
duce subjectivity which is now associated with rubric criteria
and rater performance. Empirical methods of writing evaluation,
for example, would eliminate the influence of tacit rater biases
that linguistically discriminate against cultural or linguistic
groups (Johnson & Van Brackle, 2012).
An additional advantage of discreet empirical criteria for
evaluation is the potential for use with automatic grading tech-
nology. The use of such technology would greatly increase the
potential to provide washback to EFL students anywhere, any-
time. Students could use technology to get feedback concerning
vocabulary use, grammatical accuracy, or cohesion without the
classroom constraints now imposed by instructor-evaluated ru-
brics.
A final benefit of empirical methods is that they have the po-
tential to provide EFL teacher training. Teachers may obtain
valuable feedback concerning their own personal biases em-
ployed while assessing writing quality. To facilitate the training
process, automatic assessment technology could be used to high-
light criteria of evaluation that need to be further emphasized,
or deemphasized. Teachers could then learn to provide equal
weight to each rubric category being evaluated, regardless of
factors such as language, gender, or culture.
Conclusion
Results of this study reveal that several empirically measur-
able criteria for writing related to cohesion, content, and gram-
mar can be used to predict overall writing quality of EFL learn-
ers. While some of the criteria are more accurate predictors
than others, they all appear to synergistically influence the rat-
ings of holistic scores assessed by human raters.
Empirical evaluation of writing has several advantages over
traditional methods of evaluation. It allows for the automation
of writing assessment, which opens the door to use of the tech-
nology as a means of providing both summative and formative
writing feedback for students or teachers. Not only can students
get more constant and consistent feedback, teachers can receive
valuable pre-service or in-service training to sharpen their wri-
ting evaluation skills.
Although this study is promising, more study is needed to
confirm the validity of empirical measures, as well as to discern
additional relevant criteria for empirical writing evaluation of
EFL learners. Before such methods of assessment can be used
for any summative or formative purpose, they must be thor-
oughly compared to other forms of writing assessment and
examined by a large number of highly trained raters. In addition,
empirical methods must be tested with native and non-native
English speaking populations to ensure that such techniques are
uniformly accurate. Despite the need for further research, the
potential to provide automatic EFL writing feedback is clearly
evident, and should be further explored.
REFERENCES
American Council on the Teaching of Foreign Languages (2012).
ACTFL proficiency guidelines. URL (last checked 1 October 2012).
http://actflproficiencyguidelines2012.org/writing
Beyreli, L., & Ari, G. (2009). The use of analytic rubric in the assess-
ment of writing performance: Inter-rater concordance study. Educa-
tional Sciences: Theory & Practice, 9, 105-125.
Brown, H. D. (2004). Language assessment principles and classroom
practices. White Plains, NY: Pearson Education.
Chodorow, M., & Burstein, J. (2004). Beyond essay length: Evaluating
e-rater’s performance on TOEFL essays. Princeton, NJ: ETS.
Cope, B., Kalantzis, M., McCarthey, S., Vojak, C., & Kline, S. (2011).
Technology-mediated writing assessments: Principles and processes.
Computers & Composition, 28, 79-96.
doi:10.1016/j.compcom.2011.04.007
Educational Testing Service (2008). TOEFL IBT test: Independent wri-
ting rubrics (scoring standards). URL (last checked 1 October 2012).
http://www.ets.org/Media/Tests/TOEFL/pdf/Independent_Writing_R
ubrics_2008.pdf
Fang, Z., & Wang, Z. (2011). Beyond rubrics: Using functional lan-
guage analysis to evaluate student writing. Australian Journal of
Language and Literacy, 34, 147-165.
Graham, S., Hebert, M., & Harris, K. R. (2011). Throw ‘em out or
make ‘em better? State and district high-stakes writing assessments.
Focus on Exceptional Children, 44, 1-12.
Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English. London:
Longman.
Hasan, R. (1984). Coherence and cohesive harmony. In J. Flood (Ed.),
Understanding reading comprehension (pp. 181-219). Delaware: In-
ternational Reading Association.
Hoey, M. (1996). Patterns of lexis in text. New York, NY: Oxford Uni-
versity Press.
Hunter, K., & Docherty, P. (2011). Reducing variation in the assess-
ment of student writing. Assessment & Evaluation in Higher Educa-
tion, 36, 109-124. doi:10.1080/02602930903215842
Johansson, V. (2008). Lexical diversity and lexical density in speech
and writing: A developmental perspective. Working Papers, 53, 61-
79.
Johnson, D., & Van Brackle, L. (2012). Linguistic discrimination in
writing assessment: How raters react to African American “errors,”
ESL errors, and standard English errors on a state-mandated writing
exam. Assessing Writing, 17, 35-54. doi:10.1016/j.asw.2011.10.001
Lexile Analyzer [Computer software]. URL (last checked 25 October
2012). http://lexile.com/analyzer
Mansilla, V. B., Duraisingh, E. D., Wolfe, C. R., & Haynes, C. (2009).
Targeted assessment rubric: An empirically grounded rubric for in-
terdisciplinary writing. The Journal of Higher Education, 80, 334-
353. doi:10.1353/jhe.0.0044
Peden, B. F., & Carroll, D. W. (2008). Ways of writing: Linguistic ana-
lysis of self-assessment and traditional assignments. Teaching of Psy-
chology, 35, 313-318. doi:10.1080/00986280802374419
Peregoy, S. F., & Boyle, O. F. (2005). Reading, writing, and learning
in ESL: A resource book for K-12 teachers. New York, NY: Pearson
Education.
Schulz, M. M. (2009). Effective writing assessment and instruction for
young English language learners. Early Childhood Education Jour-
nal, 37, 57-62. doi:10.1007/s10643-009-0317-0
Sévigny, S., Savard, D., & Beaudoin, I. (2009). Comparability of writ-
ing assessment scores across languages: Searching for evidence of
valid interpretations. International Journal of Testing, 9, 134-150.
doi:10.1080/15305050902880801
Text Analyser [Computer software]. URL (last checked 25 October
2012).
http://www.usingenglish.com/resources/text-statistics.php
Weigle, S. C. (2010). Validation of automated scores of TOEFL iBT
tasks against non-test indicators of writing ability. Language Testing,
27, 335-353. doi:10.1177/0265532210364406
Copyright © 2012 SciRes.
1324