Online Automated Essay Assessment:
Potentials for Writing Development

1.0              Background to the Study

Evaluating and grading student essays is often a pain to writing instructors, and giving timely pedagogic feedback is a trying challenge in many writing classrooms. Yet this diagnostic advice is necessary for student writers to learn and progress. With the advent of ICT, some writing instructors wished for an intelligent machine or software that could reduce their marking load. This wish had motivated research initiatives into automated essay assessment as early as 1966 (Whittington & Hunt, 1999; Williams, 2001).

To date, a number of such tools are being used with considerable success in education, e.g. E-rater created by Educational Testing Services and MyAccess created by Vantage Learning (Daniel and Cox, 2001). While a faction of writing practitioners is enthusiastically using these automated tools to extend their teaching capacity, another faction is doubtful about its usefulness. The debate on the value of automated essay assessment often recurs in popular journals and conferences (Hearst, 2000; Honan, 1999). The controversy points to the critical and informed adoption of an automated essay assessment tool.

At the moment, an online essay assessment tool trade-marked MyAccess is being promoted in Malaysia. This online software has built up substantial reputation in its country of creation, the USA (see http://www.vantagelearning.com). It claims high level grading accuracy that equals or exceeds human expert graders. It suggests that immediate assessment and feedback encourage learners to write and revise more frequently. A leading edge of this software that makes it shine among its competitors is its ability to assess essays in other languages in addition to English. This leading advantage seems to relate well to educational institutions that teach a number of languages.  

2.0       Identification of the Problem

A common problem among writing instructors is the difficulty in giving timely feedback to students especially those with large classes. Many instructors are reluctant to teach writing skills because of the heavy marking load. Besides writing skill courses, instructors of content courses are also reluctant to give essay or open-ended assignments. For written examinations too, under the pressure of meeting marking deadlines, instructors set more objective than open-ended questions. In the long run, students become incompetent in extended discourse and rhetoric due to insufficient practice. This is a common weakness among tertiary students or graduates that has often been highlighted in the mass media (Chapman, 2002).

In view of these problems, this research study was originally planned to explore the feasibility of using MyAccess, a web-based automated essay assessment system, to assist writing instructors in evaluating students’ written work. However during the study, it was discovered that the system also had an in-built class management tool to help a writing teacher manages a writing class such as keeping records of essays written and grades obtained, and giving comments during a student’s writing process. This asset plus the various instructional and writing support tools and resources, coupled with the immediate holistic and analytical feedback of the system,  give MyAccess the potentials in developing the writing competence of students.

3.0       MyAccess: A Brief Introduction 

To date, due to constant upgrading and updating, MyAccess has evolved into an integrated writing support environment in developing student writing competence. Vantage Learning, the creator of MyAccess, is a leading provider of online educational and assessment solutions. Its headquarter is based at the USA, and it also has offices in Taiwan, Korea, Indonesia, Thailand and Malaysia. Among the many awards won by Vantage Learning include the Codie Award Finalist 2003 & 2004, Eduventures Top 10 Breakhrough Technologies in Education, and Technology & Learning Top 10 Smart Technologies. The automated writing development environment comprises two important electronic platforms: MyAccess and IntelliMetric (see Figure 1).

MyAccess provides students of various grade levels appropriate writing tasks or prompts.    Students maintain their work in an online portfolio that contains their initial drafts, evaluation scores and subsequent revisions.  During the writing process, students have access to a variety of tools that include:

• Writer’s checklist to guide the student through the writing process.
• Scoring rubrics so the students can self-assess their writing through the process.
• My Editor to provide grammatical comments, suggestions, and explanations of rules. 
• Word counter to keep track of essay length.
• Word banks to assist in the selection of appropriate words for use in an essay of a particular genre.
• Spelling Checker to assist in the proper spelling of words used in the essay.
• Venn diagrams and other graphical writing preparation tools to assist in the formulation and organization of ideas to be included in the essay.

Teachers too have a number of tools to help them manage a writing class, for example, assigning students into various groups, and setting writing assignments. They can view any student’s portfolio, provide comments and track the overall class portfolio to manage writing instruction. Teachers can also generate letters to parents and student performance reports in a variety of formats.  In addition, the teachers have ultimate control over the tools available to the students while they write essays.  For example, if it is important that the students do not receive any help with spelling, the spell checker can be turned off for any particular assignment.

 

Figure 1: Automated Writing Development Environment

The platform that automatically scores student’s writing upon its submission to the system is trademarked IntelliMetric. It is an intelligent scoring system that emulates the process carried out by human scorers.  The system has been “trained” with various sets of previously scored responses or “known score” papers.  These papers are used as a basis for the system to infer the rubric and the pooled judgments of the human scorers. The operating principle is that IntelliMetric internalizes the characteristics of the responses associated with each score point and applies this intelligence in subsequent scoring.  The approach is consistent with typical K-12 and higher education expert writing scoring models.

4.0       Methodology

The first stage of the study was to review MyAccess and to identify potential areas where the system might help in the teaching and learning of writing skills. The study followed the single-case ethnographic research design where the instructor’s and the students’ interaction and reaction in the use of the software were observed throughout one whole semester. Thirty-four students volunteered or encouraged to participate in the exploratory study. The students wrote an in-class impromptu short essay at the beginning of the semester. This essay was used to gauge their eventual writing performance. The course instructor identified writing tasks from the pool of MyAccess writing prompts that were relevant to this group of students. Students were taught to use the various online tools through a training workshop.

During the semester, students wrote and submitted two mandatory writing assignments for the automated feedback and scoring. Besides the compulsory essays, students were encouraged to write and submit additional essays for the same online feedback and grading. When an essay was completed, the students submitted the final draft to the system to obtain a final grade. The final drafts were also printed out and submitted to the instructor for a separate evaluation.  The course instructor observed and monitored students’ reaction and progress. At the end of the semester, the instructor and the students answered a summative questionnaire. The data were then compiled and analyzed.

5.0       Discussion of Findings

5.1       Students’ Perception

After a semester of writing on the platform of MyAccess, the students were asked to write an essay describing their experience and opinions in using the system. They were also requested to answer a summative questionnaire. The following findings were extracted from their narratives and the questionnaire responses.

About 35% of the students used a computer daily. About 55% of them used a computer at least once a week, and the remaining at least once a month. 78% had experienced taking a test online, and this test was the driving or an IQ test. Slightly more than half of the students preferred writing on the computer than on paper. This information generally suggests that the students were fairly frequent computer users, and that their computer literacy is also good.

The majority of the students perceived that the website is useful as it is interactive and the feedback was almost instant and helpful. All of them commented that this was their first experience in writing online and using the accompanying writing tools, and automated essay scoring. They found the online writing resources such as the user and writer guides, writer’s model catalogue, quick reference guide, and instructional units useful. They felt that they had improved their grammar and spelling by using My Editor, and word meanings by using Thesaurus. The feedback provided by My Tutor had helped them identify their weaknesses as it explained with examples. Students were happy to know the level of their writing ability as indicated by MyAccess.  Among their positive responses on the use of the writing platform included: