Testing Computerized Writing Exams (expanded)

K-12 Testing

Zachary VanDerwerker, son of leading Virginia test reformer Mickey VanDerwerker, conducted his own test of his school's new computerized essay-scoring system, developed by Educational Testing Service (ETS). In response to a prompt, he wrote one and a quarter pages, eight paragraphs. The computer awarded him a four, the highest score, for what Mickey termed Zachary's "advanced writing skills."


For his opening paragraph, Zachary wrote:
"I just wanted to see if this computer program worked. I just wanted to see if this computer program worked. I just wanted to see if this computer program worked. I just wanted to see if this computer program worked. I just wanted to see if this computer program worked. I just wanted to see if this computer program worked. I just wanted to see if this computer program worked. I just wanted to see if this computer program worked."
The remaining seven paragraphs of this "brilliant writing" (his proud mom's words) were identical.


To further test the program, Mickey obtained access to it and pasted in the opening paragraphs of a Stephen King story, which earned a "3." When she pasted the same paragraphs twice, it earned a "4."


The computer program gives instructions to the writers:
"Please spend a few minutes planning your essay before writing. Consider the topic carefully and plan what to say. Write carefully, so that the essay is well organized. Be sure to use specific examples to support your ideas. Your essay will be judged for its overall quality. Overall quality includes topic development, support of ideas, sentence structure, word usage, and correct spelling, punctuation, and capitalization."


Ms VanDerwerker reports that, according to ETS, a "4" on any electronically submitted essay is earned for meeting the following criteria
- Develops ideas well and uses specific, relevant details throughout the essay.
- Is well organized with transitions; maintains focus.
- Contains some varied sentence structure.
- Exhibits specific word choices.
- Contains little or no errors in grammar and conventions; errors do not interfere with understanding.


Here are the first paragraphs of the Stephen King story, "Hearts in Atlantis," which earned a 3 until Mickey entered those paragraphs twice to create a longer "essay":


Bobby Garfield's father had been one of those fellows who start losing their hair in their twenties and are completely bald by the age of forty-five or so. Randall Garfield was spared this extremity by dying of a heart attack at thirty-six. He was a real-estate agent, and breathed his last on the kitchen floor of someone else's house. The potential buyer was in the living room, trying to call an ambulance on a disconnected phone, when Bobby's dad passed away. At this time, Bobby was three. He had vague memories of a man tickling him and then kissing his cheeks and forehead. He was pretty sure that man had been his dad. SADLY MISSED, it said on Randall Garfield's gravestone, but his mom never seemed at all that sad, and as for Bobby himself...well, how could you miss a guy you could hardly remember?


Eight years after his father's death, Bobby fell violently in love with a twenty-six-inch Schwinn in the window of the Harwich Western Auto. He hinted to his mother about the Schwinn in every way he knew, and then finally pointed it out to her one night when they were walking home from the movies (the show had been "The Dark at the Top of the Stairs," which Bobby didn't understand but liked anyway, especially the part where Dorothy McGuire flopped back in the chair and showed off her long legs). As they passed the hardware store, Bobby mentioned casually that the bike in the window would sure make a great eleventh-birthday present for some lucky kid.
"Don't even think about it," she said. "I can't afford the bike for your birthday. Your father didn't exactly leave us well off, you know."


The Stephen King selection was actually in response to a prompt. Note that neither the initial 3 nor the 4 awarded for doubling the number of words was given for a writing sample that in any way responded to the prompt:


"What are the qualities of a good friend? Write an essay in which you describe what it takes to be a good friend. Identify the qualities a person must have to be a good friend, and develop those ideas with specific examples and support, citing your own experiences."


Zachary's writing sample was a response to a prompt asking him to describe his school so that people who hadn't been there would be able to "see" it.


In a listserv discussion of this issue, one post said:


Tim McGee did a session at NCTE a couple years ago that dealt with this topic. He actually typed backwards an essay that received a high score, and there was nary a difference in score. He also typed in something that "grammatically" appeared correct but made no sense. That essay earned a passing score.