ETS Gender Bias Report a "Smokescreen"

University Testing

Call it a smokescreen , a red herring or even a straw (wo)man. Whatever the label, the new Educational Testing Service (ETS) research summary Gender and Fair Assessment is a textbook example of a lengthy rebuttal that nearly totally avoids the main point of a current controversy.


As anyone who has followed the debate about gender bias in testing knows, the central concern is that ETS exams, such as the SAT, Graduate Record Exam and Graduate Management Admissions Test, under-predict the academic performance of young women. Even more troublesome, when used with cut-off scores as in National Merit Scholarship semifinalist selection or admissions formulas, the tests unfairly deny females the share of academic opportunities that they have earned by superior classroom performance.


Instead of marshaling evidence to defend their products against these charges, ETS President Nancy Cole, senior researcher Warren Willingham, and their colleagues spend more than 400 pages analyzing several hundred tests to conclude that average differences in the performance of females and males were very small. But that is precisely the point made by FairTest and other critics: ETS exams are outliers which show significantly larger gender disparities than other assessments. Perhaps that is why only parts of two paragraphs are devoted to the use of the Preliminary SAT in the National Merit Scholarship process with neither term appearing in the extensive indexes. As American Association of University Women executive director Janice Weinman noted, Unfortunately, there is a big difference in demographics and rewards between the SAT, a key gate-keeper to college admissions and scholarships, and the Iowa Test of Basic Skills.


The new ETS book is not entirely without merit. It does summarize much of the research on gender in testing over the past decade. However, ETS sources are selective in a self-serving way. Conspicuous by its absence is Federal District Court Judge John Walker s ruling striking down New York s system of awarding college scholarships on the basis of SAT scores finding, . . . SAT scores capture a student s academic achievement no more than a student's yearbook photograph captures the full range of her experiences in high school. (see Examiner, Spring, 1989). Nor is there any reference to FairTest s landmark 1988 report Sex Bias in College Admissions Tests: Why Women Lose Out, which put the issue on the national assessment reform agenda. Other key studies by independent researchers are also ignored.


Nevertheless, the ETS publication does offer additional ammunition for the growing movement to eliminate gender bias in testing. For example, one table summarizes results from 27 different Advanced Placement subject exams which use a mix of multiple-choice and a variety of free-response items. On 24 of the tests, girls scores are relatively higher on the non-multiple-choice section. Of course, ETS refers to this evidence of format-related differences in performance as mixed.


The motivation behind this project was apparent at ETS Washington, DC, news conference held to release its conclusions. Copies of the actual book were not available for media or independent experts to review. Instead, initial reports had to be written from a brief executive summary and two-page news release which, not surprisingly, emphasized the part of the story ETS wanted told.


Through this cynical public relations scheme, the test makers may have confused the gender bias issue momentarily. But long after publicty about this book has faded, advocates of genuinely fair assessments will continue to challenge the biases and misuses of the SAT and other standardized tests.