New Mass. Teachers' Test Fails Professional Standards

Teacher & Employment Testing

The headlines echoed across the nation: "59 Percent of Teacher Hopefuls Flunk Mass Competency Test." Pundits and policy-makers piled on, labeling prospective classroom leaders "idiots" and calling for "abolition" of schools of education.


But a close look at the exam that produced the high failure rate shows that the results may have been significantly distorted by major substantive and administrative problems with the test. These fundamental defects may explain why those who failed include an MIT graduate and Fulbright scholar, several high-tech engineers, and dozens of experienced teachers who had passed licensing tests in other states.


Among the flaws in the Massachusetts Teacher Tests (MTT):


- Prospective teachers and their college advisers were told that simply sitting for the test at one of the initial administrations would be sufficient for certification. The official "Registration Bulletin" for the exam says, "No qualifying score will be established until after the first two administrations. . . candidates . . . eligible to participate in those first two administrations will satisfy the testing requirement by completing the tests." This means many candidates were denied their due process legal right to adequate time to prepare for a high-stakes exam. An official state "Study Guide" was withdrawn after it was found to include errors.

- The MTT included many bizarre questions unlike those on any other state's teacher licensing exams. On one, candidates were asked to transcribe a portion of The Federalist Papers as dictated from a low quality tape-recorder. Other items asked for dictionary definitions of words with questions such as "What is a preposition?" and "What is an adjective?"

- The company that produced the MTT, National Evaluation Systems (NES), has a history of making and selling defective products. Alabama dropped its NES-made state test after a lawsuit demonstrated that some evidence claimed for its accuracy was fabricated and that the passing score was set by political whim. In one case challenging that test, a Federal judge wrote, "the court concludes that the developer's procedures violated the minimum requirements for professional test development," "many items appeared on the examination even after they had been rated content invalid," and "the cut scores [passing requirements] bear no rational relationship to competence as that construct was defined by Alabama educators" (see Examiner, Spring 1987). In New York, an NES exam faces a Federal Court challenge that it is racially biased and not job related (see Examiner, Summer 1996).

- The passing levels for the MTT were set in a clearly political process in which the Mass. Board of Education rejected the test-maker's own recommendation for a lower passing level to compensate for possible measurement error in the exam.

- Massachusetts educators who were members of the official panels charged with advance review of potential MTT items say that questions they flagged as defective were used anyway.

- Many candidates were denied testing accommodations to which they were legally entitled under the Americans with Disabilities Act. When they found out -- barely a week before the test -- that cut-off scores would apply even at the first administration, students who sought extended time, readers, calculators and other assistance to which they were legally entitled were told that the "Registration Bulletin" required such requests to be made 30 days in advance. Yet, the passage in the same document stating there would be no minimum score level to pass the test was ignored.

- No technical reports on the MTT's validity, reliability or other measurement characteristics have yet been made available. Thus, test takers, policy makers and the public have to rely on the promotional claims of the exams' manufacturer and the state bureaucrats who contracted with it.


First External Study

Recently, a group of independent researchers were able to obtain sufficient data to begin to assess the MTT's quality. Drawing on test score reports from the state, colleges and test takers themselves, the study found the exam is "fundamentally flawed" because of a huge margin of error in its scores and the lack of any demonstrable relationship between test scores and initial teacher competency. As a result, the panel concluded the MTT fail to meet minimum standards of professional quality and called for suspension of its administration pending a public audit of the test and its manufacturer.


"The hallmarks of quality assessment are reliability -- the technical consistency of scores -- and validity -- the relationship between scores and the trait they claim to measure," explained Professor Walter Haney of Boston College, a nationally known measurement expert. "Based on the data we have reviewed, the Massachusetts Teacher Tests fall far short of this standard." Test results from nearly 250 prospective teachers who took the test in both April and June demonstrated huge score swings, sometimes as much as 50 points.


Since candidates only received their scores and were found out they had to retake the test a few days before the second exam, there was little chance to boost test scores by cramming. Any test that does not produce reliable results cannot be a valid measure of anything meaningful.


Next Steps

Despite the clear evidence of its flaws, state policy makers continue to defend the test. Thus, as has been the case with many other controversial standardized exams, the fate of the Massachusetts Teacher Tests will probably be decided in a courtroom. But that will take years of legal battle, during time which many competent young people will be driven out of education. Little attention has been paid to the educational consequences of that outcome, including a looming teacher shortage, particularly in urban districts.


The ongoing debate ignores another fundamental issue: there is no evidence that the MTT or any other pencil-and-paper test can fairly and accurately determine who is -- and who is not -- adequately prepared to be a competent teacher. A test score requirement for entry-level teachers is no more a "standard" for educational quality than is a pencil-and-paper exam on the Constitution a relevant way for a state to choose its Governor.