Plumbers Licensing -- A Case Study

Teacher & Employment Testing

An ongoing fiasco involving licensing of master plumbers in Westchester County, New York, clearly illustrates many of the flaws of the fixation on standardized, multiple-choice exams to make high-stakes decisions. The controversies over determining appropriate eligibility requirements for taking the new exam, its content coverage, "correct" answers, passing scores, and appeals procedures show how non-objective the testing process really is.


Until late 1995, plumbers in suburbs north of New York City were licensed on a municipality-by-municipality basis. A master plumber from Yonkers could not work in neighboring White Plains without passing another local test. Then, apparently motivated by a desire to enhance simplicity and efficiency, the Westchester County government voted to create a single county-wide exam which would allow plumbers to work across a broader region.


To supervise the new test, county officials appointed a 10-person Board of Plumbing Examiners, including several active plumbers. The board, in turn, hired Block & Associates, a Gainesville, Florida, consulting firm, to manufacture the exam.


Problems began almost immediately. In the first administration of the Westchester County Master Plumbers Examination, in April 1996, only 78 of the 257 candidates who took the test initially earned passing scores of 70 or higher. Yet, most of the test-takers were experienced plumbers with ten or more years of work experience.


Under scrutiny for the surprisingly low passing rate, Block & Associates discovered that a "mistake" in its computer grading program had failed to include a "rounding factor" for scores near the cut-off level. As a result, seven more test-takers were declared to have passed.


That did not satisfy leaders of the local plumbers association, who could not understand why so many of their members had failed. Their pressure forced Block to conduct a further review of the test. Block first revealed that three of the 130 items on the test had never been scored at all because of "candidate comments," then admitted that closer scrutiny revealed that seven of the 127 scored items on the exam were also flawed -- three had "ambiguous" answer choices; two had "more than one possible answer;" and two more were "miskeys" in which "the answer the computer used in grading the test question was the incorrect selection." These changes led to an additional 26 test-takers' scores moving above the passing level and receiving licenses.


Over the course of barely four months, at least three different scores were reported to each master plumber who took the test. Despite the changes, more than half the candidates were still declared to have failed the test. As this comedy of errors was unfolding, a local newspaper disclosed that the Westchester County Board of Plumbing Examiners had secretly voted to award automatic "grandfather" licenses to five of its members who were plumbers but who had not passed the test. Their rationale was that board members could not properly take an exam they had helped design. The County Attorney quickly ruled the Board's action illegal.


Outraged by these developments, the United Plumbers of Westchester filed suit against the County and its Plumbing Board in late 1996 seeking to have the test declared invalid and the licensing law an unconstitutional "deprivation of due process of law and equal protection." Several elected officials then intervened, asking the plumbers to hold off on their legal case until the county legislature could review the law.


As County Legislator Katherine Carsky noted, "It is imperative that the Board take immediate steps to fix this law before the flaws in the law create even more problems. . . It doesn't do anyone any good to continue along this road of trial and error."


The plumbers now face lengthy legal and political hurdles in order to gain the right to practice their chosen trade. Many are frustrated, noting that they already had passed tests to earn licenses in their home communities. Most held state certification as well.


Why, they wonder, was yet another test of experienced professionals needed? Whose needs were really served by the exam? Perhaps those fundamental questions will be addressed in upcoming hearings in a courtroom or legislative chamber.