A Draft Alternative Approach to Accountability

 

Introduction:

At its best, the idea of adequate yearly progress (AYP) is an effort to prod schools, districts and states to identify and solve two kinds of problems: schools in which most students perform well but some students do not, particularly low-income or minority-group students; and schools in which most students do not meet high standards (using standardized tests or other measures, such as their school course work, or college entry or success).

Some argue that even most students in "good" schools are actually not doing well, as seen for example in the low percentages of students reaching proficient on NAEP exams. However, NAEP linking studies to Third International Math and Science Study (TIMSS), performance of suburban schools on the second administration of TIMSS, and the success of students from "good schools" in colleges which attract fine students from around the world, all suggest this supposed problem is more a fabrication of those who seek to attack public education than it is an accurate analysis. That is, the NAEP levels are, as many researchers have found, inaccurate. Inappropriately difficult tests (tests made hard by obscure and tricky items rather than by requiring important knowledge and skills) contribute in some states to this misperception.

FairTest believes most all schools can and should improve on both academic and non-academic dimensions, and some schools need to improve a great deal for some or most all of their students. We do not think either academic achievement or the all-around ways in which schools should help children develop into adults can be reduced to test scores or will be met well in schools which overemphasize tests. If schools can improve but testing either will not help or will actively harm, what can assessment and accountability contribute to the creation of virtuous spirals in schools which continuously improve themselves as educational and social institutions? Here we will outline an answer that we hope you will consider promoting as legislation. We provide links to two more detailed versions of the basic proposal outlined here.

 

An alternative approach to evaluating school progress

Schools have dual a primary tasks of ensuring academic learning and helping students become adults capable of fully participating in all aspects of society. Evaluating school success and progress therefore requires information on all relevant aspects of schooling, not only academic success.

Contextual framework:

From a review of school improvement evidence, FairTest concludes that:
- some schools have low standards for some or all students, and that such students typically are low-income or minority-group, or have limited English proficiency or special needs, so that too many of these students are held to low expectations or "fall through the cracks";
- many schools can do better than they now do with the resources they have, but many (particularly those serving large numbers of low-income students) cannot do much better without significant new resources;
- schools cannot by themselves solve the problems and consequences of poverty, such as high mobility rates due to lack of affordable housing, inadequate nutrition, or medical and dental needs, but significant and sustained educational improvement demands that the nation address poverty;
- many schools are viewed by their students as inhospitable and alienating and the students feel disempowered and even abused, while research suggests that students are both happier and achieve more learning in environments that are hospitable and welcoming and where students feel empowered, challenged, motivated and supported;
- too few schools systematically evaluate their climate and culture or attempt to improve by using the results of such information;
- standardized assessment data helps identify some educational problems, but we do not think any existing such assessments adequately assess to high-quality standards and or provide sufficient information for educators to use to more effectively teach all their students well, and such standardized tests are not likely to be developed due to technological limitations of the standardized testing paradigm;
- in particular, standardized tests are of little use for serious academic improvement efforts because the amount of information is simply too limited to enable teachers and schools to determine what to do and how to do it;
- for assessment to best help students, it must be comprehensive and regular enough to provide fine-grained useful information about each student to guide further instruction;
- most assessment therefore must be classroom-based and used by well-prepared teachers;
- such classroom-based information should be the basis for school-improvement efforts as it can provide sufficient data;
- schools must therefore be structured to acquire and make effective use of such information;
- extensive professional development in content, pedagogy and assessment is necessary so that all teachers can successfully teach all their students and participate in school improvement efforts; and
- public information on learning and progress is important, but if it is to be effective it cannot be reduced or limited to test results or test results plus graduation/retention data, but must include richer information about student and school progress.

The test-based model for school "accountability," including "adequate yearly progress," that is embodied in the current reauthorization of ESEA and that dominates most state and district thinking, is inadequate. It cannot meet the actual needs schools have if they are to engage in continuous improvement because the information is too little. Partly this is because too little is measured, and partly because the results are typically reduced to numbers which are then often used without recourse to the information that the numbers summarize. The resulting tendency from overemphasizing too little information presented almost entirely in numerical form is to focus schools only on factors that induce short term changes in the limited data that is gathered, undermining other important learning and ignoring long-term consequences. At the extreme, this means narrow teaching to the test in ways that do improve results even on other standardized tests in the same subjects, not teaching other subjects at all, and ignoring other aspects of school climate and culture that contribute to the overall goals of schools. Even when the extremes are avoided, schools end up with too little data to be of substantive use for school improvement.

An alternative model:

FairTest has worked with the Massachusetts Coalition for Authentic Reform in Education to develop an authentic assessment and accountability program. We also have collaborated with a coalition of organizations in Chicago to develop a similar plan. We enclose descriptions of each plan. Below, we outline key elements of these proposals and then suggest how the federal government could modify the proposed ESEA accountability provisions to meet this model.

Components of the model.

1) Classroom-based information. Evidence of teaching and learning is retained by each teacher showing assignments, student work, and the teacher's observations of the learning processes, strengths and weaknesses of the students. Teachers would be expected to use a variety of appropriate assessment methods and tools, including observations, student classwork, homework, projects, essays, tests, presentations in a variety of formats, and portfolios to summarize and evaluate the evidence of learning. These are the basis for teacher reports, qualitative and numerical, as to each student's progress in meeting state, district and school standards and goals as well as goals the teacher or student may have. The summary information about each student should be supported by the classroom evidence. [This approach is found in the Learning Record http://www.learningrecord.org and the Work Sampling System http://www.rebusinc.com.]

2) Limited standardized testing. Testing should be in literacy and numeracy only and provide one means of checking on school level information. For accountability purposes, sampling could be used. Marked discrepancies between test results and school's classroom-based information should be investigated. No major decisions about students or schools should be based on test scores alone.

3) School quality reviews. Independent, well-prepared teams periodically conduct reviews of every school. Each school first prepares a self-evaluation. Then the team visits, sitting in on classrooms, shadowing students, interviewing students and educators and parents, reviewing random samples of student work, considering academic and other aspects of school life. The team then prepares a report with recommendations, which is given to the school and is available to the public in summary and complete form. Schools respond to the review and use the review in planning improvement.

Having the data does not mean schools are able to use it. Schools set up systems to summarize this information. More importantly, they develop means through which teachers look at each other's work and student work as a means of learning how to improve curriculum and teaching.

Each school prepares an annual public report. The report summarizes information from the assessments and evaluations, and then describes how the school uses information to improve, changes in processes and results over the years, and steps to be taken to improve processes and outcomes that need improvement. It would use the variety of kinds of evidence, quantitative and qualitative, to "paint a picture" of the school.

The information on learning outcomes would provide disaggregated data by race, gender, income (poverty), English proficiency and disability. The data would be reported when the sample sizes are large enough to be meaningful and when individuals are not identifiable. School reports would have to discuss the progress of the groups for whom data is disaggregated.

School reports would also include data on grade retention, suspensions, graduation/drop out, and school climate (e.g., through surveys). Over time, schools would gather information about their graduates, from how well elementary students succeeded in high school to whether high school graduates attended college and successfully graduated to employment or civic participation or satisfaction data. The goal for school, after all, is not only academic success but life success.

The plans call for intervention in schools which clearly demonstrate they are not succeeding, according to the multiple measures, for all or for significant portions of their students. The first step is an investigation. Indeed, the Quality Review process should first include schools where current information suggests low levels of performance. A review must include: factors beyond a school's control, the resources a school has, what it does with its resources, and how it might use resources better.

The investigation and support should lead to carefully targeted and planned assistance. Where a school does not make progress even with assistance, stronger interventions should take place. However, interventions such as school reconstitution or state takeovers have been tried with at best limited success (Mary Levy of Lawyers Committee did a study for DC Voice; most efforts at reconstitution failed). Too little is known about how to make such strong interventions succeed. Therefore, states should develop and implement the stronger actions with caution and keep very close track of what does and does not work so as to learn from them.

[The examples are the Massachusetts Coalition for Authentic Reform in Education (CARE) Call for an Authentic Assessment System, on the web at http://www.fairtest.org/arn/masspage.html in short and longer versions; and the New ERA plan from a coalition of Chicago groups, on the web at http://www.pureparents.org./ERAplanAug00email.htm .]

Federal implications.

The conference committee on ESEA should rewrite the legislation to adopt the provisions described here. That is, states would be required to implement comprehensive assessment, accountability and assistance plans. Schools and districts would develop plans based on the outline above. Two cautions:
- This will take substantial time to implement. The key is further education of educators and the time and resources to do it. By the end of this authorization, schools, districts and states should have the basic structures in place and operating: quality reviews, school improvement plans and processes, and a range of data emanating from classrooms that can be used to understand school progress. Thus, in most cases interventions based on this approach are not likely during this authorization.
- It will not be cheap. While all the elements of this plan exist somewhere, we have not seen any summary research on likely costs. Nonetheless, based on informal assessments of costs, we believe the cost is feasible. More importantly, it would enable a means to develop accountability that will support development of high-quality schools with strong outcomes, not just gains on test scores. The largest costs will be professional development and paying teachers for the time to engage in the sharing and reflective processes required for genuine school improvement.

Some questions and answers about the plan:

Q: Shouldn't the current test-based accountability processes be used to identify and act on schools?

A: Only very cautiously. States can use current data to determine which schools should first undergo Quality Reviews, and it can use current data to identify which schools most need additional assistance (though this is more mystification than mystery: almost all schools serving low income children need additional assistance). The final sanctions in the current ESEA versions should not be employed in any event as they will cause more harm than good (as discussed above). Disaggregated data should be compiled as soon as it can, and it should be reported and used as part of making initial determinations for QR or assistance.

Q. Without numerical targets with firm sanctions, won't schools just ignore the process?

A. This gets to the heart of the improvement issue, and thus deserves some discussion.

First, most teachers currently working have substantial knowledge, work hard, and care. They do not always have sufficient knowledge. We believe teachers are mostly willing to address that issue, but need support and resources to increase their skills. They do not need attacks or attempts to de-skill them. Too often, current accountability is, or is perceived by educators as, a means of attack and blame, while teaching to the test is de-skilling.

If it is true that teachers are less skilled or motivated than we believe, we are in a quandary that current accountability attempts cannot solve. High-stakes testing does not improve skills or increase motivation to teach well (as distinguished from seeking short cuts for boosting test scores) or to learn more. There is no reason to believe that in a high-stakes environment, high-quality applicants will flood the teaching field, which already faces severe and growing shortages. The nation has little choice but to work with current teachers and to create positive inducements to attract new teachers. The approach to school improvement we propose would be far more likely to accomplish this goal than is high-stakes tests with numerical triggers. We would add that teachers are particularly likely to leave schools which are likely to fail to make adequate improvement on tests, which will only exacerbate the problems.

In short, school improvement cannot succeed without or against teachers, and it mostly needs more skilled teachers who actively participate in improvement efforts. This cannot be well attained by threats or fear.

The second critical element is the target: is it a target of ensuring high quality educational opportunities for all students, or will low-income and minority-group students be consigned to test preparation for "basic skills" in a process that never gets beyond such basics? Evidence clearly demonstrates that most all students can in fact learn to rather high levels in most areas, but only when provided rigorous high quality intellectual environment and opportunity. Test-based reform fails to provide those opportunities, which is why progress toward higher levels is often so slow in schools with low-achieving students (see recent reports from Chicago Consortium on School Research) and why parents who believe their children have good schools are increasingly angry about tests even when most of the children pass them. Finally, the goal of focusing only on tests seems to lead to schools which ignore too many other academic and non-academic dimensions, which deprives children of opportunities and supports they need and deserve.

The argument here is that real school improvement must proceed with teachers and must have goals worth honoring. Current accountability procedures often proceed by attacking or de-skilling teachers to focus on too-limited targets. Thus, it probably won't succeed on its own narrow terms and surely cannot succeed in providing a truly high-quality education for all students. Thus, schools and teachers may pay attention – but they will pay attention to the wrong things or to inadequate things.

The plans also provide for transparency: claims of success or progress must be backed up with public and verifiable information. That it will be public will be a spur to taking the process seriously even if one is not initially motivated to do so. That it is verifiable is a spur to accuracy, rather than to inflated claims. As it is based in student work, it will actually be far more understandable to the public than are test scores, about which most people understand very little. Samples of high-quality student work can be used to help parents, students and teachers learn what such work looks like and thus seek to attain similar levels of quality.

All that said, we also recognize that for a variety of reasons, even given adequate resources, some schools may fail to do a good job and will not improve despite publicizing the situation and providing support for improvement. FairTest does not believe such schools should be left to continue to fail their students, and the assessment and accountability plans call for stronger measures if needed. That would occur based on a variety of evidence, not simply test score gains that are of questionable validity and reliability.

Q: But won't this take forever?

A: There is no reason to believe that schools serving predominantly low-income children can provide high quality opportunity for all unless they have a major infusion of resources, and no reason to believe they can succeed unless this country develops as strong a will to solve poverty as it now has to develop tests. That, barring such changes, success is not likely for more than a relatively few.

The goal of all students reaching a level of "proficiency" equivalent to "proficient" on NAEP is not feasible in the 10-12 years called for in ESEA. This is indeed unfortunate, but it will not get better by pretending the impossible can be attained. That is, the goal proposed is itself not reachable and thus will "take forever."

Schools and districts taking the approach we propose will, we conclude, be more likely to make substantive, real progress on valuable, real-world outcomes, than will schools focusing on just boosting test scores every year. The reason is that it will tap into the energy, talent, and caring of educators and students – the human factor that a test-based approach ignores and undermines. That is not to say it will happen quickly, but that the goals will be superior and reached as quickly as happens with test-based accountability. It will happen faster if the nation decides it really means to leave no children behind and takes serious measures, in education and elsewhere. We have proposed one such serious measure. It should not be dismissed in favor of narrow and illusory test-scores.

August 1, 2001