"Dimensions" Portfolio

K-12 Testing

Can a portfolio used for accountability purposes remain compatible with classroom instruction? Can an accountability portfolio allow for diversity in the materials included and still be scored reliably? Some positive, though tentative, answers to these questions have been provided by a research project in California.


From 1992-94, as part of the now-defunct California Learning Assessment System, researchers from the Educational Testing Service (ETS) worked with teachers and other educators from across California to develop an organic portfolio that would provide trustworthy information about student achievement and also support school improvement.


Using the state's curriculum frameworks, the project defined dimensions of learning in language arts (composing and expressing ideas; constructing meaning) and math (communicating math; math content; putting math to work). Each dimension has sub-parts to provide further specificity. Each sub-part is defined at four levels of accomplishment (exemplary, accomplished, developing, beginning). Thus, explains the project's Final Report, the dimensions provided a kind of map of how students progress could be evidenced in their classroom work.


The portfolio includes evidence to build the best case for demonstrating achievement in each dimension in the subject area. For example, the math portfolio includes evidence of what the student knows and can do in each of the three math dimensions. The specific material chosen was up to the student in consultation with the teacher. Thus, variety and individual choice in the work was allowed.


Because one purpose of the portfolio program was to support professional development, teachers in the project scored the portfolios in a trial scoring session in May 1995. Rather than have teachers score each piece in a portfolio, the project trained teachers to look at all the pieces together for evidence of learning in the dimension, and use that cumulative evidence to determine a score.


Given that this was a first-time effort, the results were strongly positive. A combination of statistical analyses and interviews with scorers found not only that inter-rater reliability (agreement among scorers) was high for an initial effort, but also that most readers were using sound reasoning in evaluating the portfolios.


A questionnaire showed that nearly all participating teachers reported changes in classroom practices, and two-thirds changed their beliefs and thinking about teaching. Among other things, the teachers often began involving students in developing modified dimensions and scoring guides for use in the classroom. The questionnaire results confirmed other research which shows that deeply involving teachers in evaluating performance assessments or portfolios can have a powerful, positive impact on their work. Evidence from other assessment projects suggests that improvements in reliability and even deeper impacts on teaching would have occurred had the project continued.


Some things, however, cannot be assessed well with portfolios which focus on cumulative achievement. For example, teachers in the project initially wanted to assess student progress in learning over the year. During an early trial scoring session, they realized that because these portfolios focus on achievement, an end state, they are not useful for evaluating the progress of learning during the year. Such information about progress, along with information about the process of each student s learning, is essential for teachers to work with their students, butrequires evaluation over time of the sort provided by the Primary Learning Record (see p. 7).


Though the project was successful, questions remain, many of which can only be answered if similar efforts survive and expand. Two concerns noted by the researchers are: how to ensure that the portfolio assessment system is fair for a diverse student population; and how to provide the kinds of materials and professional development that ensure both high-quality portfolios and improved learning.


More broadly, the impact on classrooms over time needs to be studied to ensure that the requirements of portfolio programs don t ultimately narrow curriculum and instruction, as appears to have occurred in Britain (see pp. 8-9).


The CLAS Portfolio Assessment Research and Development Project Final Report, by William Thomas, et al., from the Center for Performance Assessment, ETS Mail Stop 11-P, Princeton, NJ, 08541; (609) 734-5521; free.