Interpreting PISA Results: It’s Poverty, Stupid (With a Bit of the iPhone)

The results of PISA 2022 should, like all standardized test results, be filtered through a dose of skepticism about the claims of the test producers and administrators. We must also carefully scrutinize “Chicken Little” claims in the media which is notorious for manufacturing and hyping education crises. Declines in standardized test scores have been the premise for all of the failed education reforms of the past forty years, from the publication of A Nation at Risk, through No Child Left Behind, the charter school movement, and now universal vouchers and privatization. We must guard against this trap yet again.

By scrutinizing the performance of the United States versus other OECD countries, the unshocking conclusion should be that the PISA test is largely a measure of childhood poverty rates rather than academic achievement.  The United States leads the OECD in child poverty. Our rate of child poverty is approximately 26%, and higher by some measures. Thus it is not surprising that as a nation we do not perform as well as most other OECD countries on PISA. If you compare the tranche of American schools with poverty rates equal to those of other OECD countries, however, the United States does quite well.

In reading, countries at the top of the PISA list– Slovenia (499), Denmark (489), and Finland (474)–all have childhood poverty rates below 10%. If one were to measure US schools with under 10% poverty rates, the average score would be 562, good enough for first globally. In mathematics, Germany, France and the UK have child poverty rates between 15 and 18% and have scores of 475, 479 and 489 respectively.  If you measured US schools with childhood poverty rates of 10-25%, we would score a 508, good enough for 4th in the OECD.

America’s problem on PISA is poverty and inequality, not curriculum and instruction. 

PISA is a scaled score, norm-referenced, multiple-choice test. Two-thirds of all test takers globally score between 400 and 600 on a section (math, reading and science). Only 2% score over 700.  In general, these kinds of tests are set up so results will go down over a longer time frame. According to Prof. Andy Hargeaves of Boston College, “once a metric is widely used and has a competitive ranking element, gaming the system leads to overall declines in performance after an early lift, and also has negative side-effects on well being.”  Not surprisingly, during the last two decades student performance in mathematics, reading and science all significantly declined in most OECD countries.

If one were looking for an actual reason for this decline besides test design and use, the proliferation of technology and handheld device usage by 15-year olds may very well be the culprit. In response to a question on the 2022 PISA, 45% of students said they felt anxious if their phone was not near them and 65% felt distracted by them during math lessons. Prof. Sam Abrams of Teachers College, Columbia University attributes Finland’s decline in PISA scores to the introduction of the iPhone and its proliferation of use among Finnish teens.

As an aside, the United States did not do terribly relative to other OECD countries in terms of rankings. The US moved up in rankings for all three subjects (math, reading and science). And in aggregate scores the United States held its ground in reading and science pretty well from the previous administration while other countries’ scores went down.

The OECD is to be commended for attempting to analyze the extent creativity and innovation are promoted in national school systems. However, the Creativity and Innovation review did not include the United States. The PISA team found it hard to extract data about the state of creative thinking in schools in the US, because education is delegated via states to many often small districts of schools. They did cite the work of EL Education’s network of districts and public schools – a network committed to assessment via performance-based assessment– as an example of creative and innovative schooling designed to get students to think critically, communicate clearly, and create complex work. Overall the report stated that within the limitations of a snapshot review “it has not been possible to do justice to the rich variety of experiences in schools in the USA.”  

Successful attempts have been made to measure creativity, innovation and entrepreneurship in the global economy by nation.  Perhaps not surprisingly, the United States ranks near the top in those economic categories among G20 nations and has for decades.  There is a disconnect between our international testing rankings and our economic ranking based on human capital. This calls into question whether the United States should be worried about its PISA rankings at all.   Does PISA measure anything of importance?  Not really, but you wouldn’t know it from the weight policy makers and media ascribe to the results.

Tienken and Mullen (2014) found no statistically significant relationships between indicators associated with the innovation economy and PISA. Earlier studies of PISA results suggest no statistically significant relationships or weak relationships between ranks on international tests and economic output indicators such as GDP, adjusted gross income, or purchasing power parity (e.g., Baker, 2007; Rameriz, et al, 2006; Tienken, 2008). International tests do not provide meaningful information about the skills most important for full participation in society in terms of socio-civic and economic outcomes in the G20 countries (Sjoberg, 2007). The information tested on international tests is not the information children will need to compete in the innovation economy and the results do not tell us important information about student proficiency with the vocational competencies necessary to create, innovate, and pursue entrepreneurial opportunities (Zhao, 2014). (See citation below). Perhaps the correct answer is that we really shouldn’t be paying much attention to PISA results. They don’t give us particularly useful or telling information.

Finally, given the deep dislocation in schooling and trauma caused by the COVID pandemic, drops in scores from the pre-COVID administration are not surprising. The PISA scores are some evidence of what we already knew–the pandemic was bad for kids everywhere and impacted learning.

Tienken, C.H. & Mullen, C.A. (2014). The curious case of international student assessment: Rankings and realities in the innovation economy. In S. Harris & J. Mixon (Eds.), Building cultural community through global educational leadership (pp.146-164). Ypsilanti, MI: NCPEA Press