The results from the OECD PISA 2003 study of learning skills among 15-year-olds are now out. As could be expected from last year’s coverage of the PISA 2000 results, news reports have tended to misrepresent the information contained in the new report. I have already covered these misrepresentations in two previous posts, so I will not rehash that material here; however, I will spend some time looking at the new data conveyed by the report.
PISA 2003 incorporates a new (very welcome) category for student evaluation in the form of a set of questions covering Problem Solving. An analysis of this category would be worth a separate post unto itself; since my main goal here is to update the results I had obtained for PISA 2000, I will omit it from consideration in the discussion that follows.
A statistical analysis similar to that from my previous post yields, as before, four main groups with the labels shown in the table below. A new classification resulting from the PISA 2003 analysis is the subdivision of the “Substantially Below Average” group into a better-performing “High Group”, and a lower-performing “Low Group”. Countries that did not participate in PISA 2000 are highlighted in gray; countries that improved their results sufficiently to be promoted from one group to the next higher group are highlighted in green. The United Kingdom, which participated in PISA 2000, was excluded from PISA 2003 due to noncompliance with OECD response rate standards. The following table, with countries arranged in alphabetical order within groups, summarizes these results:
|
Performance of 15-Year-Old Students in
Reading, Mathematics, and Science |
|||
| Australia | Austria | Greece |
High Group
|
| Canada | Belgium | Italy | Serbia |
| Finland | Czech Republic | Portugal | Thailand |
| Hong Kong – China | Denmark | Russian Federation | Turkey |
| Japan | France | Uruguay | |
| Korea | Germany |
Low Group
|
|
| Liechtenstein | Hungary | Brazil | |
| Netherlands | Iceland | Indonesia | |
| New Zealand | Ireland | Mexico | |
| Latvia | Tunisia | ||
| Luxembourg | |||
| Macao – China | |||
| Norway | |||
| Poland | |||
| Slovak Republic | |||
| Spain | |||
| Sweden | |||
| Switzerland | |||
| United States | |||
As we can see, only four countries (Liechtenstein, Latvia, Luxembourg, and Poland) improved their results substantially from 2000. Of these, the result for Luxembourg has to be discarded from consideration, since (as noted on page 30 of the OECD report) assessment conditions in this country were changed significantly between 2000 and 2003. In the case of Liechtenstein, only 332 students were assessed, due to the small size of the country. Because of this small sample size, changes at the individual school level are just as likely to affect the final results as national policy decisions. Hence, it is difficult to ascertain the cause of the observed improvement. Finally, in the cases of Latvia and Poland, it is tempting to attribute the improvement to their respective large-scale educational reforms, which started in 1998. However, data that would allow for the determination of cause-and-effect relationships in these two cases is currently lacking.
It is unsurprising that little has changed between PISA 2000 and PISA 2003 – after all, only three years have elapsed between the two studies. However, news reports – and, I fear, some public officials – have made much of minor increases or decreases in scores that are not significant. What still stands is my conclusion from my previous post: no country can be said to have provided a solid educational floor in these categories for all of its citizens. Getting to the point where this educational floor can be guaranteed will require more than slight changes to expenditures, school year duration, or class sizes – it will require a significant rethinking of how the educational process occurs at all levels.