About a week ago, the OECD (Organisation for Economic Co-operation and Development) Education at a Glance 2003 report was released to the press. The main thrust of the report was portrayed in the press as follows:
Report: U.S. No. 1 in school spending
Test scores fall in middle of the packWASHINGTON (AP) -- The United States spends more public and private money on education than other major countries, but its performance doesn't measure up in areas ranging from high-school graduation rates to test scores in math, reading and science, a new report shows.
(taken from the September 16th article on the CNN website)
This rather damning lead was followed in the body of the article by a quote from Barry McGaw, education director for the OECD:
"There are countries which don't get the bang for the bucks, and the U.S. is one of them."
The rest of the press report cited a figure of $10,240 spent per student in the U.S., and included tables showing listings for 15-year-olds' performance in math, reading, and science that rank the US below thirteen to eighteen other countries.
Whenever I see a report from a reasonably serious organization such as the OECD described in sensationalistic terms with potential for malicious use, I get suspicious. And when I get suspicious, I go to the source and check out the numbers. Which is what I did in this case. Not to spoil the rest of the story, but while I found many interesting and worthwhile nuggets of data in the OECD report (many of which are summarized in the briefing notes for the U.S., downloadable in PDF format), I found nothing to substantiate the explicit and implicit allegations of the news report.
Let's start out with the figure for $10,240 spent per student. This figure is not as simple as might seem at first. First, the figure represents adjusted U.S. dollars - in other words, the figures that it is being compared to are not actual dollar amounts spent in each country, but have been adjusted for purchasing power parity (PPP) so as to provide a better basis for comparison. While some type of correction of this type is needed for cross-country comparisons to be meaningful, the adjustment formula used can artificially inflate or deflate the actual magnitudes involved. In other words, while the numbers obtained from this adjustment can be reasonably used to claim that country A spends more than country B per student on education, it would be foolhardy to claim that the ratio of expenditures between the two countries is more than a rough estimate.
More importantly, the $10,240 figure includes expenditures per student from primary school through college inclusive. In other words, while the performance of fifteen-year-old high school students is being used as the yardstick for educational quality comparisons, the monetary amount being referenced includes expenditures for college education. As anyone living in the U.S. knows, the ways colleges are funded differ drastically from those for high schools. To measure the "bang for the buck" being obtained would require some equivalent performance measure for college students, which is nowhere to be found in the report. A more relevant figure to the critique would be the total secondary school expenditure per student. Using Table B1.1, we obtain a figure of $8,855 - high, but far from the highest in this category (Switzerland, at $9,780), and comparable to other countries such as Austria ($8,578) and Norway ($8,476).
So much for the dollar amount. What about those tables showing the U.S. trailing the pack in the knowledge demonstrated by fifteen-year-olds in reading, mathematics, and science? As before, the story is more complex than these tables would seem to show. While the rankings published are "correct" inasmuch as they follow the published scores, they neglect to take into account the fact that in many cases, score differences between countries are too small to be significant. For instance, the U.S. indeed trails Norway in science scores - by all of 0.18%. A more useful way to think about data such as this is to look for "clusters" of countries that perform in like fashion. Using the data from Tables A5.2, A6.1, and A6.2, and the cluster analysis tools from R, I find that the data can reasonably be clustered into four groups. The first group, made up of seven countries, exhibits performance demonstrated by fifteen-year olds that is better than average. The second group, which includes the U.S., exhibits performance that is average. The third group exhibits performance below average, and the fourth group exhibits performance that is substantially below average. The following table, with countries arranged in alphabetical order within groups, summarizes these results:
Performance of 15-Year-Old Students in
Reading, Mathematics, and Science |
|||
| Australia | Austria | Greece | Brazil |
| Canada | Belgium | Italy | Mexico |
| Finland | Czech Republic | Latvia | |
| Japan | Denmark | Luxembourg | |
| Korea | France | Poland | |
| New Zealand | Germany | Portugal | |
| United Kingdom | Hungary | Russian Federation | |
| Iceland | |||
| Ireland | |||
| Liechtenstein | |||
| Norway | |||
| Spain | |||
| Sweden | |||
| Switzerland | |||
| United States | |||
While this indicates that the U.S. is not in an optimal position, it is far from indicating results as dire as those implied by the press report. Secondary school systems in the seven countries in the first group are worthwhile studying further - while the difference in performance between the first and second groups is not dramatic, it is certainly significant and noticeable.
What does this tell us, then, about the appropriateness of the adjusted expenditures? It tells us that we cannot, at this point, and based upon these numbers, make any judgment about the appropriateness of per student adjusted educational expenditures for any given country. Expenditures per secondary school student do not in any significant way correlate to the observed grouping. Nor does coupling these numbers to any other data included in the report yield any particularly insightful results: percentage of GDP spent on education, class size, number of hours of classroom instruction, and teacher pay all fail to yield any significant correlations with our observed clustering either when taken alone or when taken in groups. Again, this does not mean that none of these factors matter - rather it means that predictive models for educational success require the study of additional variables not considered in the current report.
Finally, a cautionary note about the interpretation of the results for the seven better-than-average performers: the data in the report simply points to something "interesting" happening in these seven countries, worthy of further investigation. It does not point to these countries as occupying a pinnacle that other countries should strive to achieve and then rest on their laurels. I chose the label for this group carefully: "better than average" implies just that - not an ultimate target in any sense of the word. The instruments used for the evaluation of 15-year-old student proficiency in reading, mathematics, and science are only intended to provide a rough picture of what could reasonably be expected as universal knowledge in these areas. No country even approached a near-perfect score on these tests for a majority of its tested population; thus, no country could be said to have provided a solid educational floor in these categories for all of its citizens. Getting to the point where this educational floor can be guaranteed will require more than slight changes to expenditures, school year duration, or class sizes - it will require a significant rethinking of how the educational process occurs at all levels.
Posted by Ruben at September 24, 2003 8:26 PM