About a week ago, the OECD (Organisation
for Economic Co-operation and Development) Education
at a Glance 2003 report was released to the press. The main thrust of
the report was portrayed in the press as follows:
Report: U.S. No. 1 in school spending
Test scores fall in middle of the packWASHINGTON (AP) — The United States spends more public and private money
on education than other major countries, but its performance doesn’t measure up
in areas ranging from high-school graduation rates to test scores in math,
reading and science, a new report shows.
(taken from the September
16th article on the CNN website)
This rather damning lead was followed in the body of the article by a quote from
Barry McGaw, education director for the OECD:
“There are countries which don’t get the bang for the bucks, and the U.S. is one of them.”
The rest of the press
report cited a figure of $10,240 spent per student in the U.S., and included
tables showing
listings for 15-year-olds’ performance in math, reading, and science that rank
the US below thirteen to eighteen other countries.
Whenever I see a report from a reasonably serious organization such as the
OECD described in sensationalistic terms with potential for malicious use,
I get suspicious.
And when I get suspicious, I go to the source and check out the numbers. Which
is what I did in this case. Not to spoil the rest of the story, but while I
found many interesting and worthwhile nuggets of data in the OECD report (many of which
are summarized in the briefing notes for the U.S., downloadable in PDF
format), I found nothing to substantiate
the explicit and implicit allegations of the news report.
Let’s start out with
the figure for $10,240 spent per student. This figure is not as simple as might
seem at first. First, the figure represents adjusted U.S. dollars – in other words, the figures that it is being compared to are
not actual dollar amounts spent in each country, but have been adjusted for
purchasing power parity (PPP) so as to provide a better basis for comparison. While some
type of correction of this type is needed for cross-country comparisons to
be meaningful, the adjustment formula used can artificially inflate or deflate
the actual magnitudes involved. In other words, while the numbers obtained
from this adjustment can be reasonably used to claim that country A spends more than
country B per student on education, it would be foolhardy to claim that the
ratio of expenditures between the two countries is more than a rough estimate.
More importantly,
the $10,240 figure includes expenditures per student from primary school through
college inclusive. In other words, while the performance of
fifteen-year-old high school students is being used as the yardstick for educational
quality comparisons, the monetary amount being referenced includes expenditures
for college education. As anyone living in the U.S. knows, the ways colleges
are funded differ drastically from those for high schools. To measure the “bang
for the buck” being obtained would require some equivalent performance measure
for college students, which is nowhere to be found in the report. A more relevant
figure to the critique would be the total secondary school expenditure
per student. Using Table
B1.1, we obtain a figure of $8,855 – high, but far from the highest in this category
(Switzerland, at $9,780), and comparable to other countries such as Austria
($8,578) and Norway ($8,476).
So much for the dollar amount. What about those
tables showing the U.S. trailing the pack in the knowledge demonstrated by
fifteen-year-olds in reading, mathematics,
and science? As before, the story is more complex than these tables would seem
to show. While the rankings published are “correct” inasmuch as they follow
the published scores, they neglect to take into account the fact that in many cases,
score differences between countries are too small to be significant. For instance,
the U.S. indeed trails Norway in science scores – by all of 0.18%. A more useful
way to think about data such as this is to look for “clusters” of countries
that perform in like fashion. Using the data from Tables
A5.2, A6.1, and A6.2,
and the cluster analysis tools from R,
I find that the data can reasonably be clustered into four groups. The first group, made
up of seven countries, exhibits performance demonstrated by fifteen-year olds that
is better than average. The second group, which includes the U.S., exhibits
performance that is average. The third group exhibits performance below average,
and the fourth group exhibits performance that is substantially below average. The
following table, with countries arranged in alphabetical order within groups,
summarizes these results:
|
Performance of 15-Year-Old Students in
Reading, Mathematics, and Science |
|||
| Australia | Austria | Greece | Brazil |
| Canada | Belgium | Italy | Mexico |
| Finland | Czech Republic | Latvia | |
| Japan | Denmark | Luxembourg | |
| Korea | France | Poland | |
| New Zealand | Germany | Portugal | |
| United Kingdom | Hungary | Russian Federation | |
| Iceland | |||
| Ireland | |||
| Liechtenstein | |||
| Norway | |||
| Spain | |||
| Sweden | |||
| Switzerland | |||
| United States | |||
While this indicates that the U.S. is not in an optimal position, it is far
from indicating results as dire as those implied by the press report. Secondary
school systems in the seven countries in the first group are worthwhile
studying further – while the difference in performance between the first and
second groups is not dramatic, it is certainly significant and noticeable.
What does this tell us,
then, about the appropriateness of the adjusted expenditures? It tells us that
we cannot, at this point, and based upon these numbers, make any judgment
about the appropriateness of per student adjusted educational expenditures
for any given country. Expenditures per secondary school student do not in any significant
way correlate to the observed grouping. Nor does coupling these numbers to
any other data included in the report yield any particularly insightful results:
percentage of GDP spent on education, class size, number of hours of classroom
instruction, and teacher pay all fail to yield any significant correlations
with our observed clustering either when taken alone or when taken in groups. Again,
this does not mean that none of these factors matter – rather it means that
predictive models for educational success require the study of additional variables
not considered in the current report.
Finally, a cautionary note about the interpretation
of the results for the seven better-than-average performers: the data in the
report simply points to something “interesting” happening
in these seven countries, worthy of further investigation. It does not point
to these countries as occupying a pinnacle that other countries should strive
to achieve and then rest on their laurels. I chose the label for this group
carefully: “better than average” implies just that – not an ultimate target in any sense of the
word. The instruments used for the evaluation of 15-year-old student proficiency
in reading, mathematics, and science are only intended to provide a rough picture
of what could reasonably be expected as universal knowledge in these areas.
No country even approached a near-perfect score on these tests for a majority
of its tested population; thus, no country could be said to have provided a solid
educational floor in these categories for all of its citizens. Getting to the
point where this educational floor can be guaranteed will require more than
slight changes to expenditures, school year duration, or class sizes – it will
require a significant rethinking of how the educational process occurs at all levels.