In the wake of my previous
post about the recent OECD education
report, several people have
asked for further analysis of the OECD evaluation tools. More specifically,
they have
been interested in how these tools differ from the usual standardized tests
(with their known pitfalls and blind spots), and how the results obtained
should be
interpreted. In order to discuss this, I have to preface the body of this post
with a brief definition of two terms commonly used within the educational community:
content and competency.
Broadly defined, content can be viewed as the “stuff” that
makes up a subject area. For instance, part of the content involved in a basic
health
class might be the definition of virii and bacteria, with their different characteristics
and modes of reproduction. Competencies can then be viewed as the “what
can the student do” with this content. Thus, at a very basic level of
competency, the student will be able to reproduce the definitions acquired.
However, this
level of competency is generally not the only thing that would be desired –
one would hope that the student would be able to integrate this with other
knowledge,
and thereby be able to follow simple cause-and-effect procedures supplied them
in order to keep themselves healthy. Even better, one would hope that they
would be able to create new knowledge for themselves, so that they could take
an active
role in their health decisions. Clearly, the evaluation tool to be used in
determining these progressively higher levels of competency has to go well
beyond “fill
in the blank”-type tests.
Take, for instance, the content area of basic
algebra. An evaluation tool that measures whether the student can carry out
the basic operations involved in
a standard “solve for x”-type problem is indeed providing some information
about how the student performs in this content area – but at a very low level
of competency, namely that of mimicry of a standard algorithmic procedure.
A tool that evaluates how the student performs at applying these basic operations
to a standard word problem where the necessary information is already provided
to the student corresponds to the measurement of a somewhat higher level of
competency.
Replacing the standard word problem with one using nonstandard phrasing provides
an evaluative tool for a substantially higher level of competency. At even
higher levels of competency we would find tools that evaluate how the student
performs
when presented with a problem solvable using the algebraic procedures known
to them, but where the data needed to solve the problem is not provided to
the student
a priori, and must be requested by them. Finally, at the highest level of competency,
it becomes of interest to evaluate how the student performs when applying the
tools of basic algebra to a real-world problem defined autonomously by the
student.
Most standardized tests operate on only the first two levels of our
algebra example, and hence miss a large – perhaps the most important – part
of the
competency
picture. The OECD evaluation tools are designed to provide a broad picture
of the competency spectrum within the context of an equally broad content area.
Furthermore, the “top level” of the OECD competencies corresponds
to a “competency floor” that the majority of a country’s population
can reasonably be expected to achieve. In other words, high scores on the OECD
tests
should be attainable by most people, and not just a privileged elite. The fact
that no country came close to this result indicates the distance yet to be
covered in terms of educational quality and equity throughout the world.
To
better understand the conclusions that can be derived from the OECD report,
we need to take a look at how scores relate to competency levels. A difference
of a few
points in the OECD results between two students means absolutely nothing in
terms of the relative competencies achieved; instead, the OECD provides rough
categorical
scales that correspond to broad swaths of competencies. For the three areas
studied, these categories, listed here with a few sample representative tasks
are:
| Categories and Representative Tasks |
||
| Reading Literacy | Mathematical Literacy | Scientific Literacy |
| Level 1 (335 to 407 points): locating a single piece of information; identifying the main theme of a text |
Level 1 (around 380 points): carrying out single-step mathematical processes | Level 1 (around 400 points): recalling simple scientific facts |
| Level 2 (408 to 480 points): locating straightforward information; deciding what a well-defined part of the text means |
||
| Level 3 (481 to 552 points): locating multiple pieces of information; drawing links between different parts of the text |
Level 2 (around 570 points): carrying out multiple-step mathematical processes for predefined problems |
Level 2 (around 550 points): using scientific concepts to make predictions or provide explanations |
| Level 4 (553 to 625 points): locating embedded information; critically evaluating a text |
Level 3 (around 750 points): creating new mathematical processes as required by problems |
Level 3 (around 690 points): creating new conceptual models to make predictions or provide explanations |
| Level 5 (over 625 points): locating difficult to find information; building new hypotheses based upon texts |
||
The source for this table is the OECD
Education at a Glance 2003 report, from which most of the language
describing representative tasks is drawn. More detailed information can be
found on the OECD Programme
for International Student Assessment (PISA) website – the executive
summary of
the Knowledge
and Skills for Life 2001 report is particularly useful in
this regard.
In my previous post, I had identified a group of seven countries that could
reasonably be said to perform “better than average” in the context
of these literacies. Looking at New Zealand as a representative country in
this group, we find that
its averages are 529, 537, and 528 for reading, mathematical, and scientific
literacy respectively – still a ways from providing the majority of its population
with the desired “competency floor”.
As I mentioned at the end of
my last post, remedying the deficiencies revealed by the OECD report will
take more than minor changes in expenditures or classroom
organization. Instead, it will take educational redesign that directly addresses
the idea that higher-level competencies are what is desirable and achievable
in all content areas. Interestingly, I believe that this redesign can be
accomplished within the context of existing educational structures – I have
yet to see any
data that indicates, for instance, that dramatically changing the mix of
public to private educational institutions in any given country would fundamentally
transform the results measured by the OECD.
What can play
a crucial role is the set of technological transformations that can be
brought about by the availability of networked
computers to all
participants
in the educational process. A serious discussion of why this is the case
will have to wait until a later post. However, let me provide the following
example
as a tidbit in the meantime. Returning to our earlier algebra example,
researchers working from a constructivist learning perspective have found that
in mathematics
– and in fact, in most subject areas – project-oriented learning has the
potential to work wonderfully as a way of addressing both the teaching
and evaluation
of
the highest competency level described above. However, this potential
can be effectively sabotaged in situations where a scarcity of information
resources
works against the student’s capacity to research, define, and present a
project. Additionally, at the evaluation stage, it is very important that teachers
have access to a broad range of student projects and serious collegial
discussion
across a broad range of institutions about them – it is too easy otherwise
for
evaluative inbreeding to take place. Intelligent use of networked computers
can address both these issues efficiently in ways no other resource can,
both
by
providing low-cost access to resources, as well as by allowing students
and teachers alike to share and discuss projects. Of course, just throwing
technology
at the
schools will not accomplish this result – both new research and new solutions
derived from existing research will be necessary to change the landscape
described in the OECD report. However, the early results of experiments
such as the Maine
Learning Technology Initiative make me
hopeful that this is indeed the most promising direction for change.