Authentic Evaluation And Assessment --
What Matters More Than Habits Of Mind?
Introduction
This is writing from the heart. It is more of a
journey -- or a
journal (a portfolio, perhaps?) -- than it is a paper for an end result of a
grade in a
doctoral program … although it is that, as well. As such, it is
written in the
first person, for the journey and the results of this course of
study in Fall
of 2001, are very personal … deeply meaningful studies and
reflections that
have led to a confirmation of things I’ve experienced in the
education field as
a student, an educator, and a development specialist for distance
learning.
This journey began in 1992 when I began to work
with
inner-city, at-risk youth, in a Denver High School. That is an
untruth. I had a
bunch of the hardest, “baddest” of the Gang Boys dumped in my lap.
As Media
Director with a heavy art career background, the only thing I knew
to do with
these scary kids was this: “OH! You like spray paint cans? Here!
What do you
think of THIS art?” And I proceeded to take youth with not a lick of
even
keyboarding experience, into the world of computer animation.
These young people barely knew how to string the
alphabet
together long enough to make a simple sentence, at least on paper.
Yet, their
wisdom and knowledge was incredibly deep, once you were able to wade
past the
distrust of the Gringa Library Lady, their culturally-
instigated
hatreds, and underneath it all -- their history of being told by an elitist
school
district that they were stupid and unable to ever achieve anything
in life,
simply because they had years of being assessed on a level that had
no bearing
to the reality of their lives. Or their brain power. Or their true
abilities.
For anyone, much of the underlying reality of true anger is the
feeling of
being out of control. For these youth, they’ve never been IN control
-- and
so, the anger runs deep and it runs true. THEY know they are not
stupid --but
how to convince the “white world” of that? And eventually, they
succumb to that
myth --
and then, it becomes reality. It’s the ghetto story.
At that time, I knew nothing of the world of
“education,”
nothing of evaluation or learning methods, of curriculum,
statistics, norms,
and averages. I had no “vocabulary” to apply to what I saw in these
young
people. But, I soon found out that by introducing them to something
that they could
do --
even without having the “parts and pieces,” such as typing classes
or high
grades in classes --
and get some very positive feedback, I saw the level of self-image
begin to
grow, and grow. Each project became better; some eventually became …
astounding.
Soon, these Bad Gang Boys were not hanging with the gang anymore.
Quite a few
ended up having a successful high school career. A few went on to
college. All
of them were changed. To date, not a one of them is “at-
risk.”
I was hooked. What made the difference? I truly
believe it
had something to do with a positive experience, not just in a job
performed,
but also in the feedback -- the method of evaluation -- that they
received.
Everything I did with these youth was intuitive, rather than things
I had
studied for a career. All I could see was that there was
intelligence, ability,
and even a desire to DO something with their lives. I had to find a
way to get
around their feeling that I was passing judgment on them, especially
“academic”
judgment. Years of district and national testing had already done a
tremendous
job of convincing these youth of how few methods they had for
competing in the
world. Their “reality” was very different from the reality described
by
national testing! They needed something else … in drastic
measures.
They needed real measures for skills and
knowledge that were
relevant to THEIR world.
They needed something
authentic.
This writing is about discovering a solution,
hidden in a
program of “education.” The true reason I landed “in Education” was
that the
education programs, in the early days of Internet and digital tools,
were the
only ones giving classes in things like … Multimedia! Digital Art!
Authoring
classes for the platform shells for games, kiosks! … and yes, learning
environments. That was in 1995, when I then added
architecting classes …
for building the World Wide Web. I began to envision a highly
improved method
of reaching and teaching at-risk youth on a global level. I
saw that I
could reach them with the message that they could pass a “test,” and
not fail.
This was the perfect toolkit for taking on the challenge of proving
that
authentic assessment was timely, possible, and cost-effective … for
by this
time, I DID have the beginnings of the vocabulary, the “jargon of
the trade.”
I found myself in a Master’s program for computer
science
and education … and I “met” a man named Grant Wiggins. The puzzle
pieces
clinked effortlessly into place. For these bad gang boys and for me
--
Wiggins provided the description for all the frustration -- we have
to do more than lead them “to believe that right answers matter more
than
habits of mind and the justification of one's approach and results.”
(Wiggins, 1990, ¶ 14). For
these
at-risk youth, giving “right answers” on what amounts to a contrived
instrument, is most-times impossible, but always contrary to
their
approach or their results -- in their very real “real-world”
world.
The reality behind the term, “at-risk,” is a saga
unto
itself. But the important thing, in my own journey, has been to
arrive at the
revelation that there are solutions. Many of those solutions
are found
in the various aspects of what is called authentic tasks, and
authentic
evaluation and assessment.
~
Mary B. Harris (1874–1957), U.S. prison administrator
(Bartleby.com, 2001)
When
Reformation IS
the Journey
Human beings are emotional creatures, and, as
such, their
daily actions revolve around not just skills and abilities to
perform, but also
their emotional capacities for managing the positive and negative
experiences
in life. Two very big motivators are anger and fear. Unfortunately,
these
negative emotions come from very common elements in our lives -- from
the entities that form our characters, make assessments and
judgments, and dole
out the punishments and rewards. One of these entities, of course,
is the
school system, the “nest and nursery” for most of us, for at least
twelve years
of our lives.
As young learners, most of what we learn
IS in the
hands of the educational system, and much of it has been highly
ineffective in
one supreme area: Assessment should primarily support the needs of the
learners. Traditionally, this just has not been so. Beyond any kind of
technical consideration, this is the basis for the move to reform
assessment.
(Wiggins, 1990, ¶ 11). With
so many
students falling through the gaps, it is an understatement to say
that the
system is ineffective. The current outcry is more to the point: It
is harmful.
Once learners “learn” (are conditioned or even
driven)
to approach the assessment situation with at least fear, and also with
increasing anger --
this absolutely affects the outcomes and thus the validity of the
intended
testing situation. It then obviously has affected the totality of
the learning
experience, and is no longer serving the needs of the learner. Most
people who
defend traditional tests “fail to realize that it is the form, not
the content
of the test that is harmful to learning … students come to believe
that
learning is cramming; teachers come to believe that tests are after-
the-fact,
imposed nuisances composed of contrived questions -- irrelevant to
their
intent and success. Both parties are led to believe that right
answers matter
more than habits of mind and the justification of one's approach and
results.”
(Wiggins, 1990, ¶ 14). Yet,
more than
ever, we live in a society that is results-oriented.
We must rethink, and we must reform. We have
entered an age
where traditional “proof of validity” in a testing situation is less
important
than the demonstrated results of learning. “Demonstrations of the
technical
validity of standardized tests should not be the issue in the
assessment reform
debate.” We must move on, and away from isolated tests composed of
items that
are stand-ins for true knowledge and experience, and grades that
have no
obvious meaning (to students OR to teachers!) or usefulness. These
not only
harm learning from the student’s standpoint -- they also
weaken the
teacher’s ability to improve instruction and the learner’s ability
to improve
his or her performance. (Wiggins, 1990, ¶ 10).
Recent years have seen much happen in the way of
national
reform policy and legislation, as well as national and state
initiatives to
delineate new curriculum and instruction standards. They have called
attention
to the importance of developing higher cognitive processes and
knowledge.
Interestingly, reform legislation also has set a directive for
development of
methods to evaluate the progress toward meeting national standards —
a much
more elusive goal, and more difficult task. “According to this
legislation,
acceptable assessment procedures should measure cognitive processes
and
knowledge, such as problem solving and analytic thinking and
conceptual
understanding, as well as other important educational outcomes in a
fair and
unbiased manner. Assessment tasks should reflect performance that is
authentic
…” (CLEAR, 2001, ¶ 1). The important
thing that
stands out in this statement is, how do we begin to tie “national
standards” to
an assessment process that focuses on higher cognitive processes,
with so long
a history of NOT doing this? But for me, all of this has been music
to my ears
…finally, indisputable evidence that others, many others, have been
asking the
same questions — who or what will reach the lost in this
generation?
To truly meet the needs of the learner, this
search for
authenticity must “deliver” in at least two respects: The tasks
should be
authentic with respect to the content domain; and the performances
called for
should be related to concrete demands of the student's environment
-- that
is, the task cannot be intended to simply relate to imposed marks
that meet
certain “standardized criteria.”
To really support educational reform, shouldn’t
assessment
also be based upon our knowledge of learning and of how abilities
develop in
the topics we teach? There is an alarming and obvious need for forms
of
measurement that educators can use to actually improve educational
practices.
These methods should enable us to monitor the multifaceted outcomes
of teaching
and learning. We surely know, for instance, that learning is not
merely the
accumulation of unrelated bits of knowledge and skills. We know that
learners
don’t simply absorb information in a passive way. Learning happens
primarily
with the organization and reorganization of
information. This
enables problem solving, complex thinking, transfer of knowledge,
and the
creation of new knowledge.
Effective assessment must therefore be sensitive
to how well
the organization and restructuring of knowledge is proceeding.
(CLEAR, 2001, ¶ 4). The real
question is, how
do we make traditional assessment methods compatible with the things
that
reform requires? Many have argued that assessment may actually offer a
significant tool for education reform! They say that, as traditional
standardized assessment evolves, it will be “complemented” by other
types of
assessment. I question whether changes in traditional standardized
strategies
won’t actually increase the complexity of assessment
procedures, but
this remains to be proven. It has been suggested that principles
from cognitive
psychology may possibly provide a vital structure for new forms of
educational assessment. (CLEAR, 2001, ¶
8).
So, what does cognitive psychology -- and
ultimately, cognitive
technique --
have to do with authentic assessment? Since cognitive learning is
demonstrated
through the acquisition and use of knowledge, and that we literally
construct
our own learning, this includes that it is an authentic situation
and outcome.
We must keep in mind that authenticity must be applied to both the
tasks AND
the outcomes being assessed. “A move toward more authentic tasks and
outcomes
thus improves teaching and learning: students have greater clarity
about their
obligations (and are asked to master more engaging tasks), and
teachers can
come to believe that assessment results are both meaningful and
useful for
improving instruction.” (Wiggins, 1990, ¶ 15).
The demand for results-based assessment is
actually driven
by our entry into a high-paced, knowledge-saturated technological
environment.
The arrival of rapidly advancing technologies has altered not only
the face of
learning, but also the methods of evaluation and assessment. This
was further
corroborated in my research . . . David Jonassen and
Heinz
Mandl, editors of Designing Hypermedia for Learning, say that
"if
evaluation is to address the processes as well as the products of
learning with
a new technology, a variety of methods must be used
. . . the
solution to the problems of evaluation is to take a multi-faceted
approach to
evaluation." (Jonassen & Mandl, 1989,
p. 360).
It is my opinion that the world of
hypermedia lends
one very important element to not only enhanced learning, but also
to the
assessment process. That is, that by sheer use of the computer to
study,
research, and learn — the student has literally placed herself on an
equal
level with her workplace counterpart. The computer is the tool that
performs
real tasks, both in learning and in performing work in the work
world. It is
somewhat different from the equal use of paper and pencil, or research
libraries, or any other common element. It is, more than any tool in
prior
history — an extension of ourselves. The assistance of the computer
quickly
becomes a cybernetic "right arm" for the student or the
worker — an
indivisible part of interactivity with the real world, an extension
that
increases meaning in our lives, as much as our hands or our sensory
organs. It
is almost as if the computer, with all its offerings for learning,
is part of
what must be assessed.
Jonassen and Mandl (1989) express that because of
the
personalized nature of learning (and being assessed for that
learning), it
should be realized that "the essential problem of evaluating
highly
interactive systems is in measuring both the quality of the
interaction as well
as the product of learning. Evaluations of hypermedia-based learning
must address
both the process of learning and the outcomes of learning. (Seels
&
Glasgow, 1998, p. 360).
What I see is that the process of learning
and/or
assessment also cannot be divided from the avenue for
processing
. . . and that is the computer.
Authentic Assessment --
Greater Clarity Of Learning Obligations
When Performance Is Directly In Proportion With “Real”
Tasks
Authentic assessment is one assessment
form that
perhaps is the closest to the Jonassen and Mandl proposal. The
concept of
authenticity in assessment means that performance is contemplated
directly in
proportion with "real" tasks. The opposite of authentic
assessment is
the "drill and kill," paper and pencil test (Wiggins, 1990, p. 1, as cited in Seels &
Glasgow, 1998, pp. 88-89), from which learning must
be deduced,
and from which so very much about our academic and professional
lives, are
determined. More than ever, the new Age of Technology calls upon all
of us, as
students, and ultimately, as professionals, to perform what amounts
to authentic
assessment.
Simply because we now are forced by the
advent of
high technology in commerce, to bring the real world tasks of
lifelong learning
into the workplace, as well as expect that students will
increasingly perform
according to professional standards — it is my vision that some form
of
authentic assessment will BE that ideal mode of evaluation that we
seek.
If nothing else, as instructional designers, we
must look
for a form of "test validity" that shows the test to be a
simulation
of real-world analyses of ability. "Authentic tasks involve
‘ill-structured’ challenges and roles that help students rehearse
for the
complex ambiguities of the ‘game’ of adult and professional life.
Traditional
tests are more like drills, assessing static and too-often
arbitrarily discrete
or simplistic elements of those activities."
(Wiggins, 1990,
p. 1, as cited in Seels & Glasgow, 1998,
pp. 88-89).
It is the aim of authentic assessment to evaluate learners'
abilities within
“real-world" contexts.
In other words, students learn how to apply their
skills to
bona fide tasks and projects. “Authentic assessment does not
encourage rote
learning and passive test-taking. Instead, it focuses on students'
analytical
skills; ability to integrate what they learn; creativity; ability to
work
collaboratively; and written and oral expression skills. It values
the learning
process as much as the finished product.” (Pearson
Group, 2000-2001,
¶ 2). Very frequently, this
requires learners
to work in partnership -- these assessments, based on performance,
lead students
to actually apply skills and concepts and to solve quite complex
problems, both
in short- and long-term tasks.
Authentic assessment seeks to assess many various
and
different kinds of literacy abilities in circumstances that closely
resemble
actual conditions in which those abilities are used. For example,
authentic
assessments request students to read real texts, and write for
authentic
purposes about meaningful topics. Both the material and the
assessment
tasks are presented as natural as possible. A critical element is
users’
literacy level; defined as a measure of functional reading level.
Keep in mind
that each of us experiences some degree of illiteracy when dealing
with a
subject that is beyond our present level of knowledge.
“Working on authentic tasks is a useful, engaging
activity
in itself; it becomes an "episode of learning" for the
student. From
the teacher's perspective, teaching to such tasks guarantees that we
are
concentrating on worthwhile skills and strategies. Students are
learning and
practicing how to apply important knowledge and skills for authentic
purposes.
They should not simply recall information . . . they should apply
what they
know to new tasks.” (Houghton Mifflin Co., 1997, ¶ 2).
Grant Wiggins (1990) perhaps said it the most
effectively:
Assessment is authentic when we directly
examine student
performance on worthy intellectual tasks. Traditional assessment, by
contract,
relies on indirect or proxy 'items' -- efficient, simplistic substitutes from which
we think
valid inferences can be made about the student's performance at
those valued
challenges. (¶ 2).
The essence of reform is to speak to the idea
that there is
a more effective assessment alternative. In most traditional
scenarios for
assessment, how well do multiple-choice tests really evaluate student
understanding and achievement? Authentic assessment sets great store
by the
thinking process behind the work -- the process -- as much as the finished product. The RESULTS
that are
so badly needed in our world of high technology are needed within the
processes, as well as in a final product.
Authentic assessment IS the assessment solution
that will
propel advanced stages of the teaching/learning process -- to meet the
needs of the
new millennium.
Vehicles For Authentic
Assessment
When Humans Must Construct Their Own Knowledge,
And Learners Master Only Those Activities They Actually
Practice
Constructivism is not an instructional
approach -- it
is a learning theory. We should really think of it as a way of
"growing" or enhancing instruction. Constructivists put
the learner
at the center of the equation -- with the idea that the learner
constructs
knowledge rather than absorbing it passively. The learner,
each in his
or her own way, constructs meaning based on prior experiences, mental
structures, and beliefs. These have been used to interpret objects
and events.
In many classrooms, the principal training mode
is direct
instruction, with the trainer assuming a central role for transmitting
knowledge to learners. The learner's role is to absorb that
information in a
receptive and compliant way. There is an over-reliance on rote
memorization,
which does not give the learners the skills in how to think and
solve
problems.
But learners will not make use of concepts and
ideas unless
they use them through some form of process. In other words,
learners
master only those activities they actually practice. Please
note -- this
is a supposition in both constructivism and rote learning
environments.
From this, it seems apparent to me that defenders
of the
traditional system ignore the fact that both constructivism and
instructivism
are required, since learners need to be able to solve complex
problems as well
as to be able to understand the reasons or methods they use to reach
their
conclusions. This actually follows Bloom's
Taxonomy, in that it goes from “simple learning to the
higher levels
of critical thinking.” (Clark, Developing
Instruction, 2001,
section Constructivist Theory, ¶ 5). For
a similar
highly developed level of success in assessment, the higher levels
that require
analysis, synthesis, and evaluation will stimulate interesting
interactions
that will show that learners have moved from “fact-finding” and
“knowledge-gathering” to genuine assimilation of content. (Moore,
et. al., 2001, p. 11.23).
Cognitive constructivism is based on a theory of
cognitive
development where humans cannot be "given" information,
and they
immediately --
somehow magically — understand and
use.
Instead, humans must "construct" their own knowledge. They
build
their knowledge through experience. Experiences enable them to
create mental
models in their heads. These “schemas” are -- through direct
experience
--
changed, enlarged, and made more refined through two complimentary
processes:
by being assimilated, and by accommodation. The emphasis is placed
on the student
rather than the teacher. Teachers are seen as facilitators or
coaches
who are there to help students construct their own
conceptualizations and
solutions to problems.
For me, this then is the essence of authenticity
in learning
— and in assessment of the learning. For the youth that I’ve worked
with —
finally! — a solution to deal with utter and deep dissatisfaction
with “the
system.” Finally, a method of looking at their experiences, their
knowledge and
abilities, and having a means to assess their learning, irrespective
of norms
and judgments. “Cognitive constructivism” is based on the idea that
people
learn by actively constructing new knowledge, not by having
information poured
into their heads. “Moreover, constructivism asserts that people
learn with
particular effectiveness when they are engaged in ‘constructing’
personally
meaningful artifacts.“ (Clark,
Constructivism, 2001, section
Developing Instruction, ¶ 5). What this
will mean
to many thousands of at-risk youth, in schools that have truly let
them down so
badly — is astounding to conceive.
To youth like this, the opportunity to succeed at
“the
assessment game” will turn the tide in our schools … for youth of
every
culture, locale, and ability. “Educators who view learning in this
way realize
that quantitative methods of evaluating learners do not
"measure up."
Authentic forms of assessment present a more qualitative and valid
alternative.”
(Kirka, 1995, ¶ 1). "Test
validity" should depend in part upon whether the test simulates
real-world
"tests" of ability." (Wiggins, 1990, ¶ 9).
Systems Theory -- A
Set Of Proven Procedures:
Designing Educational Programs That Work
When Scientific Method Meets Cognitive Technique
Instructional Systems Design (ISD) is one mode
that has been
clearly identified as an integral component of successful
instructional design.
Much of the literature about ISD models deals with training to do
specific
tasks. ISD typically has led to higher learning achievement, a
greater
transfer of learning from the instructional setting to the job, a
reduction in
the time required for learning, and lower per student
costs.
Let’s think about “education” for a moment.
Mostly, we still
use the same “mode” as our grandparents used — the same buildings,
the same
schedule, and the same lecture techniques. Indeed, in the
21st
century scenario, “school” might be the only place that would feel
like
familiar ground to our great-grandparents. But this is changing,
and, of recent
years, is changing very rapidly. One of the things that have brought
about this
change is Instructional Systems Development. Instructional Systems
Development,
or ISD, is a systematic process that follows a set of proven
procedures to
design educational programs that work. (Hannum,
2001).
“Systems theory requires constant self-monitoring
and
adjustment of the system. This is similar to the scientific method,
in that we
formulate hypotheses (designs) and test them, thereby supporting or
altering
our expectations. Formative evaluation is the primary means of doing
this
self-testing; at various stages, designers may try out instructional
materials
to improve their effectiveness. New methods and approaches to
formative
evaluation are based on cognitive assumptions of performance.
Whereas a
system's evaluation in the past tended to focus on learners' success
in performing
the criterion task, cognitive techniques seek to uncover thinking
processes as
they interact with the material.” (Wilson, 1992, section
Formative
Evaluation,
¶ 2).
When The Prime Goals Of ISD Are Clearly Defined
In our era of new media and an almost limitless
amount of
knowledge that is accessible, a new "mind-set" is developing
concerning, for instance, standardized instruction, with an ironclad
description of being "able to deliver instruction the same way
every
day." (Seels & Glasgow,
1998,
p. 18). If we seriously ponder where learning environments are
headed -- down
to the very setting that we, as Capella students, experience -- we
MUST ask if this is STILL a prime goal for instructional design?
“Based on cybernetic principles of general
systems theory,
the ideal design process relies on constant systemic feedback. Such an
instructional system acts something like a thermostat, always
monitoring its
own effectiveness, making revisions as needed to optimize learning
outcomes.
These cycles of self-testing and correction are repeated during the
design
process as well as during implementation and maintenance of the
system. In this
way, ID can adapt to differences in content, setting, learner
characteristics,
and other factors.” (Wilson, 1992, section New ID Methods
and
Technology, ¶ 2).
First, it seems to me that the very character of
ISD lies in
the fact that its history lies OUTSIDE of the educational system.
(Seels &
Glasgow, 1998, p. 18). This, in itself, almost describes the
collaborative
environment of the burgeoning arena of distance or online learning
environments
. . . all of which have been heavily influenced by
ISD, or at
least incorporate the components that make up a successful delivery
system for
instruction, learning, or training.
When we begin to break down the elements of
traditional ISD,
for instance, one glaring commodity is always the first to surface.
"What
will this cost?" This is almost the battle cry of educational
institutions
as they face new technologies and begin to plan for creating and
maintaining
the best possible instructional environments. This is addressed in
the cost
analysis phase. The main question still remains: How may this
instructional
tool be delivered:
·
most cost-effectively;
·
repeatedly; and,
·
to the greatest number of learners? (Seels
&
Glasgow, 1998,
p. 19).
It must be noted that it is in the EVALUATION and
ASSESSMENT
stages where the true perceived raises in cost factor is seen. It
simply is a
more thorough method — and it takes longer. Evaluation plays a major
role in
ISD by providing information to determine whether the learners are
acquiring
the desired knowledge, skills and attitudes, indicating specific
weaknesses for
individual students, determining the effectiveness of the instruction,
determining how well the total instructional system is working and
providing a
means for identifying necessary revisions.
The data collected in the evaluation form the
basis for many
crucial decisions. Thus, care must be taken in collecting these data
to ensure
accuracy and relevance. The purpose of determining whether the
objectives have
been reached is not solely to assign a grade for the learner;
program evaluation
can also help to determine the effectiveness of the educational
program, as
well as the appropriateness of training goals and objectives. (Hannum, 2001).
When
computers provide
an integrated environment for teaching a variety of
disciplines
My first experience with the changing nature of
instructional systems design, as well as the new tendency to
incorporate
training into a Web-based setting, was my a Project Lead position
with Lucent
Technologies, from May to October of 1998. I developed an
interactive Web site
that delivered training and assessment, and informational materials to
technicians, worldwide. The project was derived from instructional
content that
previously was delivered (also via the Internet) in a scrollable PDF
file.
Lucent’s desire was to "excavate" this file, which was
utilized by
few (a long, scrolling document is difficult to control or use), and
develop an
interactive, task-based website.
The overlying goal was to deliver technical
information and
instruction for equipment use, in a navigable form via the Web. The
format
intentionally was never laid out in an ISD manner. Because of the
availability
of the Internet and its research capabilities, this model was never
locked into
"the same instruction each and every time" description.
Yet, all the
while, I could see the possibilities of re-constructing certain
areas of the
content to meet the needs for testing or evaluation. It was
important for me to
see that this large corporation desired to take advantage of the
strengths of
new media, all the while leaving standard ISD possibilities open for
future
expansion of the learning environment.
Further research for this writing revealed that
the change
to Web-based delivery has been constant for quite some time,
worldwide. David
Marshall and Stephen Hurley of the University of Wales, Cardiff,
United
Kingdom, in their paper, "Delivering Hypertext-based Courseware
on the
World-Wide-Web," (1999) state:
The use of computers to provide an integrated
environment
for teaching a variety of disciplines has received much attention in
recent
years. Indeed many frameworks [this writer’s substitution:
models] have
been developed for such purposes. The material provided by such
courseware
varies greatly from the provision of lecture notes and lecture
support material
through to integrated and interactive tutorial packages. Until
recently
courseware has existed as stand alone packages; however, with the
advent of the
World Wide Web (WWW) on the Internet and accompanying WWW
(hypertext) browsers,
such as Netscape . . . the provision of courseware
has taken on
a whole new dimension.
(Introduction, ¶ 1).
Interestingly, this new dimension included one of
their
initial, calculated goals. A primary goal was that the courseware
could be used
to sustain a variety of courses, including undergraduate degree
programs in
computer science, physics, all branches of engineering, mathematics
and
electronics, as well as training courses run by computer service
departments.
The challenge of designing the learning materials for such a diverse
group was
to make the materials accessible to all classes of users, and yet
maintain a
high degree of "specialism." The authors translated
this to
mean extremely pertinent and individualized training materials. (Marshall &
Hurley, 1999).
Whether the learning environment is CBT-based, or
delivered
via Internet/Intranet, the ISD process involved for digital settings
has taken
on new directives that don’t DENY traditional methods but alter them
to fit new
learning means and methods. The IT community (and this will include
most, if
not all, instructional designers) has bent over backward to
accomplish one
goal, which is: The closer to the end user, the more powerful the
return. This
is because when response time goes down, quality goes up and costs
go down. For
the new IT, the first and primary commission is to create a
communications system
able to bring critical information to the end user in real time.
It is ONLY in this way that we can begin to
entertain a hope
that learning will continue even within the setting of the
assessment process —
as well as inside the classroom.
“Whether we are pondering the effectiveness of
the formative
value of online assessment, or evaluating the system itself -- we
no longer have to wonder if authentic assessments can incorporate a
wide
variety of techniques "designed to correspond as closely as
possible to
‘real world’ student experiences." They simply ARE compatible
with adult,
career, and vocational education. (Kirka, 1995, ¶ 1).
Changes In The
Instructional
Designer’s Role
When Researcher And Practitioner Become One
--
Knowledge Of Content And The Architecture Of The Delivery
System
It is my opinion that the line between
Researcher and
Practitioner is becoming hazy. The instructional designer can
no longer
look to previous descriptions for the role of those who develop
curriculum and
assessment, and use only those parameters. Ironically, the
"Practitioner" must then become also a
"Researcher," in
order to maintain a fully developed set of design skills. She or he
must turn
to the delivery system in order to design that which will be
delivered -- the
instructional product. In other words, one must turn to the Internet
to glean
further information, in order to be able to effectively construct a
learning
architecture that will also be Web- or Internet-based. Seels and
Glasgow (1988)
state that "as a designer, you can progress further along your
career path
if you are aware of the research associated with each step."
(p. 25).
Each of us involved in instructional design will
eventually
participate in both roles. If, in the first 40 years of the
discipline of
instructional systems design, (Seels & Glasgow, 1998, p. 25) the processes were divided into two
roles, then
the next 40 years (or less!) will bring the processes closer to a
more
holistic approach — just as the collaborative setting of
learning is
becoming more holistic. Sheer availability will provide much in the
way of
allowing the practitioner to assume more responsibility for
researching both
content and means for more effectively reaching design goals.
Designers are being called upon to be personally
informed
concerning the research leading up to the final product, as well as
knowledgeable in the actual design of the product. To effectively
design the
instructional environment, the designer must be familiar with not
just the
content and the architecture he/she is creating, but also the
architecture of
the delivery system. The reliability of the designed platform is
also the
reliability for all assessment. All of these must be taken into
account.
All of these must be nurtured as skills within ONE designer. This will
strengthen the individual as well as the team effort.
When
Keeping Pace With
Technology Is Crucial
A true reason for concern -- and the
necessity for all
involved in the current state of online learning and assessment -- is
that the goals for learning, learning environments (and thus,
Instructional
Design) are changing as rapidly as the technologies that are driving
them. As
quickly as a conclusive textbook can be copyrighted and distributed,
it has
many portions that are no longer up-to-date. There is a plethora of
information
available, concerning the generic ISD Model and the roles of
researcher and
practitioner as "specialties" of the Instructional
Designer. In the
past few years, I have worked with a number of instructional design
projects.
It is my experience that the current requirements are leaning
heavily toward
Web-based instruction and training. This incorporates a broad view of
"Internet" environments such as a company’s Intranet, as
well as password-protected
Internet-deployed training sites.
When
“Tests” Are
Learning Events
One feature that will stand out in a study of the
action of
assessment is that "research documents that tests are
learning events."
(Seels & Glasgow, 1998, p. 83). It has been my
experience
that, unfortunately, the traditional student rarely sees it this
way! The axiom
and hope of this course of study is that each student will search
out an
individual method for applying the highest level of accuracy and
usability in
their instructional design environments. As a student myself, I see
very good
reasons for NOT appreciating many forms of testing that habitually
have been
applied to a learning curve that, otherwise, was quite gratifying.
We can hope
that, as we watch the very foundation of instructional design
undergo massive
changes by the very nature of the Internet and all of its
possibilities, that
there we’ve witnessed the arrival of a new mind-set concerning
. . . assessment.
There has never been a “perfect” mode of
assessment for learners.
We all must live with the fact that most of the time, a varying
number of
methods must be considered in projects. Instructional design faces
the same
challenges as any other entity, in the face of the rapid changes
that are
happening in digital learning arenas. It must be asked: If there
were a perfect
and all-encompassing method for assessment, and the major problem
with this
particular method was that it was very labor intensive and therefore
costly,
would a suitable argument for using the method be that the increased
gains
would more than offset the increased costs? (Wiggins, 1990,
p. 1, as cited in Seels & Glasgow, 1998,
pp. 88-89).
We have come full-circle, for the answer to this question is hands-
on, experiential
learning settings and assessments. We must address the issues
of
authenticity in learning, and authentic assessment.
In comparing things learned in the study of this
chapter, I
am providing a list of a few of the instructional design projects
with which I
have been involved:
·
Math application for 4th-graders, which was available both via Web-based
technology, as
well as CD-ROM. It included games, animations, video, and many
interactive
elements. It also included automatic assessment of student activities,
providing teachers with instant, printable assessment results.
·
Two interactive "research manual"-
type
software concerning artists involved in the international space
race, and the
Hopi Tribe of the Southwestern U.S. Both were constructed for use by
kiosk,
CD-ROM, and the Web.
·
Curriculum and course design for online
college
courses, utilizing existing documentation. All projects were
constructed in
HTML, and included hypertext as well as many multimedia elements.
·
Task-based "user's manual" for
technicians
worldwide, in the Lucent Technologies communications' equipment
divisions. The
project was entirely Web-based;
·
Interactive "city guide" that
instructed
users concerning events, locations, and diversions in the Denver,
Colorado
area. The project was a kiosk, but could be transformed into a Web-
based
application.
·
Criterion-based learning environments for
assessment
and certification, for business management personnel in the legal
services
fields.
·
Online courses, in both the instructional
design, SME
(subject matter expert) and teacher/facilitator positions, with Art
Institute
Online. Assessment was both qualitative and quantitative in
nature.
This list was to show not only a variety of
instructional
design products, created with a wide variety of tools and methods,
but also to
indicate that — since 1995 — not one project was outside the realm
of the use
of the Internet and World Wide Web. Each learning structure, and all
built-in
offerings for assessment, was literally "outside" the
traditional
descriptors for "assessment." This is centered in one
word: hypertext
(or hypermedia) content. This is an arena where the rules for
assessment
change. The reason for this is that, in systems with enormous
amounts of
information, with assorted formats, one major evaluation problem
would be the
notation of what is accessed. This could be true even in a paper-
based library
system, as well as any digital system. However, in the emerging
electronic
systems that provide high levels of learner control, THE major
evaluation
problem is calculating how learners conduct this control and
freedom, as they
are learning. The next scenario is in the dynamic, collaborative
systems, where
the notable assessment enigma is who does what — as
well as how
it is done. (Jonassen & Mandl, 1989,
p. 360).
Another consideration is assessment and its
relationship
with STANDARDS. I am speaking of basically what has come to be known
as
standardized curriculum, and the emphasis that has been placed in
recent years,
on districts and teachers, for providing useful methods for
comparison on state
or even national levels. This was a very "hot" issue in
the Denver
School District, in the early- to mid-1990s. During the summer of
1993, I
worked with a group of teachers and administrators on re-developing
high school
curriculum items and issues. The project was part of a Perkins
Grant; the team
was extremely hard-working, knowledgeable experts, with a genuine
desire to
complete a prototype for use within the state, for providing all
teachers a
tool for assessing high school subject areas. The primary question,
in the
beginning was: Where To Start?
It must be pointed out that, in many ways, this
project was
doomed to fail. Much of the work was never utilized — not at the
state level —
or even within this particular high school. Recent research has led to
documentation that provided insight into WHY the project wasn't as
successful
as it might have been. The Mid-continent Regional Educational
Laboratory
(McREL) states that the process might begin with using the
"standards
document" produced by one's own state. They show that every
state except
one is developing or has developed state standards. A truly
disturbing research
fact is:
[A] study conducted by the American Federation
of
Teachers (AFT) reports that only 13 of 49 state documents are
specific enough
to be used effectively by teachers. The majority of state documents
describe
standards at levels of generality that do not provide sufficient
clarity for
classroom instruction, nor are they precise enough to serve as an
instrument of
accountability. (Marzano, 1997, section Where Will We Get
Our
Standards, ¶ 1).
My group in 1993 wished — as all schools and
districts wish
— to adopt valid standards. Furthermore, parents want to know how
their
children are performing academically — compared with standards. The
next
suggestion by McREL was to use the national "standards
documents"
such as the ones published by national associations and subject
matter groups.
Unfortunately, McREL points out . . . "these
documents
commonly embed a description of requisite knowledge and skills
within lengthy
descriptions [over 2000 pages and 14 pounds of documents] of
performance
activities, curriculum goals, instructional strategies, and the
like."
(Marzano, 1997, section Where Will We Get Our Standards, ¶ 2).
But, the assessment process is very difficult. In
1993, I
viewed much of the process as either the act of "reinventing the
wheel," or, a commitment to NOT be doing so. In the end, this
resulted in much
going ‘round and ‘round, with much loss of time and effort.
All of my early experiences as a student
indicated to me
that most assessment FEELS like the process is undeniably tied to
the act of
GRADING — it feels strictly judgmental. Seels and Glasgow made the
point that
"when we assess, we estimate or judge the value of a person,
activity, or
situation. When we approach assessment from this point of view, it
becomes
easier to understand that assessing learning is not
equivalent to
grading. While one function of assessment can be to provide a
basis for
grading, that is not its primary function." (Seels &
Glasgow, 1998, p. 82). This is an enlightening
statement. If
this is truly the issue — and, I feel that it should be — then, why
does
assessment almost always have the negative “competitive” overtones of
. . . GRADING?
For the learner, and even for the instructional
designer or
the administering teacher, the assessment process is tied closely to
psychological factors. This research led to a concept called
"knowledge
structure measures," which are based on and enriched by
psychological
theories. The postulation is that information in one's memory is
organized or
"structured" so that it facilitates the storage, recall, and
manipulation or handling of information. Much research shows that
"assessment of knowledge structures may be used to reliably
distinguish
skilled and unskilled performers in a variety of domains." (Rowe
et al., 1997, page 2).
This indicates
that this form of assessment is usable in a variety of settings. It
also
strongly shows that perhaps one very powerful missing element in
traditional
assessment methods may be the internal, psychological aspects of
learning and
evaluating the individual grasp of information.
I propose that it is because the traditional
example
disregards psychological factors, and instead, offers
. . . "drill and kill."
The interesting difference in this example is
that measures
of knowledge structures may be used to advance toward
. . . and
evaluate . . . conceptual understanding for
assessment and
training purposes. In a typical situation, the value of an
individual's
knowledge structure is decided by comparing it to a standard knowledge
structure. This would be a collection of the structures of several
experts.
Ideally, this approach indicates a positive linear relationship
between the
progress toward expert task performance levels, and the development of
knowledge. Each learner is given a "knowledge score" by
calculating
the comparison of his or her knowledge structure against the
standard knowledge
structure. This description sounded very good and reasonable, up to
this point.
Then my research source stated, "a validated knowledge
structure measure
may then be used to extract rich cognitive representations that are
useful for
assessing and diagnosing student knowledge and identifying targets
for training
intervention. This knowledge score may then be validated by
determining how well
it correlates with a recognized performance measure, such as final
exam or
final course grade." (Rowe et al., 1997, page 2).
With great dismay, this had led back to
. . . grading. The search for an elevated form of
assessment
ground to a halt.
This ultimately sounded much like the description
of the CRT
(criterion-referenced test), in which the use of the term
"criterion"
refers to the affinity between the objective and the method of
assessment, and
to the level of performance needed for mastery of a skill or
subject. The CRT
score gives information about a learner's mastery of one specific
skill
relative to the objective, and therefore shows one's ability in
performing
specific competencies. (Seels & Glasgow, 1998,
p. 83). A
CRT score lets everyone know exactly how well the student stands,
relative to a
standard. (p. 84).
But, the workplace is becoming more
technologically complex!
It demands increasingly specialized skills, workers, as well as
students who
will be entering the workplace, must rapidly become skilled in a
broad range of
cognitively complex tasks. The cognitive aspect of a task must be
spoken to,
for purposes of meaningful assessment. Since the traditional method of
assessing the cognitive domain is via a paper and pencil test
(p. 83), are
we then still stuck with grading and "drill and
kill" methods?
Reliability means that a test will
render a
dependable measure, so that if a test is repeated the same results
will be
obtained. Reliable tests have consistency and temporal dependability
(a student
will show the same level of mastery on one day of the week as on
another day).
Reliability is augmented by an architecture with items that are not
ambiguous,
and by scoring the test as objectively as possible. (Seels &
Glasgow, 1998, p. 85). Validity means that a test
will measure
what it is supposed to measure. Without reliability, there can be no
validity.
(p. 85). The best approach for substantiating a test's validity
is to
provisionally demonstrate that the test accurately discerns the
masters from
the non-masters of the test items. This is to say, items that do not
make this
distinction, should be revised or completely dropped from the test.
(p. 86).
Using guidelines from the text and outside
resources, one
can then come up with an improved understanding of assessment
procedures as a
historical process, as well as perhaps mapping out an
"ideal"
evaluation plan. It appears that using a variety of frequent
assessment techniques
in the classroom may be a preferred approach. According to McREL, in
the
traditional classroom, the teacher is responsible for assessing
students on
standards and benchmarks. They are therefore able to utilize a
variety of
techniques, including portfolios, performance tasks, traditional
tests, and
natural observation. There is a decided advantage to this approach,
in that
assessment of standards and benchmarks does not diminish or
interrupt regular
classroom instruction. The assessment process is integrated into the
regular
routine of the classroom. (Marzano, 1997).
It is a holistic and authentic approach to assessment in a real
setting.
Those of us in the broad arena of education MUST
look to the
business world while developing the tools of education and
meaningful forms of
assessment. This is the only manner in which we can train young
learners to
participate in the real time world they will soon inherit. These are
the
reasons why cognitive techniques, the ISD process, and authentic
evaluation and
assessment are so meaningful. We are training today’s learners to
not only take
control of their futures, but also to take control of the “journey,”
the
educational road to that very future. We are creating the
environment that,
through positive learning and assessment modes, these learners will
be more
likely to become LIFELONG learners. And only by instilling the
receptivity and
desire for self-assessment and hands-on, real world learning, will
we be able
to give them this roadmap to the 21st century.
The highly interactive media that is emerging
challenges the
evaluation of instructional products, as well as the learning that
happens
because of these products. This interactivity provides the learner
with access
to untold and diverse information. It gives the user a personal
control over
the process of learning, and it carries a vast potential for
collaboration with
not only the delivery system, but with other people. This tremendous
empowerment of the learner forces evaluators of learning to embrace a
broad-based set of methods and criteria. ONLY this will accommodate
self-directed learning. (Jonassen & Mandl, 1989,
p. 368). To leave this out of the
assessment
process would be to cripple the effort for advancing superior
techniques in
evaluation.
We, as designers, must heed this trend in
learning
evaluation. The Mid-continent Regional Educational Laboratory
(McREL) states it
this way:
Do not hold students accountable for specific
levels of
performance on any standards. This is the system we currently have
in place.
In virtually every state, the only standard
students must
meet to graduate is that they obtain a certain number of
"credits,"
and a credit is earned by obtaining at least a "D" in a
course. The
advantage of this approach is that it is very easy to earn a high
school
diploma. The disadvantage of this approach is that students can
graduate
without acquiring any specific skills and abilities, such as the
abilities to
read, write and compute. (Marzano, 1997, section What Will We Hold Students
Accountable For?
¶ 1).
For years, private and public sectors have cried
out that
"Johnnie cannot read," and they have asked,
"Why?" It has
been the purpose of this writing to show that the product of the
appropriate —
perfect — assessment process will have an onward, spiraling effect
on learners.
Johnnie will not only learn more
effectively, but she ( J ) will
perform at
much higher levels during evaluation.