Authentic Evaluation And Assessment ¾
What Matters More Than Habits Of Mind?
Introduction
This is writing from the heart. It is more of a journey ¾ or a
journal (a portfolio, perhaps?) ¾ than it is a paper for an end result of a grade in a
doctoral program … although it is that, as well. As such, it is written in the
first person, for the journey and the results of this course of study in Fall
of 2001, are very personal … deeply meaningful studies and reflections that
have led to a confirmation of things I’ve experienced in the education field as
a student, an educator, and a development specialist for distance learning.
This journey began in 1992 when I began to work with
inner-city, at-risk youth, in a Denver High School. That is an untruth. I had a
bunch of the hardest, “baddest” of the Gang Boys dumped in my lap. As Media
Director with a heavy art career background, the only thing I knew to do with
these scary kids was this: “OH! You like spray paint cans? Here! What do you
think of THIS art?” And I proceeded to take youth with not a lick of even
keyboarding experience, into the world of computer animation.
These young people barely knew how to string the alphabet
together long enough to make a simple sentence, at least on paper. Yet, their
wisdom and knowledge was incredibly deep, once you were able to wade past the
distrust of the Gringa Library Lady, their culturally-instigated
hatreds, and underneath it all ¾ their history of being told by an elitist school
district that they were stupid and unable to ever achieve anything in life,
simply because they had years of being assessed on a level that had no bearing
to the reality of their lives. Or their brain power. Or their true abilities.
For anyone, much of the underlying reality of true anger is the feeling of
being out of control. For these youth, they’ve never been IN control ¾ and
so, the anger runs deep and it runs true. THEY know they are not stupid ¾but
how to convince the “white world” of that? And eventually, they succumb to that
myth ¾
and then, it becomes reality. It’s the ghetto story.
At that time, I knew nothing of the world of “education,”
nothing of evaluation or learning methods, of curriculum, statistics, norms,
and averages. I had no “vocabulary” to apply to what I saw in these young
people. But, I soon found out that by introducing them to something that they could
do ¾
even without having the “parts and pieces,” such as typing classes or high
grades in classes ¾
and get some very positive feedback, I saw the level of self-image begin to
grow, and grow. Each project became better; some eventually became … astounding.
Soon, these Bad Gang Boys were not hanging with the gang anymore. Quite a few
ended up having a successful high school career. A few went on to college. All
of them were changed. To date, not a one of them is “at-risk.”
I was hooked. What made the difference? I truly believe it
had something to do with a positive experience, not just in a job performed,
but also in the feedback ¾ the method of evaluation ¾ that they received.
Everything I did with these youth was intuitive, rather than things I had
studied for a career. All I could see was that there was intelligence, ability,
and even a desire to DO something with their lives. I had to find a way to get
around their feeling that I was passing judgment on them, especially “academic”
judgment. Years of district and national testing had already done a tremendous
job of convincing these youth of how few methods they had for competing in the
world. Their “reality” was very different from the reality described by
national testing! They needed something else … in drastic measures.
They needed real measures for skills and knowledge that were
relevant to THEIR world.
They needed something authentic.
This writing is about discovering a solution, hidden in a
program of “education.” The true reason I landed “in Education” was that the
education programs, in the early days of Internet and digital tools, were the
only ones giving classes in things like … Multimedia! Digital Art! Authoring
classes for the platform shells for games, kiosks! … and yes, learning
environments. That was in 1995, when I then added architecting classes …
for building the World Wide Web. I began to envision a highly improved method
of reaching and teaching at-risk youth on a global level. I saw that I
could reach them with the message that they could pass a “test,” and not fail.
This was the perfect toolkit for taking on the challenge of proving that
authentic assessment was timely, possible, and cost-effective … for by this
time, I DID have the beginnings of the vocabulary, the “jargon of the trade.”
I found myself in a Master’s program for computer science
and education … and I “met” a man named Grant Wiggins. The puzzle pieces
clinked effortlessly into place. For these bad gang boys and for me ¾
Wiggins provided the description for all the frustration ¾ we have
to do more than lead them “to believe that right answers matter more than
habits of mind and the justification of one's approach and results.”
(Wiggins, 1990, ¶ 14). For these
at-risk youth, giving “right answers” on what amounts to a contrived
instrument, is most-times impossible, but always contrary to their
approach or their results ¾ in their very real “real-world” world.
The reality behind the term, “at-risk,” is a saga unto
itself. But the important thing, in my own journey, has been to arrive at the
revelation that there are solutions. Many of those solutions are found
in the various aspects of what is called authentic tasks, and authentic
evaluation and assessment.
~
Mary B. Harris (1874–1957), U.S. prison administrator
(Bartleby.com, 2001)
When Reformation IS
the Journey
Human beings are emotional creatures, and, as such, their
daily actions revolve around not just skills and abilities to perform, but also
their emotional capacities for managing the positive and negative experiences
in life. Two very big motivators are anger and fear. Unfortunately, these
negative emotions come from very common elements in our lives ¾ from
the entities that form our characters, make assessments and judgments, and dole
out the punishments and rewards. One of these entities, of course, is the
school system, the “nest and nursery” for most of us, for at least twelve years
of our lives.
As young learners, most of what we learn IS in the
hands of the educational system, and much of it has been highly ineffective in
one supreme area: Assessment should primarily support the needs of the
learners. Traditionally, this just has not been so. Beyond any kind of
technical consideration, this is the basis for the move to reform assessment.
(Wiggins, 1990, ¶ 11). With so many
students falling through the gaps, it is an understatement to say that the
system is ineffective. The current outcry is more to the point: It is harmful.
Once learners “learn” (are conditioned or even driven)
to approach the assessment situation with at least fear, and also with
increasing anger ¾
this absolutely affects the outcomes and thus the validity of the intended
testing situation. It then obviously has affected the totality of the learning
experience, and is no longer serving the needs of the learner. Most people who
defend traditional tests “fail to realize that it is the form, not the content
of the test that is harmful to learning … students come to believe that
learning is cramming; teachers come to believe that tests are after-the-fact,
imposed nuisances composed of contrived questions ¾ irrelevant to their
intent and success. Both parties are led to believe that right answers matter
more than habits of mind and the justification of one's approach and results.”
(Wiggins, 1990, ¶ 14). Yet, more than
ever, we live in a society that is results-oriented.
We must rethink, and we must reform. We have entered an age
where traditional “proof of validity” in a testing situation is less important
than the demonstrated results of learning. “Demonstrations of the technical
validity of standardized tests should not be the issue in the assessment reform
debate.” We must move on, and away from isolated tests composed of items that
are stand-ins for true knowledge and experience, and grades that have no
obvious meaning (to students OR to teachers!) or usefulness. These not only
harm learning from the student’s standpoint ¾ they also weaken the
teacher’s ability to improve instruction and the learner’s ability to improve
his or her performance. (Wiggins, 1990, ¶ 10).
Recent years have seen much happen in the way of national
reform policy and legislation, as well as national and state initiatives to
delineate new curriculum and instruction standards. They have called attention
to the importance of developing higher cognitive processes and knowledge.
Interestingly, reform legislation also has set a directive for development of
methods to evaluate the progress toward meeting national standards — a much
more elusive goal, and more difficult task. “According to this legislation,
acceptable assessment procedures should measure cognitive processes and
knowledge, such as problem solving and analytic thinking and conceptual
understanding, as well as other important educational outcomes in a fair and
unbiased manner. Assessment tasks should reflect performance that is authentic
…” (CLEAR, 2001, ¶ 1). The important thing that
stands out in this statement is, how do we begin to tie “national standards” to
an assessment process that focuses on higher cognitive processes, with so long
a history of NOT doing this? But for me, all of this has been music to my ears
…finally, indisputable evidence that others, many others, have been asking the
same questions — who or what will reach the lost in this generation?
To truly meet the needs of the learner, this search for
authenticity must “deliver” in at least two respects: The tasks should be
authentic with respect to the content domain; and the performances called for
should be related to concrete demands of the student's environment ¾ that
is, the task cannot be intended to simply relate to imposed marks that meet
certain “standardized criteria.”
To really support educational reform, shouldn’t assessment
also be based upon our knowledge of learning and of how abilities develop in
the topics we teach? There is an alarming and obvious need for forms of
measurement that educators can use to actually improve educational practices.
These methods should enable us to monitor the multifaceted outcomes of teaching
and learning. We surely know, for instance, that learning is not merely the
accumulation of unrelated bits of knowledge and skills. We know that learners
don’t simply absorb information in a passive way. Learning happens primarily
with the organization and reorganization of information. This
enables problem solving, complex thinking, transfer of knowledge, and the
creation of new knowledge.
Effective assessment must therefore be sensitive to how well
the organization and restructuring of knowledge is proceeding.
(CLEAR, 2001, ¶ 4). The real question is, how
do we make traditional assessment methods compatible with the things that
reform requires? Many have argued that assessment may actually offer a
significant tool for education reform! They say that, as traditional
standardized assessment evolves, it will be “complemented” by other types of
assessment. I question whether changes in traditional standardized strategies
won’t actually increase the complexity of assessment procedures, but
this remains to be proven. It has been suggested that principles from cognitive
psychology may possibly provide a vital structure for new forms of
educational assessment. (CLEAR, 2001, ¶ 8).
So, what does cognitive psychology ¾ and ultimately, cognitive
technique ¾
have to do with authentic assessment? Since cognitive learning is demonstrated
through the acquisition and use of knowledge, and that we literally construct
our own learning, this includes that it is an authentic situation and outcome.
We must keep in mind that authenticity must be applied to both the tasks AND
the outcomes being assessed. “A move toward more authentic tasks and outcomes
thus improves teaching and learning: students have greater clarity about their
obligations (and are asked to master more engaging tasks), and teachers can
come to believe that assessment results are both meaningful and useful for
improving instruction.” (Wiggins, 1990, ¶ 15).
The demand for results-based assessment is actually driven
by our entry into a high-paced, knowledge-saturated technological environment.
The arrival of rapidly advancing technologies has altered not only the face of
learning, but also the methods of evaluation and assessment. This was further
corroborated in my research . . . David Jonassen and Heinz
Mandl, editors of Designing Hypermedia for Learning, say that "if
evaluation is to address the processes as well as the products of learning with
a new technology, a variety of methods must be used . . . the
solution to the problems of evaluation is to take a multi-faceted approach to
evaluation." (Jonassen & Mandl, 1989, p. 360).
It is my opinion that the world of hypermedia lends
one very important element to not only enhanced learning, but also to the
assessment process. That is, that by sheer use of the computer to study,
research, and learn — the student has literally placed herself on an equal
level with her workplace counterpart. The computer is the tool that performs
real tasks, both in learning and in performing work in the work world. It is
somewhat different from the equal use of paper and pencil, or research
libraries, or any other common element. It is, more than any tool in prior
history — an extension of ourselves. The assistance of the computer quickly
becomes a cybernetic "right arm" for the student or the worker — an
indivisible part of interactivity with the real world, an extension that
increases meaning in our lives, as much as our hands or our sensory organs. It
is almost as if the computer, with all its offerings for learning, is part of
what must be assessed.
Jonassen and Mandl (1989) express that because of the
personalized nature of learning (and being assessed for that learning), it
should be realized that "the essential problem of evaluating highly
interactive systems is in measuring both the quality of the interaction as well
as the product of learning. Evaluations of hypermedia-based learning must address
both the process of learning and the outcomes of learning. (Seels &
Glasgow, 1998, p. 360).
What I see is that the process of learning and/or
assessment also cannot be divided from the avenue for processing
. . . and that is the computer.
Authentic Assessment ¾
Greater Clarity Of Learning Obligations
When Performance Is Directly In Proportion With “Real” Tasks
Authentic assessment is one assessment form that
perhaps is the closest to the Jonassen and Mandl proposal. The concept of
authenticity in assessment means that performance is contemplated directly in
proportion with "real" tasks. The opposite of authentic assessment is
the "drill and kill," paper and pencil test (Wiggins, 1990, p. 1, as cited in Seels &
Glasgow, 1998, pp. 88-89), from which learning must be deduced,
and from which so very much about our academic and professional lives, are
determined. More than ever, the new Age of Technology calls upon all of us, as
students, and ultimately, as professionals, to perform what amounts to authentic
assessment.
Simply because we now are forced by the advent of
high technology in commerce, to bring the real world tasks of lifelong learning
into the workplace, as well as expect that students will increasingly perform
according to professional standards — it is my vision that some form of
authentic assessment will BE that ideal mode of evaluation that we seek.
If nothing else, as instructional designers, we must look
for a form of "test validity" that shows the test to be a simulation
of real-world analyses of ability. "Authentic tasks involve
‘ill-structured’ challenges and roles that help students rehearse for the
complex ambiguities of the ‘game’ of adult and professional life. Traditional
tests are more like drills, assessing static and too-often arbitrarily discrete
or simplistic elements of those activities." (Wiggins, 1990,
p. 1, as cited in Seels & Glasgow, 1998, pp. 88-89).
It is the aim of authentic assessment to evaluate learners' abilities within
“real-world" contexts.
In other words, students learn how to apply their skills to
bona fide tasks and projects. “Authentic assessment does not encourage rote
learning and passive test-taking. Instead, it focuses on students' analytical
skills; ability to integrate what they learn; creativity; ability to work
collaboratively; and written and oral expression skills. It values the learning
process as much as the finished product.” (Pearson Group, 2000-2001,
¶ 2). Very frequently, this requires learners
to work in partnership ¾ these assessments, based on performance, lead students
to actually apply skills and concepts and to solve quite complex problems, both
in short- and long-term tasks.
Authentic assessment seeks to assess many various and
different kinds of literacy abilities in circumstances that closely resemble
actual conditions in which those abilities are used. For example, authentic
assessments request students to read real texts, and write for authentic
purposes about meaningful topics. Both the material and the assessment
tasks are presented as natural as possible. A critical element is users’
literacy level; defined as a measure of functional reading level. Keep in mind
that each of us experiences some degree of illiteracy when dealing with a
subject that is beyond our present level of knowledge.
“Working on authentic tasks is a useful, engaging activity
in itself; it becomes an "episode of learning" for the student. From
the teacher's perspective, teaching to such tasks guarantees that we are
concentrating on worthwhile skills and strategies. Students are learning and
practicing how to apply important knowledge and skills for authentic purposes.
They should not simply recall information . . . they should apply what they
know to new tasks.” (Houghton Mifflin Co., 1997, ¶ 2).
Grant Wiggins (1990) perhaps said it the most effectively:
Assessment is authentic when we directly examine student
performance on worthy intellectual tasks. Traditional assessment, by contract,
relies on indirect or proxy 'items' ¾ efficient, simplistic substitutes from which we think
valid inferences can be made about the student's performance at those valued
challenges. (¶ 2).
The essence of reform is to speak to the idea that there is
a more effective assessment alternative. In most traditional scenarios for
assessment, how well do multiple-choice tests really evaluate student
understanding and achievement? Authentic assessment sets great store by the
thinking process behind the work ¾ the process ¾ as much as the finished product. The RESULTS that are
so badly needed in our world of high technology are needed within the
processes, as well as in a final product.
Authentic assessment IS the assessment solution that will
propel advanced stages of the teaching/learning process ¾ to meet the needs of the
new millennium.
Vehicles For Authentic Assessment
When Humans Must Construct Their Own Knowledge,
And Learners Master Only Those Activities They Actually Practice
Constructivism is not an instructional approach ¾ it
is a learning theory. We should really think of it as a way of
"growing" or enhancing instruction. Constructivists put the learner
at the center of the equation ¾ with the idea that the learner constructs
knowledge rather than absorbing it passively. The learner, each in his
or her own way, constructs meaning based on prior experiences, mental
structures, and beliefs. These have been used to interpret objects and events.
In many classrooms, the principal training mode is direct
instruction, with the trainer assuming a central role for transmitting
knowledge to learners. The learner's role is to absorb that information in a
receptive and compliant way. There is an over-reliance on rote memorization,
which does not give the learners the skills in how to think and solve
problems.
But learners will not make use of concepts and ideas unless
they use them through some form of process. In other words, learners
master only those activities they actually practice. Please note ¾ this
is a supposition in both constructivism and rote learning environments.
From this, it seems apparent to me that defenders of the
traditional system ignore the fact that both constructivism and instructivism
are required, since learners need to be able to solve complex problems as well
as to be able to understand the reasons or methods they use to reach their
conclusions. This actually follows Bloom's
Taxonomy, in that it goes from “simple learning to the higher levels
of critical thinking.” (Clark, Developing Instruction, 2001,
section Constructivist Theory, ¶ 5). For a similar
highly developed level of success in assessment, the higher levels that require
analysis, synthesis, and evaluation will stimulate interesting interactions
that will show that learners have moved from “fact-finding” and
“knowledge-gathering” to genuine assimilation of content. (Moore,
et. al., 2001, p. 11.23).
Cognitive constructivism is based on a theory of cognitive
development where humans cannot be "given" information, and they
immediately ¾
somehow magically — understand and use.
Instead, humans must "construct" their own knowledge. They build
their knowledge through experience. Experiences enable them to create mental
models in their heads. These “schemas” are ¾ through direct experience
¾
changed, enlarged, and made more refined through two complimentary processes:
by being assimilated, and by accommodation. The emphasis is placed on the student
rather than the teacher. Teachers are seen as facilitators or coaches
who are there to help students construct their own conceptualizations and
solutions to problems.
For me, this then is the essence of authenticity in learning
— and in assessment of the learning. For the youth that I’ve worked with —
finally! — a solution to deal with utter and deep dissatisfaction with “the
system.” Finally, a method of looking at their experiences, their knowledge and
abilities, and having a means to assess their learning, irrespective of norms
and judgments. “Cognitive constructivism” is based on the idea that people
learn by actively constructing new knowledge, not by having information poured
into their heads. “Moreover, constructivism asserts that people learn with
particular effectiveness when they are engaged in ‘constructing’ personally
meaningful artifacts.“ (Clark, Constructivism, 2001, section
Developing Instruction, ¶ 5). What this will mean
to many thousands of at-risk youth, in schools that have truly let them down so
badly — is astounding to conceive.
To youth like this, the opportunity to succeed at “the
assessment game” will turn the tide in our schools … for youth of every
culture, locale, and ability. “Educators who view learning in this way realize
that quantitative methods of evaluating learners do not "measure up."
Authentic forms of assessment present a more qualitative and valid alternative.”
(Kirka, 1995, ¶ 1). "Test
validity" should depend in part upon whether the test simulates real-world
"tests" of ability." (Wiggins, 1990, ¶ 9).
Systems Theory ¾ A
Set Of Proven Procedures:
Designing Educational Programs That Work
When Scientific Method Meets Cognitive Technique
Instructional Systems Design (ISD) is one mode that has been
clearly identified as an integral component of successful instructional design.
Much of the literature about ISD models deals with training to do specific
tasks. ISD typically has led to higher learning achievement, a greater
transfer of learning from the instructional setting to the job, a reduction in
the time required for learning, and lower per student costs.
Let’s think about “education” for a moment. Mostly, we still
use the same “mode” as our grandparents used — the same buildings, the same
schedule, and the same lecture techniques. Indeed, in the 21st
century scenario, “school” might be the only place that would feel like
familiar ground to our great-grandparents. But this is changing, and, of recent
years, is changing very rapidly. One of the things that have brought about this
change is Instructional Systems Development. Instructional Systems Development,
or ISD, is a systematic process that follows a set of proven procedures to
design educational programs that work. (Hannum,
2001).
“Systems theory requires constant self-monitoring and
adjustment of the system. This is similar to the scientific method, in that we
formulate hypotheses (designs) and test them, thereby supporting or altering
our expectations. Formative evaluation is the primary means of doing this
self-testing; at various stages, designers may try out instructional materials
to improve their effectiveness. New methods and approaches to formative
evaluation are based on cognitive assumptions of performance. Whereas a
system's evaluation in the past tended to focus on learners' success in performing
the criterion task, cognitive techniques seek to uncover thinking processes as
they interact with the material.” (Wilson, 1992, section Formative
Evaluation,
¶ 2).
When The Prime Goals Of ISD Are Clearly Defined
In our era of new media and an almost limitless amount of
knowledge that is accessible, a new "mind-set" is developing
concerning, for instance, standardized instruction, with an ironclad
description of being "able to deliver instruction the same way every
day." (Seels & Glasgow, 1998,
p. 18). If we seriously ponder where learning environments are headed ¾ down
to the very setting that we, as Capella students, experience ¾ we
MUST ask if this is STILL a prime goal for instructional design?
“Based on cybernetic principles of general systems theory,
the ideal design process relies on constant systemic feedback. Such an
instructional system acts something like a thermostat, always monitoring its
own effectiveness, making revisions as needed to optimize learning outcomes.
These cycles of self-testing and correction are repeated during the design
process as well as during implementation and maintenance of the system. In this
way, ID can adapt to differences in content, setting, learner characteristics,
and other factors.” (Wilson, 1992, section New ID Methods and
Technology, ¶ 2).
First, it seems to me that the very character of ISD lies in
the fact that its history lies OUTSIDE of the educational system. (Seels &
Glasgow, 1998, p. 18). This, in itself, almost describes the collaborative
environment of the burgeoning arena of distance or online learning environments
. . . all of which have been heavily influenced by ISD, or at
least incorporate the components that make up a successful delivery system for
instruction, learning, or training.
When we begin to break down the elements of traditional ISD,
for instance, one glaring commodity is always the first to surface. "What
will this cost?" This is almost the battle cry of educational institutions
as they face new technologies and begin to plan for creating and maintaining
the best possible instructional environments. This is addressed in the cost
analysis phase. The main question still remains: How may this instructional
tool be delivered:
·
most cost-effectively;
·
repeatedly; and,
·
to the greatest number of learners? (Seels &
Glasgow, 1998, p. 19).
It must be noted that it is in the EVALUATION and ASSESSMENT
stages where the true perceived raises in cost factor is seen. It simply is a
more thorough method — and it takes longer. Evaluation plays a major role in
ISD by providing information to determine whether the learners are acquiring
the desired knowledge, skills and attitudes, indicating specific weaknesses for
individual students, determining the effectiveness of the instruction,
determining how well the total instructional system is working and providing a
means for identifying necessary revisions.
The data collected in the evaluation form the basis for many
crucial decisions. Thus, care must be taken in collecting these data to ensure
accuracy and relevance. The purpose of determining whether the objectives have
been reached is not solely to assign a grade for the learner; program evaluation
can also help to determine the effectiveness of the educational program, as
well as the appropriateness of training goals and objectives. (Hannum, 2001).
When computers provide
an integrated environment for teaching a variety of disciplines
My first experience with the changing nature of
instructional systems design, as well as the new tendency to incorporate
training into a Web-based setting, was my a Project Lead position with Lucent
Technologies, from May to October of 1998. I developed an interactive Web site
that delivered training and assessment, and informational materials to
technicians, worldwide. The project was derived from instructional content that
previously was delivered (also via the Internet) in a scrollable PDF file.
Lucent’s desire was to "excavate" this file, which was utilized by
few (a long, scrolling document is difficult to control or use), and develop an
interactive, task-based website.
The overlying goal was to deliver technical information and
instruction for equipment use, in a navigable form via the Web. The format
intentionally was never laid out in an ISD manner. Because of the availability
of the Internet and its research capabilities, this model was never locked into
"the same instruction each and every time" description. Yet, all the
while, I could see the possibilities of re-constructing certain areas of the
content to meet the needs for testing or evaluation. It was important for me to
see that this large corporation desired to take advantage of the strengths of
new media, all the while leaving standard ISD possibilities open for future
expansion of the learning environment.
Further research for this writing revealed that the change
to Web-based delivery has been constant for quite some time, worldwide. David
Marshall and Stephen Hurley of the University of Wales, Cardiff, United
Kingdom, in their paper, "Delivering Hypertext-based Courseware on the
World-Wide-Web," (1999) state:
The use of computers to provide an integrated environment
for teaching a variety of disciplines has received much attention in recent
years. Indeed many frameworks [this writer’s substitution: models] have
been developed for such purposes. The material provided by such courseware
varies greatly from the provision of lecture notes and lecture support material
through to integrated and interactive tutorial packages. Until recently
courseware has existed as stand alone packages; however, with the advent of the
World Wide Web (WWW) on the Internet and accompanying WWW (hypertext) browsers,
such as Netscape . . . the provision of courseware has taken on
a whole new dimension.
(Introduction, ¶ 1).
Interestingly, this new dimension included one of their
initial, calculated goals. A primary goal was that the courseware could be used
to sustain a variety of courses, including undergraduate degree programs in
computer science, physics, all branches of engineering, mathematics and
electronics, as well as training courses run by computer service departments.
The challenge of designing the learning materials for such a diverse group was
to make the materials accessible to all classes of users, and yet maintain a
high degree of "specialism." The authors translated this to
mean extremely pertinent and individualized training materials. (Marshall & Hurley, 1999).
Whether the learning environment is CBT-based, or delivered
via Internet/Intranet, the ISD process involved for digital settings has taken
on new directives that don’t DENY traditional methods but alter them to fit new
learning means and methods. The IT community (and this will include most, if
not all, instructional designers) has bent over backward to accomplish one
goal, which is: The closer to the end user, the more powerful the return. This
is because when response time goes down, quality goes up and costs go down. For
the new IT, the first and primary commission is to create a communications system
able to bring critical information to the end user in real time.
It is ONLY in this way that we can begin to entertain a hope
that learning will continue even within the setting of the assessment process —
as well as inside the classroom.
“Whether we are pondering the effectiveness of the formative
value of online assessment, or evaluating the system itself ¾ we
no longer have to wonder if authentic assessments can incorporate a wide
variety of techniques "designed to correspond as closely as possible to
‘real world’ student experiences." They simply ARE compatible with adult,
career, and vocational education. (Kirka, 1995, ¶ 1).
Changes In The Instructional
Designer’s Role
When Researcher And Practitioner Become One ¾
Knowledge Of Content And The Architecture Of The Delivery System
It is my opinion that the line between Researcher and
Practitioner is becoming hazy. The instructional designer can no longer
look to previous descriptions for the role of those who develop curriculum and
assessment, and use only those parameters. Ironically, the
"Practitioner" must then become also a "Researcher," in
order to maintain a fully developed set of design skills. She or he must turn
to the delivery system in order to design that which will be delivered ¾ the
instructional product. In other words, one must turn to the Internet to glean
further information, in order to be able to effectively construct a learning
architecture that will also be Web- or Internet-based. Seels and Glasgow (1988)
state that "as a designer, you can progress further along your career path
if you are aware of the research associated with each step." (p. 25).
Each of us involved in instructional design will eventually
participate in both roles. If, in the first 40 years of the discipline of
instructional systems design, (Seels & Glasgow, 1998, p. 25) the processes were divided into two roles, then
the next 40 years (or less!) will bring the processes closer to a more
holistic approach — just as the collaborative setting of learning is
becoming more holistic. Sheer availability will provide much in the way of
allowing the practitioner to assume more responsibility for researching both
content and means for more effectively reaching design goals.
Designers are being called upon to be personally informed
concerning the research leading up to the final product, as well as
knowledgeable in the actual design of the product. To effectively design the
instructional environment, the designer must be familiar with not just the
content and the architecture he/she is creating, but also the architecture of
the delivery system. The reliability of the designed platform is also the
reliability for all assessment. All of these must be taken into account.
All of these must be nurtured as skills within ONE designer. This will
strengthen the individual as well as the team effort.
When Keeping Pace With
Technology Is Crucial
A true reason for concern ¾ and the necessity for all
involved in the current state of online learning and assessment ¾ is
that the goals for learning, learning environments (and thus, Instructional
Design) are changing as rapidly as the technologies that are driving them. As
quickly as a conclusive textbook can be copyrighted and distributed, it has
many portions that are no longer up-to-date. There is a plethora of information
available, concerning the generic ISD Model and the roles of researcher and
practitioner as "specialties" of the Instructional Designer. In the
past few years, I have worked with a number of instructional design projects.
It is my experience that the current requirements are leaning heavily toward
Web-based instruction and training. This incorporates a broad view of
"Internet" environments such as a company’s Intranet, as well as password-protected
Internet-deployed training sites.
When “Tests” Are
Learning Events
One feature that will stand out in a study of the action of
assessment is that "research documents that tests are learning events."
(Seels & Glasgow, 1998, p. 83). It has been my experience
that, unfortunately, the traditional student rarely sees it this way! The axiom
and hope of this course of study is that each student will search out an
individual method for applying the highest level of accuracy and usability in
their instructional design environments. As a student myself, I see very good
reasons for NOT appreciating many forms of testing that habitually have been
applied to a learning curve that, otherwise, was quite gratifying. We can hope
that, as we watch the very foundation of instructional design undergo massive
changes by the very nature of the Internet and all of its possibilities, that
there we’ve witnessed the arrival of a new mind-set concerning
. . . assessment.
There has never been a “perfect” mode of assessment for learners.
We all must live with the fact that most of the time, a varying number of
methods must be considered in projects. Instructional design faces the same
challenges as any other entity, in the face of the rapid changes that are
happening in digital learning arenas. It must be asked: If there were a perfect
and all-encompassing method for assessment, and the major problem with this
particular method was that it was very labor intensive and therefore costly,
would a suitable argument for using the method be that the increased gains
would more than offset the increased costs? (Wiggins, 1990,
p. 1, as cited in Seels & Glasgow, 1998, pp. 88-89).
We have come full-circle, for the answer to this question is hands-on, experiential
learning settings and assessments. We must address the issues of
authenticity in learning, and authentic assessment.
In comparing things learned in the study of this chapter, I
am providing a list of a few of the instructional design projects with which I
have been involved:
·
Math application for 4th-graders, which was available both via Web-based technology, as
well as CD-ROM. It included games, animations, video, and many interactive
elements. It also included automatic assessment of student activities,
providing teachers with instant, printable assessment results.
·
Two interactive "research manual"-type
software concerning artists involved in the international space race, and the
Hopi Tribe of the Southwestern U.S. Both were constructed for use by kiosk,
CD-ROM, and the Web.
·
Curriculum and course design for online college
courses, utilizing existing documentation. All projects were constructed in
HTML, and included hypertext as well as many multimedia elements.
·
Task-based "user's manual" for technicians
worldwide, in the Lucent Technologies communications' equipment divisions. The
project was entirely Web-based;
·
Interactive "city guide" that instructed
users concerning events, locations, and diversions in the Denver, Colorado
area. The project was a kiosk, but could be transformed into a Web-based
application.
·
Criterion-based learning environments for assessment
and certification, for business management personnel in the legal services
fields.
·
Online courses, in both the instructional design, SME
(subject matter expert) and teacher/facilitator positions, with Art Institute
Online. Assessment was both qualitative and quantitative in nature.
This list was to show not only a variety of instructional
design products, created with a wide variety of tools and methods, but also to
indicate that — since 1995 — not one project was outside the realm of the use
of the Internet and World Wide Web. Each learning structure, and all built-in
offerings for assessment, was literally "outside" the traditional
descriptors for "assessment." This is centered in one word: hypertext
(or hypermedia) content. This is an arena where the rules for assessment
change. The reason for this is that, in systems with enormous amounts of
information, with assorted formats, one major evaluation problem would be the
notation of what is accessed. This could be true even in a paper-based library
system, as well as any digital system. However, in the emerging electronic
systems that provide high levels of learner control, THE major evaluation
problem is calculating how learners conduct this control and freedom, as they
are learning. The next scenario is in the dynamic, collaborative systems, where
the notable assessment enigma is who does what — as well as how
it is done. (Jonassen & Mandl, 1989, p. 360).
Another consideration is assessment and its relationship
with STANDARDS. I am speaking of basically what has come to be known as
standardized curriculum, and the emphasis that has been placed in recent years,
on districts and teachers, for providing useful methods for comparison on state
or even national levels. This was a very "hot" issue in the Denver
School District, in the early- to mid-1990s. During the summer of 1993, I
worked with a group of teachers and administrators on re-developing high school
curriculum items and issues. The project was part of a Perkins Grant; the team
was extremely hard-working, knowledgeable experts, with a genuine desire to
complete a prototype for use within the state, for providing all teachers a
tool for assessing high school subject areas. The primary question, in the
beginning was: Where To Start?
It must be pointed out that, in many ways, this project was
doomed to fail. Much of the work was never utilized — not at the state level —
or even within this particular high school. Recent research has led to
documentation that provided insight into WHY the project wasn't as successful
as it might have been. The Mid-continent Regional Educational Laboratory
(McREL) states that the process might begin with using the "standards
document" produced by one's own state. They show that every state except
one is developing or has developed state standards. A truly disturbing research
fact is:
[A] study conducted by the American Federation of
Teachers (AFT) reports that only 13 of 49 state documents are specific enough
to be used effectively by teachers. The majority of state documents describe
standards at levels of generality that do not provide sufficient clarity for
classroom instruction, nor are they precise enough to serve as an instrument of
accountability. (Marzano, 1997, section Where Will We Get Our
Standards, ¶ 1).
My group in 1993 wished — as all schools and districts wish
— to adopt valid standards. Furthermore, parents want to know how their
children are performing academically — compared with standards. The next
suggestion by McREL was to use the national "standards documents"
such as the ones published by national associations and subject matter groups.
Unfortunately, McREL points out . . . "these documents
commonly embed a description of requisite knowledge and skills within lengthy
descriptions [over 2000 pages and 14 pounds of documents] of performance
activities, curriculum goals, instructional strategies, and the like."
(Marzano, 1997, section Where Will We Get Our Standards, ¶ 2).
But, the assessment process is very difficult. In 1993, I
viewed much of the process as either the act of "reinventing the
wheel," or, a commitment to NOT be doing so. In the end, this resulted in much
going ‘round and ‘round, with much loss of time and effort.
All of my early experiences as a student indicated to me
that most assessment FEELS like the process is undeniably tied to the act of
GRADING — it feels strictly judgmental. Seels and Glasgow made the point that
"when we assess, we estimate or judge the value of a person, activity, or
situation. When we approach assessment from this point of view, it becomes
easier to understand that assessing learning is not equivalent to
grading. While one function of assessment can be to provide a basis for
grading, that is not its primary function." (Seels &
Glasgow, 1998, p. 82). This is an enlightening statement. If
this is truly the issue — and, I feel that it should be — then, why does
assessment almost always have the negative “competitive” overtones of
. . . GRADING?
For the learner, and even for the instructional designer or
the administering teacher, the assessment process is tied closely to
psychological factors. This research led to a concept called "knowledge
structure measures," which are based on and enriched by psychological
theories. The postulation is that information in one's memory is organized or
"structured" so that it facilitates the storage, recall, and
manipulation or handling of information. Much research shows that
"assessment of knowledge structures may be used to reliably distinguish
skilled and unskilled performers in a variety of domains." (Rowe
et al., 1997, page 2). This indicates
that this form of assessment is usable in a variety of settings. It also
strongly shows that perhaps one very powerful missing element in traditional
assessment methods may be the internal, psychological aspects of learning and
evaluating the individual grasp of information.
I propose that it is because the traditional example
disregards psychological factors, and instead, offers
. . . "drill and kill."
The interesting difference in this example is that measures
of knowledge structures may be used to advance toward . . . and
evaluate . . . conceptual understanding for assessment and
training purposes. In a typical situation, the value of an individual's
knowledge structure is decided by comparing it to a standard knowledge
structure. This would be a collection of the structures of several experts.
Ideally, this approach indicates a positive linear relationship between the
progress toward expert task performance levels, and the development of
knowledge. Each learner is given a "knowledge score" by calculating
the comparison of his or her knowledge structure against the standard knowledge
structure. This description sounded very good and reasonable, up to this point.
Then my research source stated, "a validated knowledge structure measure
may then be used to extract rich cognitive representations that are useful for
assessing and diagnosing student knowledge and identifying targets for training
intervention. This knowledge score may then be validated by determining how well
it correlates with a recognized performance measure, such as final exam or
final course grade." (Rowe et al., 1997, page 2).
With great dismay, this had led back to
. . . grading. The search for an elevated form of assessment
ground to a halt.
This ultimately sounded much like the description of the CRT
(criterion-referenced test), in which the use of the term "criterion"
refers to the affinity between the objective and the method of assessment, and
to the level of performance needed for mastery of a skill or subject. The CRT
score gives information about a learner's mastery of one specific skill
relative to the objective, and therefore shows one's ability in performing
specific competencies. (Seels & Glasgow, 1998, p. 83). A
CRT score lets everyone know exactly how well the student stands, relative to a
standard. (p. 84).
But, the workplace is becoming more technologically complex!
It demands increasingly specialized skills, workers, as well as students who
will be entering the workplace, must rapidly become skilled in a broad range of
cognitively complex tasks. The cognitive aspect of a task must be spoken to,
for purposes of meaningful assessment. Since the traditional method of
assessing the cognitive domain is via a paper and pencil test (p. 83), are
we then still stuck with grading and "drill and kill" methods?
Reliability means that a test will render a
dependable measure, so that if a test is repeated the same results will be
obtained. Reliable tests have consistency and temporal dependability (a student
will show the same level of mastery on one day of the week as on another day).
Reliability is augmented by an architecture with items that are not ambiguous,
and by scoring the test as objectively as possible. (Seels &
Glasgow, 1998, p. 85). Validity means that a test will measure
what it is supposed to measure. Without reliability, there can be no validity.
(p. 85). The best approach for substantiating a test's validity is to
provisionally demonstrate that the test accurately discerns the masters from
the non-masters of the test items. This is to say, items that do not make this
distinction, should be revised or completely dropped from the test.
(p. 86).
Using guidelines from the text and outside resources, one
can then come up with an improved understanding of assessment procedures as a
historical process, as well as perhaps mapping out an "ideal"
evaluation plan. It appears that using a variety of frequent assessment techniques
in the classroom may be a preferred approach. According to McREL, in the
traditional classroom, the teacher is responsible for assessing students on
standards and benchmarks. They are therefore able to utilize a variety of
techniques, including portfolios, performance tasks, traditional tests, and
natural observation. There is a decided advantage to this approach, in that
assessment of standards and benchmarks does not diminish or interrupt regular
classroom instruction. The assessment process is integrated into the regular
routine of the classroom. (Marzano, 1997).
It is a holistic and authentic approach to assessment in a real setting.
Those of us in the broad arena of education MUST look to the
business world while developing the tools of education and meaningful forms of
assessment. This is the only manner in which we can train young learners to
participate in the real time world they will soon inherit. These are the
reasons why cognitive techniques, the ISD process, and authentic evaluation and
assessment are so meaningful. We are training today’s learners to not only take
control of their futures, but also to take control of the “journey,” the
educational road to that very future. We are creating the environment that,
through positive learning and assessment modes, these learners will be more
likely to become LIFELONG learners. And only by instilling the receptivity and
desire for self-assessment and hands-on, real world learning, will we be able
to give them this roadmap to the 21st century.
The highly interactive media that is emerging challenges the
evaluation of instructional products, as well as the learning that happens
because of these products. This interactivity provides the learner with access
to untold and diverse information. It gives the user a personal control over
the process of learning, and it carries a vast potential for collaboration with
not only the delivery system, but with other people. This tremendous
empowerment of the learner forces evaluators of learning to embrace a
broad-based set of methods and criteria. ONLY this will accommodate
self-directed learning. (Jonassen & Mandl, 1989, p. 368). To leave this out of the assessment
process would be to cripple the effort for advancing superior techniques in
evaluation.
We, as designers, must heed this trend in learning
evaluation. The Mid-continent Regional Educational Laboratory (McREL) states it
this way:
Do not hold students accountable for specific levels of
performance on any standards. This is the system we currently have in place.
In virtually every state, the only standard students must
meet to graduate is that they obtain a certain number of "credits,"
and a credit is earned by obtaining at least a "D" in a course. The
advantage of this approach is that it is very easy to earn a high school
diploma. The disadvantage of this approach is that students can graduate
without acquiring any specific skills and abilities, such as the abilities to
read, write and compute. (Marzano, 1997, section What Will We Hold Students Accountable For?
¶ 1).
For years, private and public sectors have cried out that
"Johnnie cannot read," and they have asked, "Why?" It has
been the purpose of this writing to show that the product of the appropriate —
perfect — assessment process will have an onward, spiraling effect on learners.
Johnnie will not only learn more effectively, but she ( J ) will perform at
much higher levels during evaluation.