This is writing
from the heart. It is more of a journey ¾ or a journal (a portfolio, perhaps?) ¾ than it is a paper for an end result of a
grade in a doctoral program … although it is that, as well. As such, it is
written in the first person, for the journey and the results of this course of
study in Fall of 2001, are very personal … deeply meaningful studies and
reflections that have led to a confirmation of things I’ve experienced in the
education field as a student, an educator, and a development specialist for
distance learning.
This journey began
in 1992 when I began to work with inner-city, at-risk youth, in a Denver High
School. That is an untruth. I had a bunch of the hardest, “baddest” of the Gang
Boys dumped in my lap. As Media Director with a heavy art career background,
the only thing I knew to do with these scary kids was this: “OH! You like spray
paint cans? Here! What do you think of THIS art?” And I proceeded to take youth
with not a lick of even keyboarding experience, into the world of computer
animation.
These young people
barely knew how to string the alphabet together long enough to make a simple
sentence, at least on paper. Yet, their wisdom and knowledge was incredibly
deep, once you were able to wade past the distrust of the Gringa Library
Lady, their culturally-instigated hatreds, and underneath it all ¾ their history of being told by an elitist
school district that they were stupid and unable to ever achieve anything in
life, simply because they had years of being assessed on a level that had no
bearing to the reality of their lives. Or their brain power. Or their true
abilities. For anyone, much of the underlying reality of true anger is the
feeling of being out of control. For these youth, they’ve never been IN control
¾ and so, the anger runs deep and it runs
true. THEY know they are not stupid ¾but how to convince the “white world” of
that? And eventually, they succumb to that myth ¾ and then, it becomes reality. It’s the ghetto story.
At that time, I
knew nothing of the world of “education,” nothing of evaluation or learning
methods, of curriculum, statistics, norms and averages. I had no “vocabulary”
to apply to what I saw in these young people. But, I soon found out that by
introducing them to something that they could do ¾ even without having the “parts and pieces,”
such as typing classes or high grades in classes ¾ and get some very positive feedback, I saw
the level of self image begin to grow, and grow. Each project became better;
some eventually became … astounding. Soon, these Bad Gang Boys were not
hanging with the gang anymore. Quite a few ended up having a successful high
school career. A few went on to college. All of them were changed. To
date, not a one of them is “at-risk.”
I was hooked. What
made the difference? I truly believe it had something to do with a positive
experience, not just in a job performed, but in the feedback ¾ the method of evaluation ¾ that they received. Everything I did with
these youth was intuitive, rather than things I’d studied for a career. All I
could see was that there was intelligence, ability, and even a desire to DO
something with their lives. I had to find a way to get around their feeling
that I was passing judgment on them, especially “academic” judgment. Years of
district and national testing had already done a tremendous job of convincing
these youth of how few methods they had for competing in the world. Their
“reality” was very different from the reality described by national testing!
They needed something else … in drastic measures.
They needed real
measures for skills and knowledge that were relevant to THEIR world.
They needed something authentic.
This writing is
about discovering a solution, hidden in a program of “education.” The true
reason I landed “in Education” was that the education programs, in the early
days of Internet and digital tools, were the only ones giving classes in things
like … Multimedia! Digital Art! Authoring classes for the platform shells for
games, kiosks! … and yes, learning environments. That was in 1995, when I then
added architecting classes … for building the World Wide Web. I began to
envision a highly improved method of reaching and teaching at-risk youth on
a global level. I saw that I could reach them with the message that they
could pass a “test,” and not fail. This was the perfect toolkit for taking on
the challenge of proving that authentic assessment was timely, possible, and
cost-effective … for by this time, I DID have the beginnings of the vocabulary,
the “jargon of the trade.”
I found myself in a
Master’s program for computer science and education … and I “met” a man named
Grant Wiggins. The puzzle pieces clinked effortlessly into place. For these bad
gang boys and for me ¾ Wiggins provided the description for all the frustration ¾ we have to do more than lead them
“to believe that right answers matter more than habits of mind and the justification
of one's approach and results.” (Wiggins, 1990). For these
at-risk youth, giving “right answers” on what amounts to a contrived
instrument, is most-times impossible, but always contrary to their
approach or their results ¾ in their very real “real-world” world.
The reality behind
the term, “at-risk,” is a saga unto itself. But the important thing, in my own
journey, has been to arrive at the revelation that there are solutions.
Many of those solutions are found in the various aspects of what is called
authentic tasks, and authentic evaluation and assessment.
~ Mary B. Harris (1874–1957), U.S.
prison administrator
(Bartleby.com, 2001)
WHEN REFORMATION IS THE
JOURNEY
Human beings are
emotional creatures, and, as such, their daily actions revolve around not just
skills and abilities to perform, but also their emotional capacities for
managing the positive and negative experiences in life. Two very big motivators
are: anger and fear. Unfortunately, these negative emotions come from very
common elements in our lives ¾ from the entities that form our characters, make assessments and
judgments, and dole out the punishments and rewards. One of these entities, of
course, is the school system, the “nest and nursery” for most of us, for at
least twelve years of our lives.
As young learners,
most of what we learn IS in the hands of the educational system, and much of it
has been highly ineffective in one supreme area: Assessment should primarily
support the needs of the learners. Traditionally, this just has not been so.
Beyond any kind of technical consideration, this is the basis for the move to
reform assessment. (Wiggins, 1990). With so many students
falling through the gaps, it is an understatement to say that the system is
ineffective. The current outcry is more to the point: It is harmful.
Once learners
“learn” (are conditioned or even driven) to approach the assessment
situation with at least fear, and also with increasing anger ¾ this absolutely affects the outcomes and
thus the validity of the intended testing situation. It then obviously has
affected the totality of the learning experience, and is no longer serving the
needs of the learner. Most people who defend traditional tests “fail to realize
that it is the form, not the content of the test that is harmful to learning …
students come to believe that learning is cramming; teachers come to believe
that tests are after-the-fact, imposed nuisances composed of contrived
questions ¾ irrelevant to
their intent and success. Both parties are led to believe that right answers
matter more than habits of mind and the justification of one's approach and
results.” (Wiggins, 1990). Yet, more than ever, we live in a
society that is results-oriented.
We must rethink and
we must reform. We have entered an age where traditional “proof of validity” in
a testing situation is less important than the demonstrated results of
learning. “Demonstrations of the technical validity of standardized tests
should not be the issue in the assessment reform debate.” We must move on, and
away from isolated tests composed of items that are stand-ins for true
knowledge and experience, and grades that have no obvious meaning (to students
OR to teachers!) or usefulness. These not only harm learning from the student’s
standpoint ¾ they also
weaken the teacher’s ability to improve instruction and the learner’s ability
to improve his or her performance. (Wiggins, 1990).
Recent years have
seen much happen in the way of national reform policy and legislation, as well
as national and state initiatives to delineate new curriculum and instruction
standards. They have called attention to the importance of developing higher
cognitive processes and knowledge. Interestingly, reform legislation also has
set a directive for development of methods to evaluate the progress toward
meeting national standards – a much more elusive goal, and more difficult task.
“According to this legislation, acceptable assessment procedures should measure
cognitive processes and knowledge, such as problem solving and analytic
thinking and conceptual understanding, as well as other important educational
outcomes in a fair and unbiased manner. Assessment tasks should reflect
performance that is authentic …” (CLEAR, 2001). The important
thing that stands out in this statement is, how do we begin to tie “national
standards” to an assessment process that focuses on higher cognitive processes,
with so long a history of NOT doing this? But for me, all of this has been
music to my ears …finally, indesputable evidence that others, many others, have
been asking the same questions – who or what will reach the lost in this
generation?
To truly meet the
needs of the learner, this search for authenticity must “deliver” in at least
two respects: The tasks should be authentic with respect to the content domain;
and the performances called for should be related to concrete demands of the
student's environment ¾ that is, the task doesn’t simply correspond to imposed marks that meet
certain “standardized” criteria.
To really support
educational reform, shouldn’t assessment also be based upon our knowledge of
learning and of how abilities develop in the topics we teach? There is an
alarming and obvious need for forms of measurement that educators can use to
actually improve educational practices. These methods should enable us to
monitor the multifaceted outcomes of teaching and learning. We surely know, for
instance, that learning is not merely the accumulation of unrelated bits of
knowledge and skills. We know that learners don’t simply absorb information in
a passive way. Learning happens primarily with the organization and reorganization
of information. This enables problem solving, complex thinking, transfer of
knowledge, and the creation of new knowledge.
Effective
assessment must therefore be sensitive to how well the organization and
restructuring of knowledge is proceeding. (CLEAR, 2001). The
real question is, how do we make traditional assessment methods compatible with
the things that reform requires? Many have argued that assessment may actually
offer a significant tool for education reform! They say that, as traditional
standardized assessment evolves, it will be “complemented” by other types of
assessment. I question whether changes in traditional standardized strategies
won’t actually increase the complexity of assessment procedures, but
this remains to be proven. It has been suggested that principles from cognitive
psychology may possibly provide a vital structure for new forms of
educational assessment. (CLEAR, 2001).
So, what do
cognitive psychology ¾ and ultimately, cognitive technique ¾ have to do with authentic assessment? Since
cognitive learning is demonstrated through the acquisition and use of
knowledge, and that we literally construct our own learning, this includes that
it is an authentic situation and outcome. We must keep in mind that
authenticity must be applied to both the tasks AND the outcomes being assessed.
“A move toward more authentic tasks and outcomes thus improves teaching and
learning: students have greater clarity about their obligations (and are asked
to master more engaging tasks), and teachers can come to believe that
assessment results are both meaningful and useful for improving instruction.” (Wiggins, 1990).
The demand for
results-based assessment is actually driven by our entry into a high-paced,
knowledge-saturated technological environment. The arrival of rapidly advancing
technologies has altered not only the face of learning, but also the methods of
evaluation and assessment. This was further corroborated in my research
. . . David Jonassen and Heinz Mandl, editors of Designing
Hypermedia for Learning, say that "if evaluation is to address the
processes as well as the products of learning with a new technology, a variety
of methods must be used . . . the solution to the problems of
evaluation is to take a multi-faceted approach to evaluation." (Jonassen
& Mandl, 1989, P. 360).
It is my opinion
that the world of hypermedia lends one very important element to not only
enhanced learning, but also to the assessment process. That is, that by sheer
use of the computer to study, research, and learn — the student has literally
placed herself on an equal level with her workplace counterpart. The computer is
the tool that performs real tasks, both in learning and in performing work in
the work world. It is somewhat different than the equal use of paper and
pencil, or research libraries, or any other common element. It is, more than
any tool in prior history — an extension of our selves. The assistance of the
computer quickly becomes a cybernetic "right arm" for the student or
the worker — an indivisible part of interactivity with the real world, an
extension that increases meaning in our lives, as much as our hands or our
sensory organs. It is almost as if the computer, with all its offerings for
learning, is part of what must be assessed.
Jonassen and Mandl
express that because of the personalized nature of learning (and being assessed
for that learning), it should be realized that "the essential problem of
evaluating highly interactive systems is in measuring both the quality of the
interaction as well as the product of learning. Evaluations of hypermedia-based
learning must address both the process of learning and the outcomes of
learning. (Seels & Glasgow,
1998, P. 360).
What I see is that
the process of learning and/or assessment also cannot be divided from
the avenue for
processing . . . and that is the computer.
WHEN PERFORMANCE IS DIRECTLY IN
PROPORTION WITH “REAL” TASKS
Authentic
assessment is one assessment form that perhaps is the closest to the Jonassen
and Mandl proposal. The concept of authenticity in assessment means that
performance is contemplated directly in proportion with "real" tasks.
The opposite of authentic assessment is the "drill and kill," paper
and pencil test (Wiggins, 1990, P. 1,
as cited in Seels & Glasgow, 1998, Pp. 88-89),
from which learning must be deduced, and from which so very much about our
academic and professional lives, are determined. More than ever, the new Age of
Technology calls upon all of us, as students, and ultimately, as professionals,
to perform what amounts to authentic assessment.
Simply because we
now are forced by the advent of high technology in commerce, to bring the real
world tasks of lifelong learning into the workplace, as well as expect that
students will increasingly perform according to professional standards — it is
my vision that some form of authentic assessment will BE that ideal mode of
evaluation that we seek.
If nothing else, as
instructional designers, we must look for a form of "test validity"
that shows the test to be a simulation of real-world analyses of ability.
"Authentic tasks involve ‘ill-structured’ challenges and roles that help
students rehearse for the complex ambiguities of the ‘game’ of adult and
professional life. Traditional tests are more like drills, assessing static and
too-often arbitrarily discrete or simplistic elements of those
activities." (Wiggins, 1990, P. 1, as cited in
Seels & Glasgow, 1998, Pp. 88-89). It is the aim
of authentic assessment to evaluate learners' abilities within
“real-world" contexts.
In other words,
students learn how to apply their skills to bona fide tasks and projects.
“Authentic assessment does not encourage rote learning and passive test-taking.
Instead, it focuses on students' analytical skills; ability to integrate what
they learn; creativity; ability to work collaboratively; and written and oral
expression skills. It values the learning process as much as the finished
product.” (Pearson Group, 2000-2001). Very frequently, this
requires learners to work in partnership ¾ these assessments, based on performance,
lead students to actually apply skills and concepts and to solve quite complex
problems, both in short- and long-term tasks.
Authentic
assessment seeks to assess many various and different kinds of literacy
abilities in circumstances that closely resemble actual conditions in which
those abilities are used. For example, authentic assessments request students
to read real texts, and write for authentic purposes about meaningful
topics. Both the material and the assessment tasks are presented as natural
as possible. A critical element is users’ literacy level; defined as a measure
of functional reading level. Keep in mind that each of us experiences some
degree of illiteracy when dealing with a subject that is beyond our present
level of knowledge.
“Working on authentic tasks is a useful, engaging activity in itself; it
becomes an ‘episode of learning’ for the student. From the teacher's
perspective, teaching to such tasks guarantees that we are concentrating on
worthwhile skills and strategies. Students are learning and practicing how to
apply important knowledge and skills for authentic purposes. They should not
simply recall information . . . they should apply what they know to new tasks.”
(Houghton Mifflin Co., “Authentic…” 1997).
Grant Wiggins
perhaps said it the most effectively:
”Assessment is authentic when we directly examine student performance on
worthy intellectual tasks. Traditional assessment, by contract, relies on
indirect or proxy 'items' ¾ efficient, simplistic substitutes from which we think valid inferences
can be made about the student's performance at those valued challenges.” (Wiggins, 1990).
The essence of
reform is to speak to the idea that there is a more effective assessment
alternative. In most traditional scenarios for assessment, how well do multiple-choice
tests really evaluate student understanding and achievement? Authentic
assessment sets great store by the thinking process behind the work ¾ the process ¾ as much as the finished product. The
RESULTS that are so badly needed in our world of high technology are needed
within the processes, as well as in a final product.
Authentic
assessment IS the assessment solution that will propel advanced stages of the
teaching/learning process ¾ to meet the needs of the new millennium.
WHEN HUMANS MUST CONSTRUCT THEIR
OWN KNOWLEDGE, AND
LEARNERS MASTER ONLY THOSE ACTIVITIES THEY ACTUALLY PRACTICE
Constructivism is
not an instructional approach ¾ it is a learning theory. We should
really think of it as a way of "growing" or enhancing instruction.
Constructivists put the learner at the center of the equation ¾ with the idea that the learner constructs
knowledge rather than absorbing it passively. The learner, each in his
or her own way, constructs meaning based on prior experiences, mental
structures, and beliefs. These have been used to interpret objects and events.
In many classrooms,
the principal training mode is direct instruction, with the trainer assuming a
central role for transmitting knowledge to learners. The learner's role is to
absorb that information in a receptive and compliant way. There is an
over-reliance on rote memorization, which does not give the learners the skills
for how to think and solve problems.
But learners will
not make use of concepts and ideas unless they use them through some form of process.
In other words, learners master only those activities they actually
practice. Please note ¾ this is a supposition in both constructivism and rote learning environments.
From this, it seems apparent to me that defenders of the traditional
system ignore the fact that both constructivism and instructivism are required,
since learners need to be able to solve complex problems as well as to be able
to understand the reasons or methods they use to reach their conclusions. This
actually follows Bloom's Taxonomy, in that it goes from
simple learning to the higher levels of critical thinking. (Clark, D., Developing
Instruction, 2001). For a similar highly developed level of
success in assessment, the higher levels that require analysis, synthesis and
evaluation will stimulate interesting interactions that will show that learners
have moved from “fact-finding” and “knowledge-gathering” to genuine
assimilation of content. (Moore, et. al., 2001, P. 11.23).
Cognitive
constructivism is based on a theory of cognitive development where humans
cannot be "given" information, and they immediately – somehow
magically – understand and use it. Instead,
humans must "construct" their own knowledge. They build their
knowledge through experience. Experiences enable them to create mental models
in their heads. These “schemas” are – through direct experience – changed,
enlarged, and made more refined through two complimentary processes: by being
assimilated, and by accommodation. The emphasis is placed on the student
rather than the teacher. Teachers are seen as facilitators or coaches
who are there to help students construct their own conceptualizations and
solutions to problems.
For me, this then
is the essence of authenticity in learning – and in assessment of the learning.
For the youth that I’ve worked with – finally! – a solution to deal with utter
and deep dissatisfaction with “the system.” Finally, a method of looking at
their experiences, their knowledge and abilities, and having a means to assess
their learning, irrespective of norms and judgements. “Cognitive
constructivism” is based on the idea that people learn by actively constructing
new knowledge, not by having information poured into their heads. “Moreover,
constructivism asserts that people learn with particular effectiveness when
they are engaged in ‘constructing’ personally meaningful artifacts.“ (Clark,
D., Constructivism, 2001). What this will mean to many thousands
of at-risk youth, in schools that have truly let them down so badly – is
astounding to conceive.
To youth like this,
the opportunity to succeed at “the assessment game” will turn the tide in our
schools … for youth of every culture, locale and ability. “Educators who view
learning in this way realize that quantitative methods of evaluating learners
do not ‘measure up.’ Authentic forms of assessment present a more qualitative
and valid alternative.” (Kirka, S. 1995). "’Test
validity’ should depend in part upon whether the test simulates real-world
‘tests’ of ability." (Wiggins, 1990).
V.
SYSTEMS THEORY ¾ A
SET OF PROVEN PROCEDURES:
DESIGNING EDUCATIONAL PROGRAMS THAT WORK
WHEN SCIENTIFIC METHOD MEETS COGNITIVE
TECHNIQUE
Instructional
Systems Design (ISD) is one mode that has been clearly identified as an
integral component of successful instructional design. Much of the literature
about ISD models deals with training to do specific tasks. ISD typically
has led to higher learning achievement, a greater transfer of learning from the
instructional setting to the job, a reduction in the time required for
learning, and lower per student costs.
Let’s think about
“education” for a moment. Mostly, we still use the same “mode” as our
grandparents used – the same buildings, the same schedule, and the same lecture
techniques. Indeed, in the 21st century scenario, “school” might be
the only place that would feel like familiar ground to our great-grandparents.
But this is changing, and, of recent years, is changing very rapidly. One of
the things that has brought about this change is Instructional Systems
Development. Instructional Systems Development, or ISD, is a systematic process
that follows a set of proven procedures to design educational programs that
work. (Hannum, W., 2001).
“Systems theory requires constant self-monitoring and adjustment of the
system. This is similar to the scientific method, in that we formulate
hypotheses (designs) and test them, thereby supporting or altering our
expectations. Formative evaluation is the primary means of doing this
self-testing; at various stages, designers may try out instructional materials
to improve their effectiveness. New methods and approaches to formative evaluation
are based on cognitive assumptions of performance. Whereas a system's
evaluation in the past tended to focus on learners' success in performing the
criterion task, cognitive techniques seek to uncover thinking processes as they
interact with the material.” (Wilson, B. 1992).
WHEN THE PRIME GOALS OF ISD ARE
CLEARLY DEFINED
In our era of new
media and an almost limitless amount of knowledge that is accessible, a new
"mind-set" is developing concerning, for instance, standardized
instruction, with an iron-clad description of being "able to deliver
instruction the same way every day." (Seels & Glasgow, 1998, P. 18). If we seriously
ponder where learning environments are headed ¾ down to the very setting that we, as
Capella students, experience ¾ we MUST ask if this is STILL a prime goal for instructional design?
“Based on cybernetic principles of general systems theory, the ideal
design process relies on constant systemic feedback. Such an instructional
system acts something like a thermostat, always monitoring its own
effectiveness, making revisions as needed to optimize learning outcomes. These
cycles of self-testing and correction are repeated during the design process as
well as during implementation and maintenance of the system. In this way, ID
can adapt to differences in content, setting, learner characteristics, and
other factors.” (Wilson, B. 1992).
First, it seems to
me that the very character of ISD lies in the fact that its history lies
OUTSIDE of the educational system. (Seels & Glasgow, 1998, P. 18).
This, in itself, almost describes the collaborative environment of the
burgeoning arena of distance or online learning environments
. . . all of which have been heavily influenced by ISD, or at
least incorporate the components that make up a successful delivery system for
instruction, learning, or training.
When we begin to
break down the elements of traditional ISD, for instance, one glaring commodity
is always the first to surface. "What will this cost?" This is almost
the battle cry of educational institutions as they face new technologies and
begin to plan for creating and maintaining the best possible instructional
environments. This is addressed in the cost analysis phase. The main question
still remains: How may this instructional tool be delivered:
·
most cost-effectively;
·
repeatedly; and,
·
to the greatest number of learners? (Seels & Glasgow, 1998, P. 19).
It must be noted
that it is in the EVALUATION and ASSESSMENT stages where the true perceived
raises in cost factor is seen. It simply is a more thorough method – and it
takes longer. Evaluation plays a major role in ISD by providing information to
determine whether the learners are acquiring the desired knowledge, skills and
attitudes, indicating specific weaknesses for individual students, determining
the effectiveness of the instruction, determining how well the total
instructional system is working and providing a means for identifying necessary
revisions.
The data collected
in the evaluation form the basis for many crucial decisions. Thus, care must be
taken in collecting these data to ensure accuracy and relevance. The purpose of
determining whether the objectives have been reached is not solely to assign a
grade for the learner; program evaluation can also help to determine the
effectiveness of the educational program, as well as the appropriateness of
training goals and objectives. (Hannum, W., 2001).
When
computers provide an integrated environment
for teaching a variety of disciplines
My first experience
with the changing nature of instructional systems design, as well as the new
tendency to incorporate training into a Web-based setting, was my a Project
Lead position with Lucent Technologies, from May to October of 1998. I
developed an interactive Web site that delivered training and assessment, and
informational materials to technicians, worldwide. The project was derived from
instructional content that previously was delivered (also via the Internet) in
a scrollable PDF file. Lucent’s desire was to "excavate" this file,
which was utilized by few (a long, scrolling document is difficult to control
or use), and develop an interactive, task-based Web site.
The overlying goal
was to deliver technical information and instruction for equipment use, in a
navigable form via the Web. The format intentionally was never laid out in an
ISD manner. Because of the availability of the Internet and its research
capabilities, this model was never locked into "the same instruction each
and every time" description. Yet, all the while, I could see the
possibilities of re-constructing certain areas of the content to meet the needs
for testing or evaluation. It was important for me to see that this large
corporation desired to take advantage of the strengths of new media, all the
while leaving standard ISD possibilities open for future expansion of the
learning environment.
Further research
for this writing revealed that the change to Web-based delivery has been
constant for quite some time, worldwide. David Marshall and Stephen Hurley of
the University of Wales, Cardiff, United Kingdom, in their paper,
"Delivering Hypertext-based Courseware on the World-Wide-Web," state
that:
"The use of computers to provide an integrated environment for
teaching a variety of disciplines has received much attention in recent years.
Indeed many frameworks [this writer’s substitution: models] have been
developed for such purposes. The material provided by such courseware varies
greatly from the provision of lecture notes and lecture support material
through to integrated and interactive tutorial packages. Until recently
courseware has existed as stand alone packages; however, with the advent of the
World Wide Web (WWW) on the Internet and accompanying WWW (hypertext) browsers,
such as Netscape . . . the provision of courseware has taken on
a whole new dimension." (Marshall & Hurley, 1999).
Interestingly, this
new dimension included one of their initial, calculated goals. A primary goal
was that the courseware could be used to sustain a variety of courses,
including undergraduate degree programs in computer science, physics, all
branches of engineering, mathematics and electronics, as well as training
courses run by computer service departments. The challenge of designing the
learning materials for such a diverse group was to make the materials
accessible to all classes of users, and yet maintain a high degree of "specialism."
The authors translated this to mean extremely pertinent and individualized
training materials. (Marshall & Hurley, 1999).
Whether the
learning environment is CBT-based, or delivered via Internet/Intranet, the ISD
process involved for digital settings has taken on new directives that don’t
DENY traditional methods but alter them to fit new learning means and methods.
The IT community (and this will include most, if not all, instructional
designers) has bent over backward to accomplish one goal, which is: The closer
to the end user, the more powerful the return. This is because when response
time goes down, quality goes up and costs go down. For the new IT, the first
and primary commission is to create a communications system able to bring
critical information to the end user in real time.
It is ONLY in this
way that we can begin to entertain a hope that learning will continue even
within the setting of the assessment process – as well as inside the classroom.
“Whether we are pondering the effectiveness of the formative value of
online assessment, or evaluating the system itself ¾ we no longer have to wonder if authentic
assessments can incorporate a wide variety of techniques designed to correspond
as closely as possible to ‘real world’ student experiences. They simply ARE
compatible with adult, career, and vocational education.” (Kirka,
S. 1995).
When
Researcher and Practitioner become one ¾
Knowledge of content and the architecture of the delivery system
It is my opinion
that the line between Researcher and Practitioner is becoming
hazy. The instructional designer can no longer look to previous descriptions
for the role of those who develop curriculum and assessment, and use only
those parameters. Ironically, the "Practitioner" must then become
also a "Researcher," in order to maintain a fully developed set of
design skills. She or he must turn to the delivery system in order to
design that which will be delivered ¾ the instructional product. In other words,
one must turn to the Internet to glean further information, in order to be able
to effectively construct a learning architecture that will also be Web- or
Internet-based. Seels & Glasgow state that "as a designer, you can
progress further along your career path if you are aware of the research
associated with each step." (Seels & Glasgow, 1998, P. 25).
Each of us involved
in instructional design will eventually participate in both roles. If, in the
first 40 years of the discipline of instructional systems design, (Seels
& Glasgow, 1998, P. 25)
the processes were divided into two roles, then the next 40 years (or less!)
will bring the processes closer to a more holistic approach – just as
the collaborative setting of learning is becoming more holistic. Sheer
availability will provide much in the way of allowing the practitioner to
assume more responsibility for researching both content and means for more
effectively reaching design goals.
Designers are being
called upon to be personally informed concerning the research leading up to the
final product, as well as knowledgeable in the actual design of the product. To
effectively design the instructional environment, the designer must be familiar
with not just the content and the architecture he/she is creating, but also the
architecture of the delivery system. The reliability of the designed
platform is also the reliability for all assessment. All of these must be
taken into account. All of these must be nurtured as skills within ONE
designer. This will strengthen the individual as well as the team effort.
WHEN KEEPING PACE WITH TECHNOLOGY
IS CRUCIAL
A true reason for
concern ¾ and the
necessity for all involved in the current state of online learning and
assessment ¾ is that the
goals for learning, learning environments (and thus, Instructional Design) are
changing as rapidly as the technologies that are driving them. As quickly as a
conclusive textbook can be copyrighted and distributed, it has many portions
that are no longer up-to-date. There is a plethora of information available,
concerning the generic ISD Model and the roles of researcher and practitioner
as "specialties" of the Instructional Designer. In the past few
years, I have worked with a number of instructional design projects. It is my
experience that the current requirements are leaning heavily toward Web-based
instruction and training. This incorporates a broad view of
"Internet" environments such as a company’s Intranet, as well as
password-protected Internet-deployed training sites.
WHEN “TESTS” ARE LEARNING EVENTS
One feature that
will stand out in a study of the action of assessment is that "research
documents that tests are learning events." (Seels &
Glasgow, 1998, P. 83). It has been my experience that,
unfortunately, the traditional student rarely sees it this way! The axiom and
hope of this course of study is that each student will search out an individual
method for applying the highest level of accuracy and usability in their
instructional design environments. As a student myself, I see very good reasons
for NOT appreciating many forms of testing that habitually have been applied to
a learning curve that, otherwise, was quite gratifying. We can hope that, as we
watch the very foundation of instructional design undergo massive changes by
the very nature of the Internet and all of its possibilities, that there we’ve
witnessed the arrival of a new mind-set concerning
. . . assessment.
There has never
been a “perfect” mode of assessment for learners. We all must live with the
fact that most of the time, a varying number of methods must be considered in
projects. Instructional design faces the same challenges as any other entity,
in the face of the rapid changes that are happening in digital learning arenas.
It must be asked: If there were a perfect and all-encompassing method for
assessment, and the major problem with this particular method was that it was
very labor intensive and therefore costly, would a suitable argument for using
the method be that the increased gains would more than offset the increased
costs? (Wiggins, 1990, P. 1, as cited in Seels &
Glasgow, 1998, Pp. 88-89). We have come full-circle,
for the answer to this question is hands-on, experiential learning settings and
assessments. We must address the issues of authenticity in learning, and
authentic assessment.
In comparing things
learned in the study of this chapter, I am providing a list of a few of the
instructional design projects with which I have been involved:
·
Math
application for 4th-graders, which was available both via
Web-based technology, as well as CD-ROM. It included games, animations, video,
and many interactive elements. It also included automatic assessment of student
activities, providing teachers with instant, printable assessment results.
·
Two
interactive "research manual"-type software concerning artists
involved in the international space race, and the Hopi Tribe of the
Southwestern U.S. Both were constructed for use by kiosk, CD-ROM, and the Web.
·
Curriculum and
course design for online college courses, utilizing existing documentation. All
projects were constructed in HTML, and included hypertext as well as many
multimedia elements.
·
Task-based
"user's manual" for technicians worldwide, in the Lucent Technologies
communications' equipment divisions. The project was entirely Web-based;
·
Interactive
"city guide" that instructed users concerning events, locations, and
diversions in the Denver, Colorado area. The project was a kiosk, but could be
transformed into a Web-based application.
·
Criterion-based
learning environments for assessment and certification, for business management
personnel in the legal services fields.
·
Online
courses, in both the instuctional design, SME (subject matter expert) and
teacher/facilitotor positions, with Art Institute Online. Assessment was both
qualitative and quantitative in nature.
This list was to
show not only a variety of instructional design products, created with a wide
variety of tools and methods, but also to indicate that — since 1995 — not one
project was outside the realm of the use of the Internet and World Wide Web.
Each learning structure, and all built-in offerings for assessment, was
literally "outside" the traditional descriptors for
"assessment." This is centered in one word: hypertext (or hypermedia)
content. This is an arena where the rules for assessment change. The reason for
this is that, in systems with enormous amounts of information, with assorted
formats, one major evaluation problem would be the notation of what is
accessed. This could be true even in a paper-based library system, as well as
any digital system. However, in the emerging electronic systems that provide
high levels of learner control, THE major evaluation problem is calculating how
learners conduct this control and freedom, as they are learning. The next
scenario is in the dynamic, collaborative systems, where the notable assessment
enigma is who does what — as well as how it is done. (Jonassen
& Mandl, 1989, P. 360).
Another
consideration is assessment and its relationship with STANDARDS. I am speaking
of basically what has come to be known as standardized curriculum and the
emphasis that has been placed in recent years, on districts and teachers, for
providing useful methods for comparison on state or even national levels. This
was a very "hot" issue in the Denver School District, in the early-
to mid-1990s. During the summer of 1993, I worked with a group of teachers and
administrators on re-developing high school curriculum items and issues. The
project was part of a Perkins Grant; the team was extremely hard-working,
knowledgeable experts, with a genuine desire to complete a prototype for use
within the state, for providing all teachers a tool for assessing high school
subject areas. The primary question, in the beginning was: Where To Start?
It must be pointed
out that, in many ways, this project was doomed to fail. Much of the work was
never utilized — not at the state level — or even within this particular high
school. Recent research has led to documentation that provided insight into WHY
the project wasn't as successful as it might have been. The Mid-continent
Regional Educational Laboratory (McREL) states that the process might begin
with using the "standards document" produced by one's own state. They
show that every state except one is developing or has developed state
standards. A truly disturbing research fact is:
"[A] study conducted by the American Federation of Teachers (AFT)
reports that only 13 of 49 state documents are specific enough to be used
effectively by teachers. The majority of state documents describe standards at
levels of generality that do not provide sufficient clarity for classroom
instruction, nor are they precise enough to serve as an instrument of
accountability." (Marzano, 1997).
My group in 1993
wished — as all schools and districts wish — to adopt valid standards.
Furthermore, parents want to know how their children are performing
academically — compared with standards. The next suggestion by McREL was to use
the national "standards documents" such as the ones published by
national associations and subject matter groups. Unfortunately, McREL points
out . . . "these documents commonly embed a description of
requisite knowledge and skills within lengthy descriptions [over 2000 pages and
14 pounds of documents] of performance activities, curriculum goals,
instructional strategies, and the like." (Marzano, 1997).
But, the assessment
process is very difficult. In 1993, I viewed much of the process as either the
act of "reinventing the wheel," or, a commitment to NOT be doing so.
In the end, this resulted in much going ‘round and ‘round, with much loss of
time and effort.
All of my early
experiences as a student indicated to me that most assessment FEELS like the
process is undeniably tied to the act of GRADING – it feels strictly
judgemental. Seels and Glasgow made the point that "when we assess, we
estimate or judge the value of a person, activity, or situation. When we
approach assessment from this point of view, it becomes easier to understand
that assessing learning is not equivalent to grading. While one
function of assessment can be to provide a basis for grading, that is not its
primary function." (Seels & Glasgow, 1998, P. 82).
This is an enlightening statement. If this is truly the issue — and, I feel
that it should be — then, why does assessment almost always have the negative
“competitive” overtones of . . . GRADING?
For the learner,
and even for the instructional designer or the administering teacher, the
assessment process is tied closely to psychological factors. This research led
to a concept called "knowledge structure measures," which are based
on and enriched by psychological theories. The postulation is that information
in one's memory is organized or "structured" so that it facilitates
the storage, recall, and manipulation or handling of information. Much research
shows that "assessment of knowledge structures may be used to reliably
distinguish skilled and unskilled performers in a variety of domains." (Rowe
et al., 1997). This indicates that this form of assessment
is usable in a variety of settings. It also strongly shows that perhaps one
very powerful missing element in traditional assessment methods may be the
internal, psychological aspects of learning and evaluating the individual grasp
of information.
I propose that it
is because the traditional example disregards psychological factors, and
instead, offers . . . "drill and kill."
The interesting
difference in this example is that measures of knowledge structures may be used
to advance toward . . . and evaluate
. . . conceptual understanding for assessment and training
purposes. In a typical situation, the value of an individual's knowledge
structure is decided by comparing it to a standard knowledge structure. This
would be a collection of the structures of several experts. Ideally, this
approach indicates a positive linear relationship between the progress toward
expert task performance levels, and the development of knowledge. Each learner
is given a "knowledge score" by calculating the comparison of his or
her knowledge structure against the standard knowledge structure. This
description sounded very good and reasonable, up to this point. Then my
research source stated that "a validated knowledge structure measure may
then be used to extract rich cognitive representations that are useful for
assessing and diagnosing student knowledge and identifying targets for training
intervention. This knowledge score may then be validated by determining how well it
correlates with a recognized performance measure, such as final exam or final course
grade." (Rowe et
al., 1997).
With great dismay,
this had led back to . . . grading. The search for an elevated
form of assessment ground to a halt.
This ultimately
sounded much like the description of the CRT (criterion-referenced test), in
which the use of the term "criterion" refers to the affinity between
the objective and the method of assessment, and to the level of performance
needed for mastery of a skill or subject. The CRT score gives information about
a learner's mastery of one specific skill relative to the objective, and
therefore shows one's ability in performing specific competencies. (Seels
& Glasgow, 1998, P. 83). A CRT score lets everyone know
exactly how well the student stands, relative to a standard. (P. 84).
But, the workplace
is becoming more technologically complex! It demands increasingly specialized
skills, workers, as well as students who will be entering the workplace, must
rapidly become skilled in a broad range of cognitively complex tasks. The
cognitive aspect of a task must be spoken to, for purposes of meaningful
assessment. Since the
traditional method of assessing the cognitive domain is via a paper and pencil
test (P. 83), are we then still stuck with grading and
"drill and kill" methods?
Reliability means
that a test will render a dependable measure, so that if a test is repeated the
same results will be obtained. Reliable tests have consistency and temporal
dependability (a student will show the same level of mastery on one day of the
week as on another day). Reliability is augmented by an architecture with items
that are not ambiguous, and by scoring the test as objectively as possible. (Seels
& Glasgow, 1998, P. 85). Validity means that a test
will measure what it is supposed to measure. Without reliability, there can be
no validity. (P. 85). The best approach for substantiating a test's
validity is to provisionally demonstrate that the test accurately discerns the
masters from the non-masters of the test items. This is to say, items that do
not make this distinction, should be revised or completely dropped from the
test. (P. 86).
Using guidelines
from the text and outside resources, one can then come up with an improved
understanding of assessment procedures as a historical process, as well as
perhaps mapping out an "ideal" evaluation plan. It appears that using
a variety of frequent assessment techniques in the classroom may be a preferred
approach. According to McREL, in the traditional classroom, the teacher is
responsible for assessing students on standards and benchmarks. They are
therefore able to utilize a variety of techniques, including portfolios,
performance tasks, traditional tests, and natural observation. There is a
decided advantage to this approach, in that assessment of standards and
benchmarks does not diminish or interrupt regular classroom instruction. The
assessment process is integrated into the regular routine of the classroom. (Marzano, 1997).
It is a holistic and authentic approach to assessment in a real setting.
Those of us in the
broad arena of education MUST look to the business world while developing the
tools of education and meaningful forms of assessment. This is the only manner
in which we can train young learners to participate in the real time world they
will soon inherit. These are the reasons why cognitive techniques, the ISD
process, and authentic evaluation and assessment are so meaningful. We are
training today’s learners to not only take control of their futures, but also
to take control of the “journey,” the educational road to that very future. We
are creating the environment that, through positive learning and assessment
modes, these learners will be more likely to become LIFELONG learners. And only
by instilling the receptivity and desire for self-assessment and hands-on,
real-world learning, will we be able to give them this roadmap to the 21st
century.
The highly
interactive media that is emerging challenges the evaluation of instructional
products, as well as the learning that happens because of these products. This
interactivity provides the learner with access to untold and diverse
information. It gives the user a personal control over the process of learning,
and it carries a vast potential for collaboration with not only the delivery
system, but with other people. This tremendous empowerment of the learner
forces evaluators of learning to embrace a broad-based set of methods and
criteria. ONLY this will accommodate self-directed learning. (Jonassen &
Mandl, 1989, P. 368). To leave
this out of the assessment process would be to cripple the effort for advancing
superior techniques in evaluation.
We, as designers, must
heed this trend in learning evaluation. The Mid-continent Regional Educational
Laboratory (McREL) states it this way:
"Do
not hold students accountable for specific levels of performance on any
standards. This is the system we currently have in place.
"In
virtually every state, the only standard students must meet to graduate is that
they obtain a certain number of ‘credits,’ and a credit is earned by obtaining
at least a ‘D’ in a course. The advantage of this approach is that it is very
easy to earn a high school diploma. The disadvantage of this approach is that
students can graduate without acquiring any specific skills and abilities, such
as the abilities to read, write and compute." (Marzano, 1997).
For years, private
and public sectors have cried out that "Johnnie cannot read," and
they have asked, "Why?" It has been the purpose of this writing to
show that the product of the appropriate — perfect — assessment process will
have an onward, spiraling effect on learners.
Johnnie will not
only learn more effectively, but she ( J ) will perform at much higher levels during
evaluation.
