Home > Resources > Classroom Practice > Assessment

Three Pictures of an Exhibition: Warm, Cool, and Hard

Author(s): Joe McDonald Ordering Information

Mary McLeod Bethune High School,1 situated in a large American city, is a partner in the high school reform effort launched by the Coalition of Essential Schools in 1984. The Coalition advocates simple goals that apply to all students and are focused on intellectual achievement. It also ad vocates a pedagogy that combines a personalized and caring environment with a focus on student production -"student as worker." Crucially, it argues for "graduation by exhibition," rather than as the consequence of time served in class. This phrase leads a double life in Coalition discourse. It refers to a method of assessment, one that measures valued knowledge directly rather than indirectly; that demands performance, the application of knowledge, and metacognition; that happens in some public setting; and that guards a gate to graduation. At the same time, the phrase also refers to the Coalition's own version of outcome-based education, an ideal of schoolkeeping in which all systems are oriented toward producing graduates in the image of the school's collective vision of competent intellectual performance.

The faculty at Bethune Essential School seeks to produce graduates who demonstrate imagination, power, confidence, and compassion in intellectual matters; who communicate skillfully; who have more than textbook knowledge of the major subjects they have studied, but who can apply "content" to investigations of the important issues of their time. "Planning backwards" from this compelling image, the school's faculty have taught themselves to teach in interdisciplinary teams; they have given their students responsibility, gotten to know them well, asked them to find and solve problems, taught them how to present themselves, trusted them to work on their own, offered them time and freedom to work outside the walls of the school, and believed in them.

I visited Bethune Essential School on its first Exhibition Day, a month before its first class was to graduate. I was there to witness alternative assessment in practice. I was also there to peer into this high school's central mechanism of standard setting and, along with the rest of the school, to test for a match between its real graduating seniors and the school's image, crafted four years earlier, of what they might be. My visit, and some subsequent ruminations on it, gave me three pictures that I carry with me in my research among other schools that follow Bethune's lead -- pictures I characterize as warm, cool, and hard. Each picture conveys a key dimension of Bethune's achievements as well as some important lessons for followers.

Warm

Meeting some seniors on the morning of this first Exhibition Day, I feel the energy they feel. Though Bethune High School is new to me, I am certain from its air alone that it is different today -- crazier but more focused too, like backstage just before curtain. Perhaps the teachers are also more forgiving of interruptions: one exhibitor's search for a missing poster, another's panicky recruitment of substitute models for her fashion show, even my late appearance.

The first of the Senior Exhibitions will begin in twenty minutes, culminating a process that began formally a year ago. Each senior has defined and pursued an interdisciplinary question, that pursuit involving library research and fieldwork. Each has met the deadlines for a series of submissions: a list of research sources, an outline, notecards, an introduction, a whole draft, a finished paper. This is routine stuff for high school students -though not usually required of all kids. But now, something beyond routine: each senior has been allotted a half hour in one of the series of Exhibition sessions starting today. During this time, the senior will present highlights from his or her inquiry, then stand for questions. The questioners will include teachers, other seniors, underclass students (all of whom will attend at least one session of these "orals"), some parents, and some outside visitors like me.

I shake hands all around, appearing confident that what I am about to witness will be interesting, novel, worth my train ride. The confidence is genuine, but also a gesture of connection to the proud and friendly company. Privately I am also skeptical, the victim of some number of shoddy school drama productions, art exhibits, science fairs. These efforts become, I know, necessarily invested by a school's belief in itself. That belief is what carries everyone through the burdens of mounting such efforts outside the ordinary bounds of business, though it sometimes swamps critical judgment.

Now for the test. Can each one of the real kids fulfill the collective image? This is the assessment question. It presents a challenge interwoven with ceremony, rather like an intellectual senior prom. The challenge is real and difficult, which is why these seniors are rushing, why they are wearing dresses and ties, why they have taken such care with their graphic displays. Still, they have been well prepared for the challenge -rehearsed and coached. That is why parents, teachers, principal, sit confidently, why presenters step to the podium and speak confidently. "Good morning," they each begin.

It is a scene primed for warm attachments -those investments of belief in the performer that arise from a caring history. Without the benefit of warm attachments, most kids do not perform at peak ability. They will prefer, instead, to fall short of peak and so avoid the risk that their peak may disappoint themselves and others. Thus does assessment of any kind become inauthentic.

Of course, warm attachments can also kill authenticity by suppressing participants' critical perception. So warm attachments must be balanced by cool judgments. For example, teachers must balance their warm sense of how far each kid has come with a cool sense of what the outside expects of the young educated adult. This is tough work in any setting, but particularly in a ceremonial one.

Still, a number of teachers try on this first day of the Bethune exhibitions. They ask hard questions. I do, too, though I find myself so pleased by the presenters' fortitude in standing for such questions that I am perhaps too appreciative of the content of the answers. I ask the senior who has just spoken about African- American achievements in several fields, but who has omitted the arts, to correct the omission.

"Who are the African-American writers you admire?" I demand. "Maya Angelou, for one," he answers immediately, and goes on to say why in detail, though for my part he might well have stopped at the name alone, so impressed am I with his presence of mind. And I am so taken by the clarity and potential of another senior's exhibition question -Is our water safe to drink? -so warmly invested in my own sense of its generative power and its potential interdisciplinary sweep, that I fail to recognize that it is only my own question that moves the exhibitor's focus past chemistry and public health to an observation about water and images of purity.

Meanwhile, amid applause and the palpable pride of teachers, kids, parents, another test is under way: What do these performances reveal about the school's own systems? This is the standardsetting question. And like the assessment question, it demands both warm and cool approaches.

I wonder, as I sit observing, whether the school will dare to face it. When emotions have subsided, for example, will the faculty review with a critical eye the videos it has made of these exhibitions? Video is, as they say, a cool medium. Will younger students watch them, too? Will the portfolios that must accompany the exhibitions (documenting phases of the inquiry) inform teachers' and students' planning of next year's courses and projects? How far back will the influence of these exhibitions and portfolios run? Will the experience of next year's ninth graders be different because of what this year's seniors have done? Will there be more time set aside from now on for project work and coaching groups? Will there be more practice speaking before strangers? Will the school be materially affected -shifts of structures and resources?

Cool

It is a year later, and I have just watched a video of last May's exhibitions at Bethune High School. The video contained a selection of four exhibitions, none of them from the session I attended. The school made the selection for its own purposes -to explain itself to interested outsiders, to prepare kids to follow last May's pioneers, and to exercise its critical eye, as I suggest above.

I have watched the tape from my perspective as a researcher, interested to see by its light what I take to be some key dimensions of assessment by exhibition. Of course, sitting in my living room, far in space and time from the warm attachments I felt last May, I am primed for cool judgment.

The four exhibitions on this tape all deal with contemporary topics: substance abuse and fetal development; child abuse and the juvenile court system; recycling and the environment; cholesterol and the American diet. These are my characterizations of the topics, and I phrase them as I do -simple intersections of relatively broad subjects -to emphasize the immensity of the task the exhibitors faced and also to say that each of the exhibitions might have benefited from more focus. This is the way with contemporary topics. They are difficult to delineate, which makes them unlike the topics drawn from textbooks that high school students more commonly face. And even when one has delineated a contemporary topic reasonably well, that topic may nonetheless slide away even while one speaks.

This happened to the exhibitor who addressed child abuse and the juvenile court system, but who seemed through most of his exhibition to focus solely on the court system. Only the first questions from his teacher, intent upon directing his attention to child abuse, cued me that that had indeed been an emphasis of his rather nervous, muddled introduction. I think I must have lost track of this emphasis when I sensed the confidence that emerged as he turned, pointer in hand, to his first exhibits: a flowchart description of the juvenile court system and a collection of document specimens from key junctures in that system. He himself may have lost track there also.

In fact, I noticed as I watched the other exhibitions, too, that a strong presence sometimes overwhelmed substance. That may be a chronic problem in exhibitions generally: they require courage, and courage -being as rare in school as elsewhere -dazzles. So, one exhibitor's demonstration of how to cook lowfat crab cakes, worthy in its flair of midmorning network television, seemed to distract the exhibitor herself from attending to her main concern, the health impact of cholesterolladen foods. I think it distracted the audience, too, which got to eat the crab cakes. Another exhibitor's cleverness in presenting his own videotape on Earth Day events seemed to relieve him in his own eyes -and perhaps in the audience's eyes, too -from having to tie this exhibit to his main point. These lapses would have been far less apparent to me, of course, from a warm view, had I been in the audience, too, anticipating crab cakes, impressed by an exhibitor's use of visual media.

What can a school do to guard against such lapses? The answer, I think, is: Nothing much within the warm setting. Oral exhibitions like those at Bethune High School feature substance and presence but, I suspect, typically tip the balance of favor to the latter. For one thing, contemporary American kids, enveloped by pop culture, have enormously more access to models of presence than they have to models of substance. But, taking advantage of cool opportunities, the school can and must ensure that the assessment system, of which the public exhibition is merely an important part, builds in plenty of substance checks along the way, as well as plenty of models.

Bethune's requirement of a portfolio documenting the inquiry -including notecards, interview transcripts, drafts, a finished paper, etc. -are examples of the former. The trick in their use depends on making them "count" as much as the presentation does, designing them so that they rigorously assess the research process. Any teacher who has ever collected a novice researcher's notecards knows how easily such a process can become an exercise in assessing form rather than substance, and how impossible it is to know -- without taking the time to interview the student -what the cards really represent of the process under way. A portfolio of this kind, to be worth the trouble it takes to keep and review, will be more than the documentation of a performance; it will be the documentation of a coached performance -an artifact of teaching as well as learning.

Coaching, however, involves more than ongoing assessment and support in following some given process. It also requires the provision of models -the elaboration by various means of a vision that the student may aim for. This must be more than a procedural vision -the model of how to find and follow any compelling question. Seventeen-year-olds must also dig for substance in the gardens of mentors who can show them in various ways what excellence means (Gardner, 1989). Their teachers cannot know enough about all the fields that seniors at Bethune and elsewhere may take an interest in. They therefore cannot be the sole suppliers of question generating experience, the sole models of inquiry, or the sole prescribers and guardians of standards.

Of course, they can and must model inquiry in general, but good high school inquiry has more than generic quality. It has the whiff, as well, of incipient expertise. This is what we are used to sensing in some "extra-curricular" pursuits of teenagers -the knowledge of cars or music or perhaps retail merchandising that already exceeds what we may know about these things and which we sense is headed toward real depth. We seldom expect to find the same in school work. One reason, I think, is that we tacitly limit school work to school grounds. A school like Bethune, on the contrary, seems willing to give access to many gardens on other grounds. The success of the effort will depend on whether the kids' digging there is well supported -both on site and back at school.

Meanwhile, there are other kinds of gardens in which future exhibitors must dig. These gardens are any relevant and rich sources of data, insight, and artifact. Access can be in person again -for example, by means of long and structured visits to museums and libraries or a series of interviews; or it can be by means of technology -books and microfiche or telecommunications and CD-ROM.

But access alone is far from enough. It has become routine to talk about an information explosion and its challenge for school curricula, but to understand the real thing, take any of the four topics addressed in the Bethune exhibitions tape -let's say, substance abuse and fetal development -into a well equipped library. See for yourself what the on-line catalog and the electronic databases and the periodical indexes provide. Unless you are highly focused in your inquiry and comfortable with all the odd discourse of data in the fields this topic touches, you will surely feel overwhelmed -even perhaps yearn for the comforting information vacuum of the average high school library with its out- of-date books and its incomplete collections of Time and Science News. So, the Bethune exhibitors need access to rich information and also to teachers willing to take their coaching cues from information they did not provide and do not control. Moreover, the exhibitors need these things long before they face the pressures of Exhibition Day.

This is an enormous challenge for Bethune or any other high school, but Bethune is ahead of most others in daring to face it. Watching its own video and prompted by these exhibitions, it can begin to imagine, as I do, necessary shifts of instructional design. There are signs of what is needed in these exhibitors' occasionally shaky command of information, their apparent gaps of understanding, the unaccountable leaps they sometimes make. "Smok- ing causes low weight babies," one exhibitor declares baldly, and "fetal alcohol syndrome leads to mental retardation." ("Can lead to mental retardation," I want to add from my cool living room.) "The three causes of retardation . . ." he begins, and I wish I could interrupt coolly: "Only three?"

I offer these phrases as snapshots of a kid trying to hold his head above very deep water. He has a lively intelligence that shows even on the cool video, but he needs more time than even Bethune's unusual structures permit among adults who swim expertly in this same water. In short, he needs even more time outside the school to do school sanctioned and school-supported work. But this puts great pressures on his school: if he is so much in the field, how can he complete all his course requirements? And if the school is not to discharge its support functions through coursework, then how may it discharge these functions?

This student also needs even more access -coached access -- to the kinds of data he supplies proudly in his exhibition -city, county, and state data -but which he interprets more clumsily than one would wish. It is not just that he needs more time and opportunity to visit government offices, but probably also more coaching on how to read what such foraging turns up. This may also require even more of a shift than Bethune has already made toward coached project time and what might be called studio work.

Hard

A warm regard provides the social investment that both kids and schools need to develop, while a periodic cool regard provides the critical distance they need to reflect upon this development and shape it. But warm and cool are not enough. What is also wanted is some direction for development: something hard that exists apart from warm or cool relationships and orients them -a framework of standards.

In matters of assessment, there are three principal standards: validity, reliability, and equity. Though these terms seem alien to school life, as if they invariably involved psychometric engineering, the concerns that inhere in them are nonetheless vital to school development (Stiggins, 1991). As I say, they orient it. The third picture of Bethune's work, framed by these hard standards, tries to capture this orientation.

Validity

According to the Standards for Educational and Psychological Testing of the American Psychological Association (1985), validity refers to the appropriateness, meaningfulness, and usefulness of inferences based on assessment. I like to think that validity constitutes the interface of assessment and the world: it demands of the test that it measure what is really valued. Evidence of validity in assessment typically comes in three varieties.4 The first, "construct related," has to do with the relationship between the assessment tasks and whatever abstract phenomenon is of interest -ability to write, for example, or to conduct an inquiry or to handle questions on one's feet. The second, "content- related," has to do with the relationship between the full sample of the assessment's demands and the complexity of the domain it purports to represent. And the third, "criterion related," has to do with the relationship between an assessment's findings and some other measure -for example, between how well kids do on one test and how well they do on another or between how well they do on one test and how well they do in college.

Psychometricians weigh these kinds of evidence by means of various logical and empirical procedures involving item analyses, the testimony of experts in the domain, and tests of the test. The methods seem esoteric at first glance from the perspective of Bethune High School, but the glance is deceiving. In fact, the faculty there should and does analyze its exhibition requirements to determine their match with its real values. It should and does invite the oversight of outsiders (the example of this very paper indicates that). Finally, it should and does corroborate exhibition results with other indicators. During the past year, for example, one member of the faculty has toured the campuses where many of last May's exhibitors study now, interviewing them in order to evaluate how well their school study and exhibitions prepared them for college.

From the perspective of Bethune High School, however, a concern for validity in assessment must involve much more than the local adaptation of psychometric technique. The heart of it is the assurance of connection between school and the world. Much of what I say above -both when I write warmly and when I write coolly -- has to do with the scope and quality of this connection. Are the kids immersed in the real stuff? In the way they write and speak and handle questions, do they show the right stuff? Validity in the assessment by exhibition at Bethune High School cannot be established once and for all; it is a matter of relationship, of a willingness on the part of the school over time to open doors to the outside, so that mentors and critical friends may visit and so that kids can go out digging. It is a matter, as I suggest above, of faculty courage. Happily, this is plentiful at Bethune, so the picture is heartening.

Reliability

Packaged tests usually advertise their reliability in correlation coefficients: measures of how well results from one form of the test match with another form or how well results from even numbered items match results from odd numbered items or how well results from tests match results from retests. From the perspective of Bethune High School, this may seem unimportant -but, again, a quick glance is deceiving. Actually, the longterm success of the assessment system there will depend on reliability in at least two respects: the achievement of interrater agreement (among the multiple judges of the exhibitions and portfolios); and the achievement of consistency in scoring (across the various Exhibition Days, between one year's exhibitions and the next, between one kid's exhibition and the next one's).

The main difference, however, between achieving assessment reliability within a school and achieving it in an external test or assessment system is that the former is both a technical and a structural problem. The heart of reliability at the school level is the assurance of communication among teachers, and this depends upon structure. In fact, no standard setting devices, inter-rater agreement mechanisms, or consistency checks can work inside the ordinary cellular structure that is characteristic of nearly every U.S. high school. Without new structures, teachers will remain isolated, and assessment will remain private and unreliable. Meanwhile, restructuring may not be enough, either. A restructuring plan, for example, may address other important needs -perhaps providing collaborative opportunities for planning and teaching -yet still leave assessment private or else bump it up to some "higher" level (a supposed exchange of power for "accountability"). The latter may well leave schools and kids worse off than before. As Perrone (1990) notes, accountability, to be worthy of the name, must be rooted directly in teaching, not float above it.

In an important sense, reliability from the perspective of Bethune High School is less about seeking agreement -at least initially -than it is about seeking and sustaining productive disagreement. Before they can achieve the degree of reliability that their kids and their bold experiment deserve, the teachers at Bethune must dare to disagree with each other, must dare to abandon the safety of their own private gradebooks. They must invent methods to assess together, as well as teach and plan and advise students together. They have made great strides in this direction: team teaching arrangements and the provision of common planning and meeting time and the painful effort now under way to extend the Essential School pilot to the whole school. Again, the picture is heartening.

Equity

Equity in assessment is often equated with the elimination of bias. This is a matter again for psychometric engineering: one determines by various means which test items cause particular categories of testtakers to stumble disproportionately relative to their performance on the test as a whole. So boys have trouble with items that refer to colors, and girls have trouble with items that refer to sports. This method of bias detection works because the test items that fall under suspicion are, after all, indirect indicators of the achievement that is actually of interest. The item that refers to colors or to sports really means to measure reading comprehension or the ability to solve mathematical word problems; the content of the item -as contrasted with its objective -is irrelevant or, in these cases, misleading.

In performance based assessment, however, one strives for direct measure, which is to say that content counts. The result is that bias detection -while nonetheless important -must be merely one step in achieving equity. What does the Bethune faculty do if it discovers, in a careful review of exhibition results, that boys tend to do better in fielding questions than girls? The answer, I think, is not "eliminate the questions," as it might be if the format did not matter, if the faculty had not deliberately intended that all its graduates would be able to handle such questions successfully. Instead, I think the school's response must be to build a better scaffold for the girls.

In general, equity in assessment at Bethune High School is a matter of scaffolding. The central problem is how to get everybody to a place where some never imagined that they could get. This is an equity of results rather than merely of processes; but because what is wanted is an equity of optimal rather than minimal results, new and exceptional processes are required. To continue the example begun above, some girls will surely not get there if they are treated along the way just like boys. Nor will they get there if, for even a moment, they are less pushed than the boys or if, by various means, they are segregated from the boys. What is wanted is a community of striving in which the standards of achievement are plainly visible and plainly applicable to all, but in which support, time, and structure all vary according to need and in which membership alone moves one forward, quite apart from the effects of particular instruction.

Of course, such a community is much easier to describe in abstract terms than it is to build. The fact is, at this early point in the effort to remake the American high school, we still lack good "architectural" models. Nevertheless, we know some of the building materials we will need. For example, we will need "courses" still, but as I suggested above, something else besides -- something like "projects," something like what the Bethune kids do in their senior year. And we'll need curriculum, but a kind that is not wholly contained inside "subjects," one that links rather than isolates learning experiences, one more embedded than we can now imagine in real contexts. We will need mechanisms that enable teachers to tune their judgments to common and high standards (Wiggins, 1991); these may take the form of interrater reliability schemes within the school and between the school and the outside, which function to aim instruction high as well as keep assessment reliable.

Then we will need new systems of information management -far more responsive and capacious than school libraries, transcripts, and textbooks. We'll need grouping patterns more respectful of human diversity and potential than age grading and tracking; and we'll need management systems that can sustain community better than hierarchy can. Finally, and most importantly, we will need a commitment to democratic principles that is more than platitudinous -that can materially affect the school, that can pervade its habits. At the very least, this will involve a new conception of intelligence (Wolf et al., 1991) and an uncommon tolerance for difference and conflict.

If the last paragraph has seemed to sweep far off from issues of assessment and also from Bethune, the effect is intended. From the perspective of the school, an assessment by exhibition -in all its dimensions -is a systemic thing. It cannot be a matter of grafting something new and clever onto an old design -a new portfolio system that is supposed to make everything different, though it has been dumped onto the school's overworked and isolated English teachers; or a new "authentic" state or national exam which is supposed to force new modes of instruction but which, as Shepard (1991) points out, also implicitly cheats teachers of their professional status. From the perspective of Bethune High School specifically, the prospect of an utterly redesigned high school is a distant and daunting one, yet it is exactly what attracted the school to exhibitions in the first place. And this is the most heartening picture of all -that this and some other daring schools would be willing to scout for the rest of us.

References

Fredericksen, J. & A. Collins. 1989. "A Systems Approach to Educational Testing." Educational Researcher 18(9), pp. 27þ32.

Gardner, H. 1989. To Open Minds: Chinese Clues to the Dilemma of Contemporary Education. New York: Basic.

Perrone, V. 1990. "Testing, Teachers and Schools." In J. L.

Schwartz & K. A. Viator (eds.), The Process of Secrecy: The Social, Intellectual, and Psychological Costs of Current Assessment Practice (pp. 99þ107). Cambridge, MA: ETC, Harvard Graduate School of Education.

Shepard, L. 1991. Interview on assessment issues with Lorrie Shepard. [M. Kirst, Interviewer]. Educational Researcher 20(2), pp. 21þ23.

Standards for Educational and Psychological Testing. 1985. Washington, DC: American Psychological Association.

Stiggins, R. J. 1991. "Assessment Literacy." Phi Delta Kappan 72: 534þ539.

Wiggins, G. 1991. "Standards, Not Standardization: Evoking Quality Student Work." Educational Leadership 48(5), pp. 18þ25.

Wolf, D., J. Bixby, J. Glenn III & H. Gardner. 1991. "To Use their Minds Well: Investigating New Forms of Student Assessment." In G. Grant (ed.), Review of Research in Education 17 (pp. 31þ74). Washington, DC: American Educational Research Association.

Notes

1. The name of the school is a pseudonym, meant to protect the privacy of the exhibitors.
2. I take authenticity in assessment to be less a matter of method and more a matter of impact: does the assessment assess what it means to, and is the assessee genuinely engaged?
3. Lee Shulman describes a portfolio as the documentation of a coached performance and adds that the coaching does not compromise the use of the portfolio as an assessment device. Does the dissertation advisor, he argues, claim to have had no influence on the advisee for fear of compromising the judging of the dissertation? I would add that the juxtaposition of coaching and judging, while natural and appropriate, also demands structural arrangements that share the intellectual and emotional load -so we have dissertation committees in the university, and we need something like them in the high school.
4. Other kinds of validity than those I name above have recently been proposed. Among the most relevant of these to the issues addressed here are systemic validity (Fredericksen & Collins, 1989), which has to do with the corruptibility of the assessment -- if teachers teach to this test, will kids be better or worse off? This is related to what Lauren Resnick and others have called consequential validity, which has to do with the uses to which a test is put.

The Coalition of Essential Schools gratefully acknowledges the IBM Corporation and the UPS Foundation for their support of its research on Exhibitions.

This article is a substantially revised version of a paper presented at the annual meeting of the American Educational Research Association in Chicago, Illinois, April 1991.

Price: $5        Code: EX1
To order a hard copy of this resource you will need the title, price, and code to fill out your order form.

This resource last updated: May 30, 2002

Database Information:

About CES | CES Network | Fall Forum | Small Schools Project | Resources
My Homebase | Jobs | Search | Site Map | Contact Us | Home