The Unseen Social and Cultural Substance of Written Responses in Mathematics

Vicky Kouba, Audrey Champagne, and Zaline Roy-Campbell, O. Cezikturk, M. Benschoten, S. Sherwood, and C. Ho

Abstract

Students’ written responses to open-ended (extended constructed response) tasks show that student performance differs qualitatively based on such social and cultural factors as knowledge of valued forms of communication in mathematics, language and reading proficiency, knowledge of task-designers' implicit perspective, and type of logic. Real-world contexts also introduce greater opportunities for divergent, yet reasonable responses from students. Extended constructed response tasks are a viable means for meeting some of the challenges of equity.

Introduction

Knowing how to participate in the socially constructed forms of mathematics and science communication is an expectation of the current standards-based reform in mathematics and science (Champagne & Kouba, 1997a). Morgan (1998) argues that this expectation is a hidden, but very present aspect of assessment. She states that in many assessment situations the learner "not only needs to ‘understand’ a particular piece of school mathematics but also needs to know the forms of behaviour that will lead to recognition of this and how (and when) to display these forms of behaviour" (p. 4). The difficulty is not just that students do not heed directions to explain and justify, as Dossey, Mullis, and Jones (1993) state, but that students do not know what is implied by such directions. Morgan (1998) frames this issue as one of equity:

There is a fundamental equity issue here that is often ignored. Those students with the linguistic awareness and skills that are generally associated with advantaged, literate backgrounds are more likely to ‘pick up’ the unspoken distinctions and display the valued behaviour in the appropriate situations. Others, from less advantaged backgrounds, are less likely to come to school with these skills; they must, therefore, rely on their teachers to provide them with the necessary awareness of the forms of behaviour that will be valued. The naïve guidance (such as ‘draw a diagram’) commonly provided by teachers is not adequate for such a purpose. An important task for teachers and researchers who are concerned with equity in assessment, therefore, must be to investigate assessment practices at a level of detail that can identify which aspects of students’ behaviour are likely to be recognised as mathematical and valued as signs of mathematical understanding. Such investigations would also need to develop a language to describe these valued behaviours – a language that teachers and students can use both to help students to display the behaviours that will lead to success in the assessment process and critically to interrogate the assessment practices themselves. (p. 6)

Our current research on mathematics and science literacy, as part of the Center on English Learning and Achievement (CELA), is an investigation of students’ written explanations in mathematics and science. One of our goals is to systematically examine student responses for clues on how culture and language may be part of "disadvantaged" students’ performance, especially in terms of mathematical behaviors valued within the larger mathematics community and culture.

As we examine student responses, we also must keep in mind Tate’s (1996) admonition that any delineation of expected behaviors, especially in testing situations, ought to be done from a multicultural and social reconstructionist philosophy that allows students to "solve problems from their lived reality" (p. 195).

One of the most difficult 1996 NAEP extended-response items for eighth-grade students was:

This question requires you to show your work and explain your reasoning. You may use drawings, words, and numbers in your explanation. Your answer should be clear enough so that another person could read it and understand your thinking. It is important that you show all of your work.

Julie wants to fence in an area in her yard for her dog. After paying for the materials to build her doghouse, she can afford to buy only 36 feet of fencing.
She is considering various different shapes for the enclosed area. However, she wants all of her shapes to have 4 sides that are whole number lengths and contain 4 right angles. All 4 sides are to have fencing.

What is the largest area that Julie can enclose with 36 feet of fencing?

Support your answer by showing work that would convince Julie that your area is the largest. (NCES, 1999)

Less than 1 percent of the 1,615 eighth-grade students in the NAEP sample provided an extended or satisfactory response, 29 percent gave a partial response, 4 percent gave a minimal response, nearly 40 percent gave an incorrect response, and nearly one-third omitted the item (NCES, 1999; Kenney & Lindquist, in press).

Eighth-grade students’ responses in the NAEP sample were scored as follows:

Extended -- Correct response (Justification need not include table, but should account for all possible length-width combinations or demonstrate evidence that all combinations have been considered in the formulation of the explanation.)

Satisfactory -- Response that indicates that 9 x 9 square has maximum area (81 sq. ft.) or that another rectangle has maximum area but accompanying work contains a minor error OR Response contains all work for nine rectangles (widths 1 through 9), but maximum area is not indicated (or response indicates that a rectangle other than the 9 x 9 has the maximum area).

Partial -- Response shows at least 3 different rectangles (dimensions and areas); may indicate that one of those rectangles has maximum area
OR any response with no work that indicates that the 9 x 9 square has maximum area.

Minimal -- Response demonstrates a minimal understanding that area and perimeter formulas for rectangles are needed in the solution and may show a beginning attempt to organize the data and information. For example, this might be illustrated by generating one specific value for the length and width that fits the given information.

Incorrect – any other incorrect answer. (NCES, 1999)

Kenney and Lindquist (in press) suggest that the low performance on this item could have been a result of a limited scoring guide. They felt that the NAEP scoring guide did not account for students who had worked on similar items and "just knew" that a square yields the maximum area for a quadrilateral with a fixed perimeter. Based on our research, we also believe that narrowly interpreted scoring guides on high-stakes tests may depress reports of performance (Kouba, 1999). However, our results suggested that the low performance in our sample seemed more a societal or cultural result of students’ lack of knowledge about the expected forms of literacy (see Champagne and Kouba, 1997b for more on forms and levels of literacy in science and mathematics).

Expected Forms of Responses

We were curious about what a detailed examination of a large sample of students’ responses to the dog yard item would reveal about students’ prior knowledge and understandings of expected forms of response. We administered the dog yard item to 315 eighth-grade students across three middle schools in an urban school district that is racially, linguistically, and economically diverse. The students in our sample took the item as the last one of their year-end science exam (a situation outside the environment of the mathematics classroom, much as the NAEP assessments were administered outside of the usual mathematics classroom routine).

We did not use the NAEP scoring guide, except to look at the extended and satisfactory responses. The students in our sample faired somewhat better than the NAEP group, but the results still were low with 2 percent scoring in the extended or satisfactory level. On the encouraging side, we did have a couple of explanations where students used the expected justification structure of showing all possible areas of rectangles and concluding that the square had the largest area. We also had responses that showed an ability to argue from a more abstract perspective. For example, two students gave the following responses:

1

"No matter how many 4 sides, 4 right angled figures you try, you will find that a perfect square [a 9 x 9 square] will have the highest area. Look at [a 2 x 16 rectangle] for example, it has 32 square feet. [A 10 x 8 rectangle] has 80 square feet. [A 7 x 11 rectangle] has 77 square feet. In fact, the farther you get from having both sides equal, the more your area will reduce. So to get the most for your money, go with the square"

2

"This [a 9 x 9 square, A = 81] would be the largest area she could have because any other one would have one side bigger than the other which will end up causing it to have a smaller area than the square. It’s [a 6 x 12 rectangle, A = 72] still smaller than the square because the sides aren’t all even."

Both of these responses have the structure of a justification, a stated conclusion with a warrant (because…), and both demonstrate an understanding of the task and pattern of change in areas as length and width are altered. Both also rely on a linguistic rather than a symbolic or diagramatic explanation.

But, what of the students who did not garner a rating of Extended or Satisfactory? Was poor performance more a lack of mathematical understanding and mathematical background, or a lack of understanding the societally determined expectations for presenting a convincing argument?

Kenney and Linquist (in press) suggested that middle school students have prior experience with determining the maximum area for figures with fixed perimeters, and thus know the mathematics necessary to solve the item. For our sample of students, we checked with the schools and found that most of the eighth-grade students had done problems that looked at maximum areas possible with fixed perimeters. We also have evidence in the students’ responses of prior knowledge and experience with such items. The latter of the two responses displayed above seems more a reiteration of an established conclusion than an explanation of a relationship discovered as a result of doing this particular dog yard item. We also had students who wrote,

"She will have to enclose near a barn or house OR make the fence circular";

"Using metal fencing and enclose the yard & fence in a big circle";

"She could make it so each side has 9 feet of fencing! The shape would be a box and that would make 4 cornors [sic]. She could also make a circle that runs around 36 ft!"

The mention of a barn or house, and the suggestions of a circle also seem indicative of prior experience with similar types of items. Thus, students in our sample who answered:

"Square because it covers more area"; or

"9 ft by 9 ft is your best bet, it has the largest amount of area and you use up all the fencing,"

may have clearly understood the mathematical relationships within the item, but not the testing expectations in terms of providing a convincing justification. Based on our qualitative analysis of the students’ responses, at least a third of the eighth-grade students had the requisite mathematical understanding. Thus, we strongly agree with Morgan’s (1998) recommendation that the expectations for form of response must become part of the daily instruction in mathematics. Equitable instruction and assessment require that students get systematic instruction in the valued forms of mathematical explanations.

Linguistic Concerns Related to Equity

We also found evidence of linguistic or mathematical reading comprehension difficulties such as reading "four sides" as "four equal sides" or reading "whole number lengths" as meaning "even lengths":

"She can have 9 feet on each side. Ok you have 36 feet and 4 sides all sides equal you divide 36 by 4."

"all right angles, all even lengths, fencing on all sides, …"

Some students also thought of right angles as "squared" angles, thus requiring Julie to make "a squared fenced in area for her dog." These linguistic difficulties led students to a correct shape and area, but an incorrect justification. Other students simply wrote that they could not make sense of some of the words in this task or did not understand what the task was about.

Although language poses difficulties for all students who struggle with reading, it is a particular concern for students for whom English is not their first language. In order to be a good problem solver, one must be proficient in the language in general as well as the technical and symbolic languages of mathematics. Therefore, the limited English proficient (LEP) child may be at a disadvantage, not because he or she does not possess the necessary skills to solve the problem but because of a lack of "accessibility" in the second language (Mestre, 1981). Failure to master formal discourse styles may interfere with students’ understanding of word problems (Cummins, 1991). Mathematics has its own specific forms of discourse. Therefore LEP students must master this language as well. Students are required to combine their linguistic, cognitive, and meta-cognitive development to successfully comprehend the reading. Thus, in addition to the mathematics skills that they need to solve the problem, students must simultaneously develop the requisite comprehension skills while encountering text that is culturally biased. (Fencing in dogs is not a universal cultural activity.)

A second linguistic dimension to the interpretation of the dog-yard task emerged from our consideration of the students’ responses. Six students suggested making the dog yard some shape other than a square or rectangle (e.g., circle, hexagon, pentagon, or irregular shape using the sides of the doghouse or of Julie’s house). As we discussed these responses, we kept trying to make sense of these students’ written comments to Julie that their shapes were "what would work best." Were the students just answering the last question, "What is the largest area that Julie can enclose with 36 feet of fencing?" Or were they thinking in some other way? English education members of our research group suggested thinking of inflection as a factor in students’ reading comprehension. Julie may want to make a rectangular shape and use fence for all sides of the enclosure, but the best solution is clearly something else. In other words, the students may have interpreted the task as one of convincing Julie that her wants were not the best solution.

Social and Cultural Concerns Related to Equity

About three percent of the eighth-grade students (11 students) gave answers that indicated a divergent, yet reasonable interpretation of the context. Two students said that the problem could not be solved because they didn’t know the shape of Julie’s yard, i.e., "…I really don’t know how her yard looks and how her yard is measured;" or "[maybe] she can’t do it because her yard is too small." These responses support the concerns that Tate (1996) raises; that is, that some items are constructed from a cultural or economic perspective quite different from that of many of the students (e.g., a White, middle class, or Eurocentric perspective). The dog yard task seems designed from the perspective of having a relatively large yard. This is contrary to the "lived reality" of many of the students in our study. The lived reality for many of the students in our sample, especially those from the lower socioeconomic groups, is that they live in urban apartments or brownstones which have tiny (4-foot by 4-foot) front yards and no backyards. And those urban buildings that do have backyards often have oddly shaped ones that would not allow for a 9-foot by 9-foot square dog yard. Thus, students who placed themselves entirely within the context from their perspective may have dealt in a literal and self-situated way with the direction to find the maximum area for Julie’s dog yard.

An equitable approach to preparing students to respond in reflective ways to items such as the dog yard task might be that teachers help students to identify and solve from multiple perspectives. Tate (1996) suggests that teachers employ multicultural and social reconstructionist approaches where students ultimately are expected to solve the same question from the perspective of different members of the class, school or society. This necessitates teaching students the scientific habit of mind of always considering alternative assumptions and always suggesting solutions from alternate assumptions.

Logic, Reality, and Equity

Some divergent responses initially might be perceived as just idiosyncratic differences in people. However, we view these as indicative of child rather than adult logic, (as in Piaget’s argument that the intellectual structures of children are not the same as those for adults). Children’s thoughts, references, and logic are embedded in the details of the reality of the context. Once the mathematics has been embedded in a context, children are less able than adults to see the context as just a vehicle for understanding the mathematics. Children are less able to extract the mathematics from the context. For some students, this leads to an inability to respond as expected, because they cannot "get past" a contextual detail that the adult may have never considered. For example, some students in our sample were concerned about having a gate so that "the dog would be able to get out" and suggesting fencing around the dog house, but keeping enough pieces of fence to make a gate. Another student, who seemed to have prior knowledge about appropriate shapes of pens for dogs said a 6 by 12 rectangle should be used because that was the largest area that we could have while at the same time giving the dog room to run. One student thought it couldn’t be done with only 36 feet of fence and indicated some relationship between the height of the fence and the amount of fencing. As we discussed this response, we realized that the student seemed to be thinking that Julie had 36 board-feet of fence. Perhaps this student had experience buying lumber by the board-foot.

Although some of these types of attention to detail can be avoided by careful pilot testing and construction of tasks, there is no way to make a context free from alternative interpretations. We are brought again to the need to help students learn to provide multiple solutions from multiple assumptions (e.g., if we ignore the need for a gate, the answer is x; if we take into consideration the need for a gate, the answer is y).

Conclusion

Our work has brought us to the conclusion that the use of extended constructed response items in assessment offers a far better prognosis for equitable assessment than a return to multiple choice testing. The use of extended constructed responses opens the possibility to let students reveal their logic and argue from their lived realities (as well as from the realities of others). We see the requirement of written explanations or well constructed convincing arguments as a means to allow students to demonstrate what they know. We also see it as a means to change from a staid objective view of mathematics to what Tate (1996) argues for, mathematics "as a tool to guide social decision-making…influenced by the values of those who use it in human affairs" (p. 187). The mathematics education community is well into understanding and grappling with the challenges of using extended-constructed tasks in testing and the concomitant cultural, social and political implications.

References

Champagne, A. B., & Kouba, V. L.: 1997a, ‘Communication and reasoning in science literacy.’ In S. Sjoberg & E. Kallerud (Eds.), Science, Technology and Citizenship (pp. 75-92). Oslo, Norway: Norsk institutt for studier av forskning og utdanning (NIFU).

Champagne, A. B., & Kouba, V. L.: 1997b, ‘Assessment in the service of education: Assessment of science literacy standards.’ In Y. Bum-Hong et al. (Eds.), Globalization of Science Education: Moving Toward Worldwide Science Education Standards (pp. 167-172). Seoul, Korea: Korean Educational Development Institute.

Cummins, J.: 1981, ‘The role of primary language minority students.’ In California State Department of Education (Ed.), Schooling and Language Minority Students: A Theoretical Framework (pp. 3-49). Los Angeles: Evaluation, Dissemination and Assessment Center, California State University.

Dossey, J. A., Mullis, I. V. S., & Jones, C. O.: 1993, Can Students Do Mathematical Problem Solving? Results from Constructed-Response Questions in NAEP’s 1992 Mathematics Assessment. Washington, DC: National Center for Education Statistics.

Kenney, P. A. & Lindquist, M. M.: in press, ‘Students’ performance on thematically related NAEP tasks.’ In E. A. Silver & P. A. Kenney (Eds.), Results from the Seventh Mathematics Assessment of the National Assessment of Educational Progress. Reston, VA: National Council of Teachers of Mathematics.

Kouba, V. L.: 1999. ‘Multiple interpretations = more challenges.’ Mathematics Teaching in the Middle School, 5(4), 232-235.

Mestre, Jose P.: 1981. 'Predicting academic achievement among bilingual Hispanic college technical students.' Educational & Psychological Measurement, Vol. 41, pp. 1255-1264.

Morgan, C.: 1998, ‘Assessment of mathematical behavior: A social perspective.’ In P. Gates (webmaster), First Mathematics Education and Society Conference Proceedings. Retrieved Oct 29, 1999 from World Wide Web: http://www.nottingham.ac.uk/csme/meas/papers/morgan.html

National Center for Education Statistics (NCES): 1999, The Nation’s Report Card: What Is NAEP [Sample Questions]. Retrieved October 20, 1999 from World Wide Web: http://nces.ed.gov/nationsreportcard/ITMRLS/INTRO.SHTML

Tate, W. F.: 1996, ‘Mathematizing and the Democracy: The Need for an Education that is Multicultural and Social Reconstructionist. In C. A. Grant & M. L. Gomez (Eds.), Making Schooling Multicultural: Campus and Classroom. Englewood Cliffs, NJ: Merrill.