This report summarizes the research methods used by EDC for the units developed by EDC for the MathScape curriculum. The main goal of EDC's research was to inform the development of the curriculum units. The research process consisted of two phases. Phase One was conducted by EDC curriculum developers and involved the following: summer institutes with middle school teachers; review of relevant research on students' learning of mathematics; and early formative testing in which project staff tried out activities with groups of students. These research activities helped to define the design principles used throughout the curriculum. Phase Two was conducted by an EDC research team and Inverness Research Associates and consisted of extensive field testing by teachers in their classrooms and reviews of the unit drafts by math educators, mathematicians, and middle school teachers. Each of these research methods is described below.
These institutes in 1991 and 1992 were attended by over 50 middle school teachers from a wide range of school districts. The sessions were designed to enable EDC to gather information from teachers on the following topics: characteristics of middle school students; teaching conditions; advantages and disadvantages of current math textbooks; areas of difficulty in mathematics for many middle school students; and common students misconceptions in mathematics. Teachers were also asked to share math activities that they thought were exemplary. In the second summer institute, teachers were organized into working groups that were led by EDC curriculum developers. The teachers contributed to the development of contexts and mathematical investigations for different units.
EDC curriculum developers examined math education research to find out about students' learning, common misconceptions, and successful methods for addressing them. Some of the sources they used were: Arithmetic Teacher journal, Mathematics Teacher journal, NCTM yearbooks, and NCTM publications on middle school mathematics. Developers also reviewed research on teaching mathematics to students from different cultural and linguistic backgrounds and on middle school reform.
Curriculum developers tried out initial versions of investigations with small groups of students and whole classrooms. This helped developers get a sense of the viability of the investigation and gain insights into students' mathematical thinking. Developers used information gathered from formative testing and from relevant educational research to write a first draft of the unit. The first draft was then reviewed by MathScape project staff and revised before going into field testing.
About the field test sites
The EDC-developed units have been field-tested extensively in many Massachusetts and California school districts. In Massachusetts, middle school teachers in 9 school districts and over 20 schools have field-tested MathScape units. The research team carefully selected field-test sites to insure that there would be a diversity of teachers, students, and school settings.
The teachers who have field tested MathScape materials represent a diversity of teaching experience and styles. They include novice and veteran teachers, ranging from 2 to 30 years of teaching experience. They also include a wide range of teaching and mathematics backgrounds. Some have taught middle school math only; others have previously taught middle and high school math and/or science; several have previously taught in elementary grades; and some have previously taught or are currently teaching middle school science, language arts, and social studies. While all of the teachers have an interest in trying something innovative, they vary in terms of their experiences and expertise with several aspects of the MathScape pedagogy such as collaborative work, alternative assessment, and communication about mathematics through writing and discussion.
The students in the field-test classrooms represented a wide range of ethnic and economic backgrounds, as well as a diversity of interests and abilities in mathematics. The field-test classrooms included classrooms that were both "heterogeneously" and "homogeneously" grouped according to students' abilities. In addition, several of the classrooms included ESL and special education students. The classrooms varied widely in terms of resources available, room size, class size, and seating arrangements.
Field-testing goals and methods
EDC's field-testing efforts focused primarily on testing individual units in order to gather information about the "classroom viability" of the units, that is to determine how well the units worked in real classrooms. The research team developed a Topical Framework to guide the collection and analysis of data. This Topical Framework includes questions on math content, contexts, classroom communication, collaborative work, assessment, teacher roles, and student roles.
Prior to field-testing a unit, teachers were given an initial half -day workshop orienting them to the unit. While teachers were teaching the unit, researchers conducted regular classroom observations, two to three times a week, as well as weekly interviews with teachers. The teachers also participated in extensive initial and exit interviews. In addition to collecting and analyzing student work, researchers also conducted informal interviews with students to assess their understanding of the mathematics content.
During each unit field test, the researchers met weekly with the developers to share their classroom observations and findings. Developers also visited field test classrooms so that they could see first-hand how the units were working and to get ideas for revisions.
In California, well over 1,000 teachers have used MathScape units, primarily in conjunction with two NSF-funded teacher development projects: the Middle Grades Mathematics Renaissance project, and the Math Matters project. Inverness Research Associates in California conducted research on how nine middle school teachers in a wide range of schools used the units in their classrooms. The Inverness researchers did classroom observations, teacher interviews, and analysis of student work. In addition, researchers from Inverness and EDC conducted telephone interviews with a large number of teachers who were using the MathScape materials. EDC developers met with teacher leaders in the Renaissance project to get their feedback and suggestions on the units.
Larry Guthrie of CREATE, a California-based research organization, conducted a research project in California schools exploring how MathScape units can be implemented in classrooms with linguistically and culturally diverse students. This research focused on evaluating how well the Language of Numbers unit met EDC's goal of developing "culturally open" activities-activities in which every student, from any cultural and linguistic background, can bring his or her own experiences and interests into the math class. Data was collection through classroom observations, teacher interviews, and student work analysis. The researchers identified useful strategies for using the materials with linguistically and culturally diverse students. EDC developers incorporated these suggestions into the revision of Language of Numbers and into the development of new units.
The EDC developers prepared drafts of the units for use by field test teachers These versions were also sent out to math educators, mathematicians, and teachers for review. The reviewers were asked to respond to a reviewer questionnaire that included questions about the mathematical content, pedagogical approach, sequence of investigations, and appeal of contexts to students. The developers used these reviews and the field test results to figure out how to revise the units.
Through this research process, the project staff gained an in-depth understanding of teachers' and students' understanding of the mathematics content, as well as their responses to the MathScape approach to teaching mathematics. The findings were used to inform the revisions to the unit, as well as to inform the development of future units.
At the end of a field test, the researchers prepared a list of the major strengths and issues for the unit, and provided information on all the topics in the Topical Framework. The developers and researchers discussed different ways to address the issues and revise the unit. Then, the developers used the comments from outside reviewers and the field test results to make a revision plan. This plan was reviewed by project staff. As a final step in this process, the developers revised the unit. If major revisions were made then the unit went through another round of field testing.
In addition, to informing revisions to the overall flow of the unit, and to specific lessons the field test teachers' classroom experiences were incorporated into the final units in the form of "From the Classrooms" and "A Teacher Reflects." These teacher vignettes provide examples of different ways that lessons can be taught; teaching and classroom management strategies, student reactions to investigations, and ways of addressing common student misconceptions. Student work from field test classrooms has also been incorporated into the units.
EDC researchers collected information on teachers' and students' general responses to the MathScape materials. Here are some of the findings:
Teachers and students recognize and value MathScape's innovative approach to teaching and learning mathematics. In particular, they have identified MathScape's emphasis on making real-world connections, communicating about mathematics through writing and discussion, and engaging in collaborative work and hands-on activities as instrumental in helping students develop a new view of mathematics. Here are some quotes from field test teachers:
Teacher reports and classroom observations suggest that the MathScape units are good vehicles for engaging students in mathematical thinking, and particularly for addressing students' mathematical misconceptions. The in-depth focus of the units provides time for students to grapple with difficult often counter-intuitive concepts. In addition, the variety of activity types (i.e., hands-on, visual, writing, discussion) allows students with different learning styles to achieve success. Finally, aspects of the teacher support materials, such as the sample lesson plans, sample discussion questions, From the Classroom vignettes, and Teacher Reflection narratives, help teachers to recognize, understand and address students' mathematical understanding and misconceptions. Here are some teachers' comments on the mathematics their students learned:
Teachers have identified ways in which the MathScape materials have helped them make changes in their classroom culture. Several teachers have described how teaching the units challenged them to make important changes in their teaching styles and therefore, in their students' roles in the classroom. Some teachers say the materials as an opportunity to begin to overcome their tendency to be too directive. In one unit, for example, a teacher struggled with her desire for students to complete an activity her way, until she discovered,
"But the truth is, their methods worked! ... It tends to work better if they believe it's going to work."
Other teachers were delighted that teaching the MathScape materials allowed them and their students to have positive experiences with collaborative work and communicating about mathematics through writing and discussion. In addition, a number of teachers have commented that as math teachers they don't often have the opportunity to get to know their students. Teachers value the opportunities to learn more about their students' personalities, as a result of activities which allow students to be creative and to communicate their ideas.
"It's wonderful. As a math teacher, I very rarely get to learn about my students. It's a wonderful relief for them and me... It's interesting to get to know them!"
Teacher and student reports, as well as classroom observations, have also indicated that the MathScape materials reach a wide range of students, including those students who don't normally do well in math class. In the middle of one field-test, a student who had really been struggling with more traditional math instruction all year declared proudly, "I think I should take advanced math next year!"
Similarly, a student participating in the field-test of another unit wrote in her math journal, "I'm starting to fell more and more confident. When I came to this school I wasn't confident in these sort of things... I only knew computation. This program is definitely helping me feel better about myself in math!"
Teachers and students have identified several characteristics of the MathScape materials that enable a wide range of students to be successful:
EDC used an extensive, two-phase research process to develop units for the MathScape middle grades curriculum.
The Phase One research was used to define the design principles of the curriculum. The Phase Two research was used to inform the revisions of individual units as well to inform the development of new units.
Project 2061, the long-term math and science education reform initiative of the American Association for the Advancement of Science (AAAS) conducted a study of middle grades mathematics textbooks. MathScape was included in this study. This was the first in a series of evaluations of mathematics and science textbooks to be conducted by Project 2061.
The benchmarks-based approach to evaluation was developed with funding from the National Science Foundation. Funding for the middle grades mathematics textbook evaluation was provided by the Carnegie Corporation of New York.
Middle Grades Mathematics Textbooks: A Benchmarks-Based Evaluation can be ordered from AAAS by calling 1-800-222-7809 or visiting Project 2061.