Printable Version 

2004 NSF K-12 Math, Science, and Technology Curriculum Developers Conference

SEPUP Implementation Evaluation

Barbra Nagle

SEPUP is currently completing a middle school sequence with the development of a new sixth grade course, titled Issues in Earth Science. This course will be first in a series that will include revised editions of two existing courses: Science and Life Issues (Grade 7, initially published in 2001) and Issues, Evidence, and You (Grade 8, initially published in 1995). Current efforts to evaluate implementation focus on 1) teacher feedback and implementation of the first field test version of Issues in Earth Science and 2) student learning as a result of implementation of the commercial edition of Science and Life Issues.

Evaluation of the field test implementation of Issues in Earth Science focuses on the instructional materials themselves, teacher implementation of the curriculum, and student learning as a result of implementation. Data sources from the first year of field testing include: 1) teachers' feedback about each of the eight units of the course and about each individual activity, 2) developer and teacher associate feedback on each unit, 3) scientists' content review of each unit, 4) student work samples , 5) pretests/posttests to assess student learning for each unit, 6) the external evaluator's observations of professional development sessions and interviews with teacher participants, 7) the external evaluator's classroom observations, and 8) teachers' responses at the end of the year to a survey about the curriculum and classroom implementation.

Initial findings from the online feedback provided by the teachers after they complete each unit of the course provide insight into enactment of the curriculum. Common modifications consistent with unit goals are: use of additional time to address key concepts; additional strategies for using, reviewing, and providing examples of key vocabulary terms introduced in the materials; demonstrations to provide further examples of concepts; examples of local issues related to course content; and providing more activities on the unit topic. Modifications also included insertion of activities on topics that the course developers planned to include later in the curriculum and introduction of concepts and/or vocabulary more distantly related to course goals.

Teacher responses indicate that the majority of teachers read and use the Teachers' Guide and suggested teaching approaches. They report that the teaching materials are helpful in supplementing content area gaps and providing suggestions for how to introduce lessons and develop concepts. Common suggestions for improvement include requests for more content background and more examples of strategies such as concept maps.

For the evaluation of Science and Life Issues, a pretest/posttest has been developed to measure student learning of key concepts in the course and related to the National Science Education Standards. The test includes extended items scored with the SEPUP Scoring Guides as well as multiple choice and short answer items. Items have undergone two rounds of pilot testing. Psychometric analyses have established item fit, discrimination, and separation reliability. During the 2003-2004 academic year a pilot evaluation of student learning is being conducted at four sites (in South Carolina, New York, Texas, and California). Participating teachers are completing a survey about how they have implemented the materials and assessment system.

This project is supported by the National Science Foundation (Grant No. ESI-0352345). Any opinion, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.