2001 AGI Curriculum Leadership Institute Summer 2001

I. Introduction II. Discussion III. Conclusion

Michael J. Smith, AGI Director of Education
Matt Smith, Programs Manager
Caitlin Callahan, Programs Assistant

I. Introduction

The intent of the weeklong AGI Curriculum Leadership Institute (CLI) was to prepare people to deliver presentations and training workshops and/or serve as teacher mentors on two new Earth science curriculum programs - EarthComm and Investigating Earth Systems. Institute facilitators included Lynn Sironen (EarthComm teacher), Marti Tomko (IES teacher), Don Byerly (Professor Emeritus, UT Knoxville), Barbara Zahm (Professional Development Coordinator at It’s About Time Publishing), and Mike Smith (AGI Education Director). Most CLI participants are secondary teachers whose schools have purchased an AGI textbook; however, some university faculty and state survey geologists also attend.Several organizations provided support for the CLI.

Evaluation of the Institute includes “check point” evaluation forms completed several times during the week and an evaluation questionnaire completed at the conclusion of the Institute. AGI staff uses feedback from these sources to adjust the plan during the CLI and to make revisions for future Institutes. For example, recommendations from the 2000 CLI included devoting greater attention to workshop training methods and assessment of student learning. This report summarizes the results of the final institute evaluation questionnaire for the 2001 CLI.

In the final evaluation, participants responded to 18 statements by choosing whether they strongly disagreed, disagreed, were neutral, agreed, or strongly agreed. Participants were also asked to respond to four open-ended questions. Responses were anonymous, identified only as IES or EarthComm. Responses to the 18 statements were converted to a five-point scale, ranging from 1 (strongly disagree) to 5 (strongly agree). The questions are worded both in the affirmative and negative, which means that low scores are preferred on some items. Analysis of data included calculating the minimum, maximum, mode, median, mean, and standard deviation for each item on the questionnaire, and reviewing the content of open-ended items on the backside of the evaluation form. Statistics were calculated for the entire group (N=29), the EarthComm subgroup (N=16), and the IES subgroup (N=13). Results were also compared to evaluations of last year’s CLI. The results of this year’s final evaluation are summarized in three tables, each of which will be discussed in turn below. 

Back To Top

II. Discussion

Table 1 summarizes the overall results for the 18 items. Participants were very satisfied with the CLI. Mean scores for affirmative items ranged from 3.8 to 4.9 on a 1 to 5 scale, with modes of 5 (strongly agree) on 11 of the 15 items. Mean scores for negatively worded items ranged from 1.3 to 1.7, and modes for all three of such items were 1 (strongly disagree). Participants emerged from the Institute confident about their ability to present the programs and feeling that they understand the content of the programs. They rated the teaching of a module during the Institute (Item 17) one of the strongest aspects of the program. Item 9 (amount of time devoted to assessment) showed the greatest variance. Assessment strategies were discussed as they arose in presentations of modules and in a special session lasting 90 minutes on Tuesday morning. The variance in scores on the assessment item was expected given 1) the complexity of the issue, and 2) the wide range of participants’ understandings and attitudes towards assessment revealed during the special session. Variance was also high on item 5 (importance of providing daily feedback to staff). Comparing the means for the EarthComm and IES groups (Table 2) reveals lower scores on this item for the IES group. The difference most likely stems from forgetting to distribute feedback forms to IES group members several times during the Institute.

A sampling of responses to question 1 of the open-ended items on the evaluation illustrates how participants felt that the Institute had prepared them to go forward as EarthComm and IES Leaders:

1. Please describe your impression of the effectiveness of the institute in terms of preparing you to deliver a presentation, a training workshop, or serve as a mentor on the curriculum:

• I feel well prepared to present the format and content of the curriculum at a presentation or workshop. By practicing presenting an investigation, I feel I will be comfortable at showing how this programs gives content through inquiry.
• I have done professional development several times. I believe that this experience will only enhance my skills at presenting and training workshops. I look forward to it.
• If students learn best by hands-on or doing, then presenters become adept through practice. The institute has given me a concentrated block of time in which to observe, read, and practice. 
• Overall, I feel I am in very good shape to do this – lots of learning occurred in little time!
• Very effective. The institute provided a model for developing a workshop.
• The workshop was an excellent way to better understand how the whole EarthComm system works and how to make it an effective part of an Earth science program. I am very confident in presenting a workshop on EarthComm.
• I thought that it was very effective. The leaders had high expectations that were communicated to us.

Although responses were overwhelmingly positive, several participants asked for greater attention to Earth science content, more time to voice their opinions (especially when these opinions conflict with those held by facilitators), or the chance to present an entire chapter (rather than one activity and a chapter summary). As discussed previously, the session on assessment revealed a diversity of perspectives on this issue, including some that were out of sync with the approach used in the programs. Although it is hard to reach consensus on complex issues in a short time, facilitators need to conclude discussions in ways that ensures participants that differences in opinion are respected. 

Table 2 enables a comparison between the results from the EarthComm and IES groups. Both groups responded favorably to all items, with minor exceptions amongst individuals. The means for the two groups show a strong correlation (r = 0.97). Participants in the IES group responded less favorably (although still positively) regarding the amount of time devoted to assessment (item 9), and more favorably to the balance of content, pedagogy, and training methods (item 13). Due to the small sample size, other differences between the two groups are probably insignificant. For example, if one person strongly disagreed with an item to which everyone else strongly agreed, the overall mean would fall 0.3 points. A chart of the comparison between the two groups is provided in Figure 1. 

The second question on the evaluation asked participants to describe what they found to be most beneficial about the Institute and what topics they would have liked to be covered in greater depth. The responses were nearly unanimous that teaching and performing the activities was the most beneficial component of the Institute. Other comments included:

2. Please describe what you found to be the most beneficial component(s) of the workshop.

• First of all, there was very little content that was presented that was not of benefit, some [of it] more interesting than others. However, the best was working with my fellow educators to produce, do, and present the activity and chapter challenge.
• In addition to presenting the module, I think it was important to hear how teachers "teach" the module. It's the little tips on class management or where to find content material that would make me more comfortable with the modules.
• Being divided into groups so that I was exposed to the other books even though it was on a limited basis.
• Great inquiry curriculum, diversity of leaders, [discussion of] learning styles and information from other participants
• Sharing and comparing [experiences], web sites, seeing a module unfold into reality. [Learning about] resources that are available to everyone.

Several participants appreciated hearing from the It's About Time representatives; the Sales and Marketing staff presentation assuaged concerns about how to get information and materials for presentations or workshops and when the books would be available. In 2000, a mix of teachers and geologists were invited to the Institute. The final evaluations from 2000 indicated that it was a good idea to do so again. This year, teachers said they were grateful for the content support from the geologists and geologists commented they learned from the experience and perspective of the teachers. A selection of responses to question 3 on the evaluation provides some recommendations on what to improve for next year's Institute:

3. Please indicate what topic/content/concepts you would have preferred to see covered in greater depth.

• The pedagogy/leadership/training methods pieces seemed less organized, lacking a punch line you wanted us to leave with. They all seemed to be geared to give us an awareness level, but not much beyond that. 
• More coverage on specific structuring of workshops.
• Strategies for encouraging inquiry, cooperative group strategies and questioning skills.
• I would have liked some content sessions such as the latest on global warming, ozone hole, plate movement, glacial movement, space exploration, or any topic in Earth Science just to elevate our own knowledge base.
• The assessment conversation was incomplete so thanks for the additional reading material.
• A little more about expectations now that we've completed the Institute.

Although it is unlikely that we could address all of these concerns within a weeklong workshop, AGI will revisit the issues when planning next year’s CLI and will also ensure that the content and format of the CLI is communicated clearly to participants in advance. 

Participants at the 2001 CLI received nearly the same final evaluation as the participants at the 2000 CLI. Table 3 allows for comparisons of evaluation results for the questions that remained the same from the 2000 and 2001 Institutes. Overall, the responses improved from the 2000 Institute to the 2001 Institute. The greatest improvements over the 2000 evaluations were in statements 1, 3, 9, and 17, which dealt primarily with the design and implementation of the workshop curriculum (Items 3 and 17) as well as the scheduling of time for discussion (Items 1 and 9). More participants agreed that they had ample time to share their ideas (Item 1). Participants reported a higher degree of satisfaction about the way instructional methods were modeled for the approach being advocated (Item 3). In addition, they were more positive that the experience of teaching a module during the Institute would help them prepare others to understand the program (Item 17). 

Responses to a few questions indicated a slight decrease in workshop effectiveness in conveying some particular points, though in each case the 2000 and 2001 results are identical within statistical uncertainty. Although the mean score for statement 9, the amount of time allotted for discussion of assessment of student learning, improved from the 2000 CLI, it was one of two statements with a score below 4. The utility of daily feedback to the instructors (Item 5) also had a mean score below 4, which was slightly lower relative to the 2000 results but statistically identical. This is in part due to one very poor evaluation, but also likely accurately represents the general sentiment. Statement 5 is clearly written, but rewording statement 9 may improve by making it less subjective. The decrease in the score of statement 7 probably reflects a change in demographics of the participants (a higher ratio of geologists to teachers in the 2000 CLI compared to 2001) rather than to content or manner of presentation. There is probably no significance in the decrease of statement 8 score from 4.7 to 4.6, both of which are excellent.

Back To Top

III. Conclusions

The second annual AGI Curriculum Leadership Institute prepared 29 teachers, geologists, and science supervisors, and 14 representatives from the It's About Time publishing to serve in a leadership capacity for the EarthComm and Investigating Earth Systems curricula. The final evaluation survey results were overwhelmingly positive. Participants left with a sense of what is expected of them as a graduate of the Institute. In both 2000 and 2001, a strong component of the training was that the participants had to teach a module to their fellow participants. Based on evaluation results from the 2000 CLI, leaders incorporated more group discussions into the agenda. The effort improved the experience for some participants, while others wished for more structure to the discussions. In fairness to workshop facilitators, some of these issues are very complex, and it is unlikely that participants would achieve consensus on any one topic within several days, let alone a 1.5-hour session. That said, AGI will improve organization and planning of the special sections by budgeting more time for discussion and synthesis of results. 

Back To Top

Table 1. Final Evaluation Results – 2001 AGI Summer Leadership Institute (N = 29). 

Scoring: 1=strongly disagree, 2= disagree, 3=neutral, 4=agree, and 5=strongly agree. 

Back To Top

Table 2. EarthComm and IES Evaluation Results (EARTHCOMM N=16; IES N=13)

Scores: 1=strongly disagree, 2=agree, 3=neutral, 4=agree, and 5=strongly agree.
 

Correlation coefficient of mean scores for EarthComm and IES = 0.97. 

 

Back To Top

Figure 1. A comparison of mean evaluation scores between EarthComm and IES. Low scores are desirable for items 10, 11, and 16.

Items for which a high score is desirable
1. I had ample time to share ideas with other participants.
2. I am confident that I can give an awareness workshop to teachers because of what I learned in this institute.
3. Instructional methods were modeled in a way suitable for the approach being advocated.
4. I am satisfied with my level of participation in the institute.
5. It was important to provide written feedback to the workshop instructors each day.
6. I am now more knowledgeable about the content of the program.
7. My understanding of inquiry-based instruction was enhanced by the institute activities.
8. The concept of collaboration among students when learning science, including sharing information and discussing ideas, was reinforced during workshop activities.
9. I am comfortable with the amount of time spent discussing the assessment of student learning during the institute.
12. I understand what is expected of me as a graduate of the AGI curriculum leadership institute.
13. The institute had the right balance of content, pedagogy and training methods.
14. The performance of the institute instructors was excellent.
15. Institute staff were responsive to my needs during the institute.
17. I think that taking part in teaching a module during the institute will help me prepare others to understand the program.
18. I consider myself knowledgeable about earth science.

Items for which a low score is desirable
10. I still do not fully understand the pedagogy and goals of the program.
11. Teachers should use activities from this program to supplement to a textbook, rather than use it as a stand-alone program.
16. The field trip was not a good use of my time.

Back To Top

Table 3. Comparison of Results: 2000 and 2001 Summer Leadership Institutes.

Back To Top

Figure 2. Comparison of overall means scores on identical items on the 2000 and 2001 AGI Leadership Institute evaluations. Low scores are desirable on items 10, 11, and 16

Back To Top

Acknowledgements

The AGI Curriculum Leadership Institute was funded through grants and contributions from the American Geological Institute Foundation, The Geological Society of America, and the American Association of Petroleum Geologists Foundation.

Back To To