To assess the effectiveness of the MINDful Play Environment for learning, we are creating two test sites. The first, an exhibit called "VJDJ," located at the Oregon Museum of Science and Industry (or "OMSI"), will test math and science concepts. The second, called "Rhapsody Room" is located at Grigar's lab at Washington State University Vancouver (the MOVE Lab) and will test language concepts.
The objective of the evaluation plan is to monitor and assess the extent to which the behavioral exhibit goals are achieved. This will be accomplished through formative and summative evaluation activities in which multiple measures will be collected using qualitative and quantitative methods.
The OMSI Test Site
Evaluation activities will be led by Dr. Marcie Benne, OMSI's Senior Evaluator. The objective of the evaluation plan is to monitor and assess the extent to which the behavioral exhibit goals are achieved. This will be accomplished through formative and summative evaluation activities in which multiple measures will be collected using qualitative and quantitative methods.
The three behavioral goals for this exhibit are: 1) visitors will understand how an x, y, z coordinate system is used to program the MPE experience, 2) visitors will understand how triangulation is used to determine position, and 3) visitors will create a more interesting performance when they understand how the coordinate system is mapped in the 3-D space.
Prior work in the classroom has shown that constructivist and situated cognition educational strategies can be effective in promoting math learning (Moses and Cobb, 2001; McClintock Peck, 2005; Sutton and Krueger, 2002; Donovan and Bransford, 2006). Therefore, this exhibit is expected to have these characteristics: 1) a math rich experience within a familiar context, 2) an opportunity and encouragement to represent math in multiple ways within the activity, 3) a bridge between everyday events (informal understandings) and more formal mathematical understandings, 4) a multi-sensory, interactive experience, and 5) characteristics to promote math talk and strategy sharing among groups (such as the family-friendly exhibit characteristics of multi-user, multi-outcome, multi-modal, multi-sided, relevant, readable, and accessible, Borun et al., 1997). Facilitation may also promote these characteristics.
Throughout formative and summative evaluation, the presence of these characteristics in the exhibit will be monitored along with other characteristics related to best practices in the museum field (e.g., label guidelines, Serrell, 1996; immediate apprehendability, Allen, 2004; and safety).
The measures of success for the exhibition are that visitors 1) engage with math and have fun, 2) can state in their own words the messages about x, y, z-coordinates and triangulation, 3) experiment with x, y, z-coordinates, 4) share math discourse from three category types – answer and explain, compare and extend, generalize, and 5) make or recognize connections between mathematical thinking and their everyday lives. These measures are adapted from research on math learning in the classroom (Moses and Cobb, 2001; Sutton and Krueger, 2002; McClintock Peck, 2005; Donovan and Bransford, 2006; Kalathil, 2006).
Formative evaluation will provide the development team with feedback on the extent to which the activity and interfaces afford the desired visitor outcomes. Visitors will be observed during their interaction with the exhibit then interviewed after their experience. These data will be reported back to the team in a timely manner so they can make decisions to improve the activity and interface while it is in prototype form. In fact, team members will participate in the observations and interviews so they have first-hand knowledge of the issues.
Summative evaluation will provide final feedback on the extent to which the MPE exhibit achieved the behavioral goals. During the summative evaluation, we will consider the exhibition successful if two-thirds of the target audience demonstrate the expected measures of success. We expect the majority of the visitors to indicate positive qualitative changes in their perspective on math learning as intended by developers. For instance, visitors may state they had not realized that math learning could be so much fun.
The MOVE Lab Site
This part of the study uses both quantitative and qualitative methods. Participants (N=24) at both sites will be ages 10 to 11 (Grades 4-5) and have characteristics of low academic performance.
Evaluation activities will be led by Dr. Michael Dunn, a specialist in literacy at Washington State University Vancouver. At the MOVE Lab site, students from the Vancouver School District (WA) students will be asked to participate. The students' teachers will introduce the MOVE lab and attain verbal assent from the students choosing to participate; these students will attain their parents' signed consent. This informed consent statement will allow the researchers of this project to attain demographic information about each participant (current grade, ever retained/accelerated in a grade, recent reading achievement data, recent state assessment (WASL) data, disability (if any), race/ethnicity.
Students will participate three times per week for a total of twenty-five days (a total of twenty-five hours). Each day's hour of activities will be comprised of three twenty-minute segments. The first twenty-minute segment will consist of a writing sample given in a traditional classroom environment. The aim of this sample is to determine students' understanding of written communication. Specifically, the researchers are interested in learning students' ability to “analyze ideas, select a narrow topic, and elaborate using specific details and/or examples" ("Writing: K-10 Grade Level Expectations").
The second twenty-minute segment will have students enter the MOVE lab and complete kinesthetic activities developed for them.
In the third and final twenty-minute segment, student participants will return to the classroom and complete another writing sample. Students will also be able to post their reflections about the day's experiences to an online password-protected discussion forum such as Blackboard.
Writing samples will be scored by the Rainer Group located in Seattle, Washington and sent to research team for analysis. Data from both sites will be collected and a statistical analysis of the data will be conducted by Dr. Michael Raisinghani, a specialist in data analysis. Using SPSS 14.0 for Windows, a descriptive and inferential analysis of the data will be undertaken. A report of the findings and its implications for theory and practice will be provided for continued development of the project.