Full Text View
GSA Today
Article, pp. 4-10 | Abstract | PDF (455KB)
Increasing Undergraduate Interest to Learn Geoscience with GPS-based Augmented Reality Field Trips on Students’ Own Smartphones
Cover Image
Table of Contents
- Introduction
- Grand Canyon AR Field Trip Games
- Methods
- Data Analysis and Results
- Discussion
- Conclusions
- Acknowledgments
- References Cited
Search GoogleScholar for
Search GSA Today
Abstract
Field trips are a reliable method for attracting students into geoscience, yet for many high-enrollment college introductory courses, field trips are often impractical. Furthermore, introductory courses are often taught with a traditional lecture style that is poor at engaging students. This study examines the impact of augmented reality (AR) field trip exercises on the interest levels of students using readily accessible mobile devices (smartphones and tablets) as a means to provide simulated field trip experiences to a larger number of learners. The results of this study, involving 874 students from five different institutions, show that students who completed three geospatially oriented Grand Canyon field trip game modules were significantly more interested in learning the geosciences than control students and participants who completed only one module. More comprehensively, results from hierarchical linear modeling indicate three strong predictors of student interest in learning the geosciences: (1) the student’s initial interest, (2) being a STEM major, and (3) the number of AR field trip modules students complete. Notably, the race and gender of participants are not factors. Augmented reality field trips for mobile devices have potential to be an accessible and financially viable means to bring field trips to a diversity of students who would otherwise experience none. Results indicate these AR field trips increase student motivation to pursue geoscience learning.
Manuscript received 1 June 2016; Revised manuscript received 16 Sept. 2016, Manuscript accepted 17 Nov. 2016; Posted online 13 Feb. 2017
Introduction
There has been considerable investment in addressing low interest, poor preparedness, and the lack of student success in science, technology, engineering, and mathematics (STEM)—including the geosciences (e.g., Seymour, 2001; Ashby, 2006; Fairweather, 2010). Recent reports claim that weak college STEM participation, especially among minorities, will negatively affect the U.S. economy (Ashby, 2006; National Research Council [NRC], 2011; Chang et al., 2014). Educators naturally desire to improve the participation and completion rates of all undergraduate students pursuing STEM degrees (Chang et al., 2014).
Most students enroll in introductory geoscience courses out of the need to fulfill their science requirement for graduation rather than being interested in learning geology (Gilbert et al., 2009; van der Hoeven Kraft et al., 2011; Gilbert et al., 2012). Moving from fulfilling graduation requirements toward promoting interest is important because research has shown that the best predictor of students taking additional classes in a subject is interest rather than performance (Harackiewicz et al., 2000; Hall et al., 2011; Gilbert et al., 2012). Unfortunately, many higher-education institutions teach high-enrollment (100+ students) introductory geoscience courses using online, broadcast, or lecture-based teacher-centered approaches that are relatively ineffective at stimulating interest in further learning (Andresen et al., 1996; Mazur, 2009; Deslauriers et al., 2011). Research has shown that one of the key factors in recruiting new geoscience majors is students having an engaging and positive experience in an introductory course (Levine et al., 2007; LaDue and Pacheco, 2013; Stokes et al., 2015). There is a clear need for learning experiences in introductory classes that increase the interest of students in order to inspire them to want to learn more about geoscience.
Field trips, when practical, are typically the most engaging and impactful component of courses, because these hands-on experiences inspire students to become geoscience majors (Orion and Hofstein, 1994; Tal, 2001; McGreen and Sánchez, 2005; Fuller, 2006; Kastens et al., 2009; Mogk and Goodwin, 2012). The liability of travel and decreasing financial and administrative support at many colleges have made it so that it is becoming increasingly rare to have field trips. Furthermore, for high-enrollment lecture, online, or broadcast classes, the logistics of a field trip are just unfeasible. In contrast, smartphones and tablets are becoming ubiquitous and educational applications for them are numerous (Dahlstrom and Bichsel, 2014; Anderson, 2015). Considering students’ high comfort level with smart devices and gaming, leveraging portable devices for education could have a positive impact on student interest and engagement (Bursztyn et al., 2015). Studies have shown that gaming features contribute to greater student self-confidence and self-efficacy through increased engagement in the activity (Mayo, 2009). The game-like features of the augmented reality (AR) field trips presented in this research, in combination with convenience, low cost, and broad accessibility, are anticipated to contribute to a greater learning experience. A companion series of field-trip game modules for smart devices, now publicly and freely available, was tested for impact on students’ interests in introductory geoscience classes at a variety of post-secondary schools.
Grand Canyon AR Field Trip Games
Our field trip modules are based on relative GPS locations and conceptualized after the location-based GeePerS math games built by the IDIAS lab at Utah State University, at a time before Pokémon Go was released to the public and became the most-downloaded app of all time (GSA Data Repository 1 expanded methodology; http://idias.usu.edu/; Shelton et al., 2012). For each AR field trip the entirety of Grand Canyon has been scaled down to a 100 m playing field. The absolute geographic location of the player does not matter; however, because GPS is integrated into the application, the module must be played outside (Fig. 1). The design takes advantage of the benefits of games that provide immersion-in-context, rewards for correctness, and immediate feedback in response to student interaction. Each module takes ~20 min to play, a length of time aimed to fit within a wide range of class types and capture the typical student’s attention span (Middendorf and Kalish, 1996; Milner-Bolotin et al., 2007).
1GSA Data Repository Item 2017056, expanded description of methodology, statistics, and geoscience interest survey, is online at www.geosociety.org/datarepository/2017/.
This study uses three fundamental geoscience topics that can easily be explored within Grand Canyon as the basis for the AR field trips: (1) geologic time, (2) geologic structures, and (3) hydrologic processes. For all three AR field trips the stops run downstream from Lees Ferry to Lake Mead with photographs, videos, questions, and interactive touchscreen activities (Table 1). As of 2016, these applications (called GCX Geologic Time, GCX Geologic Structures, and GCX Hydrologic Cycles) are available on both Android (Google Play) and iOS (App Store) platforms.
Methods
Participants
Students at three educational institutions completed all three AR field trip modules to provide data for analysis (n = 391). Students at a fourth school completed two modules (n = 138), and students at a fifth school only completed one module (n = 319). Finally, additional students at two of the schools (n = 291) acted as control subjects, completing the pre- and post-tests and surveys for their regular labs without participating in the AR field trip modules. All of the classes utilized in this study were traditional lecture-based courses with accompanying labs. The data set overall represents diverse demographics and institutions (classed as teaching focus, teaching-research split, and research focus), reported in Data Repository Table S1 [see footnote 1].
Interest Index
All students, including intervention and control groups, completed a demographics survey, geoscience content questions (for the student learning component of this research, not reported in this paper), and the Geoscience Interest Survey. The evaluation instrument (the GeoIS) was used at the beginning of the semester and then after all interventions were complete. The GeoIS is a modified subset of the Motivated Strategies for Learning Questionnaire (MSLQ) using the task value component subscale and the situational interest subscale; see Data Repository Figure S1 [see footnote 1]. The MSLQ subset that comprises the GeoIS evaluates how interesting, useful, and important the course content is to the student, and should relate to student engagement by assessing changes in interest post-intervention (Pintrich et al., 1991; Harackiewicz et al., 2008). Motivation self-report subscales used to measure value beliefs (intrinsic goal orientation, extrinsic goal orientation, and task value beliefs) and self-report interest subscales (individual interest: interest in the subject residing within the individual prior to taking the course; and situational interest: emerging spontaneously in response to exposure in the environment) have been validated by the educational psychology field, and have been adapted to suit the geosciences (Pintrich and DeGroot, 1990; Pintrich et al., 1993; McConnell et al., 2006, 2009; McConnell and van Der Hoeven Kraft, 2011; Harackiewicz et al., 2008; van der Hoeven Kraft et al., 2011; Gilbert et al., 2012). The MSLQ has robust reliability data with prior studies and has both predictive validity and construct validity in the form of a confirmatory factor analysis.
Two main research questions guided the analysis of data: (1) How do these AR field trips impact student interest in learning geoscience material? and (2) Which demographic and experiential factors combined with the AR field trips best predict student motivation and interest to learn geoscience material?
Data Analysis and Results
The data analysis used three steps: (1) determining reliability and validity of the data, and generating a correlation matrix of the variables; (2) running an analysis of covariance (ANCOVA) to determine the degree of impact of the AR field trips on student interest; and (3) running a hierarchical linear model (HLM) to determine the predictors of student interest.
We assessed the inter-item reliability of the GeoIS by means of a Cronbach’s alpha analysis. While test–re-test reliability between pre- and post-tests was a possibility, we felt that inter-item reliability was more insightful given that everyone was exposed, and change was anticipated. Positive values for alpha (up to a max of 1.00) indicate that there are greater differences of opinion between learners. The observed values of 0.91 for the pre-intervention and 0.93 for the post-intervention GeoIS instrument indicate a high level of reliability (Murphy and Davidshofer, 1988). Given the established nature and prior research conducted with the MSLQ, we chose to use a confirmatory factor analysis to assess instrument validity of the GeoIS. The fifteen GeoIS items coalesced onto a single factor based on 874 observations with loadings ranging from 0.17 to 0.83. Based on this combination of observations and loading values, the adapted MSLQ instrument appears to measure a single construct at a significant level (Stevens, 1999). The correlation matrix (Data Repository Table S2 [see footnote 1]) revealed four statistically significant variables: (1) the pre-intervention survey score; (2) institution; (3) STEM major; and (4) number of AR field trips completed. Despite a lack of statistical significance, race and gender were kept as theoretically important variables for the nested regression analyses.
First-order examination of the pre- and post-intervention GeoIS scores shows a trend of increased student interest across all participants (Fig. 2). There is a distinctly greater increase in student interest among those participants who completed two and three AR field trips over those who completed only one or were in control groups (Fig. 2). In order to test for differences empirically, we used an Analysis of Covariance (ANCOVA). As recommended when students are not randomly assigned (Campbell and Stanley, 1963), we controlled for preexisting differences by using the pre-test as a covariate. The results of the ANCOVA (Table 2) indicate that the number of field trips completed does play a role in student interest: F(3, 589) = 17.55, p <0.01. Pairwise comparisons in the same table suggest that students completing three AR field trips were significantly more interested in learning geoscience in the future than students completing one or zero AR field trips.
In an effort to determine what predicts students’ interest in the geosciences, we ran a hierarchical linear model (HLM). Expanding on the basic idea of regression with a set of predictor variables and an outcome, HLM accounts for data that are nested (Raudenbush and Bryk, 2001). In this case, students came from different schools with different instructors and different regional geologic features that can play a role in curriculum decisions. The HLM adjusted for school differences by using two levels (site and student) with six predictors of geoscience interest: (1) GeoIS pre-intervention score; (2) number of AR field trips completed; (3) site classification; (4) gender; (5) race; and (6) STEM major. After a null model (Table 3) that ignored the predictors, subsequent models explored both student and site level variables. Goodness of fit (AIC and BIC) suggests that a parsimonious model with only significant predictors is a strong fit for these data. The results of the parsimonious model (Table 3) indicate that there are three strong predictor variables for student interest toward learning the geosciences: (1) GeoIS pre-intervention score at both the student and the site level; (2) being a STEM major; and (3) the number of AR field trip modules students are exposed to and complete (bolded in Table 3). The third predictor variable is of utmost importance to the study because this finding shows that interest gains associated with students completing all three AR field trips (Table 3: 3 × 1.72 = 5.16) are more than twice the gains associated with being a STEM major (Table 3: 2.18). Note that each of the values shown in bold in Table 3 represents a point value gain (out of 70) on the GeoIS post-intervention.
Discussion
The AR field trip modules tested in this study incorporate within their design two fundamental field-trip features, primarily orienteering and physically moving between geo-referenced field trip locations. The nature of this design allows for the “get out of the classroom and contemplate geology with your peers” component of the field experience to be had by all, even if just on a campus quad or soccer field (Fig. 3). The focus of this research was to determine what impact on student interest in learning geoscience material this AR field trip experience provides, because interest has been shown to be the best predictor of students pursuing additional classes in a subject area (Harackiewicz et al., 2000; Hall et al., 2011; Gilbert et al., 2012).
Exposure to and completion of all three mobile AR field trips had a significant impact on student interest to learn the geosciences. Specifically, HLM results indicate that completion of one single module increases student interest almost as much as does being a STEM major. Completion of two or three AR field trips further builds this interest.
The following factors were not at all significant: race, gender, and site classification. These results indicate that the AR field trips were effective despite variation in student demographics, which is similar to Gilbert et al. (2012), who found no variation in student motivation across gender or ethnicity in introductory geology classes. Note that the study conducted by Gilbert et al. (2012) was based on a single MSLQ survey of students at multiple institutions to ascertain who is enrolled in introductory geology courses and why they are enrolled in those classes; the authors did not measure a change in student motivation or interest after an intervention.
Furthermore, the improvement in student interest irrespective of site classification group suggests that the modules are impactful regardless of teacher, type of institution, class size, or geographic location. These findings are in contrast with Chang et al. (2014), who found students had increased persistence (less attrition) at research universities and increased motivation at liberal arts colleges over public universities and community colleges. Chang et al. (2014) used large scale survey data to track student persistence in a STEM field from their freshman year to four years into their undergraduate education; thus, these authors also did not assess a change after an intervention.
Are these AR Grand Canyon field trips useful in comparison to real on-location field trips? The gains in student interest are expected (and desired), in part because of the game-like design of the field trip modules and in part because of the interactive out-of-the-classroom experience, emulating a real field trip. Geoscience educators have long known that field trips are major attractors of students to the science, and with ubiquitous smartphones, mobile technology, games, and apps for everything, it is not surprising to find that this medium appeals to the current generation of undergraduates. The AR field trips are flexible enough to be used during a lecture period, a lab period, as homework, or as supplementary activities for online learning. One could oversimplify the hypothesis and purpose of this research by saying that since field trips are fun and games are fun, of course gamified-augmented-reality–field trips are fun! Consequently, if the students are having fun while learning the course material, there is an expectation that their level of interest and motivation to pursue study in the field will increase. In the face of economic, geographic, and/or accessibility issues that some institutions face that are prohibitive of field trips, the AR field trips are an affordable and easily implemented solution.
Conclusions
Gilbert et al. (2012) state that many post-secondary geoscience educators rank student motivation as the most important indicator for student learning. This study presents a solution not only for increasing student interest and engagement in the subject, but also the potential for increasing student learning. The AR Grand Canyon field trips for mobile smart devices are an accessible, inexpensive resource that can bring field trips to campus in lieu of students experiencing none at all. Furthermore, the findings described here are encouraging for this AR and other virtual field trip genre of pedagogy. Addressing if and how students may learn better using AR field trips is a critical question, with promising initial results (Bursztyn et al., 2016). The psychomotor aspect of AR field trips holds theoretical underpinnings that certainly require additional attention from researchers in how students remember and recall information. Teachers are experiencing the dawn of educational tools for mobile devices in the form of apps for all ages, including these Grand Canyon Expedition modules. Now that the efficacy of these AR field trips in motivating students to learn is established, the important question remaining is if they are effective at actually increasing student learning.
Acknowledgments
We thank our collaborating geoscience faculty focus group: Dr. Michelle Fleck, Richard Goode, Dr. Eric Hiatt, and Dr. Laura Triplett. We also thank all faculty and their students at our participating test institutions who provided data for this research. We are most appreciative of the NSF TUES award #DUE-1245948 that made this research possible and allowed us to hire Sovrn Creative of Boise, Idaho, USA, to do professional programming of the apps.
References Cited
- Anderson, M., 2015, Technology Device Ownership: 2015: Pew Research Center, October 2015, http://www.pewinternet.org/files/2015/10/PI_2015-10-29_device-ownership_FINAL.pdf (last accessed 12 May 2016).
- Andresen, L., Boud, D., and Cohen, R., 1996, Experience-Based Learning, in Foley, G., ed., Understanding Adult Education and Training, 2nd ed.: Sydney, Allen and Unwin, p. 225–239.
- Ashby, C., 2006, Science, technology, engineering, and mathematics: Trends and the role of federal programs. Testimony before the Committee on Education and the Workforce, House of Representatives: http://www.gao.gov/new.items/d06702t.pdf (last accessed 26 May 2015).
- Bursztyn, N., Pederson, J., Shelton, B., Walker, A., and Campbell, T., 2015, Utilizing geo-referenced mobile game technology for universally accessible virtual geology field trips: International Journal of Education in Mathematics: Science and Technology, v. 3, no. 2, p. 93–100.
- Bursztyn, N., Walker, A., Shelton, B., and Pederson, J., 2017, Assessment of student learning using virtual Grand Canyon field trips for mobile smart-devices: Geosphere (in press).
- Campbell, D.T., and Stanley, J.C., 1963, Experimental and quasi-experimental designs for research on teaching: Chicago, Rand McNally, 84 p.
- Chang, M.J., Sharkness, J., Hurtado, S., and Newman, C.B., 2014, What matters in college for retaining aspiring scientists and engineers from underrepresented racial groups: Journal of Research in Science Teaching, v. 51, no. 5, p. 555–580, doi: 10.1002/tea.21146.
- Dahlstrom, E., and Bichsel, J., 2014, Study of Students and Information Technology, 2014. Research Report: Boulder, Colorado, EDUCAUSE Center for Applied Research: http://www.educause.edu/library/resources/study-students-and-information-technology-2014 (last accessed 16 May 2015).
- Deslauriers, L., Schelew, E., and Wieman, C., 2011, Improved learning in a large-enrollment physics class: Science, v. 332, no. 6031, p. 862–864, doi: 10.1126/science.1201783.
- Fairweather, J., 2010, Linking Evidence and Promising Practices in Science, Technology, Engineering, and Mathematics (STEM) Undergraduate Education: A Status Report for The National Academies National Research Council Board on Science Education (BOSE): Washington, D.C., BOSE, https://www.nsf.gov/attachments/117803/public/Xc--Linking_Evidence--Fairweather.pdf (last accessed 16 May 2015).
- Fuller, I.C., 2006, What is the value of fieldwork? Answers from New Zealand using two contrasting undergraduate physical geography field trips: New Zealand Geographer, v. 62, p. 215–220, doi: 10.1111/j.1745-7939.2006.00072.x.
- Gilbert, L.A., Wirth, K.R., Stempien, J.A., Budd, D.A., Bykerk-Kauffman, A., Jones, M.H., Knight, C., van der Hoeven Kraft, K.J., Matheney, R.K., McConnell, D., Nell, R.M., Nyman, M., Perkins, D., and Vislova, T., 2009, What motivations and learning strategies do students bring to introductory geology? GARNET part 2, students: Geological Society of America Abstracts with Programs, v. 31, no. 7, p. 603, https://gsa.confex.com/gsa/2009AM/finalprogram/abstract_166604.htm (last accessed 6 Dec. 2016).
- Gilbert, L.A., Stempien, J., McConnell, D.A., Budd, D.A., van der Hoeven Kraft, K.J., Bykerk-Kauffman, A., Jones, M.H., Knight, C.C., Matheney, R.K., Perkins, D., and Wirth, K., 2012, Not just “rocks for jocks”: Who are introductory geology students and why are they here?: Journal of Geoscience Education, v. 60, p. 360–371, doi: 10.5408/12-287.1.
- Hall, C., Dickerson, J., Batts, D., Kauffmann, P., and Bosse, M., 2011, Are we missing opportunities to encourage interest in STEM fields?: Journal of Technology Education, v. 23, no. 1, p. 32–46, doi: 10.21061/jte.v23i1.a.4.
- Harackiewicz, J.M., Barron, K.E., Tauer, J.M., Carter, S.M., and Elliot, A.J., 2000, Short-term and long-term consequences of achievement goals: Predicting interest and performance over time: Journal of Educational Psychology, v. 92, p. 316–330, doi: 10.1037/0022-0663.92.2.316.
- Harackiewicz, J.M., Durik, A.M., Barron, K.E., Linnenbrink-Garcia, L., and Tauer, J.M., 2008, The role of achievement goals in the development of interest: Reciprocal relations between achievement goals, interest, and performance: Journal of Educational Psychology, v. 100, p. 105–122, doi: 10.1037/0022-0663.100.1.105.
- Kastens, K.A., Manduca, C.A., Cervato, C., Frodeman, R., Goodwin, C., Liben, L.S., Mogk, D.W., Spangler, T.C., Stillings, N.A., and Titus, S., 2009, How geoscientists think and learn: Eos, v. 90, no. 31, p. 265–272, doi: 10.1029/2009EO310001.
- LaDue, N.D., and Pacheco, H.A., 2013, Critical experiences for field geologists: Emergent themes in interest development: Journal of Geoscience Education, v. 61, no. 4, p. 428–436.
- Levine, R., González, R., Cole, S., Fuhrman, M., and Le Floch, K.C., 2007, The geoscience pipeline: A conceptual framework: Journal of Geoscience Education, v. 55, no. 6, p. 458–468, doi: 10.5408/1089-9995-55.6.458.
- Mayo, M., 2009, Video games: A route to large-scale STEM education?: Science, v. 323, p. 79–82, doi: 10.1126/science.1166900.
- Mazur, E., 2009, Farewell, Lecture?: Science, v. 323, p. 50–51, doi: 10.1126/science.1168927.
- McConnell, D.A., and van Der Hoeven Kraft, K.J., 2011, Affective domain and student learning in the geosciences: Journal of Geoscience Education, v. 59, p. 106–110, doi: 10.5408/1.3604828.
- McConnell, D.A., Steer, D.N., Owens, K.D., Knott, J.R., Van Horn, S., Borowski, W., Dick, J., Foos, A., Malone, M., McGrew, H., Greer, L., and Heaney, P.J., 2006, Using conceptests to assess and improve student conceptual understanding in introductory geoscience courses: Journal of Geoscience Education, v. 54, p. 61–68, doi: 10.5408/1089-9995-54.1.61.
- McConnell, D.A., Jones, M.H., Budd, D.A., Bykerk-Kauffman, A., Gilbert, L.A., Knight, C., van der Hoeven Kraft, K.J., Nyman, M., Stempien, J., Vislova, T., Wirth, K.R., Perkins, D., Matheney, R.K., and Nell, R.M., 2009, Baseline data on motivation and learning strategies of students in physical geology courses at multiple institutions: GARNET part 1, overview: Geological Society of America Abstracts with Programs, v. 41, no. 7, p. 603, https://gsa.confex.com/gsa/2009AM/finalprogram/abstract_166439.htm (last accessed 6 Dec. 2016).
- McGreen, N., and Sánchez, I.A., 2005. Mapping challenge: A case study in the use of mobile phones in collaborative, contextual learning, in Isaías, P., Borg, C., Kommers, P., and Bonanno, P., eds., Proceedings of the IADIS International Conference Mobile Learning, Qawra, Malta, 28–30 June 2005, p. 213–217.
- Middendorf, J., and Kalish, A., 1996, The “change-up” in lectures: National Teaching and Learning Forum, v. 5, no. 2, p. 1–5.
- Milner-Bolotin, M., Kotlicki, A., and Rieger, G., 2007, Can students learn from lecture demonstrations: The role and place of interactive lecture experiments in large introductory science courses: Journal of College Science Teaching, v. 36, no. 4, p. 45–49.
- Mogk, D.W., and Goodwin, C., 2012, Learning in the field: Synthesis of research on thinking and learning in the geosciences, in Kastens, K.A., and Manduca, C.A., eds., Earth and Mind II: A Synthesis of Research on Thinking and Learning in the Geosciences: Geological Society of America Special Paper 486, p. 131–163, doi: 10.1130/2012.2486(24).
- Murphy, K.R., and Davidshofer, C.O., 1988, Psychological Testing: Principles and Applications: Englewood Cliffs, New Jersey, Prentice-Hall, 602 p.
- National Research Council, 2011, Expanding underrepresented minority participation: America’s science and technology talent at the crossroads: Washington, D.C., The National Academies Press, 286 p.
- Orion, N., and Hofstein, A., 1994, Factors that influence learning during a scientific field trip in a natural environment: Journal of Research in Science Teaching, v. 31, p. 1097–1119, doi: 10.1002/tea.3660311005.
- Pintrich, P.R., and DeGroot, E., 1990, Motivational and self-regulated learning components of classroom academic performance: Journal of Educational Psychology, v. 82, p. 33–40, doi: 10.1037/0022-0663.82.1.33.
- Pintrich, P.R., Smith, D., Garcia, T., and McKeachie, W., 1991, A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ): Ann Arbor, Michigan, The University of Michigan, 76 p.
- Pintrich, P.R., Smith, D.A.F., Garcia, T., and McKeachie, W.J., 1993, Reliability and predictive validity of the motivated strategies for learning questionnaire (MSLQ): Educational and Psychological Measurement, v. 53, no. 3, p. 801–813, doi: 10.1177/0013164493053003024.
- Raudenbush, S., and Bryk, A., 2001, Hierarchical linear models: Applications and data analysis methods, v. 2: Newbury Park, California, Sage Publications, 501 p.
- Seymour, E., 2001, Tracking the processes of change in U.S. undergraduate education in science, mathematics, engineering, and technology: Science Education, v. 86, p. 79–105, doi: 10.1002/sce.1044.
- Shelton, B.E., Parlin, M.A., Jump, V., and Rowan, L., 2012, Iterative technology-based design with deaf/hard of hearing populations: Working with teachers to build a better educational game: Proceedings of the 10th International Conference of the Learning Sciences: Sydney, Australia, p. 493–494.
- Stevens, J., 1999, Intermediate statistics: A modern approach (2nd ed.): Mahway, New Jersey, Lawrence Erlbaum, 476 p.
- Stokes, P.J., Levine, R., and Flessa, K.W., 2015, Choosing the geoscience major: Important factors, race/ethnicity, and gender: Journal of Geoscience Education, v. 63, no. 3, p. 250–263, doi: 10.5408/14-038.1.
- Tal, R.T., 2001, Incorporating field trips as science learning environment enrichment—An interpretive study: Learning Environments Research, v. 4, p. 25–49, doi: 10.1023/A:1011454625413.
- van der Hoeven Kraft, K.J., Srogi, L., Husman, J., Semken, S., and Fuhrman, M., 2011, Engaging students to learn through the affective domain: A new framework for teaching in the geosciences: Journal of Geoscience Education, v. 59, p. 71–84, doi: 10.5408/1.3543934a.