Assessment choices

Barnett, S.M. and Ceci, S.J. (2002). When and where do we apply what we learn? A taxonomy for far transfer. Psychological Bulletin, 128: 612-637. This article explores the dimensions of “transfer,” or the ability of learners to apply knowledge in novel contexts. The literature typically refers to near and far transfer, but Barnett and Ceci argue that these terms have been ill-defined and used in different ways in published studies, making it difficult to determine whether, and under what conditions, far transfer occurs. They propose a taxonomy consisting of nine dimensions that describe ways in which learners may be asked to transfer knowledge. These dimensions fall into two categories: content, or what is transferred, and context, or when and where the transfer occurs. Content can differ in three ways: what skill is learned (e.g., procedure vs.  principle); what performance is measured (speed, accuracy, or approach); and memory demands placed on the learner (e.g., execute a remembered procedure vs. recall possible approaches, recognize the one to use, and execute it). Context can vary across six dimensions: knowledge domain, physical, temporal, functional, or social context, and modality. Transfer can be near or far for each dimension, with near transfer meaning that learning and testing conditions are similar and far transfer meaning that they are different. For example, within the knowledge domain, transferring knowledge from a problem involving a mouse to one involving a rat would be near transfer, while transferring from a science to an art example would be far transfer. Functional context refers to the purpose of the activity, such as using information for academic, work, or play purposes, while modality refers to the format of how learning is gained and expressed. The authors suggest that this taxonomy can be used to more fully investigate elements that promote transfer. Instructors should note that this paper provides a thought-provoking and useful tool for considering how to design assessments investigating different elements of transfer. It provides examples for each dimension that clearly illustrate how assessments can vary.

McDaniel, M.A., Cahill, M.J., Frey, R.F., Rauch, M., Doele, J., Ruvolo, D., and Daschbach, M.M. (2018). Individual differences in learning exemplars versus abstracting rules: Associations with exam performance in college science. Journal of Applied Research in Memory and Cognition, 7: 241-251. This study examines the role of individuals’ learning patterns on ability to transfer knowledge to new settings. The authors describe two kinds of learners: exemplar learners, who focus on acquiring particular exemplars and responses associated with them, and abstraction learners, who attempt to abstract underlying regularities reflected in particular exemplars. They test the hypothesis that these learning patterns lead to differential performance on transfer problems, or problems that differ significantly from examples used during training. They used a web-based learning task to differentiate abstraction (N = 351) and exemplar (N = 363) learners who were enrolled in introductory chemistry in one of three different semesters. They categorized questions from the cumulative final exam using a four-bin classification scheme, categorizing questions as (a) “solving a problem that has been exposed many times in lecture or homework,” (b) “solving a problem that has been exposed, but then applying it to a new situation (e.g., another set of variable values but same equation),” (c) “using learned tools to solve problems that may appear foreign or different from all previous practice by being described in a different application,” or (d) “requires highest level of thinking and application, often conceptual thinking beyond the scope of equations or focused lecture topic, or requiring an integration of multiple topics.” Category (a) was labelled as retention problems, and categories (c) and (d) were considered transfer problems; examples of both types of problems are provided. They found that abstraction learners demonstrated higher performance on transfer questions but not on retention questions. The results suggest that individual differences in how learners acquire and represent concepts persist from laboratory concept learning to learning complex concepts in science courses. Instructors should note that the categorization scheme and the example problems may be useful in helping them develop their own problems that require more and less transfer. 

Frey, R.F., McDaniel, M.A., Bunce, D.M., Cahill, M.J., and Perry, M.D. (2020). Using students’ concept-building tendencies to better characterize average-performing student learning and problem-solving approaches in general chemistry. CBE—Life Sciences Education, 19: ar42, 1-17. This article extends previous work that examined the impact of abstraction and exemplar approaches to learning on students’ problem solving. The authors had previously observed that abstraction learners, who learn the theory underlying examples, exhibit better performance on problems requiring transfer than exemplar learners, who rely on memory of examples. Here, they tested the hypothesis that this pattern would persist in general chemistry taught using a different approach at a different institution. They characterized learners as abstraction or exemplar learners using a previously published task unrelated to chemistry and then examined students’ performance on retention, near-transfer, and far-transfer exam questions, as characterized by a rubric they provide. They observed the same patterns exhibited in previous research, but the results did not reach significance. Their analysis suggests that the lack of statistical significance was due to small sample size (N = 82). The authors also performed think-aloud interviews with both abstraction and exemplar learners who exhibited average exam performance, using three types of problems as prompts during interviews. Specifically, they asked students to think aloud as they developed Lewis structures for a molecule that had been used in class (retention); a molecule that could be solved using an algorithm described in class (near-transfer); and a molecule that could not be solved algorithmically but required deeper conceptual understanding (far-transfer). They found that a greater number of abstraction learners relied on algorithmic thinking with understanding, while exemplar learners were more likely to rely on memory or algorithm use without understanding. Abstraction learners, even within this group that was drawn from students with average exam performance, exhibited greater accuracy. Both groups of students exhibited weaknesses in metacognitive monitoring accuracy skills. Instructors interested in assessment methods should note that this study provides a scheme for problem categorization, example problems, and a method for assessing student metacognitive monitoring accuracy. 

Belenky, D. M., & Nokes-Malach, T. J. (2012). Motivation and transfer: The role of mastery-approach goals in preparation for future learning. Journal of the Learning Sciences, 21(3), 399-432. This study examines student motivation not only as a predictor of learning and transfer, but also as an outcome influenced by instructional design. The authors focus specifically on achievement motivation, i.e., the reasons why and the manner in which students engage in learning activities. Achievement motivations are defined along two dimensions: mastery vs. performance and approach vs. avoidance orientations. Of particular interest in this study is the mastery-approach orientation, where students are driven by a desire to gain conceptual understanding and therefore engage in effortful learning strategies. A mastery-approach orientation is considered an individual trait, yet the authors hypothesized that mastery-approach goals might also be temporarily induced by the learning environment. To test this hypothesis, a 2 x 2 experimental design provided undergraduate introductory psychology students (N = 104) either invention or tell-and-practice instruction, followed by a transfer test that either was or was not preceded by an additional learning resource (a worked example). Results showed that students who entered the study with stronger mastery-approach orientations performed better on the transfer test, regardless of instruction. Crucially, a survey immediately following instruction revealed that invention (vs. tell-and-practice) instruction induced more short-term mastery-approach goal adoption. For students who entered the study with weak mastery-approach orientations, this invention-induced mastery goal adoption also supported better transfer. Instructors should note that problem solving-first instruction can positively impact assessment outcomes beyond knowledge transfer, such as motivation. Brief surveys can be used to monitor students’ learning goals over time, which can help instructors understand student motivations in their course and the effects of different instructional methods on those motivations. Positive changes in affective outcomes, like an increase in mastery orientation, may in turn support transfer and achievement. 

Taylor, J. L., Smith, K. M., van Stolk, A. P., & Spiegelman, G. B. (2010). Using invention to change how students tackle problems. CBE—Life Sciences Education, 9(4), 504-512. This study conducted in a first-year college biology course compares the impact of invention activities  versus traditionally structured problem-solving activities on several outcomes, including student opinions, engagement, and creative problem-solving behavior during a think-aloud interview. Student volunteers were randomly assigned to the invention activity (N = 170) and structured problem-solving activity (N = 170) groups, comprising 4-5 students who met weekly outside of class, with an additional 91 volunteers placed in a no-intervention control group. The IA problems were analogous to biology problems involving living cells but situated in non-biology contexts, so they did not require domain-specific knowledge; they were non-deterministic, having multiple possible solutions; and the sessions occurred prior to lecture on the analogous biology concepts, so the problems prepared students to learn future course content. In contrast, the structured problem-solving activities were presented after the introduction of relevant concepts during lecture; they were biology-specific, drawing on the course material already familiar to students; and the session wrap-ups included presentation of the correct problem solutions. When asked about the benefits of their problem-solving groups, invention activity participants were significantly more likely than structured problem-solving activity participants to mention “creativity, problem solving, and thinking skills” and “application and making knowledge connections.”  Engagement monitoring by the session facilitators revealed that invention activity groups spent less time off-task than structured problem-solving activity groups, and that off-task time in invention activity groups occurred prior to beginning the activity, whereas off-task time in structured problem-solving activity groups reflected intermittent disengagement. Finally, invention activity groups generated significantly more solution ideas than structured problem-solving activity groups, and the invention activity solutions were of comparable quality and generated in the same amount of time. Instructors should note that instructional choices can impact a range of affective and behavioral outcomes, including metacognitive awareness, engagement in activities, and creative flexibility in problem solving. This study demonstrates survey and observational techniques for assessing such effects.

Micari, M. & Pazos, P. (2021). Beyond grades: improving college students’ social-cognitive outcomes in STEM through a collaborative learning environment. Learning Environments Research, 24(1), 123-136. This study examines the impact of guided inquiry learning, in the form of supplemental peer-led collaborative learning, on social-cognitive outcomes among undergraduate students, rather than focusing on learning and transfer. Students from a range of STEM (biology, chemistry, physics, and statistics) and social science (macro- and microeconomics) courses volunteered to join peer-led study groups of 5-7 students who met weekly outside of lecture to explore course concepts and collaboratively problem solve. The study groups were led by advanced undergraduates who had previously succeeded in the target course, were screened for strong interpersonal skills, and received course-based training on small-group facilitation, student-centered learning, and group problem-solving strategies. Both study-group participants (N = 604) and non-participants (N = 676) completed a pre- and post-survey to evaluate their self-efficacy in the target course, self-regulated learning capabilities, and reliance on surface learning (i.e., memorization) rather than deep learning techniques. Results showed that study-group participants experienced improved self-efficacy over time, while non-participants exhibited a decrease in self-efficacy. The self-regulation results were similar, with study-group participants demonstrating increased self-regulation over the term, whereas self-regulation among non-participants decreased. Finally, all students showed increasing memorization over time, but that increase was significantly smaller for study-group participants. Instructors should note that collaborative, peer-led problem solving can benefit students’ social-cognitive outcomes, including their confidence in STEM courses, self-regulation skills, and choice of learning strategies (e.g., surface vs. deep). This article discusses a range of possible explanations for the positive impact of collaborative group work, such affiliation with other students, opportunities to reflect, and low-stakes practice, to name a few.