Assessment choices

  • If instructors desire to teach problem solving, they must assess using problems.
  • When choosing problems for assessment, instructors should consider how similar they want the items to be compared to instruction. 
  • Instructors often want to assess using problems that are at least somewhat different from instruction because they want students to be able to use their knowledge in novel contexts. In the problem-solving literature, this is referred to as “transfer,” and transfer is often considered “near” or “far.”
    • Problems can differ along multiple dimensions, and variation can be smaller (requiring less transfer) or larger (requiring further transfer) on each dimension.
    • These dimensions include the knowledge domain, the physical context, the temporal context, the functional context, the social context, the modality, the skill that is learned, the type of change in that skill (e.g., speed, accuracy) that is measured, and the memory demands of the task.
    • Considering these dimensions may allow instructors to identify elements of learning they want to measure and control the elements that they do not want to vary.
  • As students learn and prepare to transfer knowledge, they adopt different learning strategies, with some relying on memorization of specific exemplars (exemplar or surface learning) and others gleaning underlying patterns across different exemplars (abstract or deep learning).
    • Research shows these different learning strategies yield similar outcomes on near knowledge transfer, but abstraction learners find more success on far knowledge transfer.
    • Instructors interested in assessing different degrees of knowledge transfer can consult recent studies for example retention, near transfer, and far transfer problems, as well as categorization schemes that can be used to evaluate or design additional problems.
  • Instructors may also want to assess other cognitive and affective outcomes (e.g., motivation, creativity).
    • A growing body of evidence supports the idea that problem solving-first approaches can positively impact student motivation (e.g., confidence and engagement), metacognitive awareness and regulation, and creativity during problem solving.
    • Relatively simple surveys can be incorporated into existing course structures to help instructors understand student perceptions and experiences in their courses.
    • Other assessment strategies for monitoring these outcomes, like gathering observation data on students’ behavioral engagement over time, are effective but also labor intensive.
Barnett, S.M. and Ceci, S.J. (2002). When and where do we apply what we learn? A taxonomy for far transfer. Psychological Bulletin, 128: 612-637. This article explores the dimensions of “transfer,” or the ability of learners to apply knowledge in novel contexts. The literature typically refers to near and far transfer, but Barnett and Ceci argue that these terms have been ill-defined and used in different ways in published studies, making it difficult to determine whether, and under what conditions, far transfer occurs. They propose a taxonomy consisting of nine dimensions that describe ways in which learners may be asked to transfer knowledge. These dimensions fall into two categories: content, or what is transferred, and context, or when and where the transfer occurs. Content can differ in three ways: what skill is learned (e.g., procedure vs.  principle); what performance is measured (speed, accuracy, or approach); and memory demands placed on the learner (e.g., execute a remembered procedure vs. recall possible approaches, recognize the one to use, and execute it). Context can vary across six dimensions: knowledge domain, physical, temporal, functional, or social context, and modality. Transfer can be near or far for each dimension, with near transfer meaning that learning and testing conditions are similar and far transfer meaning that they are different. For example, within the knowledge domain, transferring knowledge from a problem involving a mouse to one involving a rat would be near transfer, while transferring from a science to an art example would be far transfer. Functional context refers to the purpose of the activity, such as using information for academic, work, or play purposes, while modality refers to the format of how learning is gained and expressed. The authors suggest that this taxonomy can be used to more fully investigate elements that promote transfer. Instructors should note that this paper provides a thought-provoking and useful tool for considering how to design assessments investigating different elements of transfer. It provides examples for each dimension that clearly illustrate how assessments can vary.
McDaniel, M.A., Cahill, M.J., Frey, R.F., Rauch, M., Doele, J., Ruvolo, D., and Daschbach, M.M. (2018). Individual differences in learning exemplars versus abstracting rules: Associations with exam performance in college science. Journal of Applied Research in Memory and Cognition, 7: 241-251. This study examines the role of individuals’ learning patterns on ability to transfer knowledge to new settings. The authors describe two kinds of learners: exemplar learners, who focus on acquiring particular exemplars and responses associated with them, and abstraction learners, who attempt to abstract underlying regularities reflected in particular exemplars. They test the hypothesis that these learning patterns lead to differential performance on transfer problems, or problems that differ significantly from examples used during training. They used a web-based learning task to differentiate abstraction (N = 351) and exemplar (N = 363) learners who were enrolled in introductory chemistry in one of three different semesters. They categorized questions from the cumulative final exam using a four-bin classification scheme, categorizing questions as (a) “solving a problem that has been exposed many times in lecture or homework,” (b) “solving a problem that has been exposed, but then applying it to a new situation (e.g., another set of variable values but same equation),” (c) “using learned tools to solve problems that may appear foreign or different from all previous practice by being described in a different application,” or (d) “requires highest level of thinking and application, often conceptual thinking beyond the scope of equations or focused lecture topic, or requiring an integration of multiple topics.” Category (a) was labelled as retention problems, and categories (c) and (d) were considered transfer problems; examples of both types of problems are provided. They found that abstraction learners demonstrated higher performance on transfer questions but not on retention questions. The results suggest that individual differences in how learners acquire and represent concepts persist from laboratory concept learning to learning complex concepts in science courses. Instructors should note that the categorization scheme and the example problems may be useful in helping them develop their own problems that require more and less transfer. 
Frey, R.F., McDaniel, M.A., Bunce, D.M., Cahill, M.J., and Perry, M.D. (2020). Using students’ concept-building tendencies to better characterize average-performing student learning and problem-solving approaches in general chemistry. CBE—Life Sciences Education, 19: ar42, 1-17. This article extends previous work that examined the impact of abstraction and exemplar approaches to learning on students’ problem solving. The authors had previously observed that abstraction learners, who learn the theory underlying examples, exhibit better performance on problems requiring transfer than exemplar learners, who rely on memory of examples. Here, they tested the hypothesis that this pattern would persist in general chemistry taught using a different approach at a different institution. They characterized learners as abstraction or exemplar learners using a previously published task unrelated to chemistry and then examined students’ performance on retention, near-transfer, and far-transfer exam questions, as characterized by a rubric they provide. They observed the same patterns exhibited in previous research, but the results did not reach significance. Their analysis suggests that the lack of statistical significance was due to small sample size (N = 82). The authors also performed think-aloud interviews with both abstraction and exemplar learners who exhibited average exam performance, using three types of problems as prompts during interviews. Specifically, they asked students to think aloud as they developed Lewis structures for a molecule that had been used in class (retention); a molecule that could be solved using an algorithm described in class (near-transfer); and a molecule that could not be solved algorithmically but required deeper conceptual understanding (far-transfer). They found that a greater number of abstraction learners relied on algorithmic thinking with understanding, while exemplar learners were more likely to rely on memory or algorithm use without understanding. Abstraction learners, even within this group that was drawn from students with average exam performance, exhibited greater accuracy. Both groups of students exhibited weaknesses in metacognitive monitoring accuracy skills. Instructors interested in assessment methods should note that this study provides a scheme for problem categorization, example problems, and a method for assessing student metacognitive monitoring accuracy. 
Belenky, D. M., & Nokes-Malach, T. J. (2012). Motivation and transfer: The role of mastery-approach goals in preparation for future learning. Journal of the Learning Sciences, 21(3), 399-432. This study examines student motivation not only as a predictor of learning and transfer, but also as an outcome influenced by instructional design. The authors focus specifically on achievement motivation, i.e., the reasons why and the manner in which students engage in learning activities. Achievement motivations are defined along two dimensions: mastery vs. performance and approach vs. avoidance orientations. Of particular interest in this study is the mastery-approach orientation, where students are driven by a desire to gain conceptual understanding and therefore engage in effortful learning strategies. A mastery-approach orientation is considered an individual trait, yet the authors hypothesized that mastery-approach goals might also be temporarily induced by the learning environment. To test this hypothesis, a 2 x 2 experimental design provided undergraduate introductory psychology students (N = 104) either invention or tell-and-practice instruction, followed by a transfer test that either was or was not preceded by an additional learning resource (a worked example). Results showed that students who entered the study with stronger mastery-approach orientations performed better on the transfer test, regardless of instruction. Crucially, a survey immediately following instruction revealed that invention (vs. tell-and-practice) instruction induced more short-term mastery-approach goal adoption. For students who entered the study with weak mastery-approach orientations, this invention-induced mastery goal adoption also supported better transfer. Instructors should note that problem solving-first instruction can positively impact assessment outcomes beyond knowledge transfer, such as motivation. Brief surveys can be used to monitor students’ learning goals over time, which can help instructors understand student motivations in their course and the effects of different instructional methods on those motivations. Positive changes in affective outcomes, like an increase in mastery orientation, may in turn support transfer and achievement. 
Taylor, J. L., Smith, K. M., van Stolk, A. P., & Spiegelman, G. B. (2010). Using invention to change how students tackle problems. CBE—Life Sciences Education, 9(4), 504-512. This study conducted in a first-year college biology course compares the impact of invention activities  versus traditionally structured problem-solving activities on several outcomes, including student opinions, engagement, and creative problem-solving behavior during a think-aloud interview. Student volunteers were randomly assigned to the invention activity (N = 170) and structured problem-solving activity (N = 170) groups, comprising 4-5 students who met weekly outside of class, with an additional 91 volunteers placed in a no-intervention control group. The IA problems were analogous to biology problems involving living cells but situated in non-biology contexts, so they did not require domain-specific knowledge; they were non-deterministic, having multiple possible solutions; and the sessions occurred prior to lecture on the analogous biology concepts, so the problems prepared students to learn future course content. In contrast, the structured problem-solving activities were presented after the introduction of relevant concepts during lecture; they were biology-specific, drawing on the course material already familiar to students; and the session wrap-ups included presentation of the correct problem solutions. When asked about the benefits of their problem-solving groups, invention activity participants were significantly more likely than structured problem-solving activity participants to mention “creativity, problem solving, and thinking skills” and “application and making knowledge connections.”  Engagement monitoring by the session facilitators revealed that invention activity groups spent less time off-task than structured problem-solving activity groups, and that off-task time in invention activity groups occurred prior to beginning the activity, whereas off-task time in structured problem-solving activity groups reflected intermittent disengagement. Finally, invention activity groups generated significantly more solution ideas than structured problem-solving activity groups, and the invention activity solutions were of comparable quality and generated in the same amount of time. Instructors should note that instructional choices can impact a range of affective and behavioral outcomes, including metacognitive awareness, engagement in activities, and creative flexibility in problem solving. This study demonstrates survey and observational techniques for assessing such effects.
Micari, M. & Pazos, P. (2021). Beyond grades: improving college students’ social-cognitive outcomes in STEM through a collaborative learning environment. Learning Environments Research, 24(1), 123-136. This study examines the impact of guided inquiry learning, in the form of supplemental peer-led collaborative learning, on social-cognitive outcomes among undergraduate students, rather than focusing on learning and transfer. Students from a range of STEM (biology, chemistry, physics, and statistics) and social science (macro- and microeconomics) courses volunteered to join peer-led study groups of 5-7 students who met weekly outside of lecture to explore course concepts and collaboratively problem solve. The study groups were led by advanced undergraduates who had previously succeeded in the target course, were screened for strong interpersonal skills, and received course-based training on small-group facilitation, student-centered learning, and group problem-solving strategies. Both study-group participants (N = 604) and non-participants (N = 676) completed a pre- and post-survey to evaluate their self-efficacy in the target course, self-regulated learning capabilities, and reliance on surface learning (i.e., memorization) rather than deep learning techniques. Results showed that study-group participants experienced improved self-efficacy over time, while non-participants exhibited a decrease in self-efficacy. The self-regulation results were similar, with study-group participants demonstrating increased self-regulation over the term, whereas self-regulation among non-participants decreased. Finally, all students showed increasing memorization over time, but that increase was significantly smaller for study-group participants. Instructors should note that collaborative, peer-led problem solving can benefit students’ social-cognitive outcomes, including their confidence in STEM courses, self-regulation skills, and choice of learning strategies (e.g., surface vs. deep). This article discusses a range of possible explanations for the positive impact of collaborative group work, such affiliation with other students, opportunities to reflect, and low-stakes practice, to name a few.

Return to Map

Cite this guide: Frey RF, Brame CJ, Fink A, and Lemons PP. (2022) Evidence Based Teaching Guide: Problem Solving. CBE Life Science Education. Retrieved from
  • Your feedback helps improve the site