When preservice teachers’ prior beliefs contradict evidence from educational research
Abstract
Background
Knowledge from educational research frequently contradicts preservice teachers’ prior beliefs about educational topics. Such contradictions can seriously affect their attitudes towards educational research and can counteract efforts taken to establish teaching as a research-based profession.
Aims
Inspired by Munro’s (2010, J. Appl. Soc. Psychol., 40, 579) work on science discounting, this study examined whether preservice teachers tend to devalue the potency of educational research when evidence contradicts their beliefs.
Sample
We used data from 145 preservice teachers from different German universities.
Methods
In an experimental design, participants indicated their prior beliefs about an educational topic (i.e., effectiveness of grade retention) before and after reading either confirming or disconfirming scientific evidence. Dependent variables were, first, whether participants devalued the potency of science to study this focal topic and whether they generalized this devaluation to further related and unrelated topics; second, whether participants preferred non-scientific over scientific sources to inform themselves about the focal topic as an indirect measure of science devaluation.
Results
Interaction effects on both outcome variables confirmed that participants devalued educational research and its sources when scientific evidence conflicted with their prior belief. Yet, results did not corroborate any generalization of devaluation to further topics. Despite the devaluation, participants indicated belief revision in the direction of the evidence read.
Conclusions
Preservice teachers may indeed critically question educational research when scientific evidence conflicts with their prior beliefs. However, they may also adapt their assumptions in light of strong evidence. More research is needed to clarify the conditions of devaluation and belief revision.
Background
When people are confronted with scientific evidence that contradicts their prior beliefs, they frequently question the evidence rather than their own beliefs (Chinn & Brewer, 1998; Munro, 2010). The effects of this bias regarding comprehending and evaluating scientific information due to individuals’ prior beliefs are well documented in the literature (Kunda, 1990). Recently, research interest in such biases has even increased (Britt, Rouet, Blaum, & Millis, 2019; Sinatra, Kienhues, & Hofer, 2014), because simplified access and the easy spread of all information through the Internet, including alternative facts and fake news, have motivated researchers and educators to seek means to address the risks of misinformation and the rejection of scientific evidence in learners (Barzilai & Chinn, 2020; Britt et al., 2019; Darner, 2019).
Unfortunately, research knowledge about learning and teaching itself often falls victim to biased reasoning and has to compete with the myths circulating continuously in the public and among practitioners (Aguilar, Polikoff, & Sinatra, 2019; Menz, Spinath, & Seifried, 2020). Evidence suggests that teachers maintain questionable beliefs about learning and teaching, even though contradicted by a solid body of research (De Bruyckere, Kirschner, & Hulshof, 2015; Sinatra & Jacobson, 2019). This is a serious problem for initial teacher education (ITE), because it should equip future teachers with research-based knowledge (Bauer & Prenzel, 2012). When encountering belief-discrepant evidence, preservice teachers may be reluctant to revise their prior assumptions and risk evolving critical stances towards educational research. This would counteract efforts to encourage research-based teaching and teacher education (Bauer & Prenzel, 2012).
Though it is well documented that preservice teachers evaluate and interpret educational research findings in favour of their beliefs (Fives & Buehl, 2012), less is known about how conflicts between beliefs and facts affect their attitudes towards educational research itself. Inspired by Munro’s (2010) work on the cognitive mechanisms underlying devaluation of science, we aimed to fill this research gap by examining whether preservice teachers question the potency of empirical educational research to deliver valid evidence on educational topics when research findings challenge their prior beliefs.
Role of prior beliefs in acquiring educational research and attitudes towards it
People’s prior beliefs critically affect how scientific evidence is processed (Sinatra et al., 2014). Specifically, when facing belief-discrepant evidence, individuals tend to discount, reinterpret, or ignore the evidence rather than revise their own assumptions (Chinn & Brewer, 1998; Lewandowsky, Ecker, Seifert, Schwarz, & Cook, 2012). Such motivated reasoning allows individuals to maintain their prior assumptions, even when scientific evidence indicates that they are questionable or false (Nauroth, Gollwitzer, Bender, & Rothmund, 2014; Sinatra et al., 2014).
The reception of educational research may be particularly prone to processes of motivated reasoning, because beliefs about education evolve through own everyday experiences and observations (Nauroth et al., 2014). These homegrown and deeply rooted personal conceptions may become especially problematic in ITE. In contrast to knowledge, personal beliefs require no external validation; they have an idiosyncratic empirical base in either one’s own or anecdotal everyday experiences and understandings (Pajares, 1992; Richardson, 1996). Conflicts between knowledge from scientific educational research and preservice teachers’ personal beliefs can have a twofold negative effect on students’ acquisition of research-based knowledge in ITE and their attitudes towards educational research. First, beliefs have a filter function that shapes attention to, interpretation of, and incorporation of new information (Fives & Buehl, 2012; Nespor, 1987). Thus, students are likely to adopt (selective) knowledge that is consistent with their prior beliefs. Indeed, findings suggest that preservice teachers’ beliefs are hard to change in ITE and can even be reinforced when faced with discrepant scientific evidence (cf. Britt et al., 2019; Hollingsworth, 1989).
Second, discrepancies between scientific knowledge from educational research and personal beliefs may pose a cognitive conflict that boils down to an issue of trust and, thus, leads to a devaluation of science (Munro, 2010). In other words, facing belief-threatening evidence may instigate a general doubt over science as a reliable process for attaining justified knowledge (see Chinn, Rinehart, & Buckland, 2014). Based on cognitive dissonance theory (Festinger, 1957), Munro (2010) coined the scientific impotence excuse claiming that people – when faced with strong belief-discrepant scientific evidence that is hard to dismiss – will tend to resist the evidence ‘by concluding that the topic of study is not amenable to scientific investigation’ (Munro, 2010, p. 597). Munro (2010) found evidence for these devaluation processes in two experiments. Participants assessed their prior belief about a specific scientific topic (i.e., the relationship between homosexuality and the prevalence of mental disorders), read evidence that either confirmed or disconfirmed their belief, and rated the potency of science to deliver valid evidence on this initial topic and further unrelated topics. Additionally, participants indicated whether they would consult a scientific source (compared to non-scientific sources) to seek topic-related information. Such source preferences are important, because selecting sources implies a decision on whether scientific evidence will be considered at all. Results confirmed that participants who read belief-discrepant evidence applied the scientific impotence excuse: they devalued the potency of research to deliver valid evidence on the focal topic and unrelated topics, and demonstrated a decreased preference for scientific sources to inform themselves about the topic.
The scientific impotence excuse in educational research
So far, there is no evidence that the scientific impotence excuse applies to the devaluation of educational research. However, we believe it does. Whereas previous studies have shown that preservice teachers contest the usefulness of research knowledge for their future practice (Allen, 2009; Fajet, Bello, Leftwich, Mesler, & Shaver, 2005; Gitlin, Barlow, Burbank, Kauchak, & Stevens, 1999), little is known about how they judge the validity of educational research findings, especially when they are belief-threatening.
Accordingly, the present study examined whether the scientific impotence excuse observed by Munro (2010) applies to the devaluation of educational research by preservice teachers. We adapted and extended Munro’s (2010) paradigm in two ways. First, we attended to both topic-specific and domain-specific differences in devaluation. Knowledge domains differ in their knowledge structures, inquiry practices, and epistemic standards (Kienhues, Thomm, & Bromme, 2018), and can therefore be perceived differently (Barzilai & Weinstock, 2015; Muis, Bendixen, & Haerle, 2006). Frequently, education is perceived as a ‘soft’ science, even among social scientists (Berliner, 2002). Therefore, we wanted to investigate how science devaluation generalizes differentially to further topics in both the same and unrelated domains. The difference between topic-related and domain-related generalization has not been considered before.
Second, we focused on a specific contrast between sources when examining source preferences. Munro (2010) examined information seeking as an indirect measure of science devaluation by asking participants to rate how probably they would use a variety of scientific and non-scientific sources to seek information about the topic. We extended the scope of potential sources to include experiential sources, because prior research indicates that (preservice) teachers specifically underscore the significance of experiential knowledge rooted in personal experiences and observations, anecdotal evidence and conventional wisdom, while being critical about the relevance and usefulness of scientific sources (Bråten & Ferguson, 2015; Gitlin et al., 1999; van Schaik, Volman, Admiraal, & Schenke, 2018). Although both scientific and experiential knowledge are essential and valuable for future teachers (Darling-Hammond & Bransford, 2005), conflicts between prior beliefs and scientific evidence may foster and stabilize a strong preference for experiential over scientific sources. Reinforcing low preferences for or even devaluation of scientific sources may critically undermine preservice teachers’ acquisition of accurate general pedagogical/psychological educational knowledge and, thus, negatively affect an important facet of their teacher competence (Asberger, Thomm, & Bauer, 2020; Heyder, Berggold, & Steinmayer, 2017; Voss, Kunter, & Baumert, 2011). Therefore, we assessed and contrasted preservice teachers’ preferences for both types of sources.
Research questions
This study aimed to examine the role of belief–evidence conflicts in preservice teachers’ devaluation of educational research. In line with Munro (2010), it pursued three research questions: (1) Do preservice teachers devalue (a) the potency of educational research and (b) scientific sources when confronted with evidence that conflicts with their prior belief about a specific educational topic? We hypothesized that preservice teachers would be more critical about the potency of educational research to study a specific educational topic when research evidence contradicted rather than supported their prior beliefs about this topic (Hypothesis 1a). Moreover, experienced discrepancies may also lead preservice teachers to refrain from consulting sources that may provide further scientific evidence. We therefore tested the hypotheses that preservice teachers who read evidence that contradicted rather than confirmed their prior beliefs would show a decreased preference for scientific sources and, conversely, an increased preference for experiential sources (Hypothesis 1b); and choose less often scientific than non-scientific (experiential) sources to seek relevant information (Hypothesis 1c).
(2) Do preservice teachers generalize the devaluation of the potency of science to carry out research to further domain-related and domain-unrelated topics? We hypothesized that preservice teachers would also devalue the potency of educational research to study further educational topics (i.e., unrelated topics from the same domain; Hypothesis 2a). However, unlike Munro (2010), we did not expect to find a generalization of devaluation to other topics in unrelated domains (e.g., from medicine; Hypothesis 2b).
(3) Is the devaluation of the potency of science expressed in resistance to belief change when facing belief-threatening evidence? Because prior results suggest that prior beliefs can be robust to change, we expected participants to maintain their prior assumptions and refrain from changing their endorsement of prior beliefs, even when evidence speaks against them (Hypothesis 3).
Methods
Design
The experiment followed the procedures described by Munro (2010) using a 2 × 2 mixed design. The specific educational topic was preservice teachers’ beliefs about the effectiveness of grade retention – that is, making pupils repeat a grade rather than moving up to the next grade. In many countries, grade retention is an active practice and considered to be an effective measure to support struggling pupils (e.g., OECD, 2020), although research findings on average do not support its effectiveness (Doyle, 1989; Jimerson & Brown, 2016). In Germany, grade retention is practised in both elementary and secondary schools. In PISA 2018, 19.6% of 15-year-old pupils were reported to have repeated at least one school year, exceeding the OECD average of 11% (OECD, 2020). Grade retention is widely known and debated by the German public. Therefore, the topic appeared suitable for this study’s purpose, as we could expect participants to indicate diverging prior beliefs on the issue, which allowed us to experimentally manipulate consistencies and inconsistencies between prior beliefs and evidence read.
The randomized between-participants factor was confirming versus disconfirming evidence on grade retention effects. Prior belief in grade retention effectiveness (measured before reading the evidence) served as the within-participant factor. We aimed at a sample size of 160 participants to have sufficient statistical power (.95) for detecting effects in the lower medium range as small as R2 = .1 with a significance level of α ≤ .05.
Participants
Participants were recruited online and from university lectures across multiple German universities. Teacher education in Germany is designed and organized separately by each of the 16 federal states. Study programmes can differ from state to state, although they all adhere to nationally shared teacher education standards. To increase the generalizability of the study results, we purposely recruited participants from universities in different federal states, attending to potential variations.1 Participation was voluntary with a lottery of vouchers as incentive. Because the study was negligible-risk research without foreseeable risk of harm or discomfort other than potential inconvenience, no formal approval from a governing or institutional review board was required. We followed AERA (2011) ethical guidelines for research with human subjects.
Overall, 162 preservice teachers completed the experiment. Data from two participants had to be deleted due to missing consent declarations and from nine further participants because they were not enrolled in a teacher education program. Finally, we removed six participants with unreasonable response times (i.e., <5 or >400 min). The final sample (N = 145, 114 female) still provided sufficient statistical power (>.90). Participants were M = 23.92 years old (SD = 4.17). All were enrolled in a teacher education programme: 48.3 % were enrolled in a bachelor’s degree programme (M = 4.73 semesters completed, SD = 2.34), 37.2 % were studying in a master’s programme (M = 3.15 semesters, SD = 2.90), and 14.5 % of participants were enrolled in traditional state examination programmes that do not distinguish between the bachelor’s and master’s level (M = 8.33 semesters, SD = 2.39; see Terhart, 2019).
Procedure
The experiment was conducted online to recruit participants from several universities. The welcome page informed participants that the study aimed to investigate their perception and appraisal of evidence from research on educational topics. The detailed goals of the study were explained later during debriefing (see below).
After giving informed consent, participants rated their prior belief on grade retention effectiveness, read the introduction, and were randomly assigned to one of the two evidence conditions. Next, they read the evidence and completed the potency and source preference ratings. Finally, they assessed anew their belief on grade retention effectiveness and gave demographic background information.
The study ended with a thorough debriefing, including information about the study’s goals, the experimental manipulation, and a concise summary of the current state of research on grade retention effectiveness. The procedure followed recommendations for online experiments (Hoerger & Currell, 2012).
Experimental manipulation
After reading an introductory text about grade retention as an intervention to foster struggling pupils’ achievement, participants read five pieces of evidence (i.e., short abstracts) on this topic. The introductory text referred to an alleged review article summarizing recent scientific studies on whether grade retention helps pupils to compensate their achievement deficits. It explained what grade retention means, and depicted the underlying assumption that the additional time for learning should help students to catch up. The introduction then told participants that they would be presented with abstracts of five selected studies presenting research methods, findings, and conclusions in grade retention research. It stated that all studies had been published in prestigious scholarly journals and had undergone peer review to assure their quality.
Participants then read abstracts of five empirical studies as pieces of evidence (all of similar length: M = 114.20 words, SD = 12.77). Each outlined the study design, main measures, results, and conclusions. Abstracts were based on authentic empirical research and depicted a range of studies describing the effects of grade retention on pupils’ school achievement across different grades, subjects, and types of school. Abstracts were identical across experimental conditions, but the passages on the results and conclusions were manipulated to provide either confirming or disconfirming evidence on the effectiveness of grade retention (i.e., positive effects vs. no or negative effects).
All materials were optimized before the experiment through cognitive interviews. A sample of nine university students studying either teacher training or psychology participated. They were instructed to think aloud while reading and evaluating the comprehensibility and logical consistency of the summaries. We purposely included psychology students based on the premise that their comprehensive methodological training would have prepared them to provide critical evaluations of the abstracts from a student perspective. To avoid contrasting effects, participants received either the confirming or disconfirming evidence summaries, but not both. The results provided information on the comprehensibility, consistency and methodological soundness of the abstracts. We adjusted the texts in accordance with this feedback in preparation for the main study. The final materials are available in Appendix S1.
Measures
At the beginning of the experiment, participants assessed their prior belief by rating their personal agreement with the statement ‘Repeating a grade helps struggling students to compensate for their achievement deficits.’ on a 9-point scale ranging from 1 (do not agree at all) to 9 (very much agree; cf. Munro, 2010). To measure potential belief change, we posed the same question again at the end of the experiment.
As a measure of science devaluation, participants first assessed their doubt over the potency of science to study the effectiveness of grade retention. Following Munro (2010), they indicated their agreement with the statement ‘The question whether grade retention helps struggling students to compensate their deficits in achievement is one that cannot be answered using scientific methods.’ on a 9-point scale ranging from 1 (do not agree at all) to 9 (very much agree).
To examine any possible generalization of science devaluation, participants subsequently assessed the potency of scientific research to study six further educational topics (e.g., impact of class size on learning outcomes) and six unrelated topics. Unrelated topics stemmed from two domains: medicine (e.g., taking vitamin D to prevent illness) and pseudo-scientific topics (e.g., existence of clairvoyance; cf. Munro, 2010). Topics were presented in two blocks (educational and unrelated) in randomized order within each block. Participants rated whether each topic could be studied scientifically (‘How far can scientific methodologies be used to determine whether [class size affects learning outcomes of school students]?’ on 9-point scales ranging from 1 (not at all) to 9 (very well). One educational topic had to be removed because a preparatory factor analysis revealed a weak loading. The averaged rating scores for each domain were sufficiently reliable (education: α = .70, medicine: α = .81, and pseudo-science: α = .86). Appendix S2 provides a complete list of topics.
To measure source preferences, participants received a list of seven scientific and non-scientific sources that they might use to inform themselves about grade retention effectiveness. Scientific sources were conceived as sources that are expected to draw on scientific evidence (i.e., research findings from educational research, statement of an educational scientist). Non-scientific sources were conceived as sources that provide the experience or opinions of stakeholders (i.e., opinion of a teacher, a school pupil, and a teacher association). Like Munro (2010), we included the positions of a proponent and an opponent of grade retention as non-scientific sources. Participants rated how probably they would use each source on a 9-point scale. Since ratings of the proponent and opponent source were highly correlated (r = .94, p < .001), we averaged them to a combined score. Preference for scientific sources yielded a reliable scale (α = .78). After dropping one item, a sufficiently reliable scale could be built for the preference for non-scientific sources (α = .71). Participants then had to choose one source they would consult (source choice). Appendix S3 provides a complete list of presented sources.
Analyses
To address Research questions 1 to 3, we used Hayes’ (2018) PROCESS macro in SPSS to analyse whether participants’ prior beliefs moderated the relationship between type of evidence read and either the scientific potency rating or their source preference. The evidence condition was included as a dummy-coded variable (0 = evidence confirming effectiveness; 1 = evidence disconfirming effectiveness). Prior belief was centred at the grand mean prior to analysis. To predict the choice of a scientific (coded 1) compared to a non-scientific source (coded 0) in Research question 3, we computed a binary logistic regression (Jaccard, 2001). Because we expected a zero effect for the unrelated domains in Research question 2, our target was the null hypothesis. Therefore, we employed a significance level of α = .20 to falsely reject an effect. Potential changes in participants’ beliefs after reading the evidence (Research question 3) were tested with a repeated-measures ANOVA. We judged effect sizes according to Cohen’s criteria (Cohen, 1988).
Prior to analyses, transformations were applied to variables exhibiting severe skew (i.e., when both P-P plots and significance tests for skew [α ≤ .01] indicated highly asymmetric distributions; Field, 2017) to prevent biased standard errors and significance tests (Fox, 2016; Tabachnick & Fidell, 2014). Log transformation was applied to variables with positive skew (i.e., topic-specific potency, potency of studying pseudo-scientific topics), and reverse score transformation to variables with negative skew (i.e., potency to study related topics, preference for scientific sources; Field, 2017; Fox, 2016).
Results
Devaluation of potency of science and of scientific sources when facing belief-threatening evidence about a specific educational topic (Research question 1)
Table 1 provides an overview of the descriptive statistics. Scale means indicated that participants had comparatively favourable beliefs about the potency of science and preferred scientific sources to inform themselves about educational topics. Note that in the following sections, the short labels confirming evidence and disconfirming evidence for the experimental conditions mean that the evidence read confirms or disconfirms the effectiveness of grade retention (not that it confirms or disconfirms participants' beliefs).
| M (SD) | Skew (SE) | Kurtosis (SE) | |
|---|---|---|---|
| Confirming evidence on grade retention effectiveness | |||
| Prior beliefs on GR (T1) | 5.15 (1.99) | −0.27 (0.29) | −0.86 (0.57) |
| Prior beliefs on GR (T2) | 6.46 (1.67) | −0.58 (0.29) | −0.50 (0.57) |
| Doubt over potency of science to study GR | 3.12 (2.07) | 0.90 (0.29)a | −0.07 (0.57) |
| Potency to study related educational topics | 6.20 (1.13) | −0.02 (0.29) | −0.67 (0.57) |
| Potency to study unrelated topics | |||
| Medicine | 5.71 (1.73) | −0.38 (0.29) | −0.09 (0.57) |
| Pseudo-science | 2.55 (2.16) | 1.67 (0.29)a | 1.99 (0.57) |
| Source preference | |||
| Scientific source GR | 6.96 (1.55) | −0.98 (0.29)a | 1.53 (0.57) |
| Non-scientific source GR | 6.25 (1.34) | −0.35 (0.29) | 0.34 (0.57) |
| Disconfirming evidence on grade retention effectiveness | |||
| Prior beliefs on GR (T1) | 5.22 (1.95) | −0.47 (0.27) | −0.90 (0.54) |
| Prior beliefs on GR (T2) | 3.48 (1.96) | 0.41 (0.27) | −0.98 (0.54) |
| Doubt over potency of science to study GR | 3.14 (2.02) | 0.80 (0.27)a | −0.42 (0.54) |
| Potency to study related educational topics | 6.38 (1.34) | −0.70 (0.27)a | 0.63 (0.54) |
| Potency to study unrelated topics | |||
| Medicine | 5.65 (1.66) | −0.22 (0.27) | 0.02 (0.54) |
| Pseudo-science | 3.26 (2.60) | 1.11 (0.27)a | −0.06 (0.54) |
| Source preference | |||
| Scientific source GR | 7.13 (1.59) | −0.84 (0.27)a | 0.05 (0.54) |
| Non-Scientific source GR | 6.01 (1.60) | −0.43 (0.27) | 0.29 (0.54) |
Note
- Values represent the untransformed data.
- GR = grade retention.
- a P-P plot and test of significance of skew (p < .01) indicated serious skewness.
To test Hypothesis 1a, we regressed doubt over the scientific potency on prior beliefs, evidence condition, and their interaction. Analyses yielded a statistically significant overall effect, F(3, 141) = 4.13, p = .008, R2 = .08. Most importantly, the interaction term confirmed a significant moderating effect of prior belief, b = 0.08, 95% CI [0.03, 0.13], SE(b) = 0.02, t = 3.24, p = .002 (Figure 1). Probing the interaction, a simple slopes analysis showed a significant effect of evidence read for both participants with high prior belief, i.e., 1 SD above the sample mean; b = 0.16, 95% CI [0.03, 0.30], SE(b) = 0.07, t = 2.40, p = .018, and low prior belief, i.e., 1 SD below the sample mean; b = −0.15, 95% CI [−0.28, −0.01], SE(b) = 0.07, t = −2.19, p = .030. That is, participants believing in the effectiveness of grade retention tended to devalue the potency of science when confronted with evidence disconfirming grade retention effects. Similarly, participants who did not believe in grade retention effectiveness significantly devalued the potency of science when facing confirming evidence on grade retention effects. No other effects were statistically significant.
Second, we tested whether prior beliefs moderated the impact of evidence on participants’ preferences for scientific sources and source choice. The overall regression model of scientific source preferences (Hypothesis 1b) on prior beliefs, evidence condition, and their interaction was statistically significant, F(3, 141) = 4.57, p = .004, R2 = .09 (Figure 2). As expected, the interaction term indicated a significant moderating impact of prior belief, b = −0.06, 95% CI [−0.09, −0.02], SE(b) = 0.02, t = −2.82, p = .006. A simple slopes analysis revealed that participants with low prior belief had a significantly lower preference for scientific sources when having to read confirming evidence on grade retention effects, b = 0.15, 95% CI [0.04, 0.25], SE(b) = 0.05, t = 2.65, p = .009. The reverse pattern for participants with high prior belief took the expected direction but failed to attain statistical significance, b = −0.07, 95% CI [−0.18, 0.04], SE(b) = 0.05, t = −1.34, p = .183. No other effect was statistically significant. Analogously, we conducted moderation analyses with participants’ preference for non-scientific sources as dependent variable. The overall regression model did not attain significance, F(3, 141) = 0.49, p = .687, R2 = .01.
Finally, the logistic regression of choosing a scientific versus a non-scientific source (Hypothesis 1c) on prior belief, evidence, and their interaction was statistically significant, χ2(3) = 9.79, p = .02 (Table 2). As expected, the interaction effect of evidence and prior beliefs on source choice was significant, b = −0.43, SE(b) = 0.19, Wald’s χ2(1) = 5.10, p = .024, OR = 0.65, 95% CI [0.45, 0.94]. No other effects were statistically significant. This interaction implied that a unit increase in belief in grade retention effectiveness led to a 0.65 change in the odds ratio of choosing a scientific source over a non-scientific one between both evidence conditions. That is, as belief in the effectiveness of grade retention increased, participants reading disconfirming evidence were less likely to choose a scientific source (compared to a non-scientific source) than participants reading confirming evidence.
| b (SE) | OR | 95% CI for OR | ||
|---|---|---|---|---|
| Lower | Upper | |||
| Non-scientific vs. scientific source | ||||
| Intercept | 0.36 (0.25) | 1.43 | ||
| Evidence | 0.12 (0.35) | 1.13 | 0.57 | 2.27 |
| Prior belief | 0.02 (0.13) | 1.02 | 0.79 | 1.30 |
| Evidence x prior belief | −0.43 (0.19)* | 0.65 | 0.45 | 0.94 |
Note
- R2 = .05 (Hosmer–Lemeshow), 0.07 (Cox–Snell), 0.09 (Nagelkerke). Model χ2(3) = 9.79, p = .02.
- * p < .05.
An inspection of the odds ratios from a simple slopes analysis helps to describe the interaction (Table 3; Jaccard, 2001). Yet, they need to be considered with due caution, as the simple slopes did not attain statistical significance. If participants indicated low belief in grade retention effectiveness and received evidence disconfirming grade retention effectiveness, they chose about 2.6 times more often a scientific source as compared to a non-scientific one, b = 0.97, SE(b) = 0.54, p = .074, OR = 2.64, 95% CI [0.90, 7.63]. However, if participants endorsed a strong belief in grade retention effectiveness and read disconfirming evidence rather than confirming evidence, they were only about half as likely to choose scientific sources compared to non-scientific ones, b = −0.72, SE(b) = 0.48, p = .139, OR = 0.49, 95% CI [0.19, 1.26].
| Evidence | Prior belief on GR effectiveness | ||
|---|---|---|---|
| −1 SD | M | +1 SD | |
| a. Predicted odds [95% CI] | |||
| Evidence disconfirming GR effectiveness (response group) | 3.64 [1.59, 8.27] | 1.62 [0.99, 2.64] | 0.72 [0.38, 1.38] |
| Evidence confirming GR effectiveness (reference group) | 1.38 [0.71, 2.68] | 1.43 [0.88, 2.34] | 1.48 [0.74, 2.95] |
| b. Odds ratios [95% CI] | |||
| Evidence disconfirming GR effectiveness/Evidence confirming GR effectiveness | 2.64 [0.90, 7.63] | 1.13 [0.57, 2.27] | 0.49 [0.19, 1.26] |
Note
- Dummy coding of source choice: 0 = non-scientific source, 1 = scientific source.
- GR = grade retention.
Generalization of devaluation of potency of science to further domain-related and domain-unrelated topics (Research question 2)
The regression model for predicting potency ratings for other educational topics (reverse score transformed; Hyothesis 2a) failed to attain significance, F(3, 141) = 2.10, p = 0.103, R2 = .04. Unexpectedly, participants did not devalue the potency of scientific methods to study topics in the same knowledge domain. For the two unrelated domains (i.e., medical and pseudo-scientific topics), we conducted separate moderation analyses. As expected, there was no generalization of devaluation to unrelated topics (Hypothesis 2b). There was no moderating impact of participants’ prior belief on assessments of scientific potency to study medical topics, F(3, 141) = 0.41, p = .745, R2 = .01, or pseudo-scientific issues (log transformed), F(3, 141) = 1.18, p = .321, R2 = .02.
Potential belief change when facing belief-threatening evidence (Research question 3)
Finally, we examined whether reading either confirming or disconfirming evidence changed participants’ prior beliefs about the effectiveness of grade retention (Hypothesis 3). A mixed ANOVA with the within-participants factor belief (before vs. after reading evidence) and the between-participants factor evidence (confirming vs. disconfirming) showed a statistically non-significant main effect of prior belief, F(1, 143) = 2.35, p = .128, η2 = .02, a significant main effect of evidence, F(1, 143) = 26.22, p < .001, η2 = .16, and a significant interaction, F(1, 143) = 117.29, p < .001, η2 = .45. Unexpectedly, follow-up t-tests for dependent samples suggested that participants changed their prior belief in the direction of the evidence read (see Figure 3). Participants who read confirming evidence reported increased belief in the effectiveness of grade retention, t(67) = −6.21, p < .001, d = 0.75, whereas participants who read disconfirming evidence decreased their belief, t(76) = 9.26, p < .001, d = 1.05.
Discussion
This study aimed to investigate whether preservice teachers devalue the potency of science to produce sound knowledge about education when experiencing conflicts between their prior beliefs and research findings. Such conflicts seem to occur frequently (De Bruyckere et al., 2015; Sinatra & Jacobson, 2019) and can be detrimental to the acquisition of scientific knowledge in teacher education. Moreover, they can lead to a devaluation of science – the focus of the present study. Such devaluation is a serious problem in light of current trends of science denial, so-called alternative facts, and post-truth political debates. It is also at odds with demands to establish teaching as a research-based profession.
In general, our participants reported a positive view on the potency of research and indicated higher appreciation of scientific than of non-scientific sources. This is of interest given the frequent reports that (preservice) teachers regard knowledge from educational research as being of little use for their professional practice (Allen, 2009; van Schaik et al., 2018). Our results add that preservice teachers seem to have favourable orientations towards the potency of research to provide valid answers on educational issues. Further disentangling teachers’ views on the utilitarian and epistemic value of educational research will require systematic surveys with representative samples (Thomm, Sälzer, Prenzel & Bauer, 2021).
Despite these favourable attitudes, we still identified the hypothesized tendencies to devalue educational research when examining Research question 1. When prior belief conflicted with scientific evidence about the effectiveness of grade retention, preservice teachers expressed more doubt on the potency of educational research. They were also less inclined to consult scientific sources – as suggested by both source preference and choice. However, it is to note that devaluation of scientific sources was mainly observed when participants believing in the effectiveness received disconfirming evidence. Participants’ preference for non-scientific knowledge sources such as experience reports was unaffected by reading belief-threatening evidence. Because preservice teachers commonly tend to prefer experiential to scientific sources (Bråten & Ferguson, 2015), this devaluation of scientific sources appears all the more critical. It can undermine knowledge acquisition in a stage as early as during teacher training and affect the knowledge basis of future teaching practices. Notably, the effects found were small to moderate in size; nevertheless, we were careful not to discard them as being of low practical importance. Even though the present study applied only a modest textual manipulation on a single topic, tendencies of devaluation could be found. Overall, findings on research question 1 are in line with Munro’s (2010) results and provide initial evidence that the scientific impotence excuse also applies to education.
Regarding Research question 2, first, results corroborated our assumption that research from unrelated domains (e.g., medicine) would not be devalued by experiencing belief–evidence conflicts on an educational topic. This is consistent with research showing that people tend to recognize differences between domains in inquiry practices and epistemic standards (Kienhues et al., 2018; Muis et al., 2006). A closer look at differences between Munro’s (2010) and this study’s materials might explain the discrepant results. Munro (2010) assessed generalization of devaluation to topics such as the effectiveness of spanking as a disciplinary measure for children or the existence of clairvoyance. In contrast, our educational topics (e.g., problem-based learning, parental support) were more neutral and less prone to trigger individuals’ endorsed values or world views. This is important, because devaluating research on such loaded topics might result from other considerations (e.g., ethical) than the epistemic ones focused on here. Second, unexpectedly, we also found no generalizing effect on other topics within education. This is encouraging, because it suggests that devaluation is primarily topic-related and does not spread immediately. Nevertheless, it would be premature to rule out generalization effects. Repeated experiences of conflict between personal beliefs and research-based knowledge may still lead to a habitual devaluation of educational research. Because educational issues are susceptible to their own experiences and assumptions, preservice teachers may well encounter such discrepancies in more than one topic and probably more than once (De Bruyckere et al., 2015), and this may also shape their attitudes towards educational research. This consideration provides an additional reason not to underestimate the small effects observed in this study, as we may have touched only the surface of preservice teachers’ experiences and perceptions of belief-evidence discrepancies.
Regarding Research question 3, contrary to our expectations, participants changed their prior belief in the effectiveness of grade retention in the direction of the evidence read. Given the frequent finding that personal beliefs are quite robust to interventions (Richardson, 1996), this result was surprising in both its nature and size. Moreover, it may appear to contradict the observed tendencies to devalue science. One explanation might be that participants use a resistance strategy to anomalous evidence called ‘accept the data and make peripheral theory changes’ (Chinn & Brewer, 1998, p. 623). Because we provided participants with five pieces of methodologically strong and unambiguous evidence, they might have perceived their validity to be difficult to ignore or refute. Hence, in the post-assessment, participants may have reported what they should believe according to the evidence without actually believing it – as expressed in their doubt over the potency of science. As a second explanation, reading belief-discrepant scientific evidence might have evoked true belief revision, but under epistemic vigilance (cf. Sperber et al., 2010). That is, faced with evidence that left hardly any other option, participants changed their beliefs in order to resolve the discrepancy. However, they did not do so blindly, but added doubt over science as a cognitive marker for the experienced epistemic conflict that had probably not been solved satisfactorily. These two explanations are similar, but differ in that belief change is espoused only in the first, and internalized more strongly in the second. So far, we can only speculate on the reasons and do not know how stable the effect is. Nevertheless, this opens up interesting directions for future research. It would be important to clarify which factors may moderate possible shifts between belief affirmation and critical evaluation and what role science devaluation plays in this context. One way to further illuminate the results pattern is to additionally control for individuals’ pre-treatment assessments of scientific potency on other topics and their source preference before facing belief-discrepant evidence. In the present study, we took only a pre-test of prior beliefs about grade retention effectiveness. Additional pre- and post-tests would enable the capture of a more comprehensive picture of potential changes in participants’ attitudes towards science and, thus, may help to better understand the extent and nature of devaluation and its relation to (the extent of) belief change.
However, several limitations need to be considered. First, conceptually, one might argue that the scientific impotence excuse is nothing more than a form of confirmation bias. Indeed, processes of motivated reasoning serve to confirm one’s beliefs (Nauroth et al., 2014). However, this study focused on the consequences of such motivated reasoning for participants’ attitudes towards research and scientific sources. Second, participants received evidence on only one specific educational topic. Including evidence on several educational topics was beyond its scope. However, doing so would be valuable in future research to further scrutinize topic-related differences. Third, within the abstracts used here, we presented authentic designs of research on grade retention, but adapted the materials to assure consistency and comparability across both evidence conditions. Future research could use abstracts of authentic studies to increase external validity. Because the present study provided a first attempt to demonstrate the causal effect implied by the scientific impotence excuse in a new context (i.e., education), we prioritized experimental control and internal validity. Finally, drawing closely on Munro’s (2010) research paradigm to ensure the comparability of the findings, we used mainly single-item measures. Although this practice may not be uncommon in the social psychology field, future studies should consider employing more elaborate multi-item scales. In addition to being substantively more informative, a scale with more items on scientific potency would also permit an estimation of the measure’s internal consistency.
To address these limitations, follow-up studies should be conducted to replicate our findings and investigate the persistence of the effects. Such studies should also examine variations in the treatment (e.g., different types of topics), the participants and the contexts (e.g., comparisons across different types of study programmes and cross-country comparisons). These limitations notwithstanding, we believe this study contributes to unravelling preservice teachers’ reservations about educational research. Our results provide indications of a devaluation of educational research when scientific evidence contradicts preservice teachers’ prior beliefs. Exploring interventions to address and manage such conflicts explicitly will be an important issue when constructing research-based ITE programmes.
Acknowledgements
We thank Juliane Ruppel and Jana Asberger for assistance in data collection, and Jonathan Harrow for advice on language editing.
Conflicts of interest
All authors declare no conflict of interest.
Author contributions
Eva Thomm (Conceptualization; Data curation; Formal analysis; Methodology; Project administration; Writing – original draft; Writing – review & editing); Bernadette Gold (Conceptualization; Writing – review & editing); Tilmann Betsch (Conceptualization; Writing – review & editing); Johannes Bauer (Conceptualization; Methodology; Resources; Visualization; Writing – review & editing).
Open Research
Data availability statement
The data is deposited at the PsychArchvies (Leibniz Institute for Psychology, Germany): http://bibliotheek.ehb.be:2168/10.23668/psycharchives.4455.
