ERIC Number: ED659502
Record Type: Non-Journal
Publication Date: 2023-Sep-28
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Investigating Systematic Reviewers' Practices: A Survey on Handsearching, Citation Chasing, and Screening Tool Adoption in Education
Qiyang Zhang; Marta Pellegrini; Francesco Marsili
Society for Research on Educational Effectiveness
Background: Searching the literature and screening studies are two key stages to build a strong foundation for a quality systematic review. Low-quality handsearching and citation chasing processes may lead to missing relevant studies, while low-quality study screening may lead to human errors and a lengthy process (Haddaway et al., 2022; Zhang & Neitzel, 2023). Technological tools specifically developed for these stages may support the reproducibility of the review and the reliability of its results. In non-educational fields, several studies compared the usability and features of screening tools were conducted in non-educational fields (Harrison et al., 2020; Van der Mierden et al., 2019). In education, Zhang and Neitzel (2023) is the first study to address this issue by examining the reviews that made explicit the use of screening tools and investigating their features. They found that only 4.19% of the studies published in Review of Educational Research reported the use of a tool for screening. Since researchers in published articles rarely report the use of tools transparently, it is difficult to assess the overall tool usage patterns through secondary data analysis. Therefore, we were motivated to collect primary data through a survey administered to educational systematic reviewers operating worldwide at different stages of their academic careers. Research Questions: (RQ1) Which software tools are most frequently used to conduct handsearching, citation searching, and screening in educational systematic reviews? (RQ2) What are the reasons behind systematic reviewers' choices in software? What are the barriers preventing them from choosing other tools? and (RQ3) Do researchers' patterns of tool usage differ by continents? Participants: Eligible participants had to have experience conducting systematic reviews or meta-analyses. We emailed eligible survey participants from authors' network, publicly available review authors in education. Figure 1 presents continent composition of potential participants we reached out through emails. We also posted our survey on Twitter with hashtags to increase exposure of the survey link. Finally, we asked relevant organizations to help distribute the survey through their mailing lists. Research Design: We used a Qualtrics survey to conduct this mixed-method study. Participants were asked to choose tools they have heard about in a list of tools (Table 1). Then they will choose and evaluate one citation chasing tool and one screening tool that they used the most frequently. The evaluation questions were adapted from the system usability scale (Brooke, 1995). Apart from these close-ended questions, we also have two sets of qualitative questions related to the use of citation chasing tools and screening tools. Data Collection and Analysis: We employed quantitative methods to analyze descriptive close-ended questions and qualitative content analysis to delineate open-ended responses. All quantitative analyses were conducted in Stata and R. Qualitative content analysis was performed using text categorization by assigning one or more labels to each answer based on the information provided (Graneheim et al., 2017). Results: From June 2022 to January 2023, we collected 175 entries (Table 2). Handsearching tools: As for current practices conducting handsearching, 61.93% of the respondents manually browse through journals' or conferences' websites, 26.14% reported using certain tools, and 11.93% never conduct handsearching. When we asked participants what tools they use to conduct handsearching, researchers mentioned their university's library tools, journal websites, databases, and looking through journals manually. Citation chasing tools: We asked researchers to select citation chasing tools they have heard about from a list generated by authors (Table 1). Databases seem to be the most heard about sources. Tools that were specifically designed for citation chasing purposes seem to remain unknown, perhaps due to the short time they exist so far. Then we inquired upon the most frequently used tool. Again, the three databases were mentioned the most. Figures 2[approximately]4 present the results. Furthermore, we asked participants to state the reasons behind choosing their preferred tool. Most individuals (n = 64) have justified their choice of tool based on ease of use. The second category is "comprehensiveness and functionality" (n = 29). Finally, we asked participants to state obstacles that stop them from using other tools. The barrier that mostly hinders the use of other citation chasing tools is the lack of knowledge of these tools (n = 36). The barriers related to technical aspects of the tools are grouped under the category lack of comprehensiveness (n = 15). Another barrier is the high cost of certain tools or the fact that their university does not provide access because it does not bear the costs (n = 14). Screening tools: We asked researchers to select the screening tools they have heard about from a list generated by authors. Citation management tools (24.72%) and spreadsheets (24.43%) seem to be the two most heard about screening assistance. Our data shows that the choice of screening tools is primarily conditioned by the scientific environment in which the researcher operates in. Two main elements explained this: the support from researchers' own institution (n = 8) and the tool usage patterns among colleagues (n = 16). Comprehensiveness and functionality (n = 22) are the second important elements to influence a researcher's tool choice. The third important aspect is the ease of use (n = 21). Cost is also an important aspect to note (n = 11). Figures 5[approximately]7 present the results. Then we analyzed the barriers preventing researchers from using other screening tools. The first reason is the lack of knowledge (n = 43) about the tools themselves. The second reason is the cost barrier (n = 16). The third reason is dissatisfaction with the functionality and technical comprehensiveness of screening tools (n = 10). Conclusions: To the best of our knowledge, this study is the first survey that collects primary data on educational systematic reviewers' tool usage status. Another innovative feature is that international participants allow us to compare researchers' tool usage patterns across different continents to generate more insights. By shedding light on tool usage patterns, this study facilitates the adoption of tools by researchers and promotes a more systematic, transparent, and reproducible review process.
Descriptors: Educational Research, Research Tools, Computer Software, Citations (References), Literature Reviews, Meta Analysis, Research Methodology, Online Searching
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A