ERIC Number: EJ1382474
Record Type: Journal
Publication Date: 2023-Jan
Pages: 16
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0958-2029
EISSN: EISSN-1471-5449
Quis Judicabit Ipsos Judices? A Case Study on the Dynamics of Competitive Funding Panel Evaluations
Research Evaluation, v32 n1 p70-85 Jan 2023
Securing research funding is essential for all researchers. The standard evaluation method for competitive grants is through evaluation by a panel of experts. However, the literature notes that peer review has inherent flaws and is subject to biases, which can arise from differing interpretations of the criteria, the impossibility for a group of reviewers to be experts in all possible topics within their field, and the role of affect. As such, understanding the dynamics at play during panel evaluations is crucial to allow researchers a better chance at securing funding, and also for the reviewers themselves to be aware of the cognitive mechanisms underlying their decision-making. In this study, we conduct a case study based on application and evaluation data for two social sciences panels in a competitive state-funded call in Portugal. Using a mixed-methods approach, we find that qualitative evaluations largely resonate with the evaluation criteria, and the candidate's scientific output is partially aligned with the qualitative evaluations, but scientometric indicators alone do not significantly influence the candidate's evaluation. However, the polarity of the qualitative evaluation has a positive influence on the candidate's evaluation. This paradox is discussed as possibly resulting from the occurrence of a halo effect in the panel's judgment of the candidates. By providing a multi-methods approach, this study aims to provide insights that can be useful for all stakeholders involved in competitive funding evaluations.
Descriptors: Foreign Countries, Research, Financial Support, Peer Evaluation, Researchers, Social Science Research, State Aid, Qualitative Research, Evaluation Criteria, Decision Making
Oxford University Press. Great Clarendon Street, Oxford, OX2 6DP, UK. Tel: +44-1865-353907; Fax: +44-1865-353485; e-mail: jnls.cust.serv@oxfordjournals.org; Web site: http://applij.oxfordjournals.org/
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: Portugal
Grant or Contract Numbers: N/A