This paper looks at the tension between the desire to claim predictive ability for Agent-Based Models (ABMs) and its extreme difficulty for social and ecological systems, suggesting that this is the main cause for the continuance of a rhetoric of prediction that is at odds with what is achievable. Following others, it recommends that it is better…

Once research questions are posed, researchers must answer many a priori questions regarding research design before analysis can be performed and any conclusions can be made, including sample selection criteria, data collection method, model specification, analysis and estimation technique. The choices made by researchers along this forking path…

This study analyzed studies done in Turkey in the context of curriculum evaluation (CE) by asking, "How is it made? The study was carried out in two stages. In the first stage, the document analysis method used 215 theses written between 1991 and 2020 on CE were analyzed according to the "thesis review form." In the second stage,…

This article argues that Agent-Based Modelling, owing to its capabilities and methodology, has a distinctive contribution to make to delivering coherent social science prediction. The argument has four parts. The first identifies key elements of social science prediction induced from real research across disciplines, thus avoiding a straw person…

Despite the rising popularity of big data, there is speculation that evaluators have been slow adopters of these new statistical approaches. Several possible reasons have been offered for why this is the case: ethical concerns, institutional capacity, and evaluator capacity and values. In this method note, we address one of these barriers and aim…

For more than a century, numerous studies examined and reported how principals allocated their time. Assuming that differences in selection, participation and satisfaction of tasks by principals resulted in different outcomes, researchers have continued to study principal time use. In 1989, Wimpelberg, Teddlie, and Stringfield indicated a need for…

The evidence continuum is a five-domain model for building evidence through needs assessment, program theory, process evaluation, outcomes and impact evaluation, and optimization studies. In this conceptual article, the first two domains of the evidence continuum are used to design a learner centered course on advanced research methods, and…

Researchers strategically assess youth mental health by soliciting reports from multiple informants. Typically, these informants (e.g., parents, teachers, youth themselves) vary in the social contexts where they observe youth. Decades of research reveal that the most common data conditions produced with this approach consist of discrepancies…

Several programs of research have sought to assess the replicability of scientific findings in different fields, including economics and psychology. These programs attempt to replicate several findings and use the results to say something about large-scale patterns of replicability in a field. However, little work has been done to understand the…

The complexity associated with how interventions result--or fail to result--in outcomes and how context matters is increasingly recognised. Logic models provide an important tool for handling complexity, with contrasting uses in programme evaluation and evidence synthesis. To reconcile these, we developed an approach that combines the strengths of…

In the context of the major influence that 'effectiveness' is having internationally, paper studies the contemporary methodological perspectives in educational research when considering teachers' practice. It shows that current trends can be boiled down to: (1) naturalistic methodology, (2) descriptive methodology, and (3) the nonmethodological…

Research impact features heavily in debates about 'the measured university' and is now formally assessed by governments in the UK and Australia. Yet clear guidance on how impact can be measured in non-monetary ways is often lacking because of confused thinking and the context-specific nature of outcomes. To help resolve this, we first propose a…

The purpose of this ITEMS module is to provide an introduction to differential item functioning (DIF) analysis using mixture item response models. The mixture item response models for DIF analysis involve comparing item profiles across latent groups, instead of manifest groups. First, an overview of DIF analysis based on latent groups, called…

We discuss the articles of this special issue with reference to an important yet previously only implicit dimension of study quality: alignment across the theoretical and methodological decisions that collectively define an approach to self-regulated learning. Integrating and extending work by leaders in the field, we propose a framework for…

Design-based methods have recently been developed as a way to analyze data from impact evaluations of interventions, programs, and policies. The impact estimators are derived using the building blocks of experimental designs with minimal assumptions, and have good statistical properties. The methods apply to randomized controlled trials (RCTs) and…