The recent commentary by Beaujean and Farmer (2020) on the original paper by Dixon et al. (2019) serves a cautionary tale of selective p-values, the law of small N sizes, and the type-II error. We believe these authors have crafted a somewhat questionable argument in which only 57% of the original Dixon et al. data were re-analyzed, based on a…

Meta-analysis and structural equation modeling (SEM) are 2 of the most prominent statistical techniques employed in the behavioral, medical, and social sciences. They each have their own well-established research communities, terminologies, statistical models, software packages, and journals ("Research Synthesis Methods" and…

Chalmers recently published a critique of the use of ordinal a[alpha] proposed in Zumbo et al. as a measure of test reliability in certain research settings. In this response, we take up the task of refuting Chalmers' critique. We identify three broad misconceptions that characterize Chalmers' criticisms: (1) confusing assumptions with…

This reflective commentary on the character and role of methodology in educational and social inquiry recounts my 45-year journey as an applied researcher and evaluator, primarily in the domain of education. The journey starts in graduate school in the early 1970s, where the methodological challenge was to master "the proper methods, properly…

This comment discusses some issues about the use and reporting of experimental studies in education, illustrated by a recently published study that claimed (i) that an educational innovation was effective despite outcomes not reaching statistical significance, and (ii) that this refuted the findings of an earlier study. The two key issues raised…

The importance of data analysis software in graduate programs in education and post-graduate educational research is self-evident. However the role of this software in facilitating supererogated statistical practice versus "cookbookery" is unclear. The need to rigorously document the role of data analysis software in students' graduate…

Danny Kaplan is DeWitt Wallace Professor of Mathematics and Computer Science at Macalester College. He received Macalester's Excellence in teaching Award in 2006 and the CAUSE/USCOTS Lifetime Achievement Award in 2017. This interview took place via email on March 4-June 17, 2017. Topics covered in the interview include: (1) the current state of…

In their essay, "Why Assessment Will Never Work...," Bacon and Stewart (2016) recommend that instead of carrying out the expensive process of experimenting themselves, many business schools would get a bigger bang for their buck if they used "published pedagogical studies that use direct measures of learning with sufficient…

In this commentary from Issue 14, n3, authors John Cadogan and Nick Lee applaud the paper by Aguirre-Urreta, Rönkkö, and Marakas "Measurement: Interdisciplinary Research and Perspectives", 14(3), 75-97 (2016), since their explanations and simulations work toward demystifying causal indicator models, which are often used by scholars…

This letter documents some problems in Ancaiani et al. (2015). Namely the evaluation of concordance, based on Cohen's kappa, reported by Ancaiani et al. was not computed on the whole random sample of 9,199 articles, but on a subset of 7,597 articles. The kappas relative to the whole random sample were in the range 0.07-0.15, indicating an…

In response to an argument by Baird, Andrich, Hopfenbeck and Stobart (2017), Michael Kane states that there needs to be a better fit between educational assessment and learning theory. In line with this goal, Kane will examine how psychometric constraints might be loosened by relaxing some psychometric "rules" in some assessment…

The authors of the focus article describe an important issue related to the use and interpretation of causal indicators within the context of structural equation modeling (SEM). In the focus article, the authors illustrate with simulated data the effects of omitting a causal indicator. Since SEMs are used extensively in the social and behavioral…

In their 2016 work, Aguirre-Urreta et al. provided a contribution to the literature on causal measurement models that enhances clarity and stimulates further thinking. Aguirre-Urreta et al. presented a form of statistical identity involving mapping onto the portion of the parameter space involving the nomological net, relationships between the…

In this commentary, Betsy McCoach and David Kenny state they are in general agreement with Bainter and Bollen (this issue) that causal indicators are not inherently unstable. Herein, they outline several similarities and differences between latent variables with reflective and causal indicators. In their examination of the two models, they find…

The author's commentary focuses more on the quantitative discussion about educational assessment of the original article than on the idea of the assessment for learning, which did not raise any substantial issues. He starts by offering some general comments on the paper. He feels the authors made a number of assumptions about quantitative…