"Re-thinking equity: the need for a multidimensional approach in evaluating educational equity through TIMSS data"

Abstract

Background

In recent years, data derived from international large-scale assessments have significantly influenced the discourse surrounding educational equity. However, the use of such data has often neglected the full spectrum of dimensions that equity encompasses, while being limited to exploring the relationship between achievement variations and student background. This approach, however, constrains our understanding of the rich notion.

Methods

This paper aims to contribute to the advancement of current research by advocating for an alternative approach that encompasses all relevant dimensions of equity and examines their impact on both achievement and motivational outcomes. A cluster analysis manifests itself as the designated method to employ, as the concept of equity remains theoretically challenged and the detection of data patterns can serve as a strong step in its multifaceted inquiry. Moreover, this method enables us to incorporate the multidimensional concept in a country-level context.

Results

Employing data from the Trends in International Mathematics and Science Study at grade four, this paper presents five distinct clusters of educational systems that illustrate diverse equity configurations. Notably, none of the identified clusters consistently scores high or low on all equity dimensions. This suggests that all of the clusters perform inconsistently to different degrees across the dimensions of equity. Furthermore, no single cluster emerges as superior to the others across both outcome measures.

Conclusions

These findings shed light on the intricate interplay between equity, achievement, and motivation within the examined educational systems. The unique configuration of the equity dimensions within each cluster underscores the importance of adopting a nuanced evaluation of equity, which can enhance our understanding of equity patterns.

Re-thinking equity

Catalyzed by an increasingly interdependent world, participation in international large-scale assessments (ILSAs) facilitates international conversations about education (Ng et al., 2020; Wagemaker, 2020). Such conversations have resulted in a growing political impetus to look beyond national borders in assessing educational systems (Dong & Cravens, 2012; Drent et al., 2013; Kyriakides & Creemers, 2018; Rutkowski et al., 2010). Armed with this rich data provided by ILSAs, research communities have revealed variations in average performance across educational systems, as well as between diverse student groups within an educational system (Addey et al., 2017; Contini & Cugnata, 2020; Oliveri & von Davier, 2014). Such revelations have made policymakers around the world aware of the level and location of existing inequalities, allowing them to draw conclusions about their educational equity (Braathe & Otterstad, 2014; Kyriakides et al., 2019; Wood et al., 2011).

ILSAs collect data on multiple dimensions of educational equity. A recent systematic review explored the conceptual and operational diversity surrounding the concept through the lens of secondary analyses of ILSA data (Appels et al., 2022b). This exploration resulted in the identification of five research approaches, each representing a different equity dimension. While the authors of the review affirmed the meaningful contributions of each approach to the understanding of the rich concept, they also found that the full complexity of educational equity was hard to capture (Braathe & Otterstad, 2014). Studies combined one or more of the five research approaches in a seemingly random manner, unveiling equity in a morphing state (Appels et al., 2022b).

Furthermore, it is worth noting that the current state of research has emphasized variations in academic achievement, overlooking other important educational outcomes (Appels et al., 2022b). This narrow focus on output variations is problematic as educational output is determined by the intersection of multiple educational dimensions (Bower, 2013; Irvine, 2010; Wood et al., 2011). Moreover, it appears that proficiency in educational quality dimensions is more strongly associated with affect-motivational outcomes, than academic achievement (Appels et al., 2024). This may be because qualitative education prioritizes addressing students according to their levels of development over placing a singular focus on high academic achievement in every subject (Moreno, 2010; Vygotsky, 1978).

Arguing that educational equity encompasses multiple dimensions and goes beyond the scope of achievement disparities, we advocate for an alternative evaluation of educational equity. This evaluation encompasses the influence of diversity on both achievement and affect-motivational outcomes of all pertinent dimensions. Taking a cluster analysis approach allows us to use the multidimensional concept in a country-level context (Michaelides et al., 2019b). Through this procedure we can explore the variety in educational equity configurations among educational systems in a detailed and complex manner.

In summary, overlooking the interconnected nature of latent dimensions when addressing equity in the international educational context risks losing sight of the bigger picture. By categorizing educational systems into clusters based on the equity dimensions they encompass, we can gain a more comprehensive perspective on the matter. The Trends in International Mathematics and Science Study (TIMSS) organized by the International Association for the Evaluation of Educational Achievement (IEA) is an ILSA that gathers extensive data related to educational equity. Hence, TIMSS enables us to explore the extent to which equity transcends geographical boundaries. This paper utilizes the TIMSS 2019 data collected from fourth-grade students worldwide.

Theoretical framework

At the heart of equity lies an inherent awareness of and sense of responsibility for diversity (Maiztegui‐Oñate and Santibáñez‐Gruber, 2008; Tomlinson, 2014). It is about the school’s effort to give each child the opportunity to reach their full potential, even though it is unrealistic to expect all children to achieve equally (OECD, 2012; UNESCO, 2007). This, however, raises questions about which degree of inequality is acceptable (Levin, 2003; OECD, 2001; Van Damme & Bellens, 2017).

Defining educational equity

Reading on educational equity, it is essential to disentangle the often interchangeable used terms equity and equality (Field et al., 2007; Secada, 1989; Zhu, 2018). Equality implies an identical treatment for all (Lafontaine et al., 2015), whereas equity acknowledges diversified needs within the educational process, and pleads to provide an unequal but equitable treatment (Maiztegui‐Oñate & Santibáñez‐Gruber, 2008; OECD, 2012; Zhu, 2018). Going further on this notion of dealing appropriately with different student backgrounds, we encounter the work of Kellaghan (2001). He highlights that students with competencies misaligned with the competences valued at school face educational disadvantages. Factors in students’ backgrounds affect the development of these competencies, and thus can or cannot facilitate the learning process (Bourdieu, 1986). Transferring this thinking to empirical research, there is mention of equity concerns, with a major focus on socioeconomic status (SES), gender, and ethnicity (Field et al., 2007; Herremans, 2012; Kyriakides & Creemers, 2018; Levin, 2003; OECD, 2012; Zhu, 2018). While this might suggest personal shortcomings, the prevailing idea is that schools bear the responsibility to provide adequate education for diverse needs (Charalambous et al., 2018; Lafontaine et al., 2015; Levin, 2003).

Equity dimensions

The work of Roemer (1998) is helpful in building a framework for the equity domains incorporated in this article. Roemer argues that the school system should ensure that a student’s knowledge is not predetermined by their circumstances but rather be the product of the level of effort they exert. Here the concepts of fair and unfair inequities enter the elaboration. This relates to attributability of educational outcomes to student background, which refers to elements beyond those within students’ control, and which hint towards their circumstances rather than their effort (Kyriakides & Creemers, 2018). An educational system will be considered more equitable when it demonstrates fewer unfair inequities in student learning. Building upon the framework from Appels and colleagues (2022b) review study (see Fig. 1), this paper defines educational equity through four domains: social equity, ethnic equity, gender equity, and equity of treatment.

Fig. 1

Approaches to the inquiry of equity drawing on ILSA data, overlap indicated by gray bar (Adapted Version, from Appels et al., 2022b)

Appels et al. (2022b) identified five approaches to educational equity. In this paper, we consider only four of these approaches, excluding ability equity from the analysis. Contemporary educational literature is inundated with discussions about the perceived achievement gap, which is often regarded as a primary indicator of educational inequity (Milner, 2010; Van Damme & Bellens, 2017). However, an exclusive emphasis on output, or ability, is limiting because it fails to address the underlying mechanisms that hinder socially, economically, or culturally diverse students from attaining academic success (Bower, 2013; Irvine, 2010; Wood et al., 2011). By neglecting these mechanisms, important aspects of equity are overlooked. Thus, our analysis is focused on those equity approaches that explore the achievement gap through such underlying mechanisms.

The four equity domains considered in this paper can be divided into two groups based on the nature of their equity concerns. As shown in Fig. 1, the equity domains in the first group—social equity, ethnic equity, and gender equity—focus on the extent to which educational outcomes can be attributed to student background. A subset of student background—the domains of ethnic equity and gender equity—captures unalterable student traits. The second group, encompassing a single dimension, equity of treatment, addresses the level of variance in the provision of quality education. Table 1 presents an overview of the selected equity dimensions and their definitions.

Table 1 Domains and definitions for educational equity

Attributability to student background

The equity domains in the first group (social, ethnic, and gender equity) pertain to factors beyond students’ control—that is, their circumstances rather than personal effort—that can affect educational outcomes (Kyriakides & Creemers, 2018). Four distinct conceptual perspectives are important to the study of the relationship between educational outcomes and student background using data from international assessments (Appels et al., 2022b).

The majority of empirical equity research on ILSA data focuses on the strength of the relationship between student background and student outcomes (Bellens et al., 2019; Benito et al., 2014; Gustafsson et al., 2018; Horn, 2009). In this approach, a higher degree of outcome attribution to background factors indicates a less equitable system. Another conceptualization defines equity as the extent to which a student is capable of counterbalancing the impact of a disadvantaged background on their educational outcomes; this ability is often referred to as resilience (e.g., Agasisti et al., 2017). A third conceptualization examines the role of the school in counterbalancing the relationship between student background and educational outcomes (e.g., López Rupérez et al., 2019). In this perspective, the responsibility for fostering equitable outcomes is shifted from the individual student to the school, emphasizing the provision of suitable education that caters to diverse student needs (Charalambous et al., 2018; Lafontaine et al., 2015; Levin, 2003). The final approach perceives equity as a gap in educational outcomes resulting from disparities in student background (e.g., Caro & Mirazchiyski, 2012).

In this paper, we will address the relationship between student background and educational outcomes as the extent to which schools can counterbalance the association between a disadvantaged student’s background and their educational outcomes (Charalambous et al., 2018; Field et al., 2007; Lafontaine et al., 2015; Levin, 2003). This framing ensures the appropriate allocation of responsibilities within the educational context. In it, social equity refers to the limited extent to which educational outcomes can be attributed to the student’s socioeconomic background, ethnic equity pertains to the limited extent to which educational outcomes can be attributed to the student’s ethnic background, and gender equity relates to the limited extent to which educational outcomes can be attributed to the student’s gender.

In the context of this study, educational outcomes encompass both academic achievement and intrinsic motivation. Existing research suggests that qualitative education may exhibit a stronger correlation with motivational outcomes than with academic achievement alone (Appels et al., 2024). This phenomenon can be attributed to the underlying principle of qualitative education, which emphasizes addressing students according to their individual levels of development, rather than solely focusing on high academic achievement across all subjects (Moreno, 2010; Vygotsky, 1978). In this pedagogical approach, the cultivation of intrinsic motivation takes precedence over a singular emphasis on academic achievement as the sole indicator of success. Intrinsic motivation then refers to a student's internal drive to learn, characterized by finding the subject interesting and enjoyable (Deci & Ryan, 1985; Hooper et al., 2017).

Variance of educational equity

The second group of equity domains contains only the dimension of equity of treatment, which has the objective of ensuring equitable treatment for all children and acknowledges that disparities in treatment can contribute to unjust inequities (Appels et al., 2022b). When equity of treatment is examined within the context of ILSAs, two distinct lines of inquiry emerge. The first pertains to immutable school characteristics such as school types or locations (e.g., Gilleece et al., 2010; Tsai et al., 2017). The second delves into the treatment students experience within schools; it addresses factors such as teacher support, adaptive instruction, and opportunities to learn (e.g., Lafontaine et al., 2015; López Rupérez et al., 2019).

Two decisions shaped the delineation of equity of treatment in this paper. First, immutable school characteristics were not employed as a means to define treatment in schools. Second, this study examined the actual treatment of children in schools with the aim to establish the undeniable connection between educational quality and equity (Kyriakides & Creemers, 2011, 2018; Kyriakides et al., 2019). As educational quality encompasses the treatment of children in the educational context, equity of treatment was defined as the variation across schools in the provision of qualitative education (e.g., teacher quality, instructional quality, quality of school life) for all children (e.g., Appels et al., 2022a; Lafontaine et al., 2015). The smaller the variation, the more equitably children are treated in the educational system.

Research questions

Although the fundamental questions addressed in this research are not new, our approach stands out by framing educational equity as a complex, multifaceted construct. Moreover, we underscore the significance of intrinsic motivation as a critical prerequisite for engagement in learning.

RQ1. What clusters of educational systems, represented by their students in grade four participating in TIMSS 2019, can be identified based on the variations in educational equity related to both academic achievement and intrinsic motivation?

RQ2. How are these clusters connected to student outcomes?

Our objective is not to provide definite answers but rather to offer insights that can contribute to our understanding of what educational equity is and how to further its understanding in an international context.

Method

Data

To establish country clusters based on educational equity, we used data collected from students in grade four in TIMSS 2019 (IEA, 2021). Among the ILSAs, TIMSS stands out due to its specific emphasis on formal mathematical and scientific knowledge (Nilsen et al., 2016). Particularly conducive to cross-linguistic comparisons, the field of mathematics serves as an interesting domain for assessment. Within the international comparative mathematics assessments TIMSS and PISA take precedence. However, as TIMSS adopts a curriculum-based evaluation approach, whereas PISA assesses students’ capacity to apply knowledge in real-world contexts, TIMSS becomes the natural choice for delving into insights regarding scholastic education. Moreover, TIMSS, like other assessments, is accompanied by its background questionnaires, gathers pertinent information on educational equity and its influence on students’ educational outcomes (Mullis & Martin, 2017). Leaving out the benchmarking participants (Moscow, Dubai, Quebec, Madrid, Ontario, and Abu Dhabi), data from all other participating educational systems were included as they all provided reliable and complete datasets regarding the variables under investigation.

The analysis of TIMSS data necessitates careful consideration of its complex sampling design (Rutkowski et al., 2010). First, the students sampled in TIMSS are not chosen individually through random selection; instead, a stratified, two-stage, cluster sampling design is employed. This involves the random selection of schools in well-defined strata, followed by the random selection of entire classes or groups of students within these schools. Second, a complexity arises from the fact that not all students respond to all items in the assessment (Martin et al., 2020).

To address the clustering inherent in the sampling design and ensure the inclusion of every student, appropriate weights were applied during the data analysis. These weights serve to signify the significance of each student within the analysis. For country-level aggregations, corresponding weights at the relevant variable level were incorporated into the data analysis. Since the variables of interest in this study were measured at the student level, the TOTWGT weight was used to account for the importance of each student in the analysis. Furthermore, it is worth noting that TIMSS does not compute an overall proficiency score for each student; instead, it relies on five plausible values to estimate a student’s proficiency level if they were to respond to all items. Prior to averaging the results of the five analyses, each of these values should be used in a separate analysis (Von Davier et al., 2009). To complete this section with the measurement of educational outcomes, it should be noted that intrinsic motivation is measured by the Students Like Learning Mathematics scale from the student TIMSS questionnaire (Hooper et al., 2017).

TIMSS indicators of educational equity

To align the conceptualization of educational equity with the measurement using TIMSS data, we translated the theoretical constructs into potential TIMSS indicators based on previous research findings. This section provides an overview of the variable selection process. Table 2 serves as a summary and guide, presenting additional information on the TIMSS codes and labels of the variables.

Table 2 Selected TIMSS variables for the operationalization of educational equity

Social equity

The examination of social equity serves as the initial focus in this elaboration, with socioeconomic status (SES) playing a significant role in its measurement. Variables that coincide with SES are diverse, and researchers have adopted different approaches in their measurement (Appels et al., 2022b). Some scholars opt for individual variables (e.g., Gilleece et al., 2010), whereas others prefer a composite measure (e.g., Gustafsson et al., 2018). Proponents of the use of a composite variable argue that capturing SES as a unidimensional construct may overlook important aspects (Gustafsson et al., 2018). However, others note that composite variables encompass all three forms of Bourdieu’s capital, resulting in a broad and non-specific measure (Kellaghan, 2001). Moreover, it is argued that the underlying components of composite measures may not always be strongly interconnected (Gilleece et al., 2010).

Additionally, the issue of measurement error arises, particularly when considering the use of student self-report measures such as the presence of books at home (Engzell, 2021). Students from book-rich homes tend to perform better academically and accumulate more books, whereas low-achieving students may underestimate the number of books in their homes. Nevertheless, this proxy has gained considerable popularity in assessments involving children who may have difficulty answering questions about their parents’ education or income. Furthermore, low response rates to parent questionnaires pose other challenges and can bias SES estimates as well to an unknown but certainly troubling extent (Engzell & Jonsson, 2015).

In summary, the selection of an appropriate SES measure involves choosing among evils. In this study, socioeconomic status was operationalized by the number of books at home. This measure provides valuable insights into assessments involving children, especially when reporting on parents’ education or income is challenging. We assessed social equity from two perspectives, treating intrinsic motivation and academic achievement as distinct outcome components. We used the degree to which intrinsic motivation, on the one hand, and academic achievement, on the other hand, could be attributed to the student’s socioeconomic background as indicators of social equity.

Ethnic to migrant equity

To measure ethnic equity, the TIMSS 2019 student questionnaire captures variables pertaining to the child’s and parents’ country of birth. This study utilizes these variables to create the student characteristic migrant status, as a proxy for ethnic background, following the approach outlined by Dockery et al. (2020). In this approach, students are categorized into two groups, natives and migrants. Natives are defined as students with one or both parents born in the country. Migrants encompass students who were born in the country but whose parents were born abroad, as well as those whose parents were born abroad and were born abroad themselves. Migrant status was chosen as a measure of ethnic equity because it is more nuanced than the alternative ethnic indicator provided within the TIMSS framework, namely the language spoken at home. Therefore, for the remainder of this paper, the term ethnic equity will be reframed as migrant equity.

We included intrinsic motivation and academic achievement as output components in exploring the extent to which educational outputs could be attributed to the student’s ethnicity. However, it is important to acknowledge that our chosen measure too is not ideal, as it does not fully capture the nuanced ethnic distinctions, especially for those who have resided in the country for multiple generations already. In this context, it is more appropriate to consider our analysis in terms of migrant equity rather than the broader ethnic equity.

Gender equity

To measure gender equity, we used a measure of student gender derived from the student questionnaire, in conjunction with the measure for intrinsic motivation and achievement. This facilitated the investigation of gender-based discrepancies in the educational outcomes.

Equity of treatment

As mentioned in the discussion of the theoretical framework, equity of treatment, which pertains to students’ perceptions of fairness and equal opportunities, was determined by within-country variations in the quality domains. To measure equity of treatment we used TIMSS indicators that Appels et al. (2024) showed were relevant for the quality domains: teacher quality, instructional quality, and quality of school life. The within-country standard deviation of the corresponding TIMSS indicators was considered an indicator of equity of treatment in this investigation.

Cluster analysis

In a cluster analysis, the aim is to identify and group together homogeneous observations within the dataset (Bartholomew et al., 2008). We chose a two-step cluster analysis. In opting for this approach, we rejected the assumption that populations are homogenous in their equity characteristics and interrelationships (von Eye & Bogat, 2006). When used to identify clusters of countries, this approach looks for patterns of associations that are similar within clusters but different between them. An advantage of the two-step cluster analysis is its capacity to handle both continuous and categorical variables in large datasets like TIMSS, making it particularly suited for our study (Michaelides et al., 2019a).

Cluster analysis is an exploratory procedure, allowing for the extraction and interpretation of different numbers of clusters. Additionally, the two-step clustering method allows for the discovery of clusters that might exhibit inconsistencies across the equity dimensions, potentially opening up interesting theoretical opportunities. In this, we restricted ourselves to the creation of a manageable number of clusters. Therefore, the fixed number of clusters was incremented between three (i.e., one less than the number of domains included) and seven (i.e., the number of indicators included). The cluster analyses were conducted using IBM SPSS Statistics (version 28.0.1.1).

Preliminary analysis

In preparation for the preliminary analyses, country-aggregated means of intrinsic motivation and achievement were computed for each group within the separate equity domains. We calculated these means using the IEA’s International Database (IDB) Analyzer, incorporating the appropriate weights (Fishbein et al., 2021). To evaluate social equity, country-specific means of intrinsic motivation and achievement were computed for students with abundant resources (option 4: 101–200 books; option 5: more than 200 books) and limited resources (option 1: 0–10 books; option 2: 11–25 books), based on the number of books at home. To measure migrant equity, country means were calculated separately for students with native status and those with migrant status. Similarly, country means were calculated separately for boys and girls in assessing gender equity. General country means were computed for each of the equity indicators included in equity of treatment. However, we focused on the associated standard deviation of these country means.

Two types Cohen’s d were calculated for the first three equity domains. Cohen’s d is a standardized effect size used to measure the difference between two group means. In this study, we examined the standardized difference in group means for intrinsic motivation and academic achievement. The use of a standardized effect size allows for more accurate comparisons across countries. In this study, a score ≥ 0.2 indicated a low effect size, a score ≥ 0.5 indicated a medium effect size, and a score ≥ 0.8 indicated a large effect size. This calculation was performed on a per-country basis by dividing the difference in average motivation or achievement between the groups by the associated standard deviation of motivation or achievement, respectively, within that country. To ensure consistency, the subtraction was carried out with the assumed less advantaged group placed first so that a negative score indicated discrimination.

Subsequently, several preliminary cluster analyses were conducted. Equal weightage was assigned to countries in the clustering process, and no additional weights were used in the actual cluster analyses. The software was initially allowed to automatically select the number of clusters, but this yielded limited information. The subsequent preliminary analyses focused primarily on the indicators of equity of treatment. Various configurations were explored, including the inclusion of all indicators, the utilization of only one indicator per quality domain, and the incorporation of only indicators derived from student questionnaires. However, these analyses obscured the clearer patterns revealed in the analyses of the other domains. Consequently, it was decided to exclude equity of treatment from further consideration in the analysis.

Equity clustering

The set of Cohen’s d calculations for the remaining three equity domains, namely, social equity, migrant equity, and gender equity, served as input for the subsequent cluster analysis. The determination of the optimal number of clusters was guided by judgment criteria. The silhouette measure of cohesion and separation needed to reach at least a fair level, indicating a reasonable-to-strong cluster structure (IBM Corporation, 2021; Kaufman & Rousseeuw, 1990). Additionally, the smallest cluster was required to contain a minimum of 7% of the sample (Michaelides et al., 2019a). Once these criteria were satisfied, ANOVAs were conducted to evaluate the significance of the remaining variables’ contributions to the finalized clusters, with the significance level set at alpha level < 0.001. The variance explained by each variable was assessed using eta squared, with a threshold of > 50% indicating substantial explanation (Milligan & Cooper, 1985).

As the clusters were derived, we considered their interpretation. The final number of clusters was determined based on the assessment of the first researcher, after ensuring consensus through discussions with the other two researchers (Michaelides et al., 2019a). The process of variable selection and cluster formation was not a linear one and involved several iterations. The interpretation of the final clustering was conducted within the framework of the theoretical model (von Eye & Bogat, 2006). Additionally, several Tukey’s Honestly Significant Difference (HSD) post-hoc tests were conducted to examine the differences between the cluster means of each variable and the means of the other clusters. The detailed statistical measures and the SPSS syntax employed for this two-step clustering process can be found in Appendices A and B.

Five distinct clusters emerged from this enterprise. To ensure the stability of the results, multiple final solutions were obtained by randomly sorting the cases in the dataset. Consistent outcomes validated the robustness of the identified clusters (IBM Corporation, 2021). Finally, the clusters were given descriptive names that best reflected the noteworthy findings obtained from the data.

Bigger picture

To gain further insight into the relationship of this clustering with educational outcomes, a cluster comparison was conducted. Effective learning occurs when instruction is tailored to the student’s zone of proximal development (Vygotsky, 1978). Tasks that are too easy or too difficult are likely to undermine students’ motivation and self-esteem (Moreno, 2010). Furthermore, motivation plays a crucial role in supporting long-term cognitive learning (Deci & Ryan, 2000). These perspectives converge to highlight the importance of motivation in educational outcomes. Motivated students engage actively in cognitive tasks, and this engagement facilitates cognitive learning at an appropriate level, leading to enhanced self-esteem (Bellens et al., 2019; Nilsen & Gustafsson, 2016).

To account for the multidimensionality of educational objectives, this study incorporated both cognitive and affect-motivational educational outcomes in the analysis (Michaelides et al., 2019a). The clusters were included as an additional variable in the country-level dataset. Moreover, country means for the outcome variables of intrinsic motivation and achievement were calculated using the IDB Analyzer and integrated into the dataset (Fishbein et al., 2021). The average educational outcomes related to the subject of mathematics were compared among the clusters.

Results

This combined evaluation resulted in the identification of five distinct equity clusters (see Appendix A), providing a comprehensive and nuanced perspective that extends across educational systems. These findings shed light on the complex interplay between equity, achievement, and motivation in the educational systems we examined. The observed variations in equity across clusters underscore the importance of understanding the multifaceted factors influencing educational outcomes. In the following sections we delve further into the results obtained from the analysis presenting descriptive and inferential statistics in Table 3. Figure 2 allows for a visual inspection of these distributions. First, it illustrates the average Cohen's d per equity dimension, along with their respective distributions represented as a standard deviation above and below the mean. In doing so, it differentiates between the attributability of either motivation or achievement to each of the three equity concerns (i.e., social, gender, ethnic). Second, it provides the cluster’s average z-score for the educational outcomes motivation and achievement themselves. Appendix C provides a deeper consideration of TIMSS country rankings and their relationships to our selected educational outcomes and clusters.

Table 3 Descriptive and inferential statistics across the five equity clusters

Fig. 2

Visual inspection of statistical differences between equity clusters. The figure illustrates the average Cohen’s d (vertical line) per equity dimension, with a distinction between the attributability of either motivation (light grey bar) or achievement (dark grey bar) to each of the three equity concerns. The distribution is visualized by a standard deviation above and down the cluster’s average. Black dots represent the average Cohen's d for each country within the cluster. In the Outcome row below, the average z-score per outcome variable is represented by a circle, with a distinction between intrinsic motivation (light grey) and achievement (dark grey)

Cluster 1—the underdogs (7%)

Cluster 1, representing a small proportion of the sample, comprised countries located in Southwest Asia and North Africa. Notably, this is the only cluster that exhibited a strong geographical connection. As implied by its name, the disadvantaged students included in it showed remarkable resilience that was reflected in both their motivation and achievement. This resilience was particularly evident in the migrant equity dimension, with migrant children outperforming native children in achievement (Cohen’s d = 0.47; SD = 0.35) and motivation (Cohen’s d = 0.14; SD = 0.04). Migrant equity was higher in this cluster than in any of the other clusters in our sample. However, this degree of equity might be interpreted as extending beyond the level of fairness to constitute a form of positive discrimination. A similar pattern was observed in gender equity related to motivation (Cohen’s d = 0.16; SD = 0.07), with girls outperforming boys. Social equity in achievement (Cohen’s d = − 0.12; SD = 0.14), social equity in motivation (Cohen’s d = 0.00; SD = 0.05), and gender equity in achievement (Cohen’s d = 0.01; SD = 0.07) were more neutral in this cluster than in the others, having no negative or tilted effects. In other words, the underdogs in this cluster had genuine educational opportunities.

The educational outcomes within this cluster presented a duality. On one hand, the students in Cluster 1 were not significantly different from those in the highest-performing cluster in intrinsic motivation (mean TIMSS score, 10.34). On the other hand, the academic achievement in Cluster 1 did not surpass the low benchmark, although it significantly differed from academic achievement in the lowest-performing cluster (mean TIMSS score, 449). Such TIMSS benchmarks permit an interpretation of achievement. The low benchmark indicates basic mathematics knowledge, whereas the advanced benchmark involves communicating conceptual understanding in various complex situations. The countries included in this cluster were Bahrain, Kuwait, Qatar, and the United Arab Emirates. It is worth noting that all these countries performed below the TIMSS centerpoint in terms of achievement, despite motivation scores that ranked in the top half, although not in the top quarter.

Cluster 2—the SES disputers (7%)

Cluster 2 was characterized by contrasts in its social equities. Gender equity in this cluster demonstrated positive discrimination in both achievement (Cohen’s d = 0.23; SD = 0.08) and motivation (Cohen’s d = 0.28; SD = 0.16), but there were negative inequities related to ethnicity in the same parameters (Cohen’s d = − 0.14; SD = 0.29 and Cohen’s d = − 0.20; SD = 0.09, respectively). The findings in social equity, however, contradicted each other to such an extent that the achievement difference between children with few and many resources at home was the greatest positive difference in the sample (Cohen’s d = 0.18; SD = 0.28), whereas the difference in motivation between these groups had, significantly, the most negative ratio (Cohen’sd = − 0.21; SD = 0.14).

Cluster 2 did not significantly differ from the highest-performing cluster in intrinsic motivation (mean TIMSS score, 10.40). However, academic achievement in Cluster 2 (mean TIMSS score, 375) was ranked lowest. The countries included in this cluster were Oman, the Philippines, Saudi Arabia, and South Africa. All of these countries scored below the TIMSS centerpoint, and none surpassed the low benchmark. However, intrinsic motivation scores in these countries, except for the Philippines, ranked in the top half, although not in the top quarter.

Cluster 3—the gender equals, yet migrant unequals (24%)

Cluster 3 stood out due to its significant migrant inequity. Migrant students in this cluster exhibited a substantial disadvantage in intrinsic motivation compared to native classmates (Cohen’s d = − 0.23; SD = 0.11). This motivational outcome attributability to migrant background was the largest in the sample. Furthermore, there was a notable disparity in achievement between migrant and native students (Cohen’s d = − 0.34; SD = 0.18). In this regard, this cluster performed second-lowest in the sample. Despite some equitable scores in social equity related to motivation (Cohen’s d = 0.10; SD = 0.05) and gender equity related to motivation (Cohen’s d = 0.08; SD = 0.07) and achievement (Cohen’s d = − 0.02; SD = 0.07), a large disparity in achievement could be attributed to the students’ social backgrounds (Cohen’s d = − 0.42; SD = 0.22). Although inequitable, this score differed significantly from the most inequitable cluster in this regard.

In terms of affect-motivational outcomes, Cluster 3 demonstrated the highest scores in the entire sample (mean TIMSS score, 10.89). Its achievement score did not significantly differ from those of the highest-performing or lowest-performing clusters (mean TIMSS score, 465), meaning that its performance was average. The countries included in this cluster were Albania, Azerbaijan, Armenia, Bosnia-Herzegovina, Georgia, the Republic of Iran, Kazakhstan, Kosovo, Montenegro, Morocco, North Macedonia, Pakistan, Serbia, and Turkey. The achievement scores of these countries clustered around the TIMSS centerpoint, ranging from the low to intermediate benchmark. It is noteworthy that all of the countries ranked in the top ten on TIMSS motivational scores were part of this cluster.

Cluster 4—the soft achievement mismatchers (50%)

Cluster 4 exhibited a degree of equity in motivational outcomes but pronounced a greater achievement inequity. Motivational outcomes showed no relationship to social (Cohen’s d = 0.06; SD = 0.07) or migrant background (Cohen’s d = 0.08; SD = 0.12). However, consideration of the role of gender on motivational outcome variations, revealed that the girls in this cluster experienced the second-highest, albeit still insignificant, level of inequality (Cohen’s d = − 15; SD = 0.10). Shifting our focus to achievement variations, significant inequities were observed in all three domains when compared to the results observed in the other clusters. Achievement was impacted by a medium effect of social background (Cohen’s d = − 0.73; SD = 0.18), as well as a small effect of migrant background (Cohen’s d = − 0.25; SD = 0.24). However, this is not the most unequal migrant attributability score in the sample, although it is not significantly different from it either. Lastly, this cluster demonstrated the least gender equity in achievement, although the effect was not substantial (Cohen’s d = − 0.12; SD = 0.06).

Surprisingly, these findings have adverse effects on the educational outcomes in this cluster. Despite exhibiting reasonable equity in motivational outcomes, the students in this cluster had the second-lowest average intrinsic motivation score (mean TIMSS score, 9.70), and this was not significantly different from the score of the lowest-performing cluster. Still, this cluster had the second-highest average score in achievement (mean TIMSS score, 531), and this score was not significantly different from that of the highest-performing cluster. Cluster 4 comprised Australia, Belgium, Bulgaria, Canada, Chile, Croatia, Cyprus, the Czech Republic, Denmark, England, Finland, Hong Kong, Hungary, Ireland, Italy, Latvia, Lithuania, Malta, the Netherlands, New Zealand, Northern Ireland, Norway, Portugal, the Russian Federation, Singapore, the Slovak Republic, Spain, Sweden, and the United States of America, making it the largest group in our sample. Almost all of these countries scored above the TIMSS centerpoint, with the scores falling between the intermediate and high benchmarks. In contrast, the intrinsic motivation scores of these countries fell in the lower half of the TIMSS ranking.

Cluster 5—the strong achievement mismatchers (12%)

Cluster 5 can best be understood as an amplification of Cluster 4, in which disparities were accentuated even further. Once again, it is essential to distinguish between the attributability of motivational outcome variations to student background and the attributability of achievement variations. The effect of social background on motivational outcomes showed the resilience of students with limited resources (Cohen’s d = 0.20; SD = 0.06). Equitable migrant treatment fostered motivation as well (Cohen’s d = − 0.04; SD = 0.12). However, motivational gender inequity was higher in this cluster than in any of the others (Cohen’s d = − 0.26; SD = 0.09). The achievement inequities related to gender were the second most unequal, although the difference between it and the other clusters was not statistically significant (Cohen’s d = − 0.10; SD = 0.07). The effects of the disparity between children with native and migrant background fell into the medium range (Cohen’s d = − 0.52; SD = 0.12), with the migrants being at a disadvantage. Finally, the achievement inequity between children from households with ample resources and those with limited resources was significant (Cohen’s d = − 0.83; SD = 0.11). Notably, these last two inequities were the largest in the sample.

In educational outcomes, Cluster 5 also mirrored Cluster 4 in a heightened manner. The cluster exhibited the lowest average intrinsic motivation score (mean TIMSS score, 9.43), although its average mathematics achievement was the highest in the sample (mean TIMSS score, 551). The countries comprising this cluster were Austria, France, Germany, Japan, Korea, Poland, and Taiwan. Most of these countries ranked towards the lower end of the TIMSS ordering in motivation, whereas the scores in achievement situated themselves between the intermediate and the advanced benchmarks.

Discussion

Despite the growing body of literature on educational equity within the realm of ILSAs, little research has explored how the multiple dimensions of equity relate to one another (Appels et al., 2022b). Consequently, equity’s underlying mechanisms and its relationship with distinct outcome variations have often been oversimplified, limiting our understanding of the rich notion. This paper advances the current body of research on educational equity. We began by presenting a definition of equity’s major dimensions and our perspective on educational goals. Subsequently, a cluster analysis was employed to map the equity indicators from the TIMSS dataset onto those dimensions to provide comprehensive clusters of various countries’ situated equity.

Before exploring the various clusters of educational equity, some considerations need to be addressed. First, our analysis relied on an existing dataset, which inherently constrained representation of equity in the theoretical framework (Biesta, 2009). Second, it is crucial to acknowledge that researchers make critical decisions during cluster analyses, including defining the number of clusters and interpreting their meaning (Michaelides, 2019a). These decisions introduce subjectivity into the analysis, relying on the researcher’s judgment and expertise. Finally, using the world as an educational laboratory can contribute to a global understanding of educational equity (IEA, 2022), but also necessitates reflection on the nuanced and enduring interplay between culture and education (Alexander, 2001; Leung, 2006; Nagengast & Marsh, 2014). Although the TIMSS study data can allow us to examine whether educational equity transcends geographical boundaries, this paper did not encompass cultural sensitivities within its scope, leaving the incorporation of a politico-educational framework for further research.

A deeper examination of individual countries and their educational policies holds promise for enhancing our understanding of the intersectionality of these concepts. While an initial exploration based on the widely recognized national cultural dimensions by Hofstede (2011) did not yield conclusive results, a more focused investigation into specific educational policies, with their similarities and differences, may provide valuable insights and policy recommendations. Moreover, besides the examination of national cultures also within-nation cultures might be an explanatory addition. Without a doubt, the availability of certain survey questions and the absence of others has influenced the patterns identified in this paper. But even more so, and again related to the topic at hand, cultural differences may have impacted responses to various questions. Although ILSAs offer unprecedented opportunities for studying education from a cross-cultural perspective, cross-cultural comparisons of constructs rest on the assumption that scales have a similar meaning in the considered cultures (Lee et al., 2011; Marsh, 2006; Nagengast & Marsh, 2014). It is, however, imperative to recognize that such scale comparability across all participating countries cannot always be taken as a given (van de Vijver & Leung, 2000) The current study did not encompass an evaluation of such measurement invariance (Nagengast & Marsh, 2014). Further exploration within the realm of cultural and politico-educational frameworks could add additional layers around the findings presented in this study.

The findings of this study shed light on the interplay between equity, achievement, and motivation in the educational systems we analyzed. Broadly speaking, Cluster 1 exhibited the highest degree of educational equity, whereas Cluster 5 displayed the least. However, none of the identified clusters consistently scored high or low on all equity dimensions. This suggests that all of the clusters performed inconsistently to different degrees across the dimensions of equity. The absence of clusters with uniform performance highlights the complex nature of equity in educational systems. Furthermore, no single cluster emerged as superior to the others across both outcome measures. Nonetheless, clusters characterized by greater inequity in achievement tended to have higher mean scores in academic achievement. In contrast, clusters demonstrating higher equity in achievement tended to exhibit higher mean scores in intrinsic motivation.

Each cluster had a unique configuration of the equity dimensions, which drew attention to the need for nuance but also offered insight into the underlying mechanisms. Generally, achievement was predominantly negatively attributed to student background for children with fewer resources (social equity) or a migrant background (migrant equity). Conversely, gender inequity appeared to be effectively countered by educational systems, which even created environments in which girls outperformed boys. Notably, gender had similar effects on achievement and motivation, whereas outcomes attributed to social background often had contradictory associations with motivation and achievement. Furthermore, the influence of gender on achievement seemed to align with the influence of SES, whereas migrant equity displayed a distinct pattern.

This paper aimed to extend the scope of equity research from focusing solely on academic achievement to incorporating motivational outcomes and a known spectrum of dimensions. This endeavor sought to contribute to a more nuanced understanding of educational equity. Despite this attempt to delve deeper into educational equity, challenges persist, echoing concerns raised by previous ILSA research on educational equity (e.g., Bellens et al., 2020; Meinck & Brese, 2019). The current operationalization too primarily centers on measurable aspects of diversity, overlooking the underlying mechanisms contributing to certain inequalities as these were not found in the dataset.

It is noteworthy that prior equity research on ILSA data heavily relied on PISA data (e.g., Lopez-Ruperez et al., 2019), potentially introducing a dataset bias that suggests PISA might offer a more comprehensive perspective on equity. Upon closer inspection, PISA assessments occasionally do touch on such underlying mechanisms through indexes providing insights into social and emotional resilience or social inclusion (Appels et al., 2022b). Consequently, an opportunity exists to enrich TIMSS questionnaires, establishing a stronger foundation for a comprehensive evaluation of educational equity within the TIMSS framework. Additionally, it should be recognized that ILSA data are not the exclusive source for studying equity within an educational system (e.g., Oates, 2021). National examinations, for instance, offer a unique lens into specific national contexts, providing valuable insights into historical contexts, closer time lags, and detailed domestic performance analyses.

The influence of this discourse on educational equity extends beyond academic discussions, as governments use research findings to inspire policy decisions aimed at improving a country’s education system (Addey et al., 2017; Dong & Cravens, 2012; Drent et al., 2013; Kyriakides & Creemers, 2018; Rutkowski et al., 2010). Understanding the complexities of equity in education is crucial for informed decision-making and effective policy implementation in order to foster positive outcomes for students across diverse educational contexts. Notably, conventional ILSA research typically delves into selected combinations of educational equity dimensions, focusing solely on its impact on achievement, requiring policymakers to comprehend this fragmentation (Authros, 2022b). In contrast, our research advocates a multidimensional approach, considering both achievement and motivation as important educational outcomes, providing a richer understanding of the complex concept that is equity.

Regarding our methodological decisions, the results of the cluster analyses, compared to the conventional analyses like multilevel analyses, revealed a more holistic understanding of the concept under consideration. The clustering uncovered a deeper and more nuanced perspective on equity configurations, providing insights into their interplay of domains within the educational realm. This multifaceted assessment of educational systems stands as evidence of the complex nature of educational equity (Levin, 2003).

Despite, this paper represents only a first exploration into the multidimensional nature of equity using TIMSS data. Alternative statistical methods could offer a more comprehensive understanding of equity’s underlying dynamics. In such further research, the element of time might be a relevant addition. Clustering might reveal that a country exhibits different equity configurations at different time points. For instance, policy changes have the potential to significantly influence a country’s educational equity, as the spirit of times may reshape the composition of an entire cluster or the distribution of countries across clusters. Additionally, we posit that combining the multiple dimensions of both quality and equity might enrich our understanding of educational systems even further (Gustafsson et al., 2018; Kyriakides et al., 2019). It is important to emphasize that this paper constitutes an initial exploration, providing a foundational groundwork for more intricate statistical inquiries and qualitative enhancement.

Conclusion

To tackle the task of precisely and validly measuring educational equity on an international scale, we introduced a cluster analysis utilizing TIMSS data. This method has proven to be a powerful tool for identifying equity clusters among countries participating in TIMSS 2019. Moreover, it provided comprehensive clusters of countries’ equity across multiple educational outcomes. The distinctiveness of each cluster’s equity profile and the fact that few overarching characteristics could be discerned underscores the need for nuance and caution when discussing educational equity. This type of equity analysis paves the way for a deeper understanding of countries’ equity patterns and how those patterns interact with educational outcomes.