A General Linear Method for Equating With Small Samples

A General Linear Function

To develop the general linear function, a more general form of the traditional identity function is first presented. Note that, in addition to assuming that X and Y do not differ in difficulty, identity equating also assumes that the scales for each form are equal. This may not always be the case. Instead, differences in the score scales for X and Y, perhaps caused by the removal or modified scoring of one or more items, may provide useful information regarding differences in their empirical distributions. A more flexible form of the identity function is formulated here to take scale differences into account. Coordinates for scores of x and y are found based on their relative positions within each scale:

(1)

Here, and are coordinates for any two points in the scales of X and Y, for example, the minimum and maximum possible scale values. The coordinate defines a point through which the line will pass. Solving 1 for y and rearranging terms produces a more general form of the identity function:

(2)

where and , and denotes the identity function applied to scores on X. In this linear form, the slope and intercept of the equating line are and , respectively, as opposed to 1 and 0 for the traditional identity line.

In its more general form, the identity function still does not estimate any parameters of the X and Y score distributions. However, the traditional assumption of no difficulty difference between forms is modified. Forms are now assumed only to differ in difficulty as a result of known differences in their scales. Because the traditional and more general identity functions do not involve parameter estimates, the SEs for the equating line will always be zero. To the extent that form difficulty differences do exist, the identity lines will both be biased. However, by accounting for form difficulty differences resulting entirely from scaling differences, the general form of the identity function can potentially be less biased than the traditional form.

The identity line in 2 can also be shifted upward or downward so as to pass through any coordinate pair ; this point on the equating line could be based on known scale values, such as , or estimates from the distributions of X and Y. Furthermore, the slope of the identity line, expressed by , can also be replaced by any estimate of the variability in Y over any estimate of the variability in X. This results in the general linear equating function:

(3)

The mean and linear equating functions, as traditionally defined, are obtained by using the means for X and Y to estimate the coordinate pair through which the line passes, and then replacing the slope with 1 in mean equating and with the ratio of standard deviations (SDs) for Y over X in linear equating.

In linear equating, it is assumed that the difficulty difference between forms is captured by their means and SDs. Any change in scale length need not be included in the equation, as its impact on the difficulty difference is captured in the SDs. Coordinates for the linear equating line are thus obtained using standardized deviation scores:

(4)

In mean equating, it is assumed that the mean difference between forms suffices to explain a constant change in difficulty across the score scale. Thus, coordinates for the mean line are obtained using mean deviations. SDs, instead of being estimated, are assumed to be equal. However, in its more general form, the mean function can account for any impact that the scales of X and Y may have on variability in the respective distributions. Coordinates for the general mean equating line are then defined by mean deviations relative to their position within each scale:

(5)

The general mean function does not estimate parameters for variability. In this way, it is similar to the traditional and general identity functions, and the traditional mean function. However, in contrast to the traditional functions, the general mean function assumes that the constant difficulty difference between forms may change, linearly, due to the difference in scaling for the forms. As with the general identity function, this may lead to less bias than the traditional mean function.

The general form of the linear function can be used to obtain other linear functions that combine information from the scales and empirical distributions of X and Y. For example, in practice the SD may be estimated reliably for one form (e.g., Y) but not the other (X), perhaps because of a small sample size; in this case, estimating the slope of 3 with and a fixed value for may be preferable to either assuming the slope to be 1, as in traditional identity and mean equating, or estimating both parts of it, as in traditional linear equating. In another situation, the mean may adequately represent average difficulty on one form (e.g., X) but not the other (Y); in this case, and a fixed value for may better estimate the central difference in form difficulty. These examples represent different combinations of identity, mean, and linear functions for X and Y in situations were sample size may limit the reliability of certain estimates.

The general linear function can also be applied with any nonequivalent-groups linear method that estimates means and SDs of X and Y in the target population T using weighted combinations of estimates from the nonequivalent-groups design. Thus, the Tucker and Levine linear methods for nonequivalent groups (Kolen & Brennan, 2004) can both be used to obtain parameters for 3 (for details see Albano, 2014). Combined forms of the resulting function can also be estimated, as described above.

Two other methods for combining equating functions have been demonstrated in the literature. The first reviewed here is referred to as circle-arc equating (Livingston & Kim, 2009); it involves a curvilinear combination of the mean and identity lines. The second is referred to as synthetic equating (Kim et al., 2008); it involves a combination of linear equating functions by averaging (see Holland & Strawderman, 2011).

Circle-Arc Functions

Nonlinear equating functions are appropriate when X and Y differ nonlinearly in difficulty, that is, when difficulty differences fluctuate across the score scale. As noted above, equipercentile equating is the most flexible method for accounting for these fluctuating difficulty differences. In equipercentile equating, no assumptions are made about how the forms relate in terms of difficulty. Instead, difficulty differences are estimated at each score point; thus, each coordinate on the equipercentile curve is estimated using information from the distributions of X and Y.

Circle-arc equating also defines a nonlinear relationship between score scales; however, only three score points in X and Y are used to do so: the low and high points, as defined above for the identity function, and a midpoint that is estimated based on the score distributions for X and Y. On their own, the low and high points define the identity linking function , a straight line. When the midpoint does not fall on the identity linking line, it can be connected to and by the circumference of a circle with center and radius r. Having obtained these values, the circle-arc function is then defined as

(6)

where the second quantity, under the square root, is added to when and subtracted when . When , the circle-arc function is the identity function.

Livingston and Kim (2010) refer to the arc connecting , , and as symmetric circle-arc equating. They also describe a simpler approach, which they refer to as simplified circle-arc equating. In this approach, the circle-arc function is decomposed into the linear component defined by and , which is the identity function, and the circumference of a circle passing through the points , , and . The low and high points in the simplified circle-arc reduce to (x₁, 0) and (x₂, 0). All three points are then used to find the center coordinates and radius of the new circle, , which is combined with the identity function to obtain the simplified circle-arc function:

(7)

Research has shown that the circle-arc method can produce significantly lower total error than other methods for equating with small samples (e.g., Livingston & Kim, 2009, 2010). A drawback to the simplified function is that it does not maintain the symmetry property of equating (for a discussion of symmetry and other properties of equating, see Holland & Dorans, 2006). The loss of symmetry may be worth the gains that the method achieves in equating accuracy (Livingston & Kim, 2011). However, given that the symmetric and simplified functions have been shown to produce very similar results (e.g., Livingston & Kim, 2010), symmetric circle-arc equating may be preferable.

Both of the circle-arc methods involve curvilinear combinations of the mean and identity lines. Note that Livingston and Kim (2009) recommend using the lowest and highest meaningful score points, i.e., Equation 2, to define the identity component of the resulting composite function. The circle-arc functions can also be formulated to combine the mean with different forms of the general linear function in 3. In each case, the identity component is simply replaced by the general linear function, and the circle-arc is redefined to pass through the low and high points of 3 as opposed to 2.

Synthetic Functions

As noted above, the circle-arc equating functions involve a curvilinear combination of the identity and mean functions, where the circle-arc overlaps with the identity function at the low and high points and with the mean function at the midpoint . Kim et al. (2008) demonstrated a different approach to incorporating information from the identity function using what they refer to as synthetic linking and equating. The synthetic function is a weighted average of the identity and another linear or nonlinear observed-score function:

(8)

where one of the functions to be combined is and the weights w₁ and w₂ are values between 0 and 1 that sum to 1.

Similar to the general circle-arc functions, synthetic functions can also involve combinations of general linear functions. Combining linear and identity linking functions, for example, results in a synthetic function with slope

(9)

and intercept

(10)

where a₁ and a₂ are the slopes for the first and second functions. The synthetic weights are used to control the influence of a given function on the final result. In the example above, when the linear weight is 1, the identity weight is 0, and ; conversely, when the linear weight is 0, the identity weight is 1, and . As shown by Kim et al. (2008), the size of the weight determines the error associated with the synthetic function. When the identity weight is .5, for example, the SE of is reduced by half compared to when the identity weight is zero; the bias is half of that for linear equating and half of that for identity equating.

Like the simplified circle-arc function, the synthetic function will generally not be symmetric. However, symmetry can be maintained for any combination of two or more linear functions when the weights are adjusted by the slopes for the linear functions being combined (Holland & Strawderman, 2011). For two linear functions, the adjusted weight W₁ is

(11)

As shown above, the synthetic, like other averaging methods, combines two functions after the parameters of each have been determined. The general linear method can also incorporate information from two separate functions; however, this information can be used to obtain the intercept and slope independently of one another. As a result, the intercept could be based on the means for two forms, while the slope is based on an average of information from the identity and linear functions. In this way, different functions, that is, different sources of information, can be used to obtain and independently of one another, and and independently of one another. This is demonstrated below.

Summary

The goal of linking and equating is to estimate the form difficulty difference from X to Y and describe how it changes across the score scale. To effectively meet this goal, any linking or equating function should take into account the known differences in the difficulty of X and Y that result from features of the test forms themselves. Linear and equipercentile equating naturally account for differences in form features, such as scale length; however, identity and mean equating, as traditionally defined, do not. This article demonstrates a more general form of the linear function for testing applications where form differences produce predictable changes in the scales for each form.

In the context of circle-arc equating, Livingston and Kim (2009) recommend incorporating information about the prespecified end-points of the X and Y scales. This idea is extended here to the traditional identity and mean functions. In identity and mean equating, difficulty is assumed to either not differ or only differ by a constant, regardless of known differences in X and Y. The general linear functions take these known differences into account. As a result, in some situations they may be better suited to describe the equivalence between X and Y. One such situation involves performance assessment, as demonstrated in the first study below (for examples of equating with performance assessment and other forms of assessment where scale length can vary, see Albano & Rodriguez, 2012; Betts, Pickart, & Heistad, 2009; Francis et al., 2008; Montague, Penfield, Enders, & Huang, 2010).

As with the synthetic and other composite methods, the main motivation for a general linear method is the reduction of SE in situations where the difficulty difference between forms is best estimated using a combination of multiple sources of information, including the score distributions on X and Y. The general linear method allows for control over which sources of information have more impact on the final equating function. In this way, it is similar to the synthetic method. However, the general linear method extends this control individually to the slope and intercept that define the equating line, and it does so while maintaining symmetry between the functions linking X to Y and Y to X. This control can be useful in situations where sample sizes differ for X and Y, as demonstrated in the second study below. (Livingston and Kim [2009] also demonstrate equating with samples of different sizes.)

Study 1

Data for Study 1 come from two forms of a reading fluency test, READ1 and READ2, where the total score was calculated as the number of words read correctly from a passage of text in 60 seconds. The reading passages were written according to the same test specifications, and were intended to be equivalent in structure and difficulty; however they differed slightly in length and in expected minimum and maximum scores. The forms were administered to 2,089 elementary students with counterbalancing in a single sitting, making this a single-group equating design. Each word read correctly was worth 1 score point. Means were 79 for READ1 and 70 for READ2, and SDs were 46 and 35, respectively. Due to the construction of the passages, the effective score ranges were [5, 250] for READ1 and [15, 200] for READ2. Note that such large score scales are expected to be more problematic than smaller ones when equating with small samples; longer score scales can result in relatively fewer individuals per score point.

Equating was performed using the equate package (Albano, 2014), within the statistical environment R (R Core Team, 2013). The equate package is open-source software for observed-score equating, supporting a variety of data collection designs and equating methods. Loglinear models were first used to smooth the distributions of READ1 and READ2, preserving the first two univariate moments of each and the first bivariate moment. Figure 1 contains the smoothed bivariate frequency distribution. The scatterplot reveals that smoothed scores on form READ1 extend to about 250, whereas smoothed scores on READ2 taper off around 200.

Linking and equating methods were compared using parametric bootstrapping where the smoothed score distributions for READ1 and READ2 served as a pseudopopulation from which samples were drawn with replacement. The equipercentile equating function based on the smoothed distributions was used as the criterion or true function. Samples of four different sizes, 30, 50, 100, 300, were drawn with replacement from the population, and linking and equating functions were estimated using each of the methods listed below, with sample sizes always matching for READ1 and READ2. The reference form was READ2.

The resampling procedure was repeated 1, 000 times, and the results were then summarized for a given method based on the SD across replications (SE), the difference between the mean estimated function across replications and the criterion equating function (bias), and the square root of the sum of SE squared plus bias squared (root mean squared error, RMSE). Weighted-mean SE, bias, and RMSE were also obtained using smoothed pseudopopulation frequencies on X as weights.

Fourteen different equating functions were compared in Study 1: the traditional and general forms of the identity, mean, and circle-arc functions; synthetic averages of the linear and traditional/general identity functions, with identity weights of .75 for function S1, .50 for S2, and .25 for S3; and linear and equipercentile functions. This resulted in two forms of each identity, mean, and circle-arc function, two forms of each of the three synthetic averages, and one form of the linear and equipercentile functions. Synthetic weights were adjusted using 11 to maintain symmetry.

The general linear equating functions differed from their respective traditional forms in the estimation of the linear slope as compared to . was fixed to 200 − 15 and was fixed to 250 − 5, resulting in . Any general identity functions were also given a midpoint based on the medians of the effective scales of X and Y, so that the equating line passed through the coordinate (127.5, 107.5). Note that the linear slope based on the pseudo-population was also .76. This suggests that the general forms of the identity, mean, and circle-arc functions, along with the three synthetic functions using Δ to define the identity slope, should better estimate the linear change in difficulty across the scale, on average, than the corresponding functions where the slope is fixed at 1.

Table 1 contains the weighted mean RMSE for each equating method by sample size. Values were all multiplied by 100 to improve readability. Figure 2 shows a subset of these results for functions with values below 1. The general identity function resulted in a weighted mean RMSE of .51, which is indicated in Figure 2 by the horizontal solid black line. Abbreviations in Table 1 and Figure 2 are the first letter of the corresponding function, where I, M, and C denote identity, mean, and circle-arc, respectively; G is added for the general form of I, M, C, S1, S2, and S3; and L and E denote the linear and equipercentile functions. Note that Figure 2 only shows the general forms of each function, along with the linear and equipercentile functions, as the remaining functions based on the traditional forms all had errors above 1.

Table 1. Study 1 Weighted Mean by Sample Size

N	I	M	C	S1	S2	S3	GI	GM	GC	GS1	GS2	GS3	L	E
30	4.49	4.12	3.60	3.40	2.38	1.52	.51	.90	.82	.49	.66	.91	1.21	1.45
50	4.49	3.98	3.48	3.39	2.33	1.39	.51	.68	.65	.45	.53	.70	.93	1.11
100	4.49	3.88	3.40	3.38	2.29	1.28	.51	.49	.50	.42	.42	.51	.65	.75
300	4.49	3.81	3.34	3.37	2.25	1.17	.51	.31	.35	.40	.32	.32	.39	.44

Note

• I, M, and C are identity, mean, and circle-arc, respectively; S1, S2, and S3 are the synthetic functions with identity weights of .75, .50, and .25; G indicates the general linear form of the corresponding function; L and E are linear and equipercentile.

Error for all of the functions decreased as sample size increased. The general synthetic functions GS1 and GS2 tended to outperform all others, with GS1 having the lowest error at samples sizes 30 and 50, and GS1 and GS2 having the lowest values at sample size 100. At sample size 300, GM had the lowest error, GS2 and GS3 were slightly larger than GM, and all the methods shown in Figure 2 were below .50. The results of Study 1 show that all of the functions begin to produce similar amounts of error as sample size increases. However, the general linear forms of each function produced noticeably lower error than their respective traditional forms, and were also lower than linear and equipercentile equating at sample sizes of 30, 50, and 100.

Study 2

Data for Study 2 are based on two forms of a large-scale certification exam, FORM1 and FORM2, containing 200 multiple-choice items each, and each administered to over 5, 000 examinees. Items were scored correct/incorrect, resulting in a maximum possible score of 200. Total and anchor test means for FORM1 were 158.5 and 28.7, with SDs of 19.5 and 4.3. Total and anchor test means for FORM2 were 160.7 and 28.7, with SDs of 19.3 and 4.3. These values indicate that the examinees taking each form were similar in ability, and that the forms were similar in difficulty.

To artificially create more of a need for equating, the total score mean for FORM2 was adjusted downward to 148.0 and the SD was increased to 25.0. The FORM2 anchor score mean was modified only slightly to 28.0, and the SD was increased to 4.8. Correlations between total and anchor tests were .86 in FORM1, FORM2, and the modified version of FORM2, referred to as FORM2R.

The bivariate distributions in FORM1 and FORM2R were smoothed to preserve the first three univariate moments and one bivariate moment. Like in Study 1, these smoothed distributions served as pseudopopulations from which samples were drawn with replacement. The sample size for FORM1 varied as 20, 50, 100, 500, whereas the sample size for FORM2R was always 500. Equating functions were estimated at each of 1, 000 replications, with FORM2R as the reference form, and error was summarized as described in Study 1. The chained equipercentile equating function based on the smoothed distributions was used as the criterion function.

Study 2 compared 13 equating functions. These included identity, Tucker mean, Tucker and chained linear, chained circle-arc, and chained and frequency estimation equipercentile. Three general linear functions were also estimated, with the X part of the linear slope, , obtained as a weighted combination of the Tucker SD for FORM1 () and for FORM2R () in the target population. In the first general linear function, G1, from Equation 3 was estimated as ; in G2, it was estimated as ; and in G3, it was estimated as . Note that with a weight of 0, the slope would be estimated as , effectively resulting in mean equating. With a weight of 1, the slope would be , resulting in linear equating. Thus, the three general linear functions examined here represent compromises between mean and linear equating in the X component of the slope; the G1 slope was closer to the mean slope of 1, the G3 slope was closer to the linear slope, and the G2 slope was midway between the two. In these and the other unchained equating methods (i.e., Tucker mean, Tucker linear, and frequency estimation) the population weights for determining T were set proportional to sample size (for details see Albano, 2014).

Three synthetic average functions were also obtained. These differed from the general linear functions in that they involved averages of the traditional Tucker linear and identity functions. As a result, both the intercepts and slopes were averaged, compared with only the slope being averaged in the general linear functions. The identity functions were weighted as in Study 1, using .75 in function S1, .50 in function S2, and .25 in S3. As a result, S1 relied on the identity the most, and S3 relied on it the least. Weights were again adjusted using 11 to maintain symmetry.

Table 2 contains the weighted mean RMSE, again multiplied by 100, for each function across the four sample sizes for Study 2. Trends in the results across sample sizes were less evident in Study 2 than in Study 1, making them difficult to visualize in a figure. Error again tended to decrease as sample size increased. At FORM1 sample size 20, error was lowest for G1, followed closely by G2; at sample size 50, error was lowest for G2, followed by G1 and G3; at sample size 100, error was again lowest for G2; at sample size 500, error was lowest for chained equipercentile, followed by G2, G3, chained linear and frequency estimation equipercentile. For identity equating, mean RMSE (i.e., bias) multiplied by 100 was constant at 3.53. The synthetic function relying the most on the identity, S1, had the next highest error overall. Results were mixed for the remaining functions, with some higher than others at different sample sizes.

Table 2. Study 2 Weighted Mean by FORM1 Sample Size

N	I	MT	LC	LT	CC	EC	EF	G1	G2	G3	S1	S2	S3
20	3.53	1.56	1.85	2.07	2.52	2.11	2.30	1.45	1.46	1.66	2.67	1.92	1.61
50	3.53	1.25	1.21	1.39	1.87	1.29	1.36	1.08	1.01	1.11	2.65	1.81	1.19
100	3.53	1.10	.93	1.09	1.58	.94	.95	.92	.81	.86	2.65	1.78	1.03
500	3.53	.95	.60	.82	1.37	.54	.60	.74	.57	.58	2.65	1.74	.87

Note

• I is identity; MT is Tucker mean; LC is chained linear; LT is Tucker linear; CC is chained circle-arc; EC and EF are chained and frequency estimation equipercentile; G1, G2, and G3 are the general linear functions, with weights of .25, .50, and .75 applied to the X component of the linear slope; and S1, S2, and S3 are the general synthetic functions with identity weights of .75, .50, and .25.