ERIC Number: ED663563
Record Type: Non-Journal
Publication Date: 2024-Sep-20
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Weighting Opt-In Surveys to Accommodate the Effects of Nonresponse
Ashani Jayasekera; Laura Stapleton
Society for Research on Educational Effectiveness
Background: A growing number of surveys are conducted online where respondents can choose to complete the questionnaire (Lehdonvirta et al., 2020). As respondents are self-selected, there is potential that the respondents will not be an accurate representation of the population. For example, white people are disproportionately more likely to participate in opt-in surveys than people of color (Jang & Vorderstrasse, 2019). Thus, researchers have explored using statistical techniques to adjust for bias that may exist. Research Questions: Research (Mercer et al., 2018) has shown that the covariates used for adjustment may be more important than the choice of weighting method. This study seeks to determine if better estimation is possible when adjustment sets include responses to items as covariates, in addition to typically-used demographics. The research questions are: (1) Can reweighting methods result in unbiased parameter estimates in opt-in survey samples of teachers? and (2) What is the best set of covariates to use with each method to yield the least biased parameter estimates? Methods: The investigated weighting methods are raking, inverse propensity score weighting, propensity score matching, and kernel weighting. In each case, the estimated survey weights are rescaled, so the sum of the weights is equal to the size of the opt-in survey sample. "Raking": Raking requires that the population marginal distributions for the adjustment variables be known (Mercer et al., 2018). Using these marginal distributions, the opt-in survey weights are adjusted iteratively such that opt-in survey's weighted marginal distributions match population levels (Debell & Krosnick, 2009). "Inverse Propensity Score Weighting": Unlike raking, inverse propensity score weighting requires that there be a sample of observations that is representative of the population. Since probabilities in opt-in surveys are unknown, they can be estimated using propensity scores (PS; Valojerdi & Janani, 2018). PS are the conditional probability that a given observation belongs to the treatment group (Z = 1) given the covariates. In this study, PS are estimated using logistic regression models where the treatment group represents the population data. Each observation was given a weight equal to the log odds of belonging to the target data. "Propensity Score Matching": PS matching matches population observations to their most similar opt-in survey observation. When matches have been found for all target cases, the unmatched opt-in observations are given weights of 0. Similarity is calculated using the nearest neighbor method (Cover & Hart, 1967). Matching is implemented using a many-to-one approach; matches to the opt-in sample are made with replacement so opt-in sample observations that are more similar to observations in the population will have a higher number of matches. The weights are the sums of the matches made to the target population (for that opt-in sample observation). "Kernel Weighting": Kernel weighting (KW) is intended to improve the external validity of survey analyses (Wang et al., 2020). A distance vector is first computed: for each target observation, the difference between its estimated PS from the PS for each opt-in sample observation is calculated. Then a zero centered kernel function is applied to smooth the distances. This study applied a Gaussian kernel function. To calculate optimal bandwidth h, Silverman's method (Silverman, 1986) is used. [sigma] is the square root of the variance, IQR is the interquartile range, and n is the length of the distance vector. The KW pseudoweight for each j in the opt-in sample is the sum of the product of the kernel weight and target sample's survey weights. Data Description: This study uses two sources of data: the 1990-1991 and 1993-1994 Schools and Staffing Surveys (SASS; Choy et al., 1993; Henke et al., 1996) which used multistage, stratified sampling designs. Table 1 and the survey codebooks (Choy et al., 1993; Henke et al., 1996) contain additional detail. Both surveys generalize to the US population of teachers and the survey estimates are taken as approximate population values. However, as we are interested in estimating convenience sample values, nonresponse is generated in the 1990 SASS to function as opt-in data and the 1993 SASS represents the target data. Informative nonresponse is generated in the 1990 SASS using several item responses, race, gender, and ethnicity information. The items used for nonresponse generation were chosen as they correlated with the outcome items in this study. More information on nonresponse generation is in the Appendix. The final size of the opt-in survey is 136 respondents while the population survey had 42,589 respondents. Research Design: A simulation study is conducted where the sample size, adjustment set, and survey weighting scheme are manipulated. Table 2 contains the specific conditions, and 100 replications are conducted in R (R Core Team, 2021) using various packages and functions (Pasek, 2018; Lumey, 2023; & Ho et al., 2010). For each condition, the weighted mean of the survey responses is compared to the population mean. The weighted true mean is calculated from SASS 1993 utilizing the teacher survey weight TCHWGT to accommodate the sampling design. Raw bias is calculated as the difference between the estimated and true mean. Three item responses are used as outcomes; each asks the respondents to rate if this is a problem at their school and the responses ranged from "serious" to "not a problem" on a four point scale: (1) Students dropping out of school (T1140); (2) Student absenteeism (T1080); and (3) Student apathy (T1145). Results: Given space restrictions, limited results are presented here. Often, researchers choose demographic variables to adjust survey estimates. Figure 9 shows that adjustment using demographic covariates outperforms naïve estimation, the case where weights are not considered when calculating the mean, across conditions except when raking is used. For raking, the "kitchen sink" adjustment set reduced the bias in the estimates across all items. The "kitchen sink" refers to the adjustment set which includes demographics as well as items related and not related to the outcome item. Conclusions: The findings demonstrate the potential of weighting adjustment schemes in reducing bias and improving estimation than if the raw estimates were to be used. We will present the results for the other conditions at the conference if selected.
Descriptors: Online Surveys, Test Construction, Test Validity, Test Reliability, Weighted Scores, Disproportionate Representation, Teachers, Statistical Bias, Teacher Participation, Response Rates (Questionnaires)
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A