Psychometric properties of scales in the General Nordic Questionnaire for Psychological and Social Factors at Work (QPS_Nordic): Confirmatory factor analysis and prediction of certified long-term sickness absence

INTRODUCTION

Among the many factors that may contribute to health problems (Zapf, Dormann & Frese, 1996), work stress has received increasing attention (European Commission, 2000). Most research in the field is based on self-report questionnaires (Rick, Briner, Daniels, Perryman & Guppy, 2001), many of which have been designed to measure workplace stressors (Hurrel, Nelson & Simmons, 1998; Rick et al., 2001). Furthermore, some questionnaires are research-oriented, with the aim of exploring hypotheses concerning how work affects health (Levi, Bartley, Marmot et al., 2000), whereas others are more comprehensive (Kristensen, Hannerz, Hogh & Borg, 2005; Spielberger & Vagg, 1998), and are intended as a basis for monitoring psychosocial work exposures or to be used in worksite health interventions (Heaney, 2003). However, in a critical review of psychological hazard measures (Rick et al., 2001), only a few comprehensive measures were identified as having sufficient information to provide detailed psychometric evaluation.

The Scandinavian countries have a long tradition of research and monitoring of the work environment. The work culture and legislation of these countries are similar, and collaboration in large-scale research projects is well established (Theorell, 1999). In 1994 the Nordic Council of Ministers decided to sponsor the development of a new questionnaire (Lindström et al., 1995, 1997), the General Nordic Questionnaire for Psychological and Social Factors at Work (QPS_Nordic), which was published in 2000 (Dallner, Elo, Gamberale et al., 2000). An important goal in the design of the questionnaire was that it could be used both in interventions at workplaces and in research, and could provide a basis for reference data. Nineteen different questionnaires concerning psychological and social factors at work were evaluated, as it was concluded that a widely usable set of scales and items was needed as a basis for further development (Lindström et al., 1995). The main principle for selecting conceptual topics for the QPS_Nordic was their relevance and importance for health and well-being. The QPS_Nordic became a broad instrument, based on theories and conceptual models of organizational behavior, work motivation, and job satisfaction, as well as theories of job stress, well-being, and health (Dallner et al., 2000; Lindström et al., 1997). Specifically, the following aspects were covered in the questionnaire: job demands, role expectations, control at work, predictability at work, mastery of work, leadership, social support, bullying and harassment at work, organizational culture, communication in the organization, work groups, organizational commitment, work motives, work centrality and interaction between work and private life. During the construction of the instrument, items were categorized into three conceptual modules; the task, the organizational and the individual modules, and in order to generate subscales, exploratory factor analysis of the QPS_Nordic item pool was performed separately within these three modules. Therefore, the QPS_Nordic should perhaps be regarded as comprising three somewhat different instruments.

Based on data used in the construction process, the test developers presented encouraging data on its reliability (Cronbach's alpha and test-retest reliability) and validity (exploratory factor analyses and correlations with concurrent self-rated health variables) (Dallner et al., 2000). Moreover, in a recent study (Wännström, Peterson, Åsberg, Nygren & Gustavsson, 2008) measurement invariance of scales in the QPS_Nordic was tested across six different occupational groups (private sector employees and health care workers from the public sector). The outcome was fairly satisfactory, as 9 of the 24 analyzed scales functioned well across all occupational groups, and a majority of the remaining scales showed measurement invariance, at least across some groups. The QPS_Nordic has subsequently been used in several large-scale projects (Bergström, Brodin, Bertilsson & Jensen, 2007; Björklund, Grahn, Jensen & Bergström).

An important underlying assumption of the QPS_Nordic is that it should be possible to use the subscales to predict health problems, and such associations can be expected, based on previous research in occupational health psychology and occupational medicine. For example, work environment factors such as repetitive work, high psychological demands, and low social support have been shown to predict decreasing self-rated health assessments (Borg, Kristensen & Burr, 2000) – a health indicator strongly associated with mental and physical health (Singh-Manoux, Martikainen, Ferrie, Zins, Marmot & Goldberg, 2006). These associations have been found to be moderate in magnitude, even after controlling for demographic factors and disease. In addition, several reviews have summarized the occupational risk factors for developing physical symptoms such as neck and back pain (Bongers, Ijmker, van den Heuvel & Blatter, 2006; Linton, 2001). These reviews suggest that work factors such as high demands, low control, monotonous or boring tasks and poor support are often related to these symptoms.

However, the strongest consistent association between psychosocial work factors and health has been found for mental health problems. For example, work factors such as quantitative demands, job control and social support (Stansfeld & Candy, 2006) have previously been found to be associated with mental health problems. In addition, differential associations between psychological work characteristics and symptoms of anxiety and depression have been suggested. In particular, demands may be specifically associated with anxiety symptoms, whereas low decision latitude may be more associated with depressive symptoms (Andrea et al., 2002, 2004).

Finally, associations between burnout and work factors have consistently been found. Work factors such as workload, role problems, and social support have proved to be differently related to the three dimensions of burnout (Lee & Ashforth, 1996; Schaufeli & Buuk, 2003). Based on results from these reviews, the expected association (in the range r= 0.30–0.50) between the QPS_Nordic scales and the three dimensions of professional burnout (MBI-GS) (Maslach, Jackson & Leiter, 1996) is that the “exhaustion” scale should be more strongly correlated to demands and resources than either “cynicism” or “efficacy”, and that the “quantitative demands”, “role clarity” and “role conflict” scales should have the strongest correlations.

Of all kinds of psychometric evaluation, predictive validity (the ability to predict criterion scores that are obtained at a later time) appears to be the most important feature for instruments measuring psychological and psychosocial factors at work. There is generally very little evidence about predictive validity of comprehensive psychosocial hazard instruments (Rick et al., 2001), and to our knowledge, this assumption has not fully been tested for the QPS instrument in a study focusing on validation. One criterion that could be used for predictive validity testing of psychosocial hazards is future sickness absence.

Studies examining psychosocial work factors as predictors of sickness absence have been dominated by studies using the demand-control-support model. In those studies, control at work has been associated with lower sickness absence (Allebeck & Mastekaasa, 2004). Beyond the demand-control model (and its components) other psychosocial factors, such as low organizational justice (Kivimaki, Elovainio, Vahtera & Ferrie, 2003), role conflict (in female samples), hiding emotions (in male samples) (Lund, Labriola, Christensen, Bultmann, Villadsen & Burr, 2005), and poor organizational climate (among blue-collar women) (Vaananen, Kalimo, Toppinen-Tanner et al., 2004) have shown associations with sickness absence. In a study using the QPS_Nordic, perceived lack of an encouraging and supportive culture was the most important work factor in terms of predicting sickness absence among nurses’ aides (Eriksen, Bruusgaard & Knardahl, 2003).

Validation is the joint responsibility of the test developer and the test user, and it is important that instruments are evaluated continuously in the new settings where the test is to be used (American Educational Research Association, American Psychological Association & National Council on Measurement in Education, 2002). Using data from a large-scale study comprising one county council in southern Sweden, the aim of the present study was thus to evaluate the psychometric properties of the QPS_Nordic scales in a public sector sample, focusing on the test criterion relationship (future certified long-tem sickness absence) and concurrent self-rated health. In addition, the study aimed to test reliability, and whether the postulated scales in the QPS_Nordic could be confirmed from associations among items in the present data.

METHODS

RESULTS

DISCUSSION

Measuring workplace stressors by questionnaire is presumably less expensive, and therefore more affordable than other methods, such as observation or interviews. Furthermore, questionnaires with a certain amount of face validity are not too difficult to design. But is a questionnaire capable of reliably and accurately measuring the perception of psychosocial hazards, and is this perception related to adverse outcomes? Among the many workplace stress questionnaires available, very few have been subjected to the detailed psychometric testing that may answer these questions (Rick et al., 2001). The current study, which aimed to test the psychometric properties of one of these questionnaires, the QPS_Nordic, demonstrated a reliability that is well in line with that obtained during its development process (Dallner et al., 2000) and the testing of similar instruments (Lindström et al., 1995; Rick et al., 2001). Lower reliability was found primarily among task-oriented scales (perhaps due to the difficulty of constructing items that are not too specific or too general). Also in line with other findings, the reliability of the scale “learning demands” was low, perhaps reflecting the concept and measurement problems in connection with demand scales, as previously discussed (Kristensen, Bjorner, Christensen & Borg, 2004).

The QPS_Nordic is a comprehensive instrument with 26 scales and 38 single items (single items are not included in this study). The instrument consists of three separate constructed modules of psychological and social phenomena at work, related to the task, the organizational and the individual. CFA was performed separately in the three modules due to the fact that factor analyses during the construction of the instrument were carried out separately in those modules. The test of measurement models, empirically demonstrated that the proposed associations between items and scales showed a good fit, at least for the scales in the organizational and individual modules (since some measurement problems were encountered for certain scales in the task module). On the whole, the findings suggest that the assessed variables are sufficiently robust to allow further testing of validity.

The associations between the QPS scales and the self-reported health scales were similar to the pattern observed during the development of the QPS_Nordic, and with what could be expected from previous studies within the field. In addition, the magnitude of the correlations between the QPS_Nordic scales, reaching 0.20 for general health, 0.30 for mental health and 0.40 for burnout, were in line with what could be expected for the different measures and generally for correlation with indicators for well-being or strain (Semmer, Zapf & Greif, 1996; van Veldhoven, de Jonge, Broersen, Kompier & Meijman, 2002; Zapf et al., 1996). This pattern of associations has generally been interpreted as evidence for the causal link between psychosocial work environment and health, but studies have also presented data suggesting a reversed or reciprocal association between job characteristics and health. For example, in a study by de Lange and co-workers (de Lange, Taris, Kompier, Houtman & Bongers, 2004) the issue of causality was scrutinized; they concluded that although results provide evidence for reciprocal associations between job characteristics and mental health, the most prominent effects were found for job characteristics predicting mental health. The present study was not designed to study causality, but as this study generally confirms previous findings in the field, the scales in the QPS_Nordic may thus be thought of as having the appropriate properties to uncover such associations. This may come as no surprise, given that these new scales mainly evolved from items taken from existent scales previously used in this field of research (Lindström et al, 1997). However, it might be appropriate to discuss a few of these findings below in some detail, in order to pinpoint some specific strengths in this instrument.

The most stringent of all tests of validity is presumably the ability to predict outcome. Since the QPS_Nordic was designed to reflect work variables thought to be related to health, its predictive validity was tested in terms of relation to certified long-term sickness absence. The significant correlations shown between several QPS_Nordic variables and prospective register data for long-term sick leave during the three years after the survey suggest evidence for a relationship to a relevant criterion: future sickness absence. Since we excluded subjects who had already started their long-term sick leave before the survey was initiated, these associations could not be exclusively accounted for by the documented effect of prior absence on future sick leave (Smulders & Nijhuis, 1999).

The social climate scale, was more strongly correlated to sickness absence than scales reflecting the theoretically well-supported concept of job demands. This is entirely in line with recent findings (Godin & Kittel, 2004; Moreau, Valente, Mak et al., 2004) and results in the literature reviewed by Allebeck and Mastekaasa (2004), who only found an association between job demands and sick leave in less than half of the studies reviewed.

Inconsistent findings of associations between demand and absence have been found (Allebeck & Mastekaasa, 2004). Several explanations of the lack of consistent findings have been discussed. For example, if studies have used too short or too long a time lag this may lead to the conclusion that no causal effect exists, or to an underestimation of the impact (Allebeck & Mastekaasa, 2004; Smulders & Nijhuis, 1999; Zapf et al., 1996). Moreover, inconsistent findings may have resulted from the different types of sick leave measurement used (Nielsen, Rugulies, Smith-Hansen, Christensen & Kristensen, 2006) and the fact that the demand scales may have had different psychometric properties (Moreau et al., 2004; Smulders & Nijhuis, 1999).

Interestingly, less studied work-related psychosocial predictors of sickness absence were also correlated with long-term sickness. These associations are in agreement with those reported in recent studies, with regard to role clarity (Vaananen et al., 2004), organizational climate (Eriksen et al., 2003), and predictability (Nielsen et al., 2006).

In a review of predictors of long-term sickness absence for psychiatric illness, Hensing and Wahlstrom (Hensing & Wahlström, 2004) found only one study (Stansfeld, Fuhrer, Head, Ferrie & Shipley, 1997) that demonstrated an association between specified factors at work and long-term sickness absence due to psychiatric problems. In our study, the correlations between the QPS scales and sickness absence for a psychiatric or a musculoskeletal disorder were sometimes somewhat higher than with our general prevalence measure of sickness absence (i.e. prevalence regardless of diagnosis). There are studies indicating the possibility of different mechanisms for work-related strain and well-being (de Jonge, Dollard, Dormann, Le Blanc & Houtman, 2000; van Veldhoven et al., 2002). Not all outcomes are affected in a similar way by a particular job characteristic. Warr (1994) suggests that certain job characteristics (job demands, job control, interpersonal contact) are differently associated with mental health. Initially the presence of those job characteristics has a beneficial effect on employee mental health, but the effect may be harmful when these job characteristics increase beyond a certain required level. To our knowledge, the finding that the different scales in the QPS_Nordic predicted different types of sickness absence (psychiatric and musculoskeletal respectively) is new.

The association is in line with recent prospective studies using the QPS_Nordic. Exposure to role conflict predicted psychological distress in nurses’ aides (Eriksen, Tambs & Knardahl, 2006), and experience of few positive challenges at work emerged as risk factors for new episodes of sick leave due to neck or back pain (Bergström et al., 2007). This suggests that the instrument is sensitive, and also that a comprehensive instrument which measures a wide range of factors may be needed to disclose different facets of the pattern of ill health caused by job stress.

There is an ongoing discussion on how many work characteristics must be included in a model or an instrument to explain employee health and well-being, and if the work characteristics must be tailored to the particular occupation. In a recent study (van Veldhoven, Taris, De Jonge & Broersen, 2005) it was concluded that three out of seven commonly used work characteristics were capable of explaining the link between work characteristics, health and well-being, and that the same patterns of predictors were found in the four branches of industry included in the study. However, in interventions and organizational development work, a wider range of work characteristics may be needed to capture modern forms of work and employment (Parker, Wall & Cordery, 2001), and these may need to be specific enough to generate useful information about potential risk factors so that more fine-grained changes can be made (Morgeson & Humphrey, 2006). However, it is important that the factors are general enough to be applicable in different settings, and that normative values can be used for benchmarking (Faragher, Cooper & Cartwright, 2004). Jones and co-workers (Jones, Bright, Searle & Cooper, 1998) concluded that: “Knowledge of a risk factor can only be useful if it is sufficiently specified to inform us about appropriate prevention actions” (p. 235).

Different measures of absence have been used in studies on the association between psychosocial work factors and sickness absence (Hensing, 2004). Normally, sick leave for more than seven days is regarded as “long”. Register data for long-term sick leave (>90 days) were used in this study. The use of long-term sick leave as a criterion is justified by findings in the literature reviewed by Allebeck and Mastekaasa (2004), where a tendency was found for psychosocial work factors to have a greater effect on long rather than short duration of absence. In a recent study (Nielsen et al., 2006), psychosocial factors generally had a stronger effect on long-term (>10 days) than on short-term sickness absence. In addition, specific work factors had an effect on short and long spells.

In the research literature, associations between psychosocial work factors and sickness absence are most often presented as odds ratios or relative risk, rather than correlation coefficients. We used correlations to avoid arbitrary dichotomization of the scale scores (MacCallum, Zhang, Preacher & Rucker, 2002), which can lead to varying results depending on where the cut-off point is placed. For example, the correlation between role clarity and long-term sickness absence due to mental disorder was −0.149. Dichotomizing the scores at two different cut-off points, at the median (4.3), or at the score of 3 on the 5-graded “role clarity” scale, yields rather different odds ratios (2.27 and 1.59 respectively).

The concurrent and predictive validity testing in this present study was performed scale-wise, and the relative importance of certain scales over others in predicting a criterion was not scrutinized. Due to the fact that the users of QPS are recommended to pick and use scales separately, and since users are not guided by the constructors in how scales are assumed to be associated, the bivariate test of concurrent and predictive ability of the scales is perhaps an appropriate first important step in the validation of the QPS. In further validation tests, specific theories are needed to design focal tests, where specific scales are selected in order to test the relative importance of scales in predicting certain criteria. Such studies should be strictly guided by theory, and as we are only in the early stages of creating a validity argument for the QPS scales, future studies are needed before such focused studies can be performed. However, the limitation of the strategy used in this study should be remembered when interpreting the associations for a single scale on several outcomes. Thus, evidence for discriminate validity is still to be addressed.

Finally, the possible influence of missing data on our results needs to be discussed. In general, internal dropout was rather low across all items, although the cumulative impact of internal dropout became quite extensive when only data from those who completed the full instrument were used in the psychometric analyses. However, when the psychometric analyses were computed on data including cases with imputed data, the results did not change and the conclusions drawn remained intact. This result is in line with findings from simulation studies where the impact of missing data on measurement models has been evaluated and not found to be a serious threat (Collins et al., 2001). The possible bias in our results, reflecting influences attributed to differential response rates across gender and profession, remains unexplored. Given that the QPS scales showed associations to external variables in an expected direction, the possible bias introduced by the differential response rates may mainly influence the magnitude of these associations (an unknown amount and direction). If non-response interacts with the research question, and individuals with bad health do not agree to participate, a somewhat restricted range of health rating may be expected. The same effect may be expected if individuals with good health choose not to participate. If these two occur at the same time, the restriction of range in health ratings is thought to have an even greater impact. In this study, non-responding was highest among those with perhaps the lowest and highest status (and control) in the health care system. Thus, a tentative hypothesis would suggest that the magnitude of the associations in our study may (if anything) be underestimations of the true associations.

In the critical review of psychological hazard measures performed by Rick and co-workers (2001), only a few comprehensive measures were identified as having sufficient information to provide detailed psychometric evaluation. However, this present study, together with other recent studies (Bergström et al., 2007; Eriksen et al., 2006; Wännström et al., 2008), provides such information for the QPS_Nordic, and the emergent conclusion regarding the quality of the QPS_Nordic is that the psychometric quality is good for the majority of the scales, and the results may even be generalized across different professions (Wännström et al., 2008).

APPENDIX