Open Journal of Psychiatry
Vol.07 No.01(2017), Article ID:72531,13 pages
10.4236/ojpsych.2017.71002

Empowerment according to Persons with Severe Mental Illness: Development of the Netherlands Empowerment List and Its Psychometric Properties

Wilma Boevink1*, Hans Kroon1, Philippe Delespaul2, Jim Van Os2,3

1Trimbos Institute, Department of Reintegration, Utrecht, The Netherlands

2Department of Psychiatry and Psychology, School of Mental Health and Neuroscience, EURON, Maastricht University Medical Centre, Maastricht, The Netherlands

3King’s College London, King’s Health Partners, Department of Psychosis Studies, Institute of Psychiatry, London, UK

Copyright © 2017 by authors and Scientific Research Publishing Inc.

This work is licensed under the Creative Commons Attribution International License (CC BY 4.0).

http://creativecommons.org/licenses/by/4.0/

Received: October 4, 2016; Accepted: December 2, 2016; Published: December 5, 2016

ABSTRACT

Objectives. It is unclear whether and to what extent purportedly empowering practices in mental health care, like rehabilitation programs, recovery-supporting mental health care environments and peer-run services, contribute to the process of empowerment. Several American empowerment questionnaires have been developed in recent years, facilitating the measurement of empowerment outcomes. Given likely major transatlantic cultural differences in a value-sensitive concept such as empowerment, this article describes the development of the Netherlands Empowerment List (NEL) and its psychometric properties. Methods. Patients in Dutch mental health services provided meaning to the empowerment concept from which the NEL was derived. Based on 531 completed questionnaires, analyses in agreement with COSMIN criteria examined aspects of internal consistency, content validity, structural validity, convergent validity, discriminant validity, reproducibility and responsiveness of the NEL. Results. The NEL is a 40-item self-report questionnaire with six subscales: Social support, Professional help, Connectedness, Confidence and purpose, Self-ma- nagement and Caring community. Internal consistency (Cronbach’s alpha = 0.94), aspects of validity, reproducibility (intraclass correlation = 0.79) and responsiveness were good. Correlation with existing scales was the highest for the Mental Health Confidence Scale (r = 0.78) and the lowest for the Boston Empowerment Scale (r = 0.61). Conclusion. The NEL appears to be a suitable instrument to capture the dimension of empowerment in European mental health settings.

Keywords:

Empowerment, Psychotic Disorder, Recovery

1. Introduction

Empowerment is a concept that is used in many sectors, including mental health care [1] [2] . Partly because of this, the concept is defined and interpreted in different ways. Rappaport describes empowerment as a process in which people, organisations or communities are able to influence events and situations that are important for them [3] . Zimmerman and colleagues describe the required ingredients to acquire such influence: access to information, knowledge and skills, the ability to take decisions, individual strength, participation in society and real control [4] . Corrigan notes the importance of hope, shared decision making, community approaches and stigma [5] . Leamey and colleagues defined empowerment as a core component of the CHIME recovery framework, together with connectedness, hope and optimism about the future, identity and meaning in life [6] . A distinction can be made between empowerment at the individual and the collective level. At the individual level, empowerment is about processes in which the person rediscovers his identity and self-esteem and “takes his life in his own hands”. At the collective level, it is about the contribution by people with lived experience to the organisation and practice of mental health care and society. The concept of empowerment and the activities it embraces have been receiving growing attention in mental health care in several European countries in recent years, including the Netherlands. The focus is on the development of so-called best practice, particularly those that are co-set up and/or co-controlled by people with lived experience [7] .

A large number of instruments to measure empowerment exist, among which a number is specifically in the area of mental health [8] (Table 1). Different instruments assess different aspects of empowerment; in fact empowerment appears to be defined so broadly that it is not possible to show convergent validity across different instruments [9] . Nevertheless, the overarching structure underlying all empowerment questionnaires appears to include domains of: patient states, experiences and capacities; patient

Table 1. Existing empowerment-related scales demonstrating wide range of conceptual approaches.

actions and behaviours; patient self-determination within the healthcare relationship and patient skills development [8] . In mental health settings, the most frequently used instrument is the Boston Empowerment Scale, which was developed in the USA [10] [11] . As cultural factors are likely to play a role in how the concept of empowerment is framed and experienced in different parts of the world, psychometric properties of assessment scales may vary across countries and cultures. For example, research in European samples suggests relatively poor psychometric properties for the Boston Empowerment Scale and relatively good properties for the Mental Health Confidence Scale [12] , which is designed to measure self-efficacy and therefore has a focus on intrapersonal aspects of empowerment [9] . Another issue is that the Boston Empowerment Scale does not tap into the domain of interactions between professionals and patients within the mental health system, and the extent of opportunities for (guided) self- management. This is important because many users, in their commitment to more self- determination, are also actively trying to get a better grip on their own mental health problems. In a pilot study, we examined which aspects people with severe mental illness related to the concept of empowerment [13] . Based on the findings of this exploratory study, a Dutch empowerment questionnaire was developed. In this article, the development of the Netherlands Empowerment List (NEL) and the results of the reliability and validity analyses are described. To conform with COSMIN criteria [14] , we describe internal consistency, reproducibility, responsiveness, construct validity (convergent and discriminant validity), content validity and structural validity.

2. Methods

2.1. Sample 1: Development of NEL and Content Validity

The items in the NEL were based on the results of a conceptual study of empowerment, using Concept Mapping, in which 56 people with psychiatric diagnoses of severe mental illness according to DSM-IV criteria mapped out the concept of empowerment [13] . Eight domains were identified based on a total of 96 statements. The domains were: 1) Emerging sense of self, 2) Recovery work, 3) Connectedness, 4) Basic conditions, 5) Social support, 6) A caring community, 7) Social security, 8) Adequate help.

Forty-five of the 96 statements were converted into questionnaire items. The selection was based on high indicated priority and low standard deviation: for each quadrant of the Concept Map we selected the items with the highest priority and a standard deviation of <2. We also screened for overlap. Finally, we assessed whether the items covered the tenor of the quadrant of origin. The items were cast in the form of concrete and unambiguous statements.

Respondents indicate on a five-point scale the extent to which they agree or disagree with the statements (“strongly disagree” [1] , “disagree” [2] , “neither agree nor disagree” [3] , “agree” [4] , “strongly agree” [5] ). Four items relate to mental health care services; these items have a sixth response category: “not applicable”.

Six peers from the Dutch psychiatric user movement evaluated the first version of the questionnaire. In addition, a supervisory committee of peers with lived experience assessed the questionnaire in particular in relation to its face validity. Finally, 33 current and former users of mental health services completed the draft questionnaire. Based on the outcome of this first test, we reformulated several items and reduced the total number of items to 42.

After completion of the test, the NEL was sent to 823 current and former users of Dutch mental health care services aged 18 - 65 years, distributed throughout the Netherlands (Table 2). We attempted to reach a representative group of current and former users, from people receiving long-term intensive care, to people who currently make little or no use of mental health care services, to people (self) employed as peer workers. The questionnaires were distributed by professionals working at mental health institutions (n = 364), staff of user-run projects (n = 224) and rehabilitation programmes (n = 235). In addition to these, a few respondents received a questionnaire directly from the researchers (n = 9). A total of 368 completed questionnaires were completed and returned by reply-paid envelope, representing a response rate of 44%.

2.2. Sample 2: RCT Sample

In addition to sample 1, data from a recently published randomised controlled trial with data collected at baseline and two follow-up points (one year and two years), were used to examine the NEL [15] . The sample was divided in a group of “early starters” (sample 2a) and a group of ‘late starters’ (sample 2b). The NEL had been collected at year 1 and year 2, allowing an analysis of the effect of TREE on NEL outcomes in those who started with TREE after year 1 [15] . The RCT examined, over a two-year period, the impact of a user-developed and user-run recovery program (Toward Recovery, Empowerment and Experiential Expertise - TREE) on outcomes in individuals with severe mental illness, as add-on to care as usual. A randomised wait-list controlled design of TREE (n = 80), added to care as usual (CAU; consisting of professional case management), versus CAU only (n = 83), was implemented in patients with severe mental illness. Follow-up was at 12 and at 24 months after inclusion [15] . The data pertaining to the RCT sample were used to examine NEL reproducibility (comparing repeated

Table 2. Sample characteristics.

Sample 2a―RCT sample group of “early starters”. Sample 2b―RCT sample group of “late starters”. #Higher educational level: higher professional/university; Lower educational level: no formal education, special or primary education, junior secondary vocational education.

measures of NEL after the start of the intervention in the RCT) and responsiveness (change in NEL after treatment). In addition, convergent validity with both the Boston Empowerment Scale [10] [11] and the Mental Health Confidence Scale [12] was analysed, hypothesizing greater convergent validity compared to the Boston Empowerment Scale, given previous research suggesting relatively poor psychometric properties for the Boston Empowerment Scale and relatively good properties for the Mental Health Confidence Scale [9] ). Finally, discriminant validity was assessed against the number of met and unmet needs as assessed by the Camberwell Assessment of Need (CAN) [16] and the Community Assessment of Psychic Experiences (CAPE) total score [17] [18] . The CAN is designed to discuss and rate 22 different needs covering all aspects of an individual’s life and mental wellbeing including accommodation, daytime activities, psychotic symptoms, childcare, money, psychological distress, physical health and relationships. Each of the 22 needs can be rated as “met” or “unmet” depending on whether the person is receiving help in that area, yielding scores of the number of met and unmet needs. The CAPE is a 42-item self-report instrument that transdiagnostically captures psychotic (20 items), depressive (8 items) and negative symptoms (14 items) [19] . Items are scored on a scale from 1 to 4, and can be added to form a total score.

2.3. Instruments

In addition to the NEL, data were collected on several other outcomes. The Boston Empowerment Scale [10] was translated to the Dutch “Making Decisions Scale” [9] . The 28-item scale consists of five factors: self-efficacy and self-esteem, power and powerlessness, community activism, righteous anger, and optimism toward and control over the future. Items are true or false statements. Possible scores range from 1 to 4, with higher scores indicating more empowerment [10] . Quality of life was assessed with the Maastricht Brief Quality of Life Scale [20] . Information about mental health was provided by the Mental Health Index (MHI-5), comprising five items from the SF-36 [21] and items on care use and use of medication. Finally, we collected socio-demo- graphic information and asked the respondents to classify their mental health problems, giving them the options of “psychotic disorder/schizophrenia”, “personality disorder”, “anxiety disorder” and “mood disorder”.

2.4. Ethics

The standing medical ethical committee approved the study. The study was conducted according to the principles expressed in the Declaration of Helsinki. All participants provided written informed consent. No incentives were offered for participation.

2.5. Analysis

Analyses were carried out in SPSS [22] and STATA, version 14 [23] . In order to examine the structural validity, a principal component analysis was carried out with listwise deletion. Principal components were selected based on the screeplot as well as on interpretability and face validity in relation to the concept mapping exercise of the pilot study [13] . Given the fact that the items on mental health services and care were missing for the sizeable proportion of participants who did not receive care, analyses were repeated without these items, in order to examine stability of the principal component structure without these items. On the basis of the principal component structure of the principal component analysis, scale scores were calculated as the weighted average of the items with high loadings (>0.4) on that particular scale. The missing data rule for the scale score analysis was that if more than 20% of the items on a particular scale were missing, the scale score was not computed.

Internal consistency of the total questionnaire as well as the subscales was determined by calculating Cronbach’s alpha. Two items were dropped due to ambiguity and low item-rest correlations (<0.40), leaving 40 items in the final instrument. Convergent validity was evaluated by calculating the Pearson correlations between the scores on the NEL on the one hand, and the scores on the Boston Empowerment Scale (both sample1 and sample 2) and the Mental Health Confidence Scale (sample 2) on the other, hypothesizing that the correlation between the NEL and the Mental health Confidence Scale would be high, and significantly higher than the correlation with the Boston Empowerment Scale. This latter hypothesis was examined in sample 2, using the MVREG (multivariate regression) procedure in STATA. Discriminant validity was examined using the Quality of Life questionnaire, the Mental Health Index, the CAPE total score and the CAN total number of met and unmet needs, hypothesizing that Pearson correlations between NEL and these scales would be low to moderate. Reproducibility and responsiveness were analysed in sample 2 as described in detail in an earlier report [15] . A multilevel random regression analysis, adjusted for location, was conducted using the XTREG routine in STATA, in order to take into account clustering of observations (level 1) within persons (level 2). In the XTREG multilevel random intercept regression model, the dependent variable was the continuous outcome measure and the independent variable was condition (TREE or CAU). Effect sizes were expressed as the regression coefficient B.

3. Results

3.1. Sample 1: Respondent Characteristics

The average age of the 368 respondents was 44 years (in a range from 18 to 82 years). Half was male, 90% was born in The Netherlands, 46% lived alone; 29% resided in a mental health institution or some form of sheltered housing. 10 percent had a regular paid job (full or part-time); 37% were engaged in voluntary work, were homemaker, studying or going to school. Twenty-seven percent of respondents had a higher educational level (higher professional/university); 33% had a lower educational level (no formal education, special or primary education, junior secondary vocational education). More than half reported they had suffered from psychosis and only 19% was never admitted to a psychiatric hospital.

3.2. Sample 2: Respondent Characteristics

Sample characteristics of the 163 participants have been described in detail elsewhere [15] . The mean age was 43.9 (years, SD = 11.4), 52% was male, 91% was born in the Netherlands. Fourteen percent were in (part/full time) paid employment, 50% were engaged in voluntary work/housemaker/study. The proportions with low, medium and high educational level were, respectively, 47%, 34% and 19%. Eighteen percent was living with parents or with children (with or without a partner); 42% lived alone, 34% were living in an institution. Forty percent suffered from a psychosis and only 17% was never admitted to a psychiatric hospital (Table 2).

3.3. Structural Validity (Sample 1)

The principal component analysis of the 40 items produced six components explaining 54% of the variance (N = 235). These components corresponded well with the domains identified in the Concept Map and were labelled Self-management; Social support; Caring community; Connectedness; Confidence and purpose and Professional help. The principal component analysis was repeated without the four items with the highest loadings on the factor Professional help. Sixty participants who did not receive care could now be included in the analysis (N = 295). Results of both analyses were highly comparable. Only two items had a higher loading on another component. Therefore, calculation of the six scale scores was based on the results of the second principal component analysis (Table 3). Correlations between subscales were low to moderate; only a single correlation-between Confidence and purpose and Self-management―had a value ≥0.8 (Table 4).

A short version with 25 items can be composed of three subscales: confidence and purpose, social support and connectedness. Together these three subscales explained 95% (sample 1) and 96% (sample 2) of the variance in the total score.

3.4. Construct Validity

Convergent validity (Table 5)

The correlation between the NEL and the Boston Empowerment Scale was moderate at 0.663 and 0.614 in sample 1 and sample 2, respectively. The correlation with the Mental Health Confidence Scale was high at 0.778, significantly higher than the correlation with the Boston Empowerment Scale (p < 0.001). Entering, in sample 2, the Boston Empowerment Scale and the Mental Health Confidence Scale in the same regression model of the NEL revealed that the association with the Boston Empowerment Scale (b = 0.010, p = 0.176) was entirely reducible to the association with the Mental Health Confidence Scale (b = 0.478, p < 0.001).

Discriminant validity (Table 6)

Pearson correlations were low to moderate for the CAPE total score, CAN unmet needs, CAN met needs, Mental Health Index and Quality of life (Table 6).

3.5. Internal Consistency (Sample 1 and Sample 2)

Cronbach’s alpha was 0.94 (sample 1) and 0.95 (sample 2) for the total scale. The outcomes for the subscales are reported in Table 4 (sample 1), showing good internal consistency.

3.6. Reproducibility (Sample 2)

In the RCT sample, repeated measures of NEL in the treatment group were available for 57 individuals. The NEL intraclass correlation coefficient in this group was 0.79.

Table 3. Results of factor analysis of scale items.

*factors 1 - 5 calculated in sample of n = 295; factor 6 calculated in sample of n = 235.

Table 4. Pearson correlations (r*) between subscales of the Netherlands Empowerment List.

*all correlations in table p-value < 0.001.

Table 5. Convergent validity: Pearson correlations between Netherlands Empowerment List and other empowerment-related scales.

#r = Pearson correlation coefficient. †Sample 1. ‡Sample 2.

Table 6. Discriminant validity: Pearson correlations between Netherlands Empowerment List and other scales.

#r = Pearson correlation coefficient. †Sample 1. ‡Sample 2.

3.7. Responsiveness (Sample 2)

In the RCT sample, the responsiveness of NEL to the TREE intervention was examined in 131 individuals who had been exposed to one or more elements of the TREE intervention in the trial (see [15] for details). Of these, 63 had NEL administered at baseline before the TREE intervention, 127 after one year of TREE treatment, and 57 after two years of TREE treatment. From year 1 to year 2, there was a small but significant effect of TREE treatment over time, similar to the effects on other outcomes reported pre- viously [15] (Table 7).

4. Summary and Discussion

The NEL questionnaire was created exclusively on the basis of contributions by people with mental disabilities from all over the Netherlands suggesting good content validity. It contains 40 statements with which respondents can agree or disagree. Principal com- ponent analysis produced six interpretable subscales: social support, professional help, connectedness, confidence and purpose, self-management and caring community. Internal consistency was good, with the items forming a coherent whole. Discriminant validity was also good. Although there was a degree of association among NEL and constructs of quality of life, mental well-being, needs and psychopathology, these correlations were not so high that they could be described as the same constructs. Convergent validity was, as expected, better for the Mental Health Confidence Scale than for the Boston Empowerment Scale, suggesting the need for a culturally sensitive scale. The NEL showed good reproducibility and was responsive to a user-led intervention.

The NEL was devised before the publication of the CHIME framework [6] , with which it clearly shows a degree of conceptual overlap. The dimensions of the NEL, similar to the CHIME framework, point to the importance of kindling strength and support for self-narrative development, promoting the role of mental health services in building inclusive communities enabling access to peer support as well as providing sanctuaries, and clinical skills that promote self-management.

Conceptually, the scale describes feelings, competencies and actions reflective of personal empowerment (confidence and purpose, self-management, connectedness), and the resources in support of this process (social support, professional help, caring community). The first factor “Confidence and purpose” describes core aspects of personal empowerment: identity, self-esteem, control, willpower and purpose. A short version of 25 items can be composed of this subscale combined with social support and connectedness.

The Netherlands Empowerment List was developed with and for adults with psychiatric disabilities and was psychometrically tested in institutionalized and non-institu- tionalized settings, and in consumer-run initiatives. To date, the scale has been used in

Table 7. Effect of TREE intervention on NEL empowerment in RCT sample (adjusted for site).

CAU = care as usual. # = the NEL was not administered at baseline. * = reference category. ¶ = regression coefficient B of TREE at year 1 and year 2. Between brackets: 95% confidence interval.

a wide range of settings in and outside mental health care in the Netherlands and Belgium, including services for homeless people, youth care, social work, visual impairments, intellectual disabilities, and physical revalidation [24] - [32] .

Study Limitations

The findings should be interpreted in the context of a number of strengths and weaknesses. An advantage is the user-led development of the NEL, thus ensuring content validity. A further strength was adherence to the full set of COSMIN criteria [14] , making NEL the exception in this regard [8] . Weaknesses are the lack of psychopathology interview data and the use of two different samples for the analyses, although the samples are quite comparable clinically and demographically. Future work should include psychopathology interview data and attempt further alignment with the CHIME framework [6] .

Acknowledgements

Made possible by a grant from the Dutch Mental Health Fund (Project No. 20015533).

Conflict of Interests

The authors have no conflict of interests.

Cite this paper

Boevink, W., Kroon, H., Delespaul, P. and Van Os, J. (2017) Empowerment according to Persons with Severe Mental Illness: Development of the Netherlands Empowerment List and Its Psychometric Properties. Open Journal of Psychiatry, 7, 18-30. http://dx.doi.org/10.4236/ojpsych.2017.71002

References

  1. 1. Pulvirenti, M., McMillan, J. and Lawn, S. (2014) Empowerment, Patient Centred Care and Self-Management. Health Expectations, 17, 303-310.
    https://doi.org/10.1111/j.1369-7625.2011.00757.x

  2. 2. Sunkel, C. (2012) Empowerment and Partnership in Mental Health. The Lancet, 379, 201-202.
    https://doi.org/10.1016/s0140-6736(11)61270-1

  3. 3. Rappaport, J. (1987) Terms of Empowerment/Exemplars of Prevention: Toward a Theory for Community Psychology. American Journal of Community Psychology, 15, 121-148.
    https://doi.org/10.1007/BF00919275

  4. 4. Zimmerman, M.A. and Rappaport, J. (1988) Citizen Participation, Perceived Control, and Psychological Empowerment. American Journal of Community Psychology, 16, 725-750.
    https://doi.org/10.1007/BF00930023

  5. 5. Corrigan, P.W. (2002) Empowerment and Serious Mental Illness: Treatment Partnerships and Community Opportunities. Psychiatric Quarterly, 73, 217-228.
    https://doi.org/10.1023/A:1016040805432

  6. 6. Leamy, M., Bird, V., Le Boutillier, C., Williams, J. and Slade, M. (2011) Conceptual Framework for Personal Recovery in Mental Health: Systematic Review and Narrative Synthesis. British Journal of Psychiatry, 199, 445-452.
    https://doi.org/10.1192/bjp.bp.110.083733

  7. 7. Delespaul, P., Milo, M., Schalken, F., Boevink, W., Van Os, J. and Goede, G.G.Z. (2016) Nieuwe Concepten, Aangepaste Taal, Verbeterde Organisatie. Diagnosis Uitgevers, Amsterdam.

  8. 8. Barr, P.J., Scholl, I., Bravo, P., Faber, M.J., Elwyn, G. and McAllister, M. (2015) Assessment of Patient Empowerment—A Systematic Review of Measures. PLoS ONE, 10, e0126553.
    https://doi.org/10.1371/journal.pone.0126553

  9. 9. Castelein, S., van der Gaag, M., Bruggeman, R., van Busschbach, J.T. and Wiersma, D. (2008) Measuring Empowerment among People with Psychotic Disorders: A Comparison of Three Instruments. Psychiatric Services, 59, 1338-1342.
    https://doi.org/10.1176/ps.2008.59.11.1338

  10. 10. Rogers, E.S., Chamberlin, J., Ellison, M.L. and Crean, T. (1997) A Consumer-Constructed Scale to Measure Empowerment among Users of Mental Health Services. Psychiatric Services, 48, 1042-1047.
    https://doi.org/10.1176/ps.48.8.1042

  11. 11. Corrigan, P.W., Faber, D., Rashid, F. and Leary, M. (1999) The Construct Validity of Empowerment among Consumers of Mental Health Services. Schizophrenia Research, 38, 77-84.
    https://doi.org/10.1016/S0920-9964(98)00180-7

  12. 12. Carpinello, S.E., Knight, E.L., Markowitz, F.E. and Pease, E.A. (2000) The Development of the Mental Health Confidence Scale: A Measure of Self-Efficacy in Individuals Diagnosed with Mental Disorders. Psychiatric Rehabilitation Journal, 23, 236-243.
    https://doi.org/10.1037/h0095162

  13. 13. Boevink, W., Kroon, H. and Giesen, F. (2008) Controle Herwinnen over Het Eigen Leven. Een conceptuele verkenning van het begrip “empowerment” van mensen met een psychische handicap. Rehabilitatie, 17, 16-26.

  14. 14. Mokkink, L.B., Terwee, C.B., Patrick, D.L., Alonso, J., Stratford, P.W., Knol, D.L., et al. (2010) The COSMIN Study Reached International Consensus on Taxonomy, Terminology, and Definitions of Measurement Properties for Health-Related Patient-Reported Outcomes. Journal of Clinical Epidemiology, 63, 737-745.
    https://doi.org/10.1016/j.jclinepi.2010.02.006

  15. 15. Boevink, W., Kroon, H., van Vugt, M., Delespaul, P. and van Os, J. (2016) A User-Developed, User Run Recovery Programme for People with Severe Mental Illness: A Randomised Control Trial. Psychosis, 8, 287-300.

  16. 16. Phelan, M., Slade, M., Thornicroft, G., Dunn, G., Holloway, F., Wykes, T., et al. (1995) The Camberwell Assessment of Need: The Validity and Reliability of an Instrument to Assess the Needs of People with Severe Mental Illness. British Journal of Psychiatry, 167, 589-595.
    https://doi.org/10.1192/bjp.167.5.589

  17. 17. Konings, M., Bak, M., Hanssen, M., Van Os, J. and Krabbendam, L. (2006) Validity and Reliability of the CAPE: A Self-Report Instrument for the Measurement of Psychotic Experiences in the General Population. Acta Psychiatrica Scandinavica, 114, 55-61.
    https://doi.org/10.1111/j.1600-0447.2005.00741.x

  18. 18. Hanssen, M., Peeters, F., Krabbendam, L., Radstake, S., Verdoux, H. and van Os, J. (2003) How Psychotic Are Individuals with Non-Psychotic Disorders? Social Psychiatry and Psychiatric Epidemiology, 38, 149-154.
    https://doi.org/10.1007/s00127-003-0622-7

  19. 19. Stefanis, N.C., Hanssen, M., Smirnis, N.K., Avramopoulos, D.A., Evdokimidis, I.K., Stefanis, C.N., et al. (2002) Evidence That Three Dimensions of Psychosis Have a Distribution in the General Population. Psychological Medicine, 32, 347-358.
    https://doi.org/10.1017/S0033291701005141

  20. 20. Van Os, J., Drukker, M., Campo, J., Meijer, J., Bak, M. and Delespaul, P. (2006) Validation of Remission Criteria for Schizophrenia. American Journal of Psychiatry, 163, 2000-2002.
    https://doi.org/10.1176/ajp.2006.163.11.2000

  21. 21. Rumpf, H.J., Meyer, C., Hapke, U. and John, U. (2001) Screening for Mental Health: Validity of the MHI-5 Using DSM-IV Axis I Psychiatric Disorders as Gold Standard. Psychiatry Research, 105, 243-253.
    https://doi.org/10.1016/S0165-1781(01)00329-8

  22. 22. IBM Corp (2013) IBM SPSS Statistics for Windows. Version 22.0. IBM Corp Armonk.

  23. 23. Stata Corp (2015) STATA Statistical Software: Release 14. College Station, Texas.

  24. 24. Antens, N. (2013) Onderzoek naar empowerment bij harten vaatpatienten. Cordiaal, 2, 54-58.

  25. 25. Van Busschbach, J.T., Swildens, W., Michon, H., Kroon, H., Koeter, M., Wiersma, D., et al. (2016) Working Effectively towards Rehabilitation Goals: Long-Term Outcome of a Randomised Controlled Trial of the Boston Psychiatric Rehabilitation Approach. Tijdschr Psychiatr, 58, 179-189.

  26. 26. Van der Aa, M.J., van den Broeke, J.R., Stronks, K., Busschers, W.B. and Plochg, T. (2016) Measuring Renewed Expertise for Integrated Care among Health- and Social-Care Professionals: Development and Preliminary Validation of the ICE-Q Questionnaire. Journal of Interprofessional Care, 30, 56-64.
    https://doi.org/10.3109/13561820.2015.1057271

  27. 27. Goossens, P.J., Beentjes, T.A., Knol, S., Salyers, M.P. and de Vries, S.J. (2016) Investigating the Reliability and Validity of the Dutch Versions of the Illness Management and Recovery Scales among Clients with Mental Disorders. Journal of Mental Health, 1-8.
    https://doi.org/10.3109/09638237.2015.1124398

  28. 28. Denig, P., Schuling, J., Haaijer-Ruskamp, F. and Voorham, J. (2014) Effects of a Patient Oriented Decision Aid for Prioritising Treatment Goals in Diabetes: Pragmatic Randomised Controlled Trial. BMJ, 349, g5651.
    https://doi.org/10.1136/bmj.g5651

  29. 29. Tambuyzer, E. and Van Audenhove, C. (2015) Is Perceived Patient Involvement in Mental Health Care Associated with Satisfaction and Empowerment? Health Expect, 18, 516-526.
    https://doi.org/10.1111/hex.12052

  30. 30. Van Gestel-Timmermans, H., Brouwers, E.P., van Assen, M.A. and van Nieuwenhuizen, C. (2012) Effects of a Peer-Run Course on Recovery from Serious Mental Illness: A Randomized Controlled Trial. Psychiatric Services, 63, 54-60.
    https://doi.org/10.1176/appi.ps.201000450

  31. 31. Cleveringa, F.G., Minkman, M.H., Gorter, K.J., van den Donk, M. and Rutten, G.E. (2010) Diabetes Care Protocol: Effects on Patient-Important Outcomes. A Cluster Randomized, Non-Inferiority Trial in Primary Care. Diabetic Medicine, 27, 442-450.
    https://doi.org/10.1111/j.1464-5491.2010.02968.x

  32. 32. De Groot, S., van der Woude, L.H., Niezen, A., Smit, C.A. and Post, M.W. (2010) Evaluation of the Physical Activity Scale for Individuals with Physical Disabilities in People with Spinal Cord Injury. Spinal Cord, 48, 542-547.
    https://doi.org/10.1038/sc.2009.178

  33. 33. Segal, S.P., Silverman, C. and Temkin, T. (1995) Measuring Empowerment in Client-Run Self-Help Agencies. Community Mental Health Journal, 31, 215-227.
    https://doi.org/10.1007/BF02188748

  34. 34. Svedberg, P., Svensson, B., Arvidsson, B. and Hansson, L. (2007) The Construct Validity of a Self-Report Questionnaire Focusing on Health Promotion Interventions in Mental Health Services. Journal of Korean Academy of Psychiatric and Mental Health Nursing, 14, 566-572.
    https://doi.org/10.1111/j.1365-2850.2007.01129.x

  35. 35. Lopez, J.E., Orrell, M., Morgan, L. and Warner, J. (2010) Empowerment in Older Psychiatric Inpatients: Development of the Empowerment Questionnaire for Inpatients (EQuIP). American Journal of Geriatric Psychiatry, 18, 21-32.
    https://doi.org/10.1097/JGP.0b013e3181b2090b

  36. 36. Oades, L.G., Law, J. and Marshall, S.L. (2011) Development of a Consumer Constructed Scale to Evaluate Mental Health Service Provision. Journal of Evaluation in Clinical Practice, 17, 1102-1107.
    https://doi.org/10.1111/j.1365-2753.2010.01474.x

  37. 37. Ortiz, G. and Schacht, L. (2012) Psychometric Evaluation of an Inpatient Consumer Survey Measuring Satisfaction with Psychiatric Care. Patient, 5, 163-173.