Psychology
Vol.09 No.10(2018), Article ID:87338,14 pages
10.4236/psych.2018.910139

Blended e-Health in Cognitive Behavioural Therapy: Usage Intensity, Attitude and Therapeutic Alliance in Clinical Practice

Janneke E. M. Aerts1, Arno van Dam1,2

1Geestelijke Gezondheidszorg Westelijk Noord Brabant, Halsteren, The Netherlands

2Tranzo (Tilburg University), Tilburg, The Netherlands

Copyright © 2018 by authors and Scientific Research Publishing Inc.

This work is licensed under the Creative Commons Attribution International License (CC BY 4.0).

http://creativecommons.org/licenses/by/4.0/

Received: June 1, 2018; Accepted: September 14, 2018; Published: September 17, 2018

ABSTRACT

Background: Though internet based cognitive behaviour therapy is proven to be effective, e-health is applied only in a minority of treatments in specialised mental health care. The low application rates are associated with therapists’ attitudes towards e-health. One of the major concerns is limitations in the therapeutic relation and communication between patient and therapist. Since therapist involvement is important in an effective e-health treatment, negative attitudes towards e-health can be a risk for effective e-health application. Aims: This study aimed to examine the relationship between patients’ e-health usage as well as patients’ attitudes towards e-health and the therapeutic alliance. The possible influence of therapists’ attitudes on patients’ e-health usage is also examined. Method: In an outpatient mental health setting, patients attended a blended treatment program for major depressive disorder and completed questionnaires on attitudes towards e-health and the therapeutic alliance. Therapists completed an alliance and attitude questionnaire as well. Results: Patients with more positive attitudes used the e-health application more intensively. Higher work alliance rates were related to sharing digital homework assignments and seeking contact with their therapist. Also, when treated by therapists with more positive attitudes towards e-health, patients shared more assignments with their therapist. Patients with high symptom rates at start, had more negative attitude rates and tended to use the e-health application less frequently. Conclusions: In a treatment cohesion in which therapists are positive about blended e-health, patients’ attitudes are positively related to e-health usage and therefore a factor of interest in improvement of effective e-health application. Patients with severe symptoms may need extra attention like active support and iteration to promote adherence to the e-health program.

Keywords:

e-Health Attitude, Therapeutic Alliance, Usage Intensity, Depression

1. Introduction

The practice of e-health is available in most domains of healthcare and there is growing evidence that e-health can be effective in the treatment of mental health problems (Andrews, Cuijpers, Craske, McEvoy, & Titov, 2010; Gainsbury & Blaszczynski, 2011; Karyotaki et al., 2015; Spek et al., 2008) . Several studies show that internet-based cognitive behaviour therapy (CBT) and face-to-face CBT for depressive, anxiety and somatic disorders are equally effective (Andersson, Cuijpers, Carlbring, Riper, & Hedman, 2014; Carlbring, Andersson, Cuijpers, Riper, & Hedman-Lagerlöf, 2018; Wagner, Horn, & Maercker, 2014) .

However, many therapists are reluctant to apply e-health interventions (Donovan, Poole, Boyes, Redgate, & March, 2015; Perle, Langsam, & Nierenberg, 2011) . Several authors suggest that this problem is related to therapists’ assumptions involving disadvantages of e-health. Psychologists, when indicating treatment methods, hardly ever prefer online treatment (Mora, Nevid, & Chaplin, 2008; Perle et al., 2011) . The most important concerns by therapists are the limitations in nonverbal communication and problems in establishing a therapeutic alliance. As seen by these therapists, e-health is applicable for specific patients with high motivation, a high self-efficacy and lower levels of hopelessness. However, this view is inconsistent with research findings and there is no evidence that the assumed drawbacks are actual threats for treatment effects.

Although therapists tend to overlook e-health, their involvement is necessary for effective e-health practice, especially when it comes to adherence (Urech et al., 2018) . Adherence problems are a threat for effective e-health application. Suboptimal usage intensity of e-health applications is related to smaller treatment effect (Donkin et al., 2011) . Specific elements in the structure of the online treatment and contact with a therapist can improve adherence and effectivity rates (Kelders, Kok, Ossebaard, & van Gemert-Pijnen, 2012) . Compared with online self-help, e-health with therapist guidance outstands treatment effect (Andersson & Cuijpers, 2009; Spek et al., 2007) . In face-to-face treatment, the therapeutic relationship is a mediating factor for therapy efficacy (Lambert & Barley, 2002) and explains a significant percentage (7% - 17%) of the variance in treatment outcome (Beutler et al., 2004) . The therapeutic relationship seems to be important in e-health too. Establishing a constructive therapeutic alliance in e-health treatment is possible, even without having face-to-face contact (Leibert, Archer, Munson, & York, 2006; Wagner, Brand, Schulz, & Knaevelsrud, 2012) .

For therapists to promote optimal e-health usage, their attitudes towards e-health may be a barrier in successfully implement e-health (Vis et al., 2018) . In comparison, research shows that lower therapists’ attitudes towards a treatment protocol have a negative impact on the effectiveness of the treatment (Wiborg, Knoop, Wensing, & Bleijenberg, 2012) . Therefore, barriers like negative attitudes towards e-health should be taken seriously. It is conceivable that this can negatively influence therapists’ capability to promote patients’ usage intensity. In addition to the therapists’ attitude, research shows also the patients’ attitude is important for e-health usage (Hardiker & Grant, 2011) . Patients’ attitudes towards technology are positively related to the frequency of e-health usage (Chiu & Eysenbach, 2010) . In order to apply e-health successfully, patients’ general e-health attitude should be taken in consideration before indicating e-health (Stallard, Velleman, & Richardson, 2010) . When e-health is prescribed to patients who are less positive about this method, the uptake of the program will presumably be suboptimal.

Both patients’ and therapists’ attitude might be important in e-health treatment, but the relationship of these factors and actual usage is not yet clear for e-health applications applied in blended e-health (a combination of an internet delivered program and face-to-face treatment). Also, little is known about the relationship between e-health usage in blended form and the therapeutic alliance. Since both expected to be important for good e-health practice and uptake, this needs to be clarified. The aim of this study is to examine the relationship between usage intensity of online modules applied as blended e-health and both e-health attitude and the therapeutic alliance. Not only the relation between patients’ attitude and e-health usage intensity, but also the possible relationship between therapists’ attitude and patients’ e-health usage is explored. In consideration of previous research, we expect to find a positive correlation between usage intensity and attitudes towards e-health. In line with face-to-face treatment, we expect the therapeutic relationship to be positively related to e-health usage.

2. Method

2.1. Participants

The study sample consisted of 50 adult patients who were registered for treatment at GGZ Westelijk Noord Brabant, an institution for mental healthcare in the Netherlands. Participants were included when they met the DSM-IV (American Psychiatric Association, 2000) criteria for a major depressive disorder with a current depressive episode, there was no language barrier and participants had internet access at home. Patients with minimal depressive symptoms according to BDI (score 0 - 13) were not included in the study. Participants suffered from moderate to severe depressive symptoms, typically treated in the specialized mental health centers, where comorbid problems are involved. Patients’ descriptive characteristics are presented in Table 1.

Therapists, experienced with CBT for depression, were asked to participate in the study and apply the blended treatment program (described below). In total, nine therapists participated voluntarily.

Table 1. Descriptive characteristics study sample.

2.2. Intervention

The blended treatment program included face-to-face group therapy performed by two cognitive behavioural therapists. The treatment consisted of 12, two-hour weekly sessions and contains psycho-education, relaxation, activation, cognitive therapy and problem solving skills.

Between sessions, patients gained access to an e-health program, consisting of eight sub-modules, matching the group sessions which provides information and includes exercises, registration assignments and cognitive therapy. These assignments could be shared with the therapist. Also, patients were able to communicate with their therapist through a message application. Each patient had one main therapist to contact.

2.3. Design and Procedure

This study was applied in an outpatient mental health setting with patients who attended a blended treatment program for major depressive disorder. Data collection was obtained from two different sources. Patients were asked to complete questionnaires three times. In order to determine the usage intensity, data of login frequency was registered by the e-health application. Also, the number of completed exercises and frequency of therapist contact were registered (Figure 1).

Prior to treatment start, patients were informed about the study through written and verbal information. If they were willing to participate, informed consent was obtained and patients received a baseline assessment to determine symptoms. Eleven patients refused to participate in the study but did participate in group therapy. When participants met the criteria for depressive disorder, they completed questionnaires on e-health attitudes. A questionnaire on treatment motivation was also provided in order to detect possible confounding effects. In order to take possible group differences into account, a group cohesion questionnaire was included as a post treatment measurement. Eight participants did not finish the program, two dropped out on practical reasons, five patients preferred individual treatment and one patient was referred to clinical treatment. There was no reported drop-out due to discontentment about e-health use.

Figure 1. Flow of data collection.

For therapists, prior to treatment they completed an e-health attitude questionnaire. Therapists who participated in the study in more than one treatment group, were asked to complete the attitude questionnaire before every new treatment cycle. For the purpose of measuring the therapeutic alliance with minimalizing confounding therapy effect, patients completed the working alliance questionnaire after the third therapy session. Previous studies reported the predictive validity of the working alliance measured early on in treatment (week 3) on treatment outcome (Wagner, Brand, Schulz, & Knaevelsrud, 2012) .

2.4. Measurements

1) e-health attitude. Attitude on the use of e-health was assessed using an edited questionnaire based on the Attitude Questionnaire introduced by Addis & Krasnow (2000) . This questionnaire is designed to measure attitudes on protocolled therapies. The questionnaire was adapted to the use of e-health, both items on familiarity with computer use and ideas on e-health effectivity were added. The patient questionnaire consists of 19 items that are scored on a 5-point Likert-scale. The scale internal consistency is good (α = .89). The same scale has been used to develop a 26 item therapists attitude scale, which consists of three components; e-health negative effects, possibilities of e-health and computer competence. Reliability scores of the scale and subscales are good (α between .83 and .89) (Aerts & van Dam, 2015) .

2) Usage intensity. Usage intensity is operationalized with login frequency as a main variable. Number of completed exercises and frequency of contact with the therapist was registered for detailed analyses.

3) Work Alliance Inventory, WAI-12. An assessment of the therapeutic alliance was obtained using the Dutch, shortened version of the Work Alliance Inventory, WAI-12 (Horvath & Greenberg, 1989; Vertommen & Vervaeke, 1990) . The questionnaire is a self-report, 12 items list on emotional bond, agreement on therapy goals and therapy tasks (Stinckens, Ulburghs, & Claes, 2009) . Both reliability and convergent validity are satisfactory (Munder, Wilmers, Leonhart, Linster, & Barth, 2010) . For this study, patients’ rating of the work alliance was obtained for the therapist couple.

4) Beck Depression Inventory (BDI) (Beck et al., 1961) . This self-report inventory measures the symptoms of a depressive disorder according to the DSM-IV present the past week. Both internal consistency and construct validity are good (van der Does, 2002) .

5) Symptom Checklist 90 (Scl-90). The presence and severity of psychopathology was assessed with the Dutch version of the Symptom Checklist 90 (Scl-90). The validity as well as the reliability of the list is good (Arrindell & Ettema, 1986) .

6) Therapy Motivation Scales (TMS-f). Treatment motivation was measured using one subscale of the Therapy Motivation Scales for forensic patients (TMS-f) (Drieschner & Boomsma, 2008) , the Motivation to Engage in Treatment (MET) scale. Considering the reference to a specific forensic problem, one item was omitted. In order to obtain an acceptable reliability rate in our study, two more items were deleted (α = .74).

7) Group cohesion. An assessment of the group cohesion was obtained by the Group Cohesion Questionnaire-23 (GCQ-23) (Trijsburg et al., 2004) . Reliability statistics are satisfactory (α = .87).

2.5. Statistical Analysis

All analyses were performed using SPSS statistics 22 for Windows. Statistical significance was assumed at p < .05.

Before hypothesis testing, some tests are performed to check on possible confounding effects. Patients with more severe or chronic problems may have more adherence problems (Donkin & Glozier, 2012) . A Kruskal-Wallis test was performed to check if the number of depressive episodes experienced was related to usage intensity. To rule out the possibility that differences in therapist couples might be a confounding factor on usage intensity, a Kruskal-Wallis test was performed. A Spearman’s correlation test was performed on group cohesion with usage intensity, in order to invest if those two factors are related and may be confounding in the relation between e-health attitude and usage intensity.

In order to investigate whether drop outs differ from patients who completed the treatment on symptom degree at start, their motivation, e-health attitude and work alliance, t-tests were performed.

To define correlations between usage intensity and symptoms at start, attitude and work alliance, Spearman correlation tests were executed. Partial correlation tests were performed on attitude and usage intensity, in order to control for symptom degree and motivation. Both factors are hypothesized to be related to usage intensity. To test the correlation between of therapists’ e-health attitude and usage intensity, the average score of both therapists was obtained to execute Spearman correlation tests. Partial correlation tests were performed on work alliance with sharing assignments, controlling for number of completed exercises.

Spearman’s correlation tests were performed on usage intensity and both symptom inventories (Scl-90 and BDI). A partial correlation test was used to control for effects of attitude on the relationship between symptom degree and usage intensity.

It should be noted that for the usage intensity variables, the assumption of normality is violated. Because transformation was not possible, non-parametric tests were performed when available.

3. Results

An overview of the mean outcome measures, standard deviations, minimum and maximum scores for both dependent and independent variables are presented in Table 2. At start patients reported moderate to severe (depression) symptoms. The average BDI score indicates that patients suffer from severe depressive disorders. There is more variation between patients’ attitude than in therapists’ attitude. The scores of these groups are not mutually comparable. There is a high variance in e-health usage intensity. During treatment period, 44% of the participants contacted their therapist by sharing digital homework assignments in the e-health portal more than 20 times.

First, analyses on confounding effects were performed. No correlations were found between usage intensity and the number of experienced depressive episodes (χ2(3) = 5.75, p > .05) or between different therapist couples (χ2(3) = 4.78, p > .05). Also, no significant correlation between usage intensity and group cohesion (r = −.02, p > .05) was found.

Drop outs and patients who completed the program did not differ on variables measured at start and early on in treatment. They did not differ significantly on symptom degree t(48) = 1.91, p > .05, motivation (t(48) = −.67, p > .05) and e-health attitude (t(48) = −.51, p > .05).

Further analysis showed that patients’ e-health attitude is correlated with login frequency (r = .28, p < .05) (Table 3). There is a positive, partial correlation between patients’ e-health attitude and usage intensity, controlling for symptoms and motivation (r = .27, p < .05). Patients with a more positive attitude towards e-health, used the online treatment module more frequently.

Table 2. Minimum and maximum scores, means and missing data of both dependent and independent variables.

Table 3. Correlation analyses (Spearmen’s rho) of symptom rate and attitude with e-health usage intensity.

*Significant at p < .05; **Significant at p < .01; 1controlled for number of completed exercises.

Although no association was found between therapists’ attitude and patients’ usage intensity, there is a positive relation between therapists’ attitude and the frequency of sharing assignments (r = .35, p < .01).

There is no significant correlation between patients’ assessed work alliance and login frequency (r = .07, p > .05). However, there is a positive correlation between the therapeutic relationship and the frequency of contacting the therapist, controlling for number of completed exercises (r = .33, p < .05).

Patients’ attitudes are negatively correlated to the overall level of psychopathology (r = −.39, p < .01). Although there is no significant correlation between the degree of depression symptoms and usage frequency (r = −.23, p > .05), the overall level of psychopathology and usage frequency are negatively correlated (r = −.28, p < .05) (Table 3).

4. Discussion

The aim of the present study was to examine the relationship between attitudes towards e-health, the therapeutic alliance in blended e-health treatment and usage intensity of online modules. This study identified a positive relation between patients’ e-health attitude and usage intensity. Patients with a more positive attitude tend to use the application more frequently. Therapists’ attitudes are partially related to patient usage. When a therapist has a positive attitude towards e-health, patients are inclined to seek more digital contact with their therapist. The expectation that the alliance quality is positively associated with e-health usage is also partially confirmed. Patients with higher alliance rates, share a higher percentage of their digital homework assignments.

These findings generate some interesting perspectives on e-health usage intensity in a blended treatment program. In line with earlier findings on stand-alone e-health usage, a positive relationship between patients’ attitudes and e-health usage appeared to be present in blended e-health. Although this study did not found causal relations, the results may suggest that patients’ attitudes can be a factor of interest in stimulating intervention exposure by increasing usage frequency. With regard to the therapists’ attitude, there is no relation with usage frequency in general, but there appeared to be a relationship with the frequency by which patients contacted their therapist. Therapists with a positive e-health attitude may stimulate patients treatment involvement. Further work needs to be done to establish whether this correlation is indeed consistent. Possibly, patients feel more invited to seek contact and share their assignments with these therapists.

Our findings are in line with a study of El Alaoui et al. (2015) which showed that patients’ beliefs and expectations are important for adherence in e-health, while the influence of the therapist is less distinct. Although a lot of researches focus on therapist variables, the present study raises the idea that patients variables should get more attention. Also, these results indicate that attention to patients’ attitude is a part of e-health application.

The relationship between the therapeutic alliance and e-health usage in blended treatment seems to be smaller compared to attitude. Like with therapists’ attitude, there is a correlation with sharing homework assignments. In a positive therapeutic alliance, patients seek more contact with their therapist and shares more assignments. This may reflect a higher treatment engagement, as described by Kelders, Bohlmeijer, & van Gemert-Pijnen (2013) . Although this seems preferable in the treatment course, it is not clear if actively seeking contact with the therapist contributes to treatment effect. Earlier studies on this topic showed positive treatment effects of guided self-help on symptom reduction, but contact frequency and support time were not correlated to this reduction (Grist & Cavanagh, 2013; Kelders, Bohlmeijer, & van Gemert-Pijnen, 2013; Tummers, Knoop, Van Dam, & Bleijenberg, 2012) . One hypotheses explaining this finding may be that, like involved patients, patients who have difficulties following the program also seek regular contact, but have less preferable outcome (Tummers et al., 2012) .

In line with previous research, this study suggests that patients with severe symptoms of depression experience more problems with adherence (Perle et al., 2011; Donkin & Glozier, 2012) . Notably, in our study this appeared to be related to e-health attitude. Patients with severe symptoms have a more negative attitude towards e-health which is related to a less frequent e-health usage. Since recent studies showed that e-health interventions can be suitable in treatment of severe depressive symptoms (Cuijpers & Riper, 2014; Meyer et al., 2015) , our findings suggest that this patient group may need extra attention. It may be valuable to evaluate if active support and iteration of psycho-education is effective in promoting adherence for patients with severe symptoms.

Overall, findings in this study show a positive relation between patients’ attitude towards e-health and e-health usage intensity. This may suggest that improving patients’ attitude may be profitable in order to promote efficient e-health use. Previous research showed it is possible to influence attitudes toward e-health (Ebert et al., 2015; Mitchell & Gordon, 2007) . Positive effects of a short intervention to improve acceptance of e-health for patient with depressive symptoms were reported. After displaying a video with an expert and patient providing information about internet based therapy, e-health acceptance rates increased (Ebert et al., 2015) .

This study has several methodological limitations. First, due to the small sample size, the power of the study is limited and results should be interpreted with caution. Second, the sample was relatively low educated. Though it is not yet clear which mediating factors are of significant importance, education level may affect e-health usage and the ability to benefit from it. Third, it is not clear if in blended treatment, WAI recordings are applicable in the context of an e-health program. It may be primary related to the face-to-face treatment. Jasper et al. (2014) noted differences in credibility of different aspect of the therapeutic relationship between face-to-face treatment and e-health. For the fist, “bond” seems to be more important and for the latter “tasks” seem to be prominent. Finally, the therapists in our study were practically consentient in their attitude, so possible effects were difficult to detect. The therapists in this study had positive attitudes towards e-health in comparison with a general group of therapists (Aerts & van Dam, 2015) . Therefore, findings in this study are established in a treatment climate with therapists whom were positive about blended e-health. To explore effects of therapists’ attitude, a more diverse therapist sample should be included.

Taken together, some recommendations towards future research arise from this study. Firstly, for a better understanding of the contribution of therapists’ attitudes, the relationship between therapists’ attitude and therapists’ attention for e-health should be investigated. As an additive on attitude and self-rated usage, it may be interesting to use log-data of therapists’ e-health usage and rate face-to-face contacts on e-health attention and possible negative signals about e-health. Also, in order to get a comprehensive view on the contribution of the therapist’s attitude, a more diverse therapist group should be included.

This study on e-health usage intensity provides some insight in the correlation between e-health attitudes and patients’ e-health usage. It has identified patients’ e-health attitude as a factor related to e-health usage intensity, when applied by therapists with positive attitudes on e-health applications. Hence, we suggest keeping therapists’ attitudes in mind when implementing e-health, in order to enable patients to get access to qualitative e-health treatment and effective support by using it. Also, patients with more severe symptoms may need extra attention in order to promote adherence. Since adherence is important in efficient e-health application, further work needs to be done to invest if improving patient’s attitude before start of treatment is a good option in enhancing adherence. This may be credible, especially in patients with severe disorders. Therefore implementing and evaluating interventions on attitude could be the next step in improving e-health applications.

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

Cite this paper

Aerts, J. E. M., & van Dam, A. (2018). Blended e-Health in Cognitive Behavioural Therapy: Usage Intensity, Attitude and Therapeutic Alliance in Clinical Practice. Psychology, 9, 2422-2435. https://doi.org/10.4236/psych.2018.910139

References

  1. 1. Addis, M. E., & Krasnow, A. D. (2000). A National Survey of Practicing Psychologists’ Attitudes toward Psychotherapy Treatment Manuals. Journal of Consulting and Clinical Psychology, 68, 331-339. https://doi.org/10.1037/0022-006X.68.2.331 [Paper reference 1]

  2. 2. Aerts, J. E. M., & van Dam, A. (2015). E-health Attitudelijst; Wat Drijft de Behandelaar? [E-Health Attitude Questionnaire: What Drives the Health Care Professional?] Psychopraktijk, 7, 26-30. https://doi.org/10.1007/s13170-015-0090-2 [Paper reference 2]

  3. 3. American Psychiatric Association (2000). Diagnostic and Statistical Manual of mental disorders: DSM-IV-TR. Washington DC: American Psychiatric Pub. [Paper reference 1]

  4. 4. Andersson, G., & Cuijpers, P. (2009). Internet-Based and Other Computerized Psychological Treatments for Adult Depression: A Meta-Analysis. Cognitive Behavior Therapy, 38, 196-205. https://doi.org/10.1080/16506070903318960 [Paper reference 1]

  5. 5. Andersson, G., Cuijpers, P., Carlbring, P., Riper, H., & Hedman, E. (2014). Guided Internet-Based vs. Face-to-Face Cognitive Behavior Therapy for Psychiatric and Somatic Disorders: A Systematic Review and Meta-Analysis. World Psychiatry, 13, 288-295. https://doi.org/10.1002/wps.20151 [Paper reference 1]

  6. 6. Andrews, G., Cuijpers, P., Craske, M. G., McEvoy, P., & Titov, N. (2010). Computer Therapy for the Anxiety and Depressive Disorders Is Effective, Acceptable and Practical Health Care: A Meta-Analysis. PLoS One, 5, el13196. https://doi.org/10.1371/journal.pone.0013196 [Paper reference 1]

  7. 7. Arrindell, W. A., & Ettema, J. H. M. (1986). SCL-90: Handleiding bij een Multidimensionele Psychopathologie-Indicator. Lisse: Swets & Zeitlinger. [Paper reference 1]

  8. 8. Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An Inventory for Measuring Depression. Archives of General Psychiatry, 4, 561-571. https://doi.org/10.1001/archpsyc.1961.01710120031004 [Paper reference 1]

  9. 9. Beutler, L. E., Malik, M., Alimohamed, S., Harwood, T. M., Talebi, H., Noble, S., & Wong, E. (2004).Therapist Variables. In M. J. Lambert (Eds.), Bergin and Garfield’s Handbook of Psychotherapy and Behaviour Change (5th ed., pp. 227-306). New York: Wiley. [Paper reference 1]

  10. 10. Carlbring, P., Andersson, G., Cuijpers, P., Riper, H., & Hedman-Lagerlöf, E. (2018). Internet-Based vs. Face-to-Face Cognitive Behavior Therapy for Psychiatric and Somatic Disorders: An Updated Systematic Review and Meta-Analysis. Cognitive Behaviour Therapy, 47, 1-18. https://doi.org/10.1080/16506073.2017.1401115 [Paper reference 1]

  11. 11. Chiu, T. M., & Eysenbach, G. (2010). Stages of Use: Consideration, Initiation, Utilization, and Outcomes of an Internet-Mediated Intervention. BMC Medical Informatics and Decision Making, 10, 73. https://doi.org/10.1186/1472-6947-10-73 [Paper reference 1]

  12. 12. Cuijpers, P., & Riper, H. (2014). Internet Interventions for Depressive Disorders: An Overview. Revista de Psicopatología y Psicología Clínica, 19, 209-216. https://doi.org/10.5944/rppc.vol.19.num.3.2014.13902 [Paper reference 1]

  13. 13. Donkin, L., & Glozier, N. (2012). Motivators and Motivations to Persist with Online Psychological Interventions: A Qualitative Study of Treatment Completers. Journal of Medical Internet Research, 14, e91. https://doi.org/10.2196/jmir.2100 [Paper reference 3]

  14. 14. Donkin, L., Christensen, H., Naismith, S. L., Neal, B., Hickie, I. B., & Glozier, N. (2011). A Systematic Review of the Impact of Adherence on the Effectiveness of e-Therapies. Journal of Medical Internet Research, 13, e52. https://doi.org/10.2196/jmir.1772

  15. 15. Donovan, C. L., Poole, C., Boyes, N., Redgate, J., & March, S. (2015). Australian Mental Health Worker Attitudes towards cCBT: What Is the Role of Knowledge? Are There Differences? Can We Change Them? Internet Interventions, 2, 372-381. https://doi.org/10.1016/j.invent.2015.09.001 [Paper reference 1]

  16. 16. Drieschner, K. H., & Boomsma, A. (2008). The Treatment Motivation Scales for Forensic Outpatient Treatment (TMS-F): Construction and Psychometric Evaluation. Assessment, 15, 224-241. https://doi.org/10.1177/1073191107311650 [Paper reference 1]

  17. 17. Ebert, D. D., Berking, M., Cuijpers, P., Lehr, D., Pörtner, M., & Baumeister, H. (2015). Increasing the Acceptance of Internet-Based Mental Health Interventions in Primary Care Patients with Depressive Symptoms. A Randomized Controlled Trial. Journal of Affective Disorders, 176, 9-17. https://doi.org/10.1016/j.jad.2015.01.056 [Paper reference 2]

  18. 18. El Alaoui, S., Ljótsson, B., Hedman, E., Kaldo, V., Andersson, E., Rück, C., Andersson, G., & Lindefors, N. (2015). Predictors of Symptomatic Change and Adherence in Internet-Based Cognitive Behaviour Therapy for Social Anxiety Disorder in Routine Psychiatric Care. PLoS ONE, 10, e0124258. [Paper reference 1]

  19. 19. Gainsbury, S., & Blaszczynski, A. (2011). A Systematic Review of Internet-Based Therapy for the Treatment of Addictions. Clinical Psychology Review, 31, 490-498. https://doi.org/10.1016/j.cpr.2010.11.007 [Paper reference 1]

  20. 20. Grist, R., & Cavanagh, K. (2013). Computerised Cognitive Behavioural Therapy for Common Mental Health Disorders, What Works, for Whom under What Circumstances? A Systematic Review and Meta-Analysis. Journal of Contemporary Psychotherapy, 43, 243-251. https://doi.org/10.1007/s10879-013-9243-y [Paper reference 1]

  21. 21. Hardiker, N. R., & Grant, M. J. (2011). Factors That Influence Public Engagement with eHealth: A Literature Review. International Journal of Medical Informatics, 80, 1-12. https://doi.org/10.1016/j.ijmedinf.2010.10.017 [Paper reference 1]

  22. 22. Horvath, A. O., & Greenberg, L. S. (1989). Development and Validation of the Working Alliance Inventory. Journal of Counseling Psychology, 36, 223-233. https://doi.org/10.1037/0022-0167.36.2.223 [Paper reference 1]

  23. 23. Jasper, K., Weise, C., Conrad, I., Andersson, G., Hiller, W., & Kleinstäuber, M. (2014). The Working Alliance in a Randomized Controlled Trial Comparing Internet-Based Self-Help and Face-to-Face Cognitive Behavior Therapy for Chronic Tinnitus. Internet Interventions, 1, 49-57. https://doi.org/10.1016/j.invent.2014.04.002 [Paper reference 1]

  24. 24. Karyotaki, E., Kleiboer, A., Smit, F., Turner, D. T., Pastor, A. M., Andersson, G., Berger, T., Botella, C., Breton., J. M., Carlbring, P., Christensen, H., Graaf, E., Griffiths, K., Donker, T., Farrer, L., Huibers, M. J. H., Lenndin, J., Machinnon, A., Meyer, B., Moritz, S., Riper, H., Spek, V., Vernmark, K., & Cuijpers, P. (2015). Predictors of Treatment Dropout in Self-Guided Web-Based Interventions for Depression: An “Individual Patient Data” Meta-Analysis. Psychological Medicine, 45, 2717-2726. https://doi.org/10.1017/S0033291715000665 [Paper reference 1]

  25. 25. Kelders, S. M., Bohlmeijer, E. T., & van Gemert-Pijnen, J. E. W. C. (2013). Participants, Usage, and Use Patterns of a Web-Based Intervention for the Prevention of Depression within a Randomized Controlled Trial. Journal of Medical Internet Research, 15, e172. https://doi.org/10.2196/jmir.2258 [Paper reference 3]

  26. 26. Kelders, S. M., Kok, R. N., Ossebaard, H. C., & van Gemert-Pijnen, J. E. W. C. (2012). Persuasive System Design Does Matter: A Systematic Review of Adherence to Web-Based Interventions. Journal of Medical Internet Research, 14, e152. https://doi.org/10.2196/jmir.2104

  27. 27. Lambert, M. J., & Barley, D. E. (2002). Research Summary on the Therapeutic Relationship and Psychotherapy Outcome. In J. C. Norcross (Ed.), Psychotherapy Relationships That Work (pp. 17-32). Oxford: Oxford University Press. [Paper reference 1]

  28. 28. Leibert, T., Archer Jr., J., Munson, J., & York, G. (2006). An Exploratory Study of Client Perceptions of Internet Counseling and the Therapeutic Alliance. Journal of Mental Health Counseling, 28, 69-83. https://doi.org/10.17744/mehc.28.1.f0h37djrw89nv6vb [Paper reference 1]

  29. 29. Meyer, B., Bierbrodt, J., Schröder, J., Berger, T., Beevers, C. G., Weiss, M., Jacob, G., Späth, C., Andersson, G., Lutz, W., Hautzinger, M., Löwe, B., Rose, M., Hohagen, F., Caspar, F., Greiner, W., Moritz, S., & Klein, J. P. (2015). Effects of an Internet Intervention (Deprexis) on Severe Depression Symptoms: Randomized Controlled Trial. Internet Interventions, 2, 48-59. https://doi.org/10.1016/j.invent.2014.12.003 [Paper reference 1]

  30. 30. Mitchell, N., & Gordon, P. K. (2007). Attitudes towards Computerized CBT for Depression amongst a Student Population. Behavioural and Cognitive Psychotherapy, 35, 421-430. https://doi.org/10.1017/S1352465807003700 [Paper reference 1]

  31. 31. Mora, L., Nevid, J., & Chaplin, W. (2008). Psychologist Treatment Recommendations for Internet-Based Therapeutic Interventions. Computers in Human Behaviour, 24, 3052-3062. https://doi.org/10.1016/j.chb.2008.05.011 [Paper reference 1]

  32. 32. Munder, T., Wilmers, F., Leonhart, R., Linster, H. W., & Barth, J. (2010). Working Alliance Inventory-Short Revised (WAI-SR): Psychometric Properties in Outpatients and Inpatients. Clinical Psychology & Psychotherapy, 17, 231-239. [Paper reference 1]

  33. 33. Perle, J. G., Langsam, L. C., & Nierenberg, B. (2011). Controversy Clarified: An Updated Review of Clinical Psychology and Tele-Health. Clinical Psychology Review, 31, 1247-1258. https://doi.org/10.1016/j.cpr.2011.08.003 [Paper reference 3]

  34. 34. Spek, V., Cuijpers, P., Nyklícek, I., Riper, H., Keyzer, J., & Pop, V. (2007). Internet-Based Cognitive Behaviour Therapy for Symptoms of Depression and Anxiety: A Meta-Analysis. Psychological Medicine, 37, 319-328. https://doi.org/10.1017/S0033291706008944 [Paper reference 1]

  35. 35. Spek, V., Cuijpers, P., Nyklícek, I., Smits, N., Riper, H., Keyzer, J., & Pop, V. (2008). One-Year Follow-up Results of a Randomized Controlled Clinical Trial on Internet-Based Cognitive Behavioural Therapy for Subthreshold Depression in People over 50 Years. Psychological Medicine, 38, 635-639. https://doi.org/10.1017/S0033291707002590 [Paper reference 1]

  36. 36. Stallard, P., Velleman, S., & Richardson, T. (2010). Computer Use and Attitudes towards Computerised Therapy amongst Young People and Parents Attending Child and Adolescent Mental Health Services. Child and Adolescent Mental Health, 15, 80-84. https://doi.org/10.1111/j.1475-3588.2009.00540.x [Paper reference 1]

  37. 37. Stinckens, N., Ulburghs, A., & Claes, L. (2009). De Werkalliantie als Sleutelelement in het Therapiegebeuren: Meting met behulp van de WAV-12, de Nederlandstalige Verkorte Versie van de Working Alliance Inventory. Tijdschrift voor Klinische Psychologie, 39, 44-60. [Paper reference 1]

  38. 38. Trijsburg, R. W., Bogaerds, H., Letiche, M., Bidzjel, l., & Duivenvoorden, H. J. (2004). De Ontwikkeling van de Group Cohesion Questionnaire (GCQ). Amsterdam/Rotterdam: Universiteit van Amsterdam/Erasmus Universiteit Rotterdam. [Paper reference 1]

  39. 39. Tummers, M., Knoop, H., Van Dam, A., & Bleijenberg, G. (2012). Implementing a Minimal Intervention for Chronic Fatigue Syndrome in a Mental Health Centre: A Randomized Controlled Trial. Psychological Medicine, 42, 2205-2215. https://doi.org/10.1017/S0033291712000232 [Paper reference 2]

  40. 40. Urech, A., Krieger, T., Möseneder, L., Biaggi, A., Vincent, A., Poppe, C., Meyer, B., Riper, H., & Berger, T. (2018). A Patient Post Hoc Perspective on Advantages and Disadvantages of Blended Cognitive Behaviour Therapy for Depression: A Qualitative Content Analysis. Psychotherapy Research, 1-13. https://doi.org/10.1080/10503307.2018.1430910 [Paper reference 1]

  41. 41. van der Does, A. J. W. (2002). BDI-II-NL. Handleiding. De Nederlandse Versie van de Beck Depression Inventory (2nd ed.). Lisse: Harcourt Test Publishers. [Paper reference 1]

  42. 42. Vertommen, H., & Vervaeke, G. A. C. (1990). Werkalliantievragenlijst (WAV). Vertalingvoorexperimenteelgebruik van de WAI (Horvath and Greenberg: 1986). [Paper reference 1]

  43. 43. Vis, C., Mol, M., Kleiboer, A., Bührmann, L., Finch, T., Smit, J., & Riper, H. (2018). Improving Implementation of eMental Health for Mood Disorders in Routine Practice: Systematic Review of Barriers and Facilitating Factors. JMIR Mental Health, 5, e20. https://doi.org/10.2196/mental.9769 [Paper reference 1]

  44. 44. Wagner, B., Brand, J., Schulz, W., & Knaevelsrud, C. (2012). Online Working Alliance Predicts Treatment Outcome for Posttraumatic Stress Symptoms in Arab War-Traumatized Patients. Depression and Anxiety, 29, 646-651. https://doi.org/10.1002/da.21962 [Paper reference 2]

  45. 45. Wagner, B., Horn, A. B., & Maercker, A. (2014). Internet-Based versus Face-to-Face Cognitive-Behavioral Intervention for Depression: A Randomized Controlled Non-Inferiority Trial. Journal of Affective Disorders, 152, 113-121. https://doi.org/10.1016/j.jad.2013.06.032 [Paper reference 1]

  46. 46. Wiborg, J. F., Knoop, H., Wensing, M., & Bleijenberg, G. (2012). Therapist Effects and the Dissemination of Cognitive Behaviour Therapy for Chronic Fatigue Syndrome in Community-Based Mental Health Care. Behaviour Research and Therapy, 50, 393-396. https://doi.org/10.1016/j.brat.2012.03.002 [Paper reference 1]