- I was recently asked by the Norwegian editor of The Nordic Psychiatrist, Anne Kristine Bergem, if I could write a brief piece (400 words) outlining what I think is the most salient present issue for the field of psychotherapy. There are a host of salient issues pertaining to psychotherapy, but I decided to write a brief commentary entitled “The limited evidence for effectiveness of psychotherapy”. A less provoking and more precise title, however, would probably be “Can we be confident about estimates of treatment effects achieved in psychotherapy trials?” The entire issue of The Nordic Psychiatrist can be accessed here.
It is often assumed that the evidence base for psychotherapy is relatively good and that psychotherapy is relatively effective for a number of mental disorders. For example, Wampold and Imel [1] recently argued that we know for sure that psychotherapy is remarkably effective (my italics). However, as I will argue in this brief commentary, the evidence for the effectiveness of psychotherapy is not as satisfactory as is often assumed. To the contrary, there is substantial uncertainty regarding the effect estimate of psychotherapy.
In order to attest the effectiveness of any treatment, we need clinical trials of adequate methodological quality. Due to the potential bias introduced in trials of inadequate methodological quality, we need to interpret effect size estimates obtained in these studies with great caution. And it is a rather consistent finding from clinical trials in general, that trials with low or poor methodological quality (and thus high risk of bias) tend to overestimate the effects of any given treatment.
A recently published umbrella review of meta-analyses of psychotherapy randomized controlled trials found that out of a total of 247 identified meta-analyses, only 16 (7%) met methodological criteria necessary for providing what the authors defined as convincing evidence for the effectiveness of psychotherapy [2]. In other words, the majority of meta-analyses of psychotherapy do not provide adequate evidence that psychotherapy has the effects that are often assumed. This does not necessarily indicate that psychotherapy is ineffective, but that there is substantial uncertainty regarding the effect estimates achieved in most studies.
In another informative study, Cuijpers and colleagues [3] investigated how many randomized controlled trials of psychotherapy for depression that met eight rather basic methodological quality criteria. Of 115 clinical trials identified, only 11 trials met all eight quality criteria. Thus, only a minority of clinical trials investigating the effects of psychotherapy for depression has adequate methodological quality. The effect size for the trials not meeting all the quality criteria was 0.75. However, the effect size estimate from the trials meeting the quality criteria was substantially lower (SMD=0.22).
Meta-analyses of psychotherapy assessing methodological quality consistently find that (1) the majority of psychotherapy trials have suboptimal methodological quality making them susceptible to bias, and (2) there is a negative association between the methodological quality and the estimated effect size of treatment, i.e. that higher methodological quality is associated with lower effect size estimates [e.g. 4-11].
A number of different challenges pertaining to psychotherapy clinical trials also contributes to uncertainty regarding the effect size estimates; publication bias [12, 13], flexibility in the collection and analyses of data especially in the light of the low number of trials prospectively registered in clinical trials databases [14, 15] , low statistical power [16, 17], that patients enrolled in clinical trials of psychotherapy are allowed to stay on stable doses of psychiatric medications [18] and so forth.
Although we acknowledge that conducting methodological stringent and rigorous randomized controlled trials of psychotherapy is excruciatingly difficult, we should not ignore the fact that the majority of clinical trials of psychotherapy is characterized by inadequate methodological quality leading to substantial uncertainty regarding the effect estimates obtained in most psychotherapy trials. As pinpointed by Dal-Ré and colleagues [19], we need to interpret the findings from clinical trials of psychotherapy with prudence.
References
1. Wampold, B.E. and Z.A. Imel. What do we know about psychotherapy?—and what is there left to debate? 2015 [cited 2017 02.04.]; Available from: http://societyforpsychotherapy.org/what-do-we-know-about-psychotherapy-and-what-is-there-left-to-debate/.
2. Dragioti, E., et al., Does psychotherapy work? An umbrella review of meta-analyses of randomized controlled trials. Acta Psychiatrica Scandinavica, 2017: p. n/a-n/a.
3. Cuijpers, P., et al., The effects of psychotherapy for adult depression are overestimated: A meta-analysis of study quality and effect size. Psychological Medicine: A Journal of Research in Psychiatry and the Allied Sciences, 2010. 40(2): p. 211-223.
4. Cristea, I.A., et al., Efficacy of psychotherapies for borderline personality disorder: A systematic review and meta-analysis. JAMA Psychiatry, 2017.
5. Cuijpers, P., et al., How effective are cognitive behavior therapies for major depression and anxiety disorders? A meta-analytic update of the evidence. World Psychiatry, 2016. 15(3): p. 245-258.
6. Gerber, A.J., et al., A quality-based review of randomized controlled trials of psychodynamic psychotherapy. The American Journal of Psychiatry, 2011. 168(1): p. 19-28.
7. Thoma, N.C., et al., A quality-based review of randomized controlled trials of cognitive-behavioral therapy for depression: An assessment and metaregression. The American Journal of Psychiatry, 2012. 169(1): p. 22-30.
8. Lynch, D., K.R. Laws, and P.J. McKenna, Cognitive behavioural therapy for major psychiatric disorder: Does it really work? A meta-analytical review of well-controlled trials. Psychological Medicine, 2010. 40(1): p. 9-24.
9. Svartberg, M. and T.C. Stiles, Comparative effects of short-term psychodynamic psychotherapy: A meta-analysis. Journal of Consulting and Clinical Psychology, 1991. 59(5): p. 704-714.
10. Jauhar, S., et al., Cognitive-behavioural therapy for the symptoms of schizophrenia: systematic review and meta-analysis with examination of potential bias. The British Journal of Psychiatry, 2014. 204(1): p. 20-29.
11. Wykes, T., et al., Cognitive behavior therapy for schizophrenia: Effect sizes, clinical models, and methodological rigor. Schizophrenia Bulletin, 2008. 34(3): p. 523-537.
12. Cuijpers, P., et al., Efficacy of cognitive–behavioural therapy and other psychological treatments for adult depression: meta-analytic study of publication bias. The British Journal of Psychiatry, 2010. 196(3): p. 173-178.
13. Driessen, E., et al., Does Publication Bias Inflate the Apparent Efficacy of Psychological Treatment for Major Depressive Disorder? A Systematic Review and Meta-Analysis of US National Institutes of Health-Funded Trials. PLOS ONE, 2015. 10(9): p. e0137864.
14. Simmons, J.P., L.D. Nelson, and U. Simonsohn, False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 2011. 22(11): p. 1359-1366.
15. Cybulski, L., E. Mayo-Wilson, and S. Grant, Improving transparency and reproducibility through registration: The status of intervention trials published in clinical psychology journals. Journal of Consulting and Clinical Psychology, 2016. 84(9): p. 753-767.
16. Ioannidis, J.P.A., Why most discovered true associations are inflated. Epidemiology, 2008. 19(5): p. 640-648 10.1097/EDE.0b013e31818131e7.
17. Ioannidis, J.P.A., Why science is not necessarily self-correcting. Perspectives on Psychological Science, 2012. 7(6): p. 645-654.
18. Skapinakis, P., et al., Pharmacological and psychotherapeutic interventions for management of obsessive-compulsive disorder in adults: a systematic review and network meta-analysis. The Lancet Psychiatry, 2016. 3(8): p. 730-739.
19. Dal-Ré, R., J. Bobes, and P. Cuijpers, Why prudence is needed when interpreting articles reporting clinical trial results in mental health. Trials, 2017. 18(1): p. 143.
