What does the replication crisis mean for psychotherapy?

March 6, 2020 8:44 AM

The evidence for evidence-based therapy is not as clear as we thought (Aeon): "Is the credibility of the evidence for ESTs [empirically supported treatments] as strong as that designation suggests? Or does the evidence-base for ESTs suffer from the same problems as published research in other areas of science?" Of the 70 ESTs listed by the Society of Clinical Psychology, researchers found that 20% performed well, 30% had mixed results, and 50% had subpar outcomes.

Around 20 per cent of ESTs performed well across a majority of our metrics (eg, problem-solving therapy for depression, interpersonal psychotherapy for bulimia nervosa, the aforementioned exposure therapy for specific phobias). This means not only that the therapies have been subjected to clinical trials, but that the evidence produced from these clinical trials seems credible and supports the claim that the EST will help people. We also found a ‘murky middle’: 30 per cent of ESTs had mixed results across metrics, performing neither consistently well nor poorly (eg, cognitive therapy for depression, interpersonal psychotherapy for binge-eating disorder).

That leaves 50 per cent of ESTs with subpar outcomes across most of our metrics (eg, eye-movement desensitisation and reprocessing for PTSD, interpersonal psychotherapy for depression). In other words, although these ESTs seemed to work based on the claims of the clinical trials cited by the Society of Clinical Psychology, we found the evidence from these trials lacked statistical credibility. For these ESTs, the relevant research results are sufficiently ambiguous that we cannot be sure that they really do work better than other forms of therapy.

Evaluating the evidential value of empirically supported psychological treatments (ESTs): A meta-scientific review (Journal of Abnormal Psychology abstract)

posted by not_the_water (6 comments total)

Wait... so figuring out how to solve your problems works better for depression than simply changing your thoughts about your problems?

It sounds obvious when it's put like that, but I've been operating for years on the assumption that CBT was the gold standard of depression treatment.
posted by clawsoon at 9:11 AM on March 6, 2020 [9 favorites]

Very interesting, thanks for posting!

This bit from the Aeon piece seems to get to the heart of the matter:

Ethical therapists can continue to engage in practice that is evidence-based, not eminence-based, rooting their therapies in scientific evidence rather than their own conjecture or that of senior colleagues. They can also continue the routine outcome measurement many already employ: solicit therapy clients’ feedback early and often, be open to surprise about what’s working and what’s not, and adjust accordingly. Clients can ask their therapists upfront if they will offer the opportunity for such mutual assessment of their progress.

Surely any replicability at all is pretty surprising in the context of a system as complex as the human brain and a process as complex as therapy. If ESTs are understood as "here is a thing that research suggests is relatively likely to help, so let's start there" rather than "here is a Thing That Works," then a lack of statistical power behind clinical trial results becomes a lot less troubling IMO.
posted by Not A Thing at 9:26 AM on March 6, 2020 [11 favorites]

Wow, it's almost like human beings are unique individuals within mindbogglingly specific social contexts (not to mention massively different cultural and familial experiences, personalities, and emotional patterns AND a range of identities that might be marginalized/oppressed) and trying to treat them like interchangeable machines that simply need the same set of protocols doesn't actually make very much sense.
posted by overglow at 9:28 AM on March 6, 2020 [26 favorites]

Blood letting is an effective treatment for polycythemia, but it's not going to be very effective at generally treating numbness and itching. Unfortunately our understanding of psychological disorders are very symptom based, we don't really understand the pathogenesis. It's going to be very hard to evaluate the treatment-diagnosis combinations in a scientific way under those conditions.
posted by betaray at 1:19 PM on March 6, 2020 [2 favorites]

50 per cent of ESTs with subpar outcomes across most of our metrics (eg, eye-movement desensitisation and reprocessing for PTSD, interpersonal psychotherapy for depression).

Interpersonal psychotherapy works fine when doctor and patient have a good rapport. It doesn't when they don't, and can cause additional damage if the doctor, deliberately or not, undermines the patient's confidence or progress. I don't see any sign that they tried to measure doctor-patient rapport as part of the trial results.

Not mentioned in the article: the fact that, unlike surgery or bonesetting, treatments in the psych field are never administered exactly the same way with different patients. The guide can say "Express empathy: Validate the patient's experience," but that doesn't mean every therapist is actually doing so.

Also not mentioned: That clinical trials for psych treatments may have standards that are not repeated in real practice. "Sessions 2x/week for 12 weeks" are outside a lot of people's budget, both cash and time. And the studies don't go into "so... how useful is this if the patient only makes 1/3 of the sessions because their insurance won't cover more than that?"
posted by ErisLordFreedom at 2:00 PM on March 6, 2020 [14 favorites]

I admit to not having time to read the full article yet, but doing a quick cntl-f for "common factors"... yeah, this article seems to have buried the lede... most good therapists who put any thought into their practice and the reseach are well aware that "evidence-based" treatment is less critical to good therapy outcomes than one would intuitively imagine. Psychotherapy isn't medication (hell, even medication's effects are influenced by the same common factors that influence psychotherapy)! You don't have discrete variables where if you do intervention A and C you'll get outcomes B and D. Its much more complex (and vastly more fascinating!) than that. Unfortunately people think this is how therapy works (even many bad therapists) and you often see this play out here in Metafilter when folks are adamant that it was CBT homework that allieved their anxiety symptoms when really the homework and psychoedication on cognitive distortions was more like the window dressing (though still important!) around the real agents of change in therapy which are things like the alliance, relationship etc.

Anyway, I encourage anyone who's into this debate check out The Heart and Soul of Change and other research on the common factors. It will change how you see the helping professions including straight up medicine.
posted by flamk at 5:22 PM on March 6, 2020 [4 favorites]