Publication bias in studying the efficacy of psychotherapy.

October 5, 2015 9:17 AM

"The efficacy of antidepressant medication has been shown empirically to be overestimated due to publication bias [...] The efficacy of psychological interventions for depression has been overestimated in the published literature, just as it has been for pharmacotherapy. Both are efficacious but not to the extent that the published literature would suggest"
posted by OmieWise (41 comments total)

And how does that make you feel?
posted by jedicus at 9:20 AM on October 5, 2015 [4 favorites]

Well, as a therapist, not that great!
posted by OmieWise at 9:21 AM on October 5, 2015 [12 favorites]

Good. Maybe it's about time we stop pathologizing people with accurate perceptions of reality.

(Antidepressant medication allows me to get out of bed in the morning, but has yet to provide me with a reason why doing so would be a good idea.)
posted by Faint of Butt at 9:26 AM on October 5, 2015 [60 favorites]

Is there a third option that I'm unaware of or should people just give up entirely?
posted by GuyZero at 9:32 AM on October 5, 2015 [3 favorites]

option 3 as always is existential darkness.
posted by poffin boffin at 9:34 AM on October 5, 2015 [25 favorites]

Fascinating.
posted by grobstein at 9:34 AM on October 5, 2015

Is there a third option that I'm unaware of or should people just give up entirely?

Sure. Realize that -- like restoring a house -- you can't do the whole job with just one tool, and not only is that okay, it's actually exciting to have a range of options tailored to your house's/brain's needs.
posted by Celsius1414 at 9:43 AM on October 5, 2015 [3 favorites]

Yeah, nope. I'm not putting too much stock in an article that appears in a pay-for-publish journal.
posted by batbat at 9:46 AM on October 5, 2015 [1 favorite]

MetaFilter: Maybe it's about time we stop pathologizing people with accurate perceptions of reality.
posted by ZenMasterThis at 9:49 AM on October 5, 2015 [8 favorites]

It's not really news that one type of therapy, one medication, one size, doesn't fit all. That doesn't mean they don't work for subgroups. What would be nice is a study testing some theories on a) which groups respond to what type of treatment, and b) what's the mechanism for this.

I don't take my meds I don't have a fucking accurate perception of reality.
posted by Braeburn at 9:52 AM on October 5, 2015 [4 favorites]

I know that some disorders (e.g. BPD) are most likely to get better with the passing of time (e.g. decades).

Is that also true for anxiety/depression/etc.? ("Ya see, what happened, what happened was that I forgot what I was worried about...")
posted by clawsoon at 9:59 AM on October 5, 2015

Well then, to paraphrase a wise man, my advice to you all is to start drinking heavily.
posted by jonmc at 10:00 AM on October 5, 2015 [3 favorites]

I attempted to RTFA but I can't parse through the technical language. Can someone sum up the statistics?
posted by joeyjoejoejr at 10:02 AM on October 5, 2015

I know that some disorders (e.g. BPD) are most likely to get better with the passing of time (e.g. decades).

Is that also true for anxiety/depression/etc.?

Well, I'm well into 50+ years of depression, so...no, not really.
posted by Thorzdad at 10:18 AM on October 5, 2015 [10 favorites]

a word of caution:
Talk therapy, once the go to "science" got supplanted by Meds, usually now combined with a bit of talk therapy too. I am in no position to say with any evidence which or either works best, but I do know that talk therapy was limited in the amount of time insurance would cover such therapy because the insurance industry did NOT consider talk therapy as real science.

Therapists, if they had the appropriate credentials, then switched to issuing meds, which the insurance industry does consider science and thus will pay out for over a longer period of time.
In sum, no matter what works or works best, the insurance coverage is a key point in what takes place.

I was told this first hand by a therapist in discussing the field with him.

ps: a similar thing took place with chiropractic. Insurance did not pay for visits to chiropractors and so not that many people went to them for treatment. When chiropractors sued, however, they finally got accepted as a viable treatment, and insurance companies began paying for numerous visits for help. Chiropractic is now mainstream and very lucrative for those in the field.
posted by Postroad at 10:18 AM on October 5, 2015 [1 favorite]

The big story here is publication bias, though I'll admit that I've long put great faith in the claim that both talk and drug therapies were equally effective, and so it's a bit of a blow there as well.

Think about this for a second:

Of the 13 unpublished studies, only two were submitted for review; neither was accepted for publication. Six other studies were never submitted for review, although, in three instances, the investigators still hoped to do so. For the remaining five studies, it was unclear whether the authors tried to submit their findings for publication. Explanations that the investigators gave for not submitting manuscripts included that they did not think the findings were interesting enough to warrant publication, that they got distracted by other obligations or that they had practical problems.

We've all been walking around with the wrong beliefs for a decade because you can't publish null or uninteresting results, and so the study authors got bored. We live in a world where only "Man Bites Dog" studies can get published, or at least more in that world than is healthy.
posted by anotherpanacea at 10:40 AM on October 5, 2015 [9 favorites]

Dunno. Not all studies are equal
posted by MisantropicPainforest at 10:55 AM on October 5, 2015

batbat: “Yeah, nope. I'm not putting too much stock in an article that appears in a pay-for-publish journal.”

The trouble is that the other kind is just as bad.
posted by koeselitz at 10:56 AM on October 5, 2015 [1 favorite]

We've all been walking around with the wrong beliefs for a decade because you can't publish null or uninteresting results, and so the study authors got bored. We live in a world where only "Man Bites Dog" studies can get published, or at least more in that world than is healthy.

Also people are going back and finding that they cannot replicate results of many psychology studies. Man bites dog, this one time, in our lab, where we had everything set up just how he likes it.
posted by thelonius at 11:04 AM on October 5, 2015 [1 favorite]

batbat: Yeah, nope. I'm not putting too much stock in an article that appears in a pay-for-publish journal.

Aren't basically all journals pay-for-publish? Like, even the good ones?
posted by Mitrovarr at 11:53 AM on October 5, 2015 [4 favorites]

Yeah, nope. I'm not putting too much stock in an article that appears in a pay-for-publish journal.

You want to expand on that? Does that include all journals that have page charges?
posted by one_bean at 11:54 AM on October 5, 2015 [2 favorites]

...talk therapy was limited in the amount of time insurance would cover such therapy because the insurance industry did NOT consider talk therapy as real science.

I don't think "real science" has anything to do with it. I mean, many insurers cover chiropractors, ffs. I think insurers' limited the number of visits to therapists because talk therapy tends to be very open-ended, with the very real possibility of it going on for years. That's more what insurers have an issue with, and not whether talk-therapy is a "real science".
posted by Thorzdad at 12:09 PM on October 5, 2015 [7 favorites]

Well then, to paraphrase a wise man, my advice to you all is to start drinking heavily.

I, of course, concur.
posted by booooooze at 12:54 PM on October 5, 2015 [4 favorites]

Yeah, nope. I'm not putting too much stock in an article that appears in a pay-for-publish journal.

This is absolute noise. Many, many academic journals charge fees to the authors of an article - including some top-tier ones such as PNAS. PLoS One is a totally legitimate and respected peer-reviewed scientific publication.
posted by kickingtheground at 12:56 PM on October 5, 2015 [10 favorites]

talk therapy tends to be very open-ended, with the very real possibility of it going on for years.

This is true, but if it works then it should be obvious to cover it; My insurance pays a little bit less than $3000/year for me to get a year of weekly talk therapy, which is certainly not cheap, but if it makes a real difference in my quality of life then it doesn't seem that much different from them paying thousands of dollars when I had to have hand surgery to remove a benign bone tumor (which was, subjectively, not as bad as anxiety for my quality of life); it's just that the expense is spread out over a much longer period of time.

I do see where it's a lot harder, with talk therapy, to establish a point of "OK, you're fine, you're done"; it's hard to establish the line between depression that needs treatment and ordinary existential gloominess; I guess I'm speaking mostly from my own experience, which was that the 8 weeks that many insurance companies used to cover was just barely enough to get out of the immediate crisis, and if you stopped the evaluation right then you would say that talk therapy was awfully ineffective for me.

(Do some insurance companies still only cover 8 weeks? That's one of the things that changed because of the ACA, right?)
posted by Jeanne at 1:00 PM on October 5, 2015 [2 favorites]

>Yeah, nope. I'm not putting too much stock in an article that appears in a pay-for-publish journal.

Uh, Open Access *does not* mean Pay-To-Publish-Because-"Reputable"-Journals-Wouldn't-Accept-It. Open Access is a publishing alternative to the Paywall-Mode of publishing which restricts access to information to those organizations with the (significant!) funds to pay for it. My research group was looking into launching a niche journal a few years ago, and we focused on OA for that reason.

Anyway, the real test of a journal is whether it has effective peer review, and the test for that is in the pudding, what other articles are published by the journal. And how frequently are they cited by others?
posted by insert.witticism.here at 1:08 PM on October 5, 2015 [7 favorites]

Oh, yeah, it is PLOS One. That is a top tier journal, highly ranked.
posted by insert.witticism.here at 1:15 PM on October 5, 2015 [5 favorites]

Also, it's unclear how strictly treatment protocols were adhered to in the trials, or what difference treatment fidelity (or lack thereof) would make in a naturalistic therapeutic setting.

I know I've seen some cross-cultural studies that have suggested that the particular therapy hardly matters, and that it's more about the relationship with the therapist (or equivalent), but that sort of takes the shine off clinical psychology as an endeavour with predictive authority (and might possibly mean that the small effects seen are down to placebo).
posted by cotton dress sock at 1:43 PM on October 5, 2015 [1 favorite]

third option

If you're not too far down the hole, talking with friends, finding meaningful work or leisure activities (or directing energy to anger vs. sadness when those opportunities aren't there); engaging with community, politics, art; Vit D and fish oil supps, burpees/running/swimming/walks in the park; recognizing when you're headed down the hole, or asking for a heads-up from friends who notice it happening; maybe living in a totally different culture than the one we've got.
posted by cotton dress sock at 1:50 PM on October 5, 2015 [2 favorites]

Do some insurance companies still only cover 8 weeks? That's one of the things that changed because of the ACA, right?)

Officially: No, and yes. Unofficially: Insurance companies can start putting a lot of pressure on therapists who see clients for an extended period of time (which can sometimes be anything more than 12 weeks).

I admit that I skimmed everything other than the intro and discussion, but I'm not seeing if the researchers compared length of time in therapy (I would assume not, since that's not what they were looking at). There's been such a push toward short-term therapy and I seem to be seeing more studies showing that longer-term therapy (that is, 6-24 months) is more effective, but I don't know what the research consensus is, if any.
posted by jaguar at 1:59 PM on October 5, 2015

SAMHSA's Working Definition of Recovery from Mental Disorders and/or Substance Use Disorders is a nice holistic view of the various forms "treatment" or support might come in.
posted by jaguar at 2:01 PM on October 5, 2015 [3 favorites]

Lobotomies for everyone!

I know that was a joke, but apparently, physical methods for treatment-resistant depression are being seriously looked at again (i.e. deep brain stimulation).

(Also - ayahuasca [previously], MDMA, & nitrous oxide)
posted by cotton dress sock at 2:35 PM on October 5, 2015

I would much rather hug a puppy than puke up yaje.
posted by poffin boffin at 3:38 PM on October 5, 2015 [4 favorites]

Also people are going back and finding that they cannot replicate results of many psychology studies.

Kind of a derail, but I actually was... kind of encouraged by the results from that study! 36% is clearly less than you would expect if all the effects were real, but it's also not quite as bad as it sounds: even if all of the effects were totally as described, you wouldn't actually expect 100% perfect replication, especially for the smaller effect sizes where your power (ability to find a true and statistically significant difference) is going to be the worst. Other people seem to agree. I'm inclined to agree with those who have said that the real issue is not so much the percentage and more that this massive replication-a-thon was a one-off, not a regularly scheduled/funded event.
posted by en forme de poire at 6:33 PM on October 5, 2015

Anyway, I am a complete dilettante in this field so take the following with a grain of salt, but: my understanding is that while people who don't receive any treatment at all don't tend to show a lot of improvement, even placebo treatment actually seems to help quite a bit. (You can get a sense of that comparing Fig. 1 vs Fig. 2 in this study: in the first one the comparison group is "no treatment" and in the second it's placebo. The effect size of psychotherapy is significantly different from zero in both cases, but in the first figure it's around 1.0 vs. ~0.25 in the second.) It seems like if you wanted to be cute you could flip the question around and frame it as "why are placebos so effective for anxiety and depression?" From that editorial: "More than 30 years of double-blind placebo-controlled antidepressant efficacy studies have consistently shown that 30 to 40% of moderately to severely depressed patients improve with placebo treatment (Klerman and Cole 1965; Stark and Hardison 1985; Brown et al. 1988)." The rates for mild depression are even better (FWIW, I think even people like Irving Kirsch who are pessimistic about antidepressants for mild depression accept that for major depression, antidepressants are significantly and substantially more effective than placebo).

But beyond that, it's also worth repeating, I think, that this meta-analysis still shows 1. a statistically-significant benefit from psychotherapy over placebo, 2. no hugely-obvious differences between therapy and antidepressants in terms of effect size, and 3. evidence of synergy between psychotherapy and antidepressants (in that combining both appears to be more effective than ADs alone). They're not enormous effect sizes, but they do seem to be statistically significant.

So to respond to anotherpanacea's (eponysterical?) comment above, "I've long put great faith in the claim that both talk and drug therapies were equally effective," this study doesn't seem to actually contradict that conclusion -- if there is a difference, it wasn't detectable from this meta-analysis and would probably end up being pretty small. (Personally, since each approach has big fans, I wouldn't necessarily expect there to be publication bias in a way that systematically favors one vs. the other, so it makes sense to me that this paper wouldn't present a drastically different answer to that particular question -- but like I said, it's not my field.)
posted by en forme de poire at 7:22 PM on October 5, 2015 [3 favorites]

I attempted to RTFA but I can't parse through the technical language. Can someone sum up the statistics?

In addition to what anotherpanacea said, if it's still unclear, you can get a pretty okay idea of what's going on by skipping the text entirely and just looking at the figures. Figure 1 is just a flowchart giving how they picked the studies they included in this meta-analysis, and Table 1 has a bunch of details about the grants, so you can skip them if you just want to get to the results.

Figures 2-7 summarize a bunch of individual studies in basically the same way. Each study is shown as a horizontal bar. The dot in the bar is that study's estimate of how big the effect was (higher numbers = more reduction in depression), and the bar width shows the 95% confidence interval (CI). 95% of the time, the "real" effect size should lie within its 95% CI. The published ones are on top and the unpublished ones (ones that were registered in an NIH database but never made it to publication -- the authors got the data by talking to the people who did the study directly) are on the bottom. Finally, the result the authors get after integrating all the studies is the diamond on the bottom (the two intermediate diamonds are for just the published and just the unpublished studies, so you can compare the two).

Figure 2 has studies comparing psychotherapy to any type of control, Figure 3 has only comparisons to no treatment whatsoever, Figure 4 compares to pill or psychological placebos, Figure 5 compares to miscellaneous other types of controls, Figure 6 compares to antidepressants, and Figure 7 compares psychotherapy plus ADs to ADs alone. Finally, Table 2 summarizes everything in one table ("g" is on the same scale as the figures and "95% CI" is the 95% confidence interval for g) along with some other statistics.
posted by en forme de poire at 12:46 AM on October 6, 2015

Now that I think about it, I'm wondering whether they need to control for the differential strength of the placebo effect. Like, this study says that psychotherapy is still more effective than placebo, but what if people who are susceptible to placebo are also more susceptible to psychotherapy (which is a reasonable guess). Combined with the fact that, in the population, there are more people susceptible to placebo than not. Or maybe this is something already taken into account with standard statistical calculations re: placebo effect, I have no idea.
posted by unknowncommand at 1:21 PM on October 6, 2015

That makes me wonder: what's the placebo for psychotherapy? Having coffee with a friend? Going to see a psychic? Jungian or Lacanian analysis? :-)
posted by anotherpanacea at 1:33 PM on October 6, 2015

That makes me wonder: what's the placebo for psychotherapy?

From what I can tell, it's going to see a supportive professional once a week for an hour and talking about your problems, but the professional doesn't do any specific interventions other than being supportive. But the professional needs to act like a therapist, because the point of studying a placebo is that the participant is not supposed to know they're on placebo treatment, so they need to think they're seeing therapist. So it seems that placebo-psychotherapy is very similar to regular psychotherapy, stripped of the bells-and-whistles that various proponents of various types of therapy claim are what makes therapy effective, except that placebo-psychotherapy being very effective rather undermines those claims.
posted by jaguar at 2:41 PM on October 6, 2015 [3 favorites]

So I'm reading this paper, and wondering more about this question. Are studies of psychotherapy really offering nonspecific supportive talk therapy by a professional, with the placebo therapist refusing to offer advice or basic cognitive therapy recommendations?

I confess that I always thought that the placebo in studies of psychotherapy was a sugar pill; you'd know you'd been given a pill, but not the contents. And in that paper, it looks more like the authors are referring to the placebo in something like a metaphorical sense, not actually trying to produce a replicable process for supplying placebo therapy.
posted by anotherpanacea at 3:13 PM on October 6, 2015

Anotherpanacea, it looks like there are a mixture of different types of controls that this meta-analysis deals with. Some are indeed an inert pill (but even this still involves some ongoing contact with doctors and monitoring of symptoms, I think), some are "treatment as usual" and some are a "psychological placebo". I think this last one is what jaguar is talking about a few comments ago (is that right?). I'm not sure what "treatment as usual" means in this context or even if it means anything particularly consistent.
posted by en forme de poire at 3:59 PM on October 6, 2015