"P-hacking is widespread."

March 17, 2015 3:01 PM

P-hacking, or inflation bias, is the selective inclusion of experimental results that suggest statistical significance, as well as the selective exclusion of results which argue against the hypothesis. Skewing work towards positive results helps investigators publish in high-profile journals, which in turn improves access to funding. In a recent PLOS publication, Michael Jennions and Megan Head use text-mining and meta-analyses to determine the extent to which this influences a broad array of published research, offering recommendations on how to reduce this practice.
posted by a lungful of dragon (7 comments total)

A moderating note from the abstract:

while p-hacking is probably common, its effect seems to be weak relative to the real effect sizes being measured. This result suggests that p-hacking probably does not drastically alter scientific consensuses drawn from meta-analyses.

I probably will skim the text of the article because I'm intrigued about how you can "use text-mining" to detect p-hacking, which is totally opaque (to me) from the abstract.

If they're right that p-hacking doesn't critically affect results being discussed in their population of studies, I wonder what a similar analysis of different populations.
posted by grobstein at 3:49 PM on March 17, 2015

This Has been at the heart of criticisms of frequentist statistics: the p value that one gets is the chance you would see the data you have if the bull is true, but deciding when to stop getting/modeling the data is completely opaque. Chalk one up for the bayesians.
posted by MisantropicPainforest at 4:13 PM on March 17, 2015 [2 favorites]

In my field at least, it's pretty common to report the significant p-values in-text as numbers (e.g., p=0.032 or whatever). But if they aren't significant we often just write it verbally, saying something like "all other comparisons were non-significant" or "all other p-values were greater than 0.05."

The text-mining approach followed in this paper would mis-characterise this practice as not reporting the non-significant correlations: it only identifies p-values when they are reported in a certain format (see the S1 appendix), which does not correspond to this verbal one.

So, while I totally agree that p-hacking can be an issue, I'm not sure that I trust this result or method for its ability to accurately pick that up.
posted by forza at 5:26 PM on March 17, 2015 [1 favorite]

They actually address that in the paper, forza:

Both p-hacking and selective publication bias predict a discontinuity in the p-curve around 0.05, but only p-hacking predicts an overabundance of p-values just below 0.05
posted by Zalzidrax at 5:35 PM on March 17, 2015 [3 favorites]

Ah - I see, thanks for pointing that out, Zalzidrax.
posted by forza at 6:17 PM on March 17, 2015

A moderating note from the abstract:

while p-hacking is probably common, its effect seems to be weak relative to the real effect sizes being measured. This result suggests that p-hacking probably does not drastically alter scientific consensuses drawn from meta-analyses.

I read that in the same voice as I'd read John Wilkins careful accounting of how his comprehensive list of animals could all fit on Noah's Ark.
posted by ethansr at 9:56 AM on March 18, 2015

with enough space "to receive all the dung that should proceed from them for a whole year"
posted by ethansr at 10:01 AM on March 18, 2015 [1 favorite]