An impending train wreck in social psychology.
May 19, 2014 11:53 PM Subscribe
The current critique of experimental social science is coming mainly from the inside. Strohminger, Simmons, and a handful of other mostly young researchers are at the heart of a kind of reform movement in their field. Together with a loose confederation of crusading journal editors and whistle-blowing bloggers, they have begun policing the world of experimental research, assiduously rooting out fraud and error, as if to rescue the scientific method from embarrassment—and from its own success. The Reformation: Can Social Scientists Save Themselves?
"By convention, the standard for statistical significance in psychology—enforced by the editors of professional journals in deciding whether to accept a paper—is a p value of less than five percent (p < 0.05). In other words, if you did 100 repetitions of the experiment, you should get a similar result in at least 95 of them."That is not at all what this means.
The p-value and its importance is notoriously difficult to explain to laymen, but the best shot at it that I've ever seen can be found here.
posted by Blasdelb at 1:20 AM on May 20, 2014 [29 favorites]
I like the quote about glamorizing the single study - something that bothers me a lot about the popular perception of empirical science. It is most evident in mainstream science journalism, where it's normal to hype sexy results without giving their context.
There are a lot of deep, serious issues within the sciences in general when it comes to the integrity of the research - it is not just social science, which I think gets unfairly singled out in part because (ironically) there are people within social science making noise about them.
The article focuses on deliberate fraud, but I don't think that this is as big an issue as mistakes that are simply not caught. Scientists are using increasingly sophisticated statistical methods in order to analyze their data, but most of them are not statisticians. If you don't understand the methods that you're using very well, it is easy to misapply them or misinterpret the results. (Case in point with the p-values.)
As a concrete example - I have a strong background in mathematics and just completed an intensive, year-long graduate level course in statistics. This is all of the formal statistical training I will be able to manage, and I know that it isn't enough. I should have begun my statistics training earlier, and continued statistics training should be an expected part of my career for the next several years, rather than an extra I'd struggle to squeeze into an already brutal timetable. Barring that, we need more institutional support for collaboration with actual statisticians.
And then, once the mistakes are made, there are not many incentives for other scientists to attempt to reproduce your work, which is a whole nother barrel of problems.
posted by Kutsuwamushi at 1:44 AM on May 20, 2014 [8 favorites]
There are a lot of deep, serious issues within the sciences in general when it comes to the integrity of the research - it is not just social science, which I think gets unfairly singled out in part because (ironically) there are people within social science making noise about them.
The article focuses on deliberate fraud, but I don't think that this is as big an issue as mistakes that are simply not caught. Scientists are using increasingly sophisticated statistical methods in order to analyze their data, but most of them are not statisticians. If you don't understand the methods that you're using very well, it is easy to misapply them or misinterpret the results. (Case in point with the p-values.)
As a concrete example - I have a strong background in mathematics and just completed an intensive, year-long graduate level course in statistics. This is all of the formal statistical training I will be able to manage, and I know that it isn't enough. I should have begun my statistics training earlier, and continued statistics training should be an expected part of my career for the next several years, rather than an extra I'd struggle to squeeze into an already brutal timetable. Barring that, we need more institutional support for collaboration with actual statisticians.
And then, once the mistakes are made, there are not many incentives for other scientists to attempt to reproduce your work, which is a whole nother barrel of problems.
posted by Kutsuwamushi at 1:44 AM on May 20, 2014 [8 favorites]
Psychologist here.
Speaking from my own experience with statistical analysis (which is considerable), the concerns about the state of affairs in social psychology are broadly justified, but an insistence on a more rigorous approach to statistics misses the more fundamental problems in the field. That is to say: Only good things can come from the field becoming more quantitatively sophisticated, but so long as the underlying problems persist, studies will continue to be published whose long-term theoretical validity are questionable.
At its most fundamental, the biggest affliction that social psychology has is the reflex that "everyone is like the folks in my town" (where the "town" is the insular world of academia). Nearly all research psychologists have good motives (whether those be basic scientific curiosity or a desire to improve the world in some way), but their theoretical intuitions are powerfully shaped by social norms and current trends. To date, following those intuitions has seemed to pay off experimentally at first, only to grow shakier a decade or two down the line.
This is not, in my view, solely a function of poor analysis. For example, if you routinely use college undergraduates as your sample, the problems of biased sampling are much more serious than the problems of p-hacking. Although tightening the analytic standards of the field will help to some extent, and making the original data more transparent will also do so, the true problem in my view is the pursuit of ideas that are merely fashionable, rather than working to develop theories that have durability and depth.
posted by belarius at 1:45 AM on May 20, 2014 [21 favorites]
Speaking from my own experience with statistical analysis (which is considerable), the concerns about the state of affairs in social psychology are broadly justified, but an insistence on a more rigorous approach to statistics misses the more fundamental problems in the field. That is to say: Only good things can come from the field becoming more quantitatively sophisticated, but so long as the underlying problems persist, studies will continue to be published whose long-term theoretical validity are questionable.
At its most fundamental, the biggest affliction that social psychology has is the reflex that "everyone is like the folks in my town" (where the "town" is the insular world of academia). Nearly all research psychologists have good motives (whether those be basic scientific curiosity or a desire to improve the world in some way), but their theoretical intuitions are powerfully shaped by social norms and current trends. To date, following those intuitions has seemed to pay off experimentally at first, only to grow shakier a decade or two down the line.
This is not, in my view, solely a function of poor analysis. For example, if you routinely use college undergraduates as your sample, the problems of biased sampling are much more serious than the problems of p-hacking. Although tightening the analytic standards of the field will help to some extent, and making the original data more transparent will also do so, the true problem in my view is the pursuit of ideas that are merely fashionable, rather than working to develop theories that have durability and depth.
posted by belarius at 1:45 AM on May 20, 2014 [21 favorites]
Ethnographer here, sort of, waving what is starting to feel like a tired old flag - not all social science or knowledge is determined by experimental design. Other empirical methods and epistemologies have value. Anthropology has its own demons, sure, but these here are not those. [Looks at flag, puts it down, wanders off.]
posted by YouRebelScum at 2:04 AM on May 20, 2014 [13 favorites]
posted by YouRebelScum at 2:04 AM on May 20, 2014 [13 favorites]
It seems to me that the difficulty of psychology is that there is still no working theory of where the mind comes from and how it works. It's like doing physics experiments in a world where people don't know what energy, momentum, gravity, force, work, etc are. Sure you can get some numbers, but if you don't know how all these different things relate to each other, you're not getting anything useful or really enlightening anyone.
posted by empath at 2:27 AM on May 20, 2014 [5 favorites]
posted by empath at 2:27 AM on May 20, 2014 [5 favorites]
empath, Steven Pinker seems to believe that we are on the way towards such a framework. See the reply to "If you were trying to get someone interested in this field today, what would you say?" in the interview here.
posted by Gyan at 2:38 AM on May 20, 2014 [1 favorite]
posted by Gyan at 2:38 AM on May 20, 2014 [1 favorite]
I'm part of the effort to analyze pre-registered replications of papers, associated the Open Science Framework. I can't give the details here (some of them will be presented at APS meeting, I'm told, but I can't be there) but I can tall you that MANY of them did not work out. It is embarrassing for the field.
But I love it as a methodologist and statistician, because it means they need people like me!
posted by Philosopher Dirtbike at 2:53 AM on May 20, 2014 [6 favorites]
But I love it as a methodologist and statistician, because it means they need people like me!
posted by Philosopher Dirtbike at 2:53 AM on May 20, 2014 [6 favorites]
There are really deep invalidating problems with social psychology, and its good that there really is a new generation now that is unwilling to tolerate them any more than is still absolutely necessary for a career. I was an experimental psych major before leaving it for harder stuff, and there is a reason I left. All of the old giants in social psychology today came into the field inspired by the original Candid Camera pranks, pulled for the benefit of television audiences in the fifties and sixties, and collectively never gained a meaningful understanding of statistics or an appreciation for science as the study of the natural world, physics, rather than mere stamp collecting. They never felt the need to go beyond simply collecting neat seeming pranks with plausible just-so stories like they were stamps to move towards actually connecting their theories to findings from other disciplines. With egos like these, what use is there for giants to stand on?
The problems with social psychology as a discipline go a lot deeper that just the appalling statistical illiteracy, and "illiteracy", that is normative among the people who most need to know better. Its the old boys club of mediocre hacks who write profitable books, and are just smart enough to fool undergrads in introductory courses, that have allowed the illiteracy to fester unchallenged. Its the Zimbardos who think that wowing people with their exciting lack of basic ethics can turn experiments with trivial findings into science, the Baumeisters whose really fundamental lack of understanding of even basic concepts only goes a little way towards explaining the old timey racism that pervades their work, and the even less esteemed professors who traveled in drunken packs loudly looking for female graduate students at the conferences I went to after hours. Its how bizarrely welcome the woo woo bullshit deep end of evolutionary psychology is. Its the reasons why the couple of younger professors whose work demonstrates the profound weirdness and non-generalizability of undergraduates studying psychology, who form social psychology's only model system, don't publish nearly as well as they should and why the whole field is basically just continuing as if they didn't exist. Its the reasons why I can now say these things as strongly as I fucking please from a safe distance while at conferences other students and I knew we could only exchange horrified whispers if we ever wanted to publish.
Eventually the old fuckers who form the root problem are going to die and their lackeys are going to find themselves quite alone but, until that slowly happens, the issues are just simple enough for the internet skeptic machine to understand and eventually it is going to latch on and not let go.
posted by Blasdelb at 3:10 AM on May 20, 2014 [30 favorites]
The problems with social psychology as a discipline go a lot deeper that just the appalling statistical illiteracy, and "illiteracy", that is normative among the people who most need to know better. Its the old boys club of mediocre hacks who write profitable books, and are just smart enough to fool undergrads in introductory courses, that have allowed the illiteracy to fester unchallenged. Its the Zimbardos who think that wowing people with their exciting lack of basic ethics can turn experiments with trivial findings into science, the Baumeisters whose really fundamental lack of understanding of even basic concepts only goes a little way towards explaining the old timey racism that pervades their work, and the even less esteemed professors who traveled in drunken packs loudly looking for female graduate students at the conferences I went to after hours. Its how bizarrely welcome the woo woo bullshit deep end of evolutionary psychology is. Its the reasons why the couple of younger professors whose work demonstrates the profound weirdness and non-generalizability of undergraduates studying psychology, who form social psychology's only model system, don't publish nearly as well as they should and why the whole field is basically just continuing as if they didn't exist. Its the reasons why I can now say these things as strongly as I fucking please from a safe distance while at conferences other students and I knew we could only exchange horrified whispers if we ever wanted to publish.
Eventually the old fuckers who form the root problem are going to die and their lackeys are going to find themselves quite alone but, until that slowly happens, the issues are just simple enough for the internet skeptic machine to understand and eventually it is going to latch on and not let go.
posted by Blasdelb at 3:10 AM on May 20, 2014 [30 favorites]
There are really deep invalidating problems with social psychology
Don't fool yourself; these issues exist across many sciences. Hopefully, there will be a reckoning; also hopefully, it doesn't take the good science down with the bad.
posted by Philosopher Dirtbike at 3:42 AM on May 20, 2014 [3 favorites]
Don't fool yourself; these issues exist across many sciences. Hopefully, there will be a reckoning; also hopefully, it doesn't take the good science down with the bad.
posted by Philosopher Dirtbike at 3:42 AM on May 20, 2014 [3 favorites]
Holy shit is that takedown referenced at the end of the article brutal. Clear, concise, and damning as fuck.
Positive affect and the complex dynamics of human flourishing.posted by Blasdelb at 3:52 AM on May 20, 2014 [9 favorites]
Extending B. L. Fredrickson's (1998) broaden-and-build theory of positive emotions and M. Losada's (1999) nonlinear dynamics model of team performance, the authors predict that a ratio of positive to negative affect at or above 2.9 will characterize individuals in flourishing mental health. Participants (N=188) completed an initial survey to identify flourishing mental health and then provided daily reports of experienced positive and negative emotions over 28 days. Results showed that the mean ratio of positive to negative affect was above 2.9 for individuals classified as flourishing and below that threshold for those not flourishing. Together with other evidence, these findings suggest that a set of general mathematical principles may describe the relations between positive affect and human flourishing.
The complex dynamics of wishful thinking: the critical positivity ratio.
We examine critically the claims made by Fredrickson and Losada (2005) concerning the construct known as the "positivity ratio." We find no theoretical or empirical justification for the use of differential equations drawn from fluid dynamics, a subfield of physics, to describe changes in human emotions over time; furthermore, we demonstrate that the purported application of these equations contains numerous fundamental conceptual and mathematical errors. The lack of relevance of these equations and their incorrect application lead us to conclude that Fredrickson and Losada's claim to have demonstrated the existence of a critical minimum positivity ratio of 2.9013 is entirely unfounded. More generally, we urge future researchers to exercise caution in the use of advanced mathematical tools, such as nonlinear dynamics, and in particular to verify that the elementary conditions for their valid application have been met.
Updated thinking on positivity ratios.
This article presents my response to the article by Brown, Sokal, and Friedman (2013), which critically examined Losada’s conceptual and mathematical work (as presented in Losada, 1999; Losada & Heaphy, 2004; and Fredrickson & Losada; 2005) and concluded that mathematical claims for a critical tipping point positivity ratio are unfounded. In the present article, I draw recent empirical evidence together to support the continued value of computing and seeking to elevate positivity ratios. I also underscore the necessity of modeling nonlinear effects of positivity ratios and, more generally, the value of systems science approaches within affective science and positive psychology. Even when scrubbed of Losada’s now-questioned mathematical modeling, ample evidence continues to support the conclusion that, within bounds, higher positivity ratios are predictive of flourishing mental health and other beneficial outcomes. (PsycINFO Database Record (c) 2013 APA, all rights reserved)
"Don't fool yourself; these issues exist across many sciences. Hopefully, there will be a reckoning; also hopefully, it doesn't take the good science down with the bad."There is that moment of horror we all have after grokking graduate level statistics and then seeing bullshit, bullshit everywhere, but social psychology is dominated by a special kind of awful. So much of it isn't even science done badly, but not even science anymore.
posted by Blasdelb at 3:58 AM on May 20, 2014 [2 favorites]
Two decades later, abstruse postmodern theory is passé, thanks in no small part to the embarrassment that Sokal’s hoax inflicted on it. Today’s intellectual fashions tend instead toward the empirical, the data-driven, and the breezily counterintuitive.
Sadly, this isn't true.
The intellectual rot at the core of postmodernism is primarily attributable to its slack, free-associative conception of reasoning, its worshipful attitude toward almost meaninglessly vague terminological shibboleths, and the fact that it is highly politicized and seems to make almost no effort at all to be objective. It's like bad, highly-political free-form poetry pretending to be philosophy... And those tendencies are still rampant in parts of the humanities and social sciences, and in middle-brow public discussions. People say that we've moved on from postmodernism, but that's only true if you identify postmodernism very narrowly, with certain very narrowly-defined fads. The corrupt essence of the thing is just as influential today as it was 20 years ago.
It's not that we used to have humanities nonsense and now we have social science nonsense. We've had both for a long time.
(On the social-science side, however: My recognition that I was flat-out entitled--and perhaps obligated--to ignore single psychological studies that were patently idiotic came with that ridiculous "Blonde Like Me" study. ( http://jeanneleroy.files.wordpress.com/2008/11/bry-et-al-jesp-2008.pdf ) It's pretty bad when an experimental conclusion is dumb enough that I'd immediately be willing to bet every penny I have in the bank against it, and be ecstatic to have that opportunity...)
posted by Fists O'Fury at 4:18 AM on May 20, 2014 [12 favorites]
Sadly, this isn't true.
The intellectual rot at the core of postmodernism is primarily attributable to its slack, free-associative conception of reasoning, its worshipful attitude toward almost meaninglessly vague terminological shibboleths, and the fact that it is highly politicized and seems to make almost no effort at all to be objective. It's like bad, highly-political free-form poetry pretending to be philosophy... And those tendencies are still rampant in parts of the humanities and social sciences, and in middle-brow public discussions. People say that we've moved on from postmodernism, but that's only true if you identify postmodernism very narrowly, with certain very narrowly-defined fads. The corrupt essence of the thing is just as influential today as it was 20 years ago.
It's not that we used to have humanities nonsense and now we have social science nonsense. We've had both for a long time.
(On the social-science side, however: My recognition that I was flat-out entitled--and perhaps obligated--to ignore single psychological studies that were patently idiotic came with that ridiculous "Blonde Like Me" study. ( http://jeanneleroy.files.wordpress.com/2008/11/bry-et-al-jesp-2008.pdf ) It's pretty bad when an experimental conclusion is dumb enough that I'd immediately be willing to bet every penny I have in the bank against it, and be ecstatic to have that opportunity...)
posted by Fists O'Fury at 4:18 AM on May 20, 2014 [12 favorites]
Given the increasing competition for fewer decent academic positions and less research funding, engaging in fraud seems like a pretty decent proposition (or at least it was before there was a concerted effort to check on those sorts of things). If you're a competent but uninspired scientist, you can look forward to years of grinding out postdocs until you hit your numbers and get a TT position. Or you can try to shortcut all that with a bit of sexy bullshit research; if you get caught, you have to leave academics, but that was likely going to happen anyway.
So while it's probably a good thing that social science is trying to do better about policing itself, it doesn't really address the underlying problems that are contributing to a willingness to publish fraudulent research.
That said, if you ever have the misfortune of being stuck in a room discussing research with a social psychologist, or one of their bastard social neuroscience offspring, it's wise to assume that, without extensive evidence to the contrary, everything that comes out of their mouth is a falsehood. And to keep at least one hand on your wallet.
posted by logicpunk at 4:28 AM on May 20, 2014 [2 favorites]
So while it's probably a good thing that social science is trying to do better about policing itself, it doesn't really address the underlying problems that are contributing to a willingness to publish fraudulent research.
That said, if you ever have the misfortune of being stuck in a room discussing research with a social psychologist, or one of their bastard social neuroscience offspring, it's wise to assume that, without extensive evidence to the contrary, everything that comes out of their mouth is a falsehood. And to keep at least one hand on your wallet.
posted by logicpunk at 4:28 AM on May 20, 2014 [2 favorites]
Metafilter: An impending train wreck in social psychology.
posted by The 10th Regiment of Foot at 4:32 AM on May 20, 2014
posted by The 10th Regiment of Foot at 4:32 AM on May 20, 2014
So much of [social psychology] isn't even science done badly, but not even science anymore.
Back in my undergrad psych days, I started feeling this way when I noticed that just about every 'discovery' was explained by a novel 'theory', matched pretty much to explain that discovery and that discovery only.
Ever since then I can't think of social psychology as anything but the least elegant disaster; I mean what is even the point of having theories if you need a new one for every damn thing you see
posted by obliterati at 4:33 AM on May 20, 2014 [3 favorites]
Back in my undergrad psych days, I started feeling this way when I noticed that just about every 'discovery' was explained by a novel 'theory', matched pretty much to explain that discovery and that discovery only.
Ever since then I can't think of social psychology as anything but the least elegant disaster; I mean what is even the point of having theories if you need a new one for every damn thing you see
posted by obliterati at 4:33 AM on May 20, 2014 [3 favorites]
It's like bad, highly-political free-form poetry pretending to be philosophy...
Future scholars taking an intertextual approach to my MeFi commenting history please note I will be stealing this.
posted by Dr Dracator at 4:44 AM on May 20, 2014 [2 favorites]
Future scholars taking an intertextual approach to my MeFi commenting history please note I will be stealing this.
posted by Dr Dracator at 4:44 AM on May 20, 2014 [2 favorites]
Academic research scientist here. The field of statistics as used in my field of psycholinguistics has undergone a revolution in the last five years, moving from more straightforward analysis to model building. The problem is that the statistics training for most of us that we received in our graduate training was of the former rather than the latter, and re-tooling statistics knowledge is a bitch. Moreover, the statistics courses currently taught at many psychology departments rely on the older statistics rather than the newer statistical approaches and so the system perpetuates itself. Why does this matter? the newer statistics (linear mixed models) being used in my field require that a researcher carefully looks at their whole data (not just means), decisions about selecting outliers for removal (a data massaging tack that can be abused to find significant results) is much more principled and less open for abuse, and the kinds of inferences I think are more robust. Why does this matter here? The statistics training in all fields of psychology is woefully insufficient, and I think experiments are often designed to demonstrate effects rather than to test hypotheses. The state of affairs in social psychology has been a wakeup call for the entire field but I also detect a certain smugness coming from psychology as a whole towards social psychology (as in we knew it was all bullshit all along). No field can police fraud at the level of Sokol or Hauser but it does have a responsibility to train PhDs with sophistication in the tools of their trade - experimental design and statistics.
posted by bluesky43 at 4:52 AM on May 20, 2014 [8 favorites]
posted by bluesky43 at 4:52 AM on May 20, 2014 [8 favorites]
Issues like this are why I stick to the hard sciences and stuff we really understand. Like quantum mechanics.
posted by professor plum with a rope at 5:16 AM on May 20, 2014 [7 favorites]
posted by professor plum with a rope at 5:16 AM on May 20, 2014 [7 favorites]
curuinor:
The p-value and its importance is notoriously difficult to explain to laymen, but the best shot at it that I've ever seen can be found here.
From the article you link to:
posted by lunasol at 5:22 AM on May 20, 2014 [2 favorites]
"By convention, the standard for statistical significance in psychology—enforced by the editors of professional journals in deciding whether to accept a paper—is a p value of less than five percent. In other words, if you did 100 repetitions of the experiment, you should get a similar result in at least 95 of them."That is not at all what this means.
The p-value and its importance is notoriously difficult to explain to laymen, but the best shot at it that I've ever seen can be found here.
From the article you link to:
Here’s how it’s [the p-value] defined:What's the critical difference here that makes the first definition incorrect? The only difference I see is that the definition from the OP is a bit vaguer when it comes to defining the nature of the effect you'll see 95% of the time, but based on my own statistical knowledge I wouldn't call it wrong.
The P value is defined as the probability, under the assumption of no effect or no difference (the null hypothesis), of obtaining a result equal to or more extreme than what was actually observed.
posted by lunasol at 5:22 AM on May 20, 2014 [2 favorites]
There is that moment of horror we all have after grokking graduate level statistics
What graduate level statistics courses/subjects is most useful for this type of stuff?
posted by MisantropicPainforest at 5:34 AM on May 20, 2014
What graduate level statistics courses/subjects is most useful for this type of stuff?
posted by MisantropicPainforest at 5:34 AM on May 20, 2014
What's the critical difference here that makes the first definition incorrect?
The p-value is a probablity, and at a 5% significance, level, it means that if no effect were happening, there is only a 5 percent chance of getting the significant results that you did.
one part of why the first is wrong and the second is right is that 'similar result' should be 'at least as large'--for example, the result could be dissimilar but at least as large, say 0.0000001.
In other words, and this mimics how statistics is written out, the p-value says: given the data we have, what is the probability that there is no effect?
posted by MisantropicPainforest at 5:39 AM on May 20, 2014
The p-value is a probablity, and at a 5% significance, level, it means that if no effect were happening, there is only a 5 percent chance of getting the significant results that you did.
one part of why the first is wrong and the second is right is that 'similar result' should be 'at least as large'--for example, the result could be dissimilar but at least as large, say 0.0000001.
In other words, and this mimics how statistics is written out, the p-value says: given the data we have, what is the probability that there is no effect?
posted by MisantropicPainforest at 5:39 AM on May 20, 2014
When I was in grad school I had to take a mandatory stats class, and could pick any that qualified at the university (it wasn't taught in my own department). Purely by chance I ended up in the class populated mostly by people from psych, also doing their one mandatory class. Not only was it a fairly cursory overview of a wide and complicated field, but the psych grad students openly cheated. The rest of us muddled along and probably didn't learn anything more, but at least did our own work.
It was my first year and the cheating was so blatant and open that it was clear that the prof knew and didn't care, so I just put my head down and finished the class; I still regret not having said something, though. It's the only dishonesty I saw in my entire grad school experience other than undergrads plagiarizing (and hearing about one dumbass who got caught plagiarizing his entire dissertation).
So that is just one anecdote, but it has left me feeling profoundly unsurprised when I read about ethical and statistical problems in that field.
posted by Dip Flash at 5:40 AM on May 20, 2014
It was my first year and the cheating was so blatant and open that it was clear that the prof knew and didn't care, so I just put my head down and finished the class; I still regret not having said something, though. It's the only dishonesty I saw in my entire grad school experience other than undergrads plagiarizing (and hearing about one dumbass who got caught plagiarizing his entire dissertation).
So that is just one anecdote, but it has left me feeling profoundly unsurprised when I read about ethical and statistical problems in that field.
posted by Dip Flash at 5:40 AM on May 20, 2014
Former social psychologist here (I'm now in IT. Long story), and there's no argument from me that stats taught in psych faculties are lightweight. That being said, I was also taught to look at published research very skeptically (one daily assignment was to find all the statistical errors in a randomly selected social psych paper), and am saddened to hear that this wasn't universal.
posted by Mogur at 5:53 AM on May 20, 2014
posted by Mogur at 5:53 AM on May 20, 2014
The line about Stapel's dread of the "null finding", because a study isn't exciting if it doesn't prove something new, took me back to my first year in grad school trying to play catch-up in philosophy of science.
It seems like the popularized version of many studies makes the mistake of assuming that scientists are able to prove new things ... Single study or no, I think if we're careful we read scientific results as trying and failing to disprove hypotheses, right? And the more times one cannot disprove a hypothesis, the closer it comes to being a sound theory, but it never can be proved "true".
I'm just an old country philosopher, so I can't help but wonder whether Popper's insistence on the limits of induction is more telling on this issue than p-values and other statistical magic.
posted by allthinky at 5:55 AM on May 20, 2014 [2 favorites]
It seems like the popularized version of many studies makes the mistake of assuming that scientists are able to prove new things ... Single study or no, I think if we're careful we read scientific results as trying and failing to disprove hypotheses, right? And the more times one cannot disprove a hypothesis, the closer it comes to being a sound theory, but it never can be proved "true".
I'm just an old country philosopher, so I can't help but wonder whether Popper's insistence on the limits of induction is more telling on this issue than p-values and other statistical magic.
posted by allthinky at 5:55 AM on May 20, 2014 [2 favorites]
This is a pretty good overview of a real problem, but it also makes it seem universal and like a huge crisis for the field. I don't think this is fair, for some of the reasons given by others. First, throughout, most social psychologists have been doing good work, and the same terrible cases keeping being brought out to tar everyone. Second, these scandals are real, but in the short time since they happened, there has been real effort to address them, as one would expect in science. Buried in the article, for example, are the results of replication projects that back up many of the key studies in the field. Even priming, the article's whipping-boy, has a large number of good studies showing effects - but it also seems to be very dependent on conditions, and some studies are better than others.
Meanwhile, half the thread seems to be engaging in exactly the sort of bad statistical reasoning critiqued here: "For X personal reason, social psychologists [don't know statistics/are lying/are doing bad work]." Seriously?
For what it is worth, I am not a social psychologist, but I am in a closely related field (economic sociology) and I work with many of them. There is the usual mix of amazing work/mediocre work/bad work. A lot of time is spent sorting these out from each other in seminars and by reviewers - a typical paper in a top journal will take 2-3 years to publish and be subject to many revisions by three anonymous reviewers and two editors, if not more. Sometimes they mess up, sometimes people lie, etc. but the article makes the whole thing seem like a vague conspiracy. It is not. Additionally, the statistical training for people who want to do top tier work is usually pretty rigorous. No one is reporting raw p-values based on naive OLS regressions and just getting things published in good journals.
Now, at the same time, the critiques about technique are very true - there is lots of wiggle room in how social science research is done, and that allows both errors and biases to creep in. The lack of places to publish null results is problematic - though I love that this journal is trying. We try to be vigilant and careful, but social science is messy and highly contingent. While we need to get better about this, science is (hopefully) progressive, and we are getting better at both improving method and discovering (in fits and starts) how people operate, which is exactly what the scientific method is about.
posted by blahblahblah at 6:13 AM on May 20, 2014 [6 favorites]
Meanwhile, half the thread seems to be engaging in exactly the sort of bad statistical reasoning critiqued here: "For X personal reason, social psychologists [don't know statistics/are lying/are doing bad work]." Seriously?
For what it is worth, I am not a social psychologist, but I am in a closely related field (economic sociology) and I work with many of them. There is the usual mix of amazing work/mediocre work/bad work. A lot of time is spent sorting these out from each other in seminars and by reviewers - a typical paper in a top journal will take 2-3 years to publish and be subject to many revisions by three anonymous reviewers and two editors, if not more. Sometimes they mess up, sometimes people lie, etc. but the article makes the whole thing seem like a vague conspiracy. It is not. Additionally, the statistical training for people who want to do top tier work is usually pretty rigorous. No one is reporting raw p-values based on naive OLS regressions and just getting things published in good journals.
Now, at the same time, the critiques about technique are very true - there is lots of wiggle room in how social science research is done, and that allows both errors and biases to creep in. The lack of places to publish null results is problematic - though I love that this journal is trying. We try to be vigilant and careful, but social science is messy and highly contingent. While we need to get better about this, science is (hopefully) progressive, and we are getting better at both improving method and discovering (in fits and starts) how people operate, which is exactly what the scientific method is about.
posted by blahblahblah at 6:13 AM on May 20, 2014 [6 favorites]
Apologies in advance for getting a bit rant-y
Psych is not the same thing as social psych. Serial position effect - the idea that you remember things from the beginning and end of lists better than things in the middle? There are obviously moderators on that, but it's a pretty darn replicable psych effect. Human psychophysics, operant and classical conditioning, visual processing, prospect theory, parallel distributed processing, child development, language acquisition - all can fall under psych. "Behavioral econ", as Kahneman has argued, is really pretty much psych done by people in another department, occasionally under the (questionable) assumption that people are rational. Look at a psych department today and you'll find neuroscientists, clinicians, sociologists, statisticians AND social psychologists. There are many exciting ways to study both how AND why people do what they do, and it can be done well.
Also, I'm terribly embarrassed to Godwin the thread, but one of the major roots of social psychology arose from the second world war, both in terms of designing propaganda, but also in terms of understanding why people who had seemed not-evil would go along with the Nazis. That's why so many of the early experiments were on conformity to social pressures and persuasion. (Also, Cialdini's results replicate pretty well.) I'm not at all arguing against the idea that many of the studies in the field are broken (they are, they suck, and it's frustrating), but there's a difference between a bad trend in a useful field and a bad field.
Many people have touched on one of the biggest problems - often papers that do it right, use novel techniques, or don't follow the "I totally predicted this from the start" narrative get stuffed before getting published because they don't look as pretty. Real science is ugly as hell.
(ooh, on preview, also what blahblahblah said)
posted by synapse at 6:18 AM on May 20, 2014 [6 favorites]
Psych is not the same thing as social psych. Serial position effect - the idea that you remember things from the beginning and end of lists better than things in the middle? There are obviously moderators on that, but it's a pretty darn replicable psych effect. Human psychophysics, operant and classical conditioning, visual processing, prospect theory, parallel distributed processing, child development, language acquisition - all can fall under psych. "Behavioral econ", as Kahneman has argued, is really pretty much psych done by people in another department, occasionally under the (questionable) assumption that people are rational. Look at a psych department today and you'll find neuroscientists, clinicians, sociologists, statisticians AND social psychologists. There are many exciting ways to study both how AND why people do what they do, and it can be done well.
Also, I'm terribly embarrassed to Godwin the thread, but one of the major roots of social psychology arose from the second world war, both in terms of designing propaganda, but also in terms of understanding why people who had seemed not-evil would go along with the Nazis. That's why so many of the early experiments were on conformity to social pressures and persuasion. (Also, Cialdini's results replicate pretty well.) I'm not at all arguing against the idea that many of the studies in the field are broken (they are, they suck, and it's frustrating), but there's a difference between a bad trend in a useful field and a bad field.
Many people have touched on one of the biggest problems - often papers that do it right, use novel techniques, or don't follow the "I totally predicted this from the start" narrative get stuffed before getting published because they don't look as pretty. Real science is ugly as hell.
(ooh, on preview, also what blahblahblah said)
posted by synapse at 6:18 AM on May 20, 2014 [6 favorites]
I just read the most fascinating study. If you expose a social psychologist to bull feces, he or she is twice as likely to fabricate research results for the next several hours!
posted by vorpal bunny at 6:23 AM on May 20, 2014 [1 favorite]
posted by vorpal bunny at 6:23 AM on May 20, 2014 [1 favorite]
blahblahblah, I hear what you're saying, that it isn't as widespread as the critics deem it to be, but has any other field experienced anything like Fredrickson and Losada positivity ratio debacle? Their 'research' was prima facie absurd, and it was published not just in a peer-reviewed journal, but the flagship publication of the field. I simply can't imagine anything like that happening in, say, the American Political Science Review.
posted by MisantropicPainforest at 6:23 AM on May 20, 2014 [1 favorite]
posted by MisantropicPainforest at 6:23 AM on May 20, 2014 [1 favorite]
I wrote a whole chapter about p-values and their perils in the book I just finished, and now I see how hard it is to explain! But on the upside, it gave me the opportunity to title a section "Doctor, it hurts when I p."
posted by escabeche at 6:38 AM on May 20, 2014 [9 favorites]
posted by escabeche at 6:38 AM on May 20, 2014 [9 favorites]
Also, I'm terribly embarrassed to Godwin the thread
Why? You didn't.
posted by thelonius at 6:44 AM on May 20, 2014 [1 favorite]
Why? You didn't.
posted by thelonius at 6:44 AM on May 20, 2014 [1 favorite]
The p-value is a probablity, and at a 5% significance, level, it means that if no effect were happening, there is only a 5 percent chance of getting the significant results that you did.
one part of why the first is wrong and the second is right is that 'similar result' should be 'at least as large'--for example, the result could be dissimilar but at least as large, say 0.0000001.
In other words, and this mimics how statistics is written out, the p-value says: given the data we have, what is the probability that there is no effect?
Nope. It's rather, given that there is no effect, what is the probability of observing our data (or something more extreme).
These are different probabilities. Given that you are a natural-born U.S. citizen, the probability that you are the President is infinitesimal. Given that you are the President, the probability that you are a natural-born U.S. citizen is 1.
I tried to explain p-values here.
posted by a snickering nuthatch at 6:47 AM on May 20, 2014 [6 favorites]
one part of why the first is wrong and the second is right is that 'similar result' should be 'at least as large'--for example, the result could be dissimilar but at least as large, say 0.0000001.
In other words, and this mimics how statistics is written out, the p-value says: given the data we have, what is the probability that there is no effect?
Nope. It's rather, given that there is no effect, what is the probability of observing our data (or something more extreme).
These are different probabilities. Given that you are a natural-born U.S. citizen, the probability that you are the President is infinitesimal. Given that you are the President, the probability that you are a natural-born U.S. citizen is 1.
I tried to explain p-values here.
posted by a snickering nuthatch at 6:47 AM on May 20, 2014 [6 favorites]
Let this be a lesson to not confuse bayesian and frequentist statistics.
posted by MisantropicPainforest at 6:52 AM on May 20, 2014 [2 favorites]
posted by MisantropicPainforest at 6:52 AM on May 20, 2014 [2 favorites]
I studied psych as an undergrad and used to read experiments in my free time and so many of the studies were so flawed that I wondered how they ever got that far in the first place. It didn't take an expert to see something was fishy with certain experiments. The ones that were good where great and the ones that were bad were shockingly bad.
Then I went into journalism and I saw a lot of the same problems dog it, too. I think researching is a delicate science and being able to separate lies from truth takes a different approach as does understanding that opinion and beliefs are not the same as truth.
The impression I got when reading so many of those studies is that the experimenter had something to prove rather than merely having something to discover (a hypothesis that could be tested, no doubt).
But it's not just social science that needs a refresher in how to search for truths -- that needs to be a discipline all on its own...
posted by Alexandra Kitty at 7:06 AM on May 20, 2014 [1 favorite]
Then I went into journalism and I saw a lot of the same problems dog it, too. I think researching is a delicate science and being able to separate lies from truth takes a different approach as does understanding that opinion and beliefs are not the same as truth.
The impression I got when reading so many of those studies is that the experimenter had something to prove rather than merely having something to discover (a hypothesis that could be tested, no doubt).
But it's not just social science that needs a refresher in how to search for truths -- that needs to be a discipline all on its own...
posted by Alexandra Kitty at 7:06 AM on May 20, 2014 [1 favorite]
Interestingly, I've been thinking in the opposite direction. The methodological and ethical problems in trying to construct causal models among variables within slushy, uncontrollable, and chaotic environments are likely intractable. Rather than attempt to put sandbags into attempts to build correlations that work in this context but are swamped by other effects in that context, social sciences should embrace the qualitative. Treat contexts as singletons and build rich descriptions using multiple sources and lines of evidence.
posted by CBrachyrhynchos at 7:09 AM on May 20, 2014 [4 favorites]
posted by CBrachyrhynchos at 7:09 AM on May 20, 2014 [4 favorites]
Let this be a lesson to not confuse bayesian and frequentist statistics.
For most of the social psych studies that have been or will be invalidated in the coming years, this distinction won't be that important. Most (although certainly not all) highly significant results will also display substantial Bayes Factors (relative to a reasonable alternative hypothesis), for example. This is especially true of the many thousands of studies whose samples consist of fewer than a hundred college undergrads. Sampling bias and cohort effects in such studies are much, much more problematic. Even if social psychology entirely cleans house in terms of its statistical acumen and methodological rigor, its ubiquitous use of convenience samples damningly limits the validity of its findings.
posted by belarius at 7:10 AM on May 20, 2014 [2 favorites]
For most of the social psych studies that have been or will be invalidated in the coming years, this distinction won't be that important. Most (although certainly not all) highly significant results will also display substantial Bayes Factors (relative to a reasonable alternative hypothesis), for example. This is especially true of the many thousands of studies whose samples consist of fewer than a hundred college undergrads. Sampling bias and cohort effects in such studies are much, much more problematic. Even if social psychology entirely cleans house in terms of its statistical acumen and methodological rigor, its ubiquitous use of convenience samples damningly limits the validity of its findings.
posted by belarius at 7:10 AM on May 20, 2014 [2 favorites]
Not so much the validity as the generalizability, i.e. the results might be perfectly valid...but only for the narrow population known as American college students.
posted by obliterati at 7:18 AM on May 20, 2014 [4 favorites]
posted by obliterati at 7:18 AM on May 20, 2014 [4 favorites]
When it comes to academic journals my rule of thumb is - does it involve IV estimators? Skip it and move on.
posted by playertobenamedlater at 7:23 AM on May 20, 2014
posted by playertobenamedlater at 7:23 AM on May 20, 2014
Not so much the validity as the generalizability, i.e. the results might be perfectly valid...but only for the narrow population known as American college students.
Fair. In practice, the claims such studies make tend to be vastly more general than the narrowness of the sample justifies (not just American college students, but students from that institution in a particular historical moment, which are more homogeneous than the wider sample of college students and recent graduates). My intent was to question was the validity of those general claims.
posted by belarius at 7:24 AM on May 20, 2014
Fair. In practice, the claims such studies make tend to be vastly more general than the narrowness of the sample justifies (not just American college students, but students from that institution in a particular historical moment, which are more homogeneous than the wider sample of college students and recent graduates). My intent was to question was the validity of those general claims.
posted by belarius at 7:24 AM on May 20, 2014
No argument from me on that point. It's always baffled me how casually that serious issue gets brushed aside. I understand that there are a lot of practical constraints in terms of time and resources, but qualifying your results doesn't seem difficult or outrageous.
posted by obliterati at 7:30 AM on May 20, 2014
posted by obliterati at 7:30 AM on May 20, 2014
I think it remains to be seen to what degree the use of convenience samples is problematic. It may be that for certain small effects, with unknown moderators, they will only work with a given subset of american college students. Most of the effects I've seen in various cultural psych studies are tiny. For instance, the whole west=individualist east=collectivist thing is a tiny effect and it struck me as ludicrous to think that this is any grounds to argue that psychology on americans won't generalize. Just because you can find differences across cultures or age cohorts, doesn't mean they matter for all psychological phenomena. We don't need a different theory of visual perception, attention, memory or motor control for every age/culture/gender group. Social psychology may be more sensitive to these things, but it's a bit unscientific to write off the field on the hunch that it will be.
posted by Smegoid at 7:48 AM on May 20, 2014 [1 favorite]
posted by Smegoid at 7:48 AM on May 20, 2014 [1 favorite]
...as if to rescue the scientific method from embarrassment...
No, to rescue their field from embarrassment. The scientific method has no reason to be embarrassed.
posted by Mental Wimp at 8:11 AM on May 20, 2014
No, to rescue their field from embarrassment. The scientific method has no reason to be embarrassed.
posted by Mental Wimp at 8:11 AM on May 20, 2014
I simply can't imagine anything like that happening in, say, the American Political Science Review.
Your PhD program will beat this tendency out of you.
posted by ROU_Xenophobe at 8:33 AM on May 20, 2014 [5 favorites]
Your PhD program will beat this tendency out of you.
posted by ROU_Xenophobe at 8:33 AM on May 20, 2014 [5 favorites]
We don't need a different theory of visual perception, attention, memory or motor control for every age/culture/gender group.
It's not that we need different theories for each group. But our theories should be informed by and inclusive enough to handle group differences.
For example, this illusion probably works on you (in that the lines appear to have different lengths). But it doesn't work on people from certain cultures. Any account of how our visual system works to produce this result must contend with that fact, so it's a good thing we checked.
posted by a snickering nuthatch at 8:39 AM on May 20, 2014 [1 favorite]
It's not that we need different theories for each group. But our theories should be informed by and inclusive enough to handle group differences.
For example, this illusion probably works on you (in that the lines appear to have different lengths). But it doesn't work on people from certain cultures. Any account of how our visual system works to produce this result must contend with that fact, so it's a good thing we checked.
posted by a snickering nuthatch at 8:39 AM on May 20, 2014 [1 favorite]
I simply can't imagine anything like that happening in, say, the American Political Science Review.
One of my first jobs was writing comprehensive reviews of a few subdiciplines over a period of three or four years. I looked at hundreds of papers, reworking their results, etc... I'd guess that at least a quarter or so had serious, this-should-have-not-been-published errors in their data analysis. Non-peer reviewed reports were worst, but these were common even in the flagship journals.
posted by bonehead at 8:47 AM on May 20, 2014 [1 favorite]
One of my first jobs was writing comprehensive reviews of a few subdiciplines over a period of three or four years. I looked at hundreds of papers, reworking their results, etc... I'd guess that at least a quarter or so had serious, this-should-have-not-been-published errors in their data analysis. Non-peer reviewed reports were worst, but these were common even in the flagship journals.
posted by bonehead at 8:47 AM on May 20, 2014 [1 favorite]
(I should have said "illusion" or "misconception." Either way, they will peel back the foreskin of ignorance and apply the wire brush of enlightenment.)
posted by ROU_Xenophobe at 9:35 AM on May 20, 2014 [1 favorite]
posted by ROU_Xenophobe at 9:35 AM on May 20, 2014 [1 favorite]
This is also a good place to say that, for all the deserved lumps the guy has gotten, Jonah Lehrer was, I think, the first guy to really talk about this stuff in a big high-profile US publication, and deserves credit for that.
posted by escabeche at 10:19 PM on May 20, 2014
posted by escabeche at 10:19 PM on May 20, 2014
assiduously rooting out fraud and error, as if to rescue the scientific method from embarrassment
It's almost... self-correcting.
posted by Rykey at 5:36 PM on May 21, 2014 [1 favorite]
It's almost... self-correcting.
posted by Rykey at 5:36 PM on May 21, 2014 [1 favorite]
« Older The Racoon Not Only Talks, He Also Adjusts Himself | There is one argument for doing a thing; the rest... Newer »
This thread has been archived and is closed to new comments
You sit there and hear people talk about the derivation of latent structure in psychometrics with PCA and you become very disappointed. Especially given that they always give names to them and stuff.
More grousing, technical in nature, here.
Notably, though, in this article, this guy put some effort into telling people what p-value actually is, and he gets close ("effect at least as large" seems to conflate effect size and p-value: you can get low p-values from small effect, you just have tons of data, but he may not be using the technical definition of "effect").
It seems like people act like they're doing Bayesian statistics when they're doing the normal stuff, a lot of the time, which of course leads to terribad stuff.
posted by curuinor at 12:37 AM on May 20, 2014 [1 favorite]