"Reject – More holes than my grandad’s string vest!"

December 12, 2014 9:50 AM

Shit My Reviewers Say (SLTumblr)
posted by capricorn (20 comments total)

My favorite so far (I read this in my internal Herzog voice): "The findings are not novel and the solution induces despair. "
posted by el io at 9:56 AM on December 12, 2014 [3 favorites]

What do the authors mean by one standard error?

And

The reported mean of 7.7 is misleading because it appears that close to half of your participants are scoring below that mean

Statistics- how do they work?
posted by damayanti at 10:00 AM on December 12, 2014 [10 favorites]

I can only read 7 before the crying outweighs the laughing. What is wrong with people?! The reviewing process needs an overhaul.
posted by hydrobatidae at 10:16 AM on December 12, 2014 [1 favorite]

The rest of this review operates from the assumption that this paper is a sincere attempt at scientific evidence and argument.

I'm keeping that one in my back pocket.
posted by backseatpilot at 10:29 AM on December 12, 2014 [4 favorites]

"What do the authors mean by one standard error?"

Augghhh, what?! In what universe is this person qualified to review, well, basically anything with statistics involved?
posted by dialetheia at 10:32 AM on December 12, 2014 [2 favorites]

Many of the comments that were posted actually seem reasonable. Is it the authors themselves who are posting the comments? I'd be embarrassed.

For example:

The statistical analyses were not correct. Actually they were so confused that I lost all confidence in the analyses and data presentation.

That's totally a reasonable comment for a referee to make in a report.

And

This kind of prose simply borders on cruelty against the reader. And finally comes the conclusion, which is the intellectual equivalent of bubblegum.

is maybe not as diplomatically phrased as might be ideal, but is also totally a legitimate response (if indeed the paper is poorly written).
posted by leahwrenn at 10:45 AM on December 12, 2014 [3 favorites]

I did an article review today, thankfully of a good paper. This is not as common as one might hope. Some idiots aside my sympathy is very much with the reviewers while reading these, I have felt the way some of these reviewers feel while looking at papers - the appalling English, the ludicrously over-aggrandising claims for global importance of results based on data from one location, the insane claims that a premise is proven since it is hinted at in some obscure online journal and thus can form the basis for incredibly wide-ranging suppositions. I do tend to start being diplomatic with poorer papers but hacking through 10-15,000 words of drivel is wearing when you know you can be doing something else. My urge to be helpful can morph into a completionist streak that means I keep adding criticism when sometimes I need to just say "This doesn't work for X,Y and Z reasons". The competent work-a-day stuff is good for keeping up to speed on but can be quite quick to review. The really interesting stuff can often take as long as the crap, a really exciting set of ideas forces you to really dig in and engage and can be fascinating but is not very common, certainly not as common as the crap from article mills at some institutions.

I liked this line especially:

“I am afraid this manuscript may contribute not so much towards the field’s advancement as much as towards its eventual demise.”

I cannot tell you how fast and how long I would howl for if I got this one on feedback:

“I understand that Wikipedia is not the best source for my information, however, I don’t have access to the [peer-reviewed] literature you cite, and based on the information from Wikipedia, your hypothesis breaks down."

It makes me wonder how lousy a journal it must be for the editor to have let that get back to the author.

The worst feedback I have had was some one who thought my already overly long paper needed to consider some history underlying the policy it was all about. They then recommended five papers that would fit the bill, only one of them really saying anything about the history, the rest highly technical engineering papers. All, perhaps not coincidentally, were by the same author. And not an important author either. When I looked them up, one had been withdrawn by its publisher (the journal I was submitting to) for plagiarism. So yup, I will agree that the review system needs an overhaul.

To be honest, worse than article reviews are bid reviews. Loads of work, all chances of promotion hang on them and they send them to someone whose comments indicate they know about as much as your cat about the specifics of your field and the context of the ideas you want to develop.
posted by biffa at 11:06 AM on December 12, 2014 [3 favorites]

Like biffa, I guffawed a lot at a bunch of these, but others struck me as everything from reasonable to at least understandable. (If the data or statistical analysis is wrong, then...it's wrong?) I've refereed articles that reinvented the wheel because the author hadn't done even the most cursory research on the topic, and the most sugar-coated response to that kind of work still won't make the author happy. Some of the quotations on that site fell under the heading of the sort of thing I might communicate separately to the editor, but not to the author.

I've been pretty lucky: I've benefited from most readers' reports I've received, even if they have made me say "ow" on occasion. That being said, I remain extremely fond of what one referee from Extremely Prestigious Press had to say about Book One: "too convincing for the target audience."
posted by thomas j wise at 11:38 AM on December 12, 2014 [3 favorites]

I don't think the intent of this is supposed to be 100% "look at how dumb/bad these reviewers are" -- some of them seem to be legitimate admission from the paper's author that the reviewer schooled them.
posted by axiom at 12:16 PM on December 12, 2014 [1 favorite]

If this topic were not dear to my heart, I would perhaps have struggled to follow your logic. I applaud the effort, and I find the data reviews relatively interesting and useful.



Shaaaaaaade.
posted by TheWhiteSkull at 12:25 PM on December 12, 2014

That's totally a reasonable comment for a referee to make in a report.

Reasonable but shitty. Doing peer review well is really fucking hard, precisely and pretty much exclusively because of this problem — it's so, so much easier to just note flaws unhelpfully than it is to figure out how to fix them. Writing helpful rejections is one of the hardest, and least rewarded, things in academia.

But yeah, there are still a few mixed in here that seem both reasonable and helpful and presumably some of the point is just that there's still a bunch of potential for hurt feelings even then.
posted by RogerB at 12:28 PM on December 12, 2014 [1 favorite]

I don't think the intent of this is supposed to be 100% "look at how dumb/bad these reviewers are" -- some of them seem to be legitimate admission from the paper's author that the reviewer schooled them.

I think it's more "peer review comments are amusing out of context" - at least that's why I posted the FPP!
posted by capricorn at 12:37 PM on December 12, 2014 [1 favorite]

Some of these a completely inane (the best examples are already picked out above), and some of these are things I could see myself writing. For example:

``This needs some rephrasing—it’s loaded with the assumption that there is a real world" is a completely legitimate criticism in some fields.
posted by yeolcoatl at 1:38 PM on December 12, 2014

Most gratifying reader review I ever got: "Not sexist enough."

That was over a year ago, and I still glow with pride when I think of it.
posted by scaryblackdeath at 1:46 PM on December 12, 2014 [2 favorites]

Ah, referee reports. I work in a field where preprints are the main way to disseminate results. Until the last two or three years, I thought that substantive referee reports were gone -- then I received a few helpful ones that actually improved my work (may there be more in the future).

Myself I think peer reviewing is tremendously important, because it is how science corrects itself; it's part of the entire process and I really do put effort into doing it well. But doing it well takes time, and when that time is being wasted on a complete crap paper-- well, there are times I really want to respond "please don't make me read another paper by these authors because I will break down and cry". Crap papers *do* get submitted (derivative work, not up to the standards of the discipline, bad English, etc). How bad does a paper have to be before it's just not worth my (or anyone's) time trying to make it better?
posted by nat at 6:44 PM on December 12, 2014

Ah, and also, Mr. Nat (also an academic) and I have taken to calling the state when you have finished all your referee reports, and not yet been assigned any new papers to review, "the state of grace". Right now I am in the state of grace, and it is luxurious.
posted by nat at 6:45 PM on December 12, 2014

" You will see that Reviewer 2 has slightly missed the point, so please don’t pay too much attention to their comments in your revision. " The Editor.


#iwilllickthetoecheeseofaneditorthisglorious
posted by lalochezia at 7:00 PM on December 12, 2014

This needs to be standardized!!!

I've written something like this so many times it makes me think SPSS licenses should be as hard to get as driver's licenses.
posted by aaronetc at 9:28 PM on December 12, 2014

Find your inner nerd—it must be a big part of you—bind and gap it and then dump it in the ocean tied to a large rock.
- Referee 1

We regret that some of the remarks made by Referee 1 were not edited before being sent to you.
- Editorial Assistant on behalf of the Editor

Perfection.
posted by Ragini at 10:15 PM on December 12, 2014 [1 favorite]

This discussion is missing a link to Dear Journal Editor, It's Me Again.
posted by parudox at 11:23 PM on December 12, 2014 [3 favorites]