Here’s an interesting paper (may require login) from the Journal of the Royal Society of Medicine. From the abstract:
Design 607 peer reviewers at the BMJ were randomized to two intervention groups receiving different types of training (face-to-face training or a self-taught package) and a control group. Each reviewer was sent the same three test papers over the study period, each of which had nine major and five minor methodological errors inserted.
Results The number of major errors detected varied over the three papers.The interventions had small effects. At baseline (Paper 1) reviewers found an average of 2.58 of the nine major errors, with no notable difference between the groups.The mean number of errors reported was similar for the second and third papers, 2.71 and 3.0, respectively. Biased randomization was the error detected most frequently in all three papers, with over 60% of reviewers rejecting the papers identifying this error. Reviewers who did not reject the papers found fewer errors and the proportion finding biased randomization was less than 40% for each paper.
The thing is, i am having a relatively difficult time convincing myself that the comparison they made is the interesting one. When reviewing a paper, are we really ever looking for all of the errors in the piece or just enough to sufficiently determine whether to accept/reject the article? So, how interesting is the difference in the “number of errors found” among those who rejected the paper? To me, not very. This doesn’t undermine their conclusion:
Conclusions Editors should not assume that reviewers will detect most major errors, particularly those concerned with the context of study. Short training packages have only a slight impact on improving error detection.
My question is do you find the question interesting, or would you have sliced the data a different way?
(HT: Michelle Poulin)
Permutations 
Interesting post! I think the interesting thing about this paper have little to do with the experiment. Its the descriptive part that is valuable.
To address your question, while there is only so much work a reviewer can put into a review, and while putting less work into rejection recommendations is understandable, it certainly seems like the more errors the reviewer finds the better – its a chance for the authors of the paper to learn something and stop a mistake from being published at another journal.
Personally, I think we can really improve on the way peer review is done, and I’ve given some thought to a better system, but that I shouldn’t get into that right now.
Yeah, we’ve all got our “how i’d overhaul peer review” ideas…maybe for another time.
But as for how that relates to this post, “the more errors the reviewer finds the better” …the better for whom? From a reviewer’s point of view i’m not sure i agree. If i’ve reached a point of having found enough holes that convince me a paper is unpublishable, why should i continue hunting for more?
I’m not saying that it always worth the additional effort to hunt for more errors. I’m just making the seemingly simple observation that, all else equal, since authors benefit from people identifying their errors, finding more errors is better.
i get your point, i just don’t think it’s necessarily that simple. It all comes down to whether you think the primary goal of peer review is (a) to shepherd “good” research through to publication or (b) to gatekeep the bad stuff out. Thus the “all else equal” bit has a very different angle depending on which is the aim. It is increasingly my opinion* (with some, limited, experience now in both roles) that editors’ business in the peer review process is (a), and thus for them, i would agree with your statement. But for reviewers, i think it’s primarily (b), and if so “all else equal” i’d argue their aim would be to stop as soon as the it’s clear the bar can’t be cleared, allowing their time open for more reviews (or more importantly, their own work).
*Though i am more than willing to have this view changed.