a case for single-blind review

(Cross posted from here)
When i was in grad school, at one of the academic meetings i regularly participate in, it became regular fare for 2 particular folks in my circles to engage in a prolonged debate about how we should overhaul the academic publishing system. This was so regular (i recall them having portions of this debate for 3 consecutive years over dinner) that the grad students in the bunch thought of this as a grenade in our back pockets we could toss into the fray if ever conversations took an unwelcome turn to the boring. I bring this up because there are lots of aspects of this process that i have quite a few thoughts on, but have never really formalized them too much more than is required for such elongated dinner conversations. And one particular aspect of that was raised on Facebook yesterday by a colleague – asking about the merits of single blind review. I started my answer there, but wanted to engage this a little more fully. So, i’m going to start a series of posts (not sure how many there will be at this point) on the publication/review process here, that i think could be interesting discussions. I hope others will chime in with opinions, questions, etc. These posts will likely be slightly longer than typical fare around here. I expect that some of my thoughts on these will be much more formulated than others.

So, let’s start with a case for single blind review. I think think there are quite a few merits to single blind review (for a few other takes, see here and here). I won’t presume to cover them all here, but i will get a start. Feel free to add others, or tell me i’m completely off my rocker in the comments.

Advertisement

Professor Quality and Professor Evaluation

If you wanted to be more objective about student and professor evaluation, you would have standardized measures of student performance across professors.  In the rare case in which this is done, we learn all sorts of fascinating things, including things which raise questions about the unintended consequences of our evaluation systems.

Tyler Cowen points me to a paper in the Journal of Political Economy, by Scott E. Carrell and James E. West [ungated version].

In the U.S. Airforce Academy students are randomly assigned to professors but all take the same final exam.  What makes the data really interesting is that there are mandatory follow-up courses so you can see the relationship between which Calculus I professor you had, and your performance in Calculus II!  Here’s the summary sentence that Tyler quotes:

The overall pattern of the results shows that students of less experienced and less qualified professors perform significantly better in the contemporaneous course being taught.  In contrast, the students of more experienced and more highly qualified introductory professors perform significantly better in the follow-on courses.

Here’s a nice graph from the paper:

Student evaluations, unsurprisingly, laud the professors who raise performance in the initial course.  The surprising thing is that this is negatively correlated with later performance.  In my post on Babcock’s and Marks’ research, I touched on the possible unintended consequences of student evaluations of professors.  This paper gives new reasons for concern (not to mention much additional evidence, e.g. that physical attractiveness strongly boosts student evaluations).

That said, the scary thing is that even with random assignment, rich data, and careful analysis there are multiple, quite different, explanations.

The obvious first possibility is that inexperienced professors, (perhaps under pressure to get good teaching evaluations) focus strictly on teaching students what they need to know for good grades.  More experienced professors teach a broader curriculum, the benefits of which you might take on faith but needn’t because their students do better in the follow-up course!

But the authors mention a couple other possibilities:

For example, introductory professors who “teach to the test” may induce students to exert less study effort in follow-on related courses.  This may occur due to a false signal of one’s own ability or from an erroneous expectation of how follow-on courses will be taught by other professors.  A final, more cynical, explanation could also relate to student effort.  Students of low value added professors in the introductory course may increase effort in follow-on courses to help “erase” their lower than expected grade in the introductory course.

Indeed, I think there is a broader phenomenon.  Professors who are “good” by almost any objective measure, will have induced their students to put more time and effort into their course.  How much this takes away from students efforts in other courses is an essential question I have never seen addressed.  Perhaps additional analysis of the data could shed some light on this.

Carrell, S., & West, J. (2010). Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors Journal of Political Economy, 118 (3), 409-432 DOI: 10.1086/653808

Added: Jeff Ely has an interesting take: In Defense of Teacher Evaluations.

Added 6/17: Another interesting take from Forest Hinton.

Babcock replies on College Slackers

Philip Babcock was kind enough to reply to my previous post about his research.  This is the second time a scholar I don’t know personally has responded to a blog post I wrote.*  How excellent!  Let me take this occasion to say explicitly something I was thinking, and should have emphasized, when I initially wrote the post.**  I believe Babcock’s and Marks’s central finding, that college students spend much less time studying than they did in the past, is an important discovery.  Sure, some scholars of education must have had an idea that study time has been declining, but when one considers how many numbers have been crunched and how much ink has been spilled in the name of understanding education, it is shocking to realize that a question as fundamental as the amount of time students spend studying has been paid so little attention.  The authors deserve a great deal of credit for tracking down multiple datasets in an attempt to answer an important question.  Important follow-up questions include: why? and, how should we feel about it?  See the old post for a little discussion of those issues.

*Should I email someone every time I discuss their work?  I tried that for one of the posts on this blog and got no reply.

**I think it is enormously important to criticize and attach qualifications to other people’s research, in fact, I think social science suffers from too little good criticism.  But too little appreciation may be an equally big problem.

Quantum Psychology?

Let me be frank; I think “The conjunction fallacy and interference effects” (ungated version) is a horrible misuse of math and indicates an embarrassing failure of peer review.

The author, Riccardo Franco, introduces a parameter that does doesn’t have any foundation in the phenomena it is trying to explain, nor is it shown to aid in modeling.

Please tell me I’m missing something.

What?  You’ve never heard of the conjunction fallacy? It is yet another cognitive bias studied by Amos Tversky and Daniel Kahneman.  They gave people the following problem (quoting from Wikipedia):

Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.

Which is more probable?

1. Linda is a bank teller.
2. Linda is a bank teller and is active in the feminist movement.

Forced to Defend Rational Choice Theory

I’m finally responding to Eli Thorkelson, who asked for my comments on an article by feminist economist Julie Nelson.  The article is a critique of rational choice theory (RCT) and I think it has some omissions and misleading claims.

I regularly come across particular instances of rational choice theorizing that I dislike, but non-economists, including sociologists, often dismiss rational choice theory without understanding it, so when the topic comes up among non-economists, I almost inevitably find myself defending it.  My claim is that rational choice theory, broadly construed, is an important, though certainly not the only, useful framework for understanding human behavior.  This should be considered an utterly boring claim.  What is interesting is how any social scientist could deny it…

My Old Book Review of Six Degrees

I wrote a review of Duncan Watts book on social networks, for a class and thought I might as well share it here, even if it is a little out of date:

Networks are everywhere.  In the first chapter of Six Degrees Duncan Watts notes that gossip, power outages, epidemics, even properties of the human brain such as consciousness all emerge from the interaction of their constituent elements.  Having provided this motivation, Watts spends much of first half of the book discussing what he knows best, “small world” networks.  In the second half he presents a network perspective for a wide range of topics such as…