If you wanted to be more objective about student and professor evaluation, you would have standardized measures of student performance across professors. In the rare case in which this is done, we learn all sorts of fascinating things, including things which raise questions about the unintended consequences of our evaluation systems.
Tyler Cowen points me to a paper in the Journal of Political Economy, by Scott E. Carrell and James E. West [ungated version].
In the U.S. Airforce Academy students are randomly assigned to professors but all take the same final exam. What makes the data really interesting is that there are mandatory follow-up courses so you can see the relationship between which Calculus I professor you had, and your performance in Calculus II! Here’s the summary sentence that Tyler quotes:
The overall pattern of the results shows that students of less experienced and less qualified professors perform significantly better in the contemporaneous course being taught. In contrast, the students of more experienced and more highly qualified introductory professors perform significantly better in the follow-on courses.
Here’s a nice graph from the paper:
Student evaluations, unsurprisingly, laud the professors who raise performance in the initial course. The surprising thing is that this is negatively correlated with later performance. In my post on Babcock’s and Marks’ research, I touched on the possible unintended consequences of student evaluations of professors. This paper gives new reasons for concern (not to mention much additional evidence, e.g. that physical attractiveness strongly boosts student evaluations).
That said, the scary thing is that even with random assignment, rich data, and careful analysis there are multiple, quite different, explanations.
The obvious first possibility is that inexperienced professors, (perhaps under pressure to get good teaching evaluations) focus strictly on teaching students what they need to know for good grades. More experienced professors teach a broader curriculum, the benefits of which you might take on faith but needn’t because their students do better in the follow-up course!
But the authors mention a couple other possibilities:
For example, introductory professors who “teach to the test” may induce students to exert less study effort in follow-on related courses. This may occur due to a false signal of one’s own ability or from an erroneous expectation of how follow-on courses will be taught by other professors. A final, more cynical, explanation could also relate to student effort. Students of low value added professors in the introductory course may increase effort in follow-on courses to help “erase” their lower than expected grade in the introductory course.
Indeed, I think there is a broader phenomenon. Professors who are “good” by almost any objective measure, will have induced their students to put more time and effort into their course. How much this takes away from students efforts in other courses is an essential question I have never seen addressed. Perhaps additional analysis of the data could shed some light on this.
Carrell, S., & West, J. (2010). Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors Journal of Political Economy, 118 (3), 409-432 DOI: 10.1086/653808
Added: Jeff Ely has an interesting take: In Defense of Teacher Evaluations.
Added 6/17: Another interesting take from Forest Hinton.