Professor Quality and Professor Evaluation

If you wanted to be more objective about student and professor evaluation, you would have standardized measures of student performance across professors.  In the rare case in which this is done, we learn all sorts of fascinating things, including things which raise questions about the unintended consequences of our evaluation systems.

Tyler Cowen points me to a paper in the Journal of Political Economy, by Scott E. Carrell and James E. West [ungated version].

In the U.S. Airforce Academy students are randomly assigned to professors but all take the same final exam.  What makes the data really interesting is that there are mandatory follow-up courses so you can see the relationship between which Calculus I professor you had, and your performance in Calculus II!  Here’s the summary sentence that Tyler quotes:

The overall pattern of the results shows that students of less experienced and less qualified professors perform significantly better in the contemporaneous course being taught.  In contrast, the students of more experienced and more highly qualified introductory professors perform significantly better in the follow-on courses.

Here’s a nice graph from the paper:

Student evaluations, unsurprisingly, laud the professors who raise performance in the initial course.  The surprising thing is that this is negatively correlated with later performance.  In my post on Babcock’s and Marks’ research, I touched on the possible unintended consequences of student evaluations of professors.  This paper gives new reasons for concern (not to mention much additional evidence, e.g. that physical attractiveness strongly boosts student evaluations).

That said, the scary thing is that even with random assignment, rich data, and careful analysis there are multiple, quite different, explanations.

The obvious first possibility is that inexperienced professors, (perhaps under pressure to get good teaching evaluations) focus strictly on teaching students what they need to know for good grades.  More experienced professors teach a broader curriculum, the benefits of which you might take on faith but needn’t because their students do better in the follow-up course!

But the authors mention a couple other possibilities:

For example, introductory professors who “teach to the test” may induce students to exert less study effort in follow-on related courses.  This may occur due to a false signal of one’s own ability or from an erroneous expectation of how follow-on courses will be taught by other professors.  A final, more cynical, explanation could also relate to student effort.  Students of low value added professors in the introductory course may increase effort in follow-on courses to help “erase” their lower than expected grade in the introductory course.

Indeed, I think there is a broader phenomenon.  Professors who are “good” by almost any objective measure, will have induced their students to put more time and effort into their course.  How much this takes away from students efforts in other courses is an essential question I have never seen addressed.  Perhaps additional analysis of the data could shed some light on this.

Carrell, S., & West, J. (2010). Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors Journal of Political Economy, 118 (3), 409-432 DOI: 10.1086/653808

Added: Jeff Ely has an interesting take: In Defense of Teacher Evaluations.

Added 6/17: Another interesting take from Forest Hinton.

Advertisement

Babcock replies on College Slackers

Philip Babcock was kind enough to reply to my previous post about his research.  This is the second time a scholar I don’t know personally has responded to a blog post I wrote.*  How excellent!  Let me take this occasion to say explicitly something I was thinking, and should have emphasized, when I initially wrote the post.**  I believe Babcock’s and Marks’s central finding, that college students spend much less time studying than they did in the past, is an important discovery.  Sure, some scholars of education must have had an idea that study time has been declining, but when one considers how many numbers have been crunched and how much ink has been spilled in the name of understanding education, it is shocking to realize that a question as fundamental as the amount of time students spend studying has been paid so little attention.  The authors deserve a great deal of credit for tracking down multiple datasets in an attempt to answer an important question.  Important follow-up questions include: why? and, how should we feel about it?  See the old post for a little discussion of those issues.

*Should I email someone every time I discuss their work?  I tried that for one of the posts on this blog and got no reply.

**I think it is enormously important to criticize and attach qualifications to other people’s research, in fact, I think social science suffers from too little good criticism.  But too little appreciation may be an equally big problem.

Preparing the next generation.

If you’re a regular reader or contributor to this blog, you probably agree that mathematics have an indispensable role in the social sciences. Lately, however, I’ve been thinking a lot about something: what kind of mathematical tools do sociologists of the future require?

The reason I’ve been thinking about this is a talented undergraduate who is strongly considering going to grad school in sociology. Interestingly, part of what has moved him in that direction seems to have been two classes he’s taken with me. The first was the required undergraduate statistics class and the second a substantive class that includes a lot of network analysis, structural theory, and associated material. As it turns out, it’s entirely possible to teach Mayhew & Levinger to undergraduates. Who knew? In any case, in this second class he’s gotten a strong sense that sociology involves a lot of math and this excites him. I’m all for it, since this student is very smart and we need more mathematically-gifted grad students. Yesterday during a conversation, though, he asked me what sorts of math he should be thinking about taking as he prepares for graduate school. I gave him my answers- and an regression analysis textbook to work through during winter break- but I wonder what others think.

If you could somehow start over again in the field, what areas of mathematics would you make sure you learned right from the start?

The Frog Pond Effect in Schools

Appearing in the same issue of the ASR as the Condron article I previously discussed, Robert Crosnoe publishes evidence that lower income students suffer some negative academic and psychosocial consequences from attending higher income schools.  He uses propensity score weighting (no silver bullet, but probably the best methodology you could ask for with this data) in an attempt attempt to reduce possible confounding due to different students selecting into different schools.  Putting that issue aside, my question is, how are students’ academic and psychosocial outcomes changing over time?

Most of the outcomes Crosnoe uses (GPA, negative self-image, social isolation, depression) are measured more than once.  He is predicting the later measure, which is appropriate, but why not run models with the lagged dependent variable?

 

Does Class Exist?

In Dennis Condron’s new ASR article on academic achievement he writes:

“The second contribution [of this article] lies in conceptualizing and analyzing social class rather than socioeconomic status (SES).  This reflects the view that children growing up in different positions in the stratification hierarchy have categorically unequal and qualitatively different (rather than continuously graded) life and educational experiences.”

What evidence exists for this claim?  To me, this is the sort of claim that the paper should explore empirically, but let’s go ahead and see how he measures social class:

“I [Condron] code children as middle/upper class if either parent has a bachelor’s degree or higher or works in an executive, administrative, or managerial position and their household income is above the federal poverty line.  Children are coded working class if both parents have less than a bachelor’s degree and do not work in an executive, administrative, or managerial position and their household income is above the poverty line.  Finally, I code students as poor if their household income is below the poverty line, regardless of parents’ education levels and job positions.”

Then he examines the relationship between the class categories he created and academic achievement.

The standard objection to this approach is that it is throwing away data, the class categories hide important differences.  Whether you have one or two parents with a B.A. does not enter the analysis.  If your parents’ income is under the poverty line, their education is ignored.  Income differences above the poverty line are ignored, etc.

If I reanalyzed the data using parental income, education, and occupation, instead of class, I would be able to explain more variation in academic achievement.  There are times when I am more willing to “throw away” data, for example when I’m desperate for more degrees of freedom, but even then I feel bad about it.

This provocative blog post title: “Does class exist?” drew inspiration from Daniel Little.  I’m curious to see if it attracts more clicks.  It would probably be more to the point to ask whether the concept of discrete social classes is useful.  If it does not help one predict outcomes of interest, then I think not.