Neil Gross, the University of British Columbia researcher who studies the political outlooks of U.S. faculty members, this week published two co-authored papers that address the sources of the liberal-left domination of the American academy. In “Unnatural Selection,” I commented on one of those two papers. Here I comment on the other, “Political Bias in the Graduate Admissions Process: A Field Experiment.” Peter Schmidt has already covered this story thoroughly in The Chronicle of Higher Education, and I refer the reader to Schmidt’s account of the details.
Gross’s work is never less than imaginative and insightful. His paper on the graduate admissions process, however, comes across to me as an ingenious idea for an experiment that foundered on the details. The original idea was intriguing. Why not find out whether graduate schools in certain disciplines treat applicants differently depending on the applicants’ evident political orientation? In principle, that’s a good question, and I would be among many interested in the answer.
The trouble begins with finding a way to collect meaningful data on the question. Asking the faculty members in graduate programs whether their programs discriminate among candidates for admissions on the basis of the candidates’ political views is not an approach that is likely to get any answer beyond, “Of course not!”
To work around this, Gross and his colleagues Ethan Fosse and Joseph Ma devised a procedure in which they sent letters from fake applicants to directors of graduate study asking whether they would be a “good fit” for the program in question. The letters were brief but offered a few details about the supposed applicant, indicating an area of topical interest that matched the program. Some of the letters also included the sentence, “When I was a sophomore I also spent a few intense months working for the ___________ campaign, which was quite a learning experience.” The blank in some letters was filled in with “Obama,” and in others with “McCain.”
The goal was to see if the letters that mentioned working on McCain’s campaign were treated differently from those that mentioned working on the Obama campaign. The results showed that there were in fact some differences: There were “slightly more” responses to the Obama letters and the responses came “slightly more quickly.” But the researchers did not find these differences statistically significant.
Their main finding was negative: The study found no evidence of bias against conservative students sending letters of inquiry to graduate departments.
I accept the finding as entirely valid, given the protocol. The study appears to me to have been scrupulously conducted.
But I also see it as essentially worthless. To use work on the John McCain presidential campaign as a proxy for a student’s conservatism is to vitiate the study from the outset. Few American conservatives consider John McCain a “conservative.” He is a Republican Senator who styled himself during his presidential campaign as a “maverick,” and who is mostly known as a political moderate. On most of the social and economic issues that define contemporary conservatism, McCain took positions that run counter to the conservative mainstream.
Gross and his colleagues were aware of this problem, and explained their decision as follows:
It is true that McCain is seen by some as more a moderate than a true conservative, but our sense is that the vast majority of professors code him as being on the right, particularly given his association in the 2008 campaign with Sarah Palin. We worried that a stronger conservative prompt, such as being a George W. Bush supporter, might—if claims about the extent of hostility to conservatism in academe are true—lead some respondents to question the legitimacy of the email.
In other words, they presumed that the political bias they were setting out to study is so strong that to make their fake inquiries plausible they had to vitiate their own premise.
They return to this compromise near the end of the paper when considering “methodological limitations,” where they write:
For example, had our treatment stimulus been different—had our conservative student worked for the election of George Bush, say, or not merely indicated his personal conservatism but also stated his intention to do conservatively themed research—we might well have found more bias.
This “methodological limitation” is astonishing and a lot more than a “limitation.” All by itself it renders the project incapable of finding out what the researchers say they wanted to find out. Bad faith? I don’t think so. All I can imagine is that Gross and his colleagues are so tone deaf to conservatism that they looked at McCain and said, “close enough.”
But the methodological problems don’t stop there. They have built into the study another even larger one: their choice of using a preliminary letter addressed to the director of graduate studies as something that would plausibly show, depending on the response, the presence or absence of bias. This is nonsense. Directors of graduate studies are not in the business of turning away potential applicants at the stage of initial letters of inquiry unless the author of the letter is manifestly unqualified or simply mistaken about the nature of the program. Gross and his colleagues carefully crafted fake letters that passed these two tests. Why then would a director of graduate admissions write back anything other than a rote letter adjusted as need be to the individual case?
The researchers tried to control for rote letters by looking for oft-repeated phrasings with plagiarism detection software, but that is hardly going to filter out the entirely routine nature of these transactions. If an applicant is going to encounter bias, it will be further down the line among members of the graduate admissions committee.
Again, the researchers are more or less aware of this, but explain that they had ethical qualms and practical doubts about attempting to carry out a more elaborate ruse. Well they should, but that doesn’t make a flawed experimental protocol any less flawed. This study has something of the drunk looking under the streetlight for his lost keys because that is where the light is. Just because the director of graduate studies is available to take and answer letters doesn’t mean that’s where the keys to academic bias will be found.
The McCain-as-proxy for conservatism and find-bias-at-the-preliminary-letter strategy give us a study that is well-intentioned but incapable of illuminating the problem at hand. Is there political bias in graduate-school admissions? I suspect there is, but this study doesn’t bear on the question one way or another.
Gross and his colleagues include an important footnote in the paper—footnote 8—to the paragraph about methodological limitations, in which they acknowledge a still deeper problem: that “many prominent researcher paradigms in the social sciences and humanities are fundamentally at odds with certain strands of conservative ideology—in their assumptions, casual claims, and policy implications.” Indeed.
From this he allows that scholars who “reject these paradigms” get negative reviews from their peers:
But it is not clear to us whether this reflects political bias or discrimination or simply the same sort of reasoned intellectual judgment/disconfirmation bias that would lead supporters of any paradigm to be skeptical toward champions of an opposing paradigm.
So, at a deep level, bias and intellectual judgment may be indistinguishable? The lines at least, in Gross’s and his colleagues’ view, “can be blurry,” and that means “leaving open the possibility that disadvantages for conservative students and scholars may obtain as a consequence of scholarly judgment.”
Let me offer an emendation. Any attempt to disadvantage conservative students and conservative scholarship can always be framed as a matter of “scholarly judgment.” That, in fact, is the essence of what the AAUP has been promoting the last several years though documents such as “Freedom in the Classroom,” and its new draft, “Ensuring Academic Freedom in Politically Controversial Academic Personnel Decisions.” These documents rely on a doctrine that amounts to “scholarship is whatever we say scholarship is.” Conservatives know what that means.
This article was originally posted in the Chronicle of Higher Education's Innovations blog.
