Choosing The Best
Eight years ago, I wrote a post about a colleague’s protest about the lack of women in a conference program. Since then, the post has been sitting in the penalty box — the place where volatile posts go to cool off. (Regular readers may be astonished that I possess such a thing.)
Of course, that conference has been over for years. I think, though, that there’s a useful idea here, one that casts some light on what I call the Treaty For Web Science, about which I hope to write soon. So I’ve rewritten and extended the post here.
2005: My friend had written that
There is no such thing as selection from strict quality criteria and nothing else.
Here, I think we've wandered into the swamp or stepped off the end of the pier. If there's no such thing, for example, as selecting from strict academic quality, then universities are just social clubs where some lucky people get to distribute lots of money to their attractive and well-connected friends. That can't be right.
One could, I think, assemble a technical conference program from purely objective criteria that would likely correlate with "quality". We might need to fine-tune our metrics; that's why this is hard. It doesn't mean it can't be done.
Is it possible to select the best baseball player ever, selecting strictly from on-field performance and nothing else? I think so. Can we ask, "Was Babe Ruth a better player than Willie Mays?" We can, and the answer is yes -- even though most people seem to like Mays and lots of people thought Ruth was a jerk. (Update: Eight years later, a more effective comparison would be Barry Bonds and Mays. Or load the deck even more: Barry Bonds or Jackie Robinson. Jackie’s number 42 has been retired from baseball and Bonds might never get into the Hall, but no one is going to argue that Bonds wasn’t a better player.)
Is it possible to select the best 5 novels of the year, arguing strictly from literary quality and nothing else? Most people think this is a plausible enterprise, though it's bound to be difficult. The National Book Award, the Booker, the Pulitzer – they'd mean nothing if people thought they were rigged or jobbed or arbitrary.
As it happens, the last National Book Award (i.e. 2004 or 2005) ended up short-listing five novels. All five were written by women. All were "small" novels. None sold very well. A number of other novelists (Philip Roth, Tom Wolfe) wrote books that were eligible, but weren't nominated. If we neglect the questions raised by Middlesex, we'd expect that all five books on the short list would be written by people of one gender or the other about once every fifteen years, just from luck. It's possible that Roth's maleness worked against him, it's possible that judges thought he was already sufficiently famous, or that having already won the prize, he didn't need another shiny object. It's possible that the judges simply liked the other books more.
Writing in The Believer, National Book Award chairman Rick Moody – no slouch of a writer – said that's just the way it turned out. Moody thinks the resentment is, at core, anti-intellectual: famous writers should create the best books, right? He's got a nice polemic on how anti-intellectual spleen has no place in the National Book Award, and how the media furor surrounding the award infantilizes the American book-loving public.
2013: What I didn’t appreciate sufficiently in 2005 is the way this disagreement illuminates a disciplinary boundary. My friend is a humanist steeped in postmodern thought. My background lies in the physical sciences. We seem to be arguing politics, but we’re really arguing disciplinary faith.
My friend’s position, I think, is that all these judgments are necessarily embedded in social contexts and understandings. We can’t truly know which novel was the best of 2013; it’s not really a question that makes any sense. The best we can hope to do is suggest which novel would be the best one for you to read right now. Someone else, at some other time, might find it dull or trite or impenetrable. And if we can’t choose the best novel, how can we choose the very best conference speaker? And might not being female sometimes in itself make one person a more effective speaker than another?
Suppose you’re having a dinner party. Seven guests have been invited; your table can manage eight. Is there one best person to invite? Context is everything here, and it’s entirely possible that balancing genders, personalities, and interests will lead to the best answer.
But science cannot work this way. As Curie said, in science we talk about things, not people. Considering a talk at a scientific conference, we can easily ask (and, one hopes, answer) questions that would confound us in literature:
- Is the work completely correct?
- Is it completely original?
- Does it suggest exciting new avenues for research?
- Might it have important practical consequences?
No one can read a new novel and tell you with confidence whether it’s going to inspire lots of novels or not. For plenty of computer science papers, on the other hand, this is immediately apparent. In literature, it might be interesting to hear someone with talent expound a position that’s almost certainly wrong: Edmund Wilson’s case against The Lord Of The Ring, or Jane Smiley’s rejection of Huckleberry Finn. This is even more true in History, which thrives on energetic defenses of such seemingly-indefensible positions as “our sympathies should lie with Sparta, not Athens” or “it might have been better for everyone if Britain had let Germany win WW1.” Even if it turns out that the new argument doesn’t quite hold up, the attempt may well repay some time and effort by giving us a broader understanding and deeper sympathy.
But in science, wrong is wrong. And few things would be more wrong than preferring paper A to paper B because the author of A, though he’s clearly made a blunder this time, is an important fellow while the author of B is an unknown student from a backwater. To take the speaker’s podium away from B and give it to A would, in the sciences, be a revolting crime and a scandal. It’s unthinkable.
A fairly precise parallel can be found in the Anglo-American legal tradition. Suppose Smith, a beloved movie star, has committed a serious crime. He is immensely wealthy. He is head of prominent charities and is considering running for office. Thousands of workers depend on him and would lose their jobs if he weren’t available to make his next film. May we excuse the crime? The Romans would have answered without hesitation, “yes.” But the Anglo-American tradition is unambiguous: though the sky fall, let justice be done.
Now, even in the sciences we may have tough decisions. We might not catch a mistake. We might not know that something has already been published, especially if the first publication was obscure or if it used a different notation. Reasonable people can disagree over whether a given result is intriguing or rather dull. Committees can err. But, obviously, they must not commit crimes.
Now, scientists are not (always) dim or parochial. They understand that people are fallible, and they understand that in other fields to ask for a judgment of whether a conclusion is wrong is to ask too much. It’s impossible to apply the standards of physical chemistry to a paper on ethics or narratology. But to consider persons, not facts, when choosing conference papers is going to make scientists very, very uncomfortable.