Write only? The Program Committee Problem
I received a (fairly) nice referee report yesterday, so a paper I co-authored with Claus Atzenbeck, Stacey Mason, and Marwa Al-Shafey will be presented at Hypertext ’13. I’m a little miffed, though, because it was clear that none of the reviewers were familiar with my recent papers for Hypertext, a series I think of as my Big Three:
These, along with the Genre piece on Designing a New Media Economy, were a ton of work. As far as I can see, they’ve had no impact at all.
I'm tempted to wrap these up between paper covers. If I do and you’d like a copy, Email me. Thanks.
This is frustrating not just because of the “do you know who I am!” aspect, but because it’s becoming very difficult to have a discussion amongst papers. Because people don’t really know the literature, they restate their opinions when they ought to be responding to, and building on, the work you and they have already done.
One of the reviewers pointed to a paper from Hypertext ’12 that we ought to have cited. I nodded. And then I realized that I myself haven’t read that paper.
This could easily be blamed on “kids today,” on declining standards of scholarship. Perhaps it’s a complain of an old fogey who imagines things used to be better.
But I think the real problem is that we don’t have program committees that actually meet anymore. In the old days, you’d have ten or twenty people in a room, in person, and you’d discuss every paper. That meant you heard your colleagues talk all day about papers, about their strengths and weaknesses, about whether the work was really original or really convincing. It instilled a habit of argument, of knowing what’s been done and what’s new. At the same time, because some papers were mentioned time and again, it built a body of shared knowledge for the field.
This helps teach everyone How To Do It. You can learn a lot from good discussions of bad papers, and even more from thorough vetting of fairly good papers.
Now, program committee members never see each other. They write reviews and submit them to EasyChair and the wheels go round. For many conferences, I’m not sure that reviewers always read other reviews, or that anyone pays much attention when the reviewers basically agree.
This needs to be fixed. I’m not sure how to do it, or if it can be done, but if we don’t manage to address this problem, we’re not going to have a research discipline and we won’t have any claim to being a science.