Web Science 2013 has now received a terrific (and large!) array of research submissions. The program committee’s task is to sort these out, to choose the best for the program and also to direct researchers toward ways to improve their results.
In Computer Science, conference papers matter — both in terms of directing research and in shaping careers. Moreover, we have only three days in Paris; we need to select the best and most interesting research and arrange for its presentation in the most interesting and efficient form.
I’ve written a guide to reviewing conference papers. I also direct the attention of my fellow computer scientists to Bertrand Meyer’s essay in the current CACM on the incidence of wrong and mean-spirited reviews.
Web Science is wildly interdisciplinary, ranging from computation to sociology, literature, philosophy, law, and the arts. The program committee contains lots of people who aren’t computer scientists; these remarks are meant for them.
Conference review is not grading papers, or jurying a literary prize, or writing a book review. Our core concern is good science and sound engineering. If we have a paper:
THE WHIFFLESTONE TRUSS
The Pemberly approach to bridge design has always been appreciated for its economy and simplicity of construction. Unfortunately, Pemberly bridges have frequently turned out to be unstable. The Whifflestone adds an Anson Cap to each Cobb Tie; simulation and analysis both confirm that the resulting structure lacks the characteristic Pemberly failure modes while adding less than 3% to the structure’s weight.
Your first care in reviewing this paper is simply this: if readers rely on the paper to build a bridge, is that bridge liable to fall down?
The program chairs and subchairs sent you this paper because some aspect of the work intersects your expertise. Perhaps the argument depends on traditional Chinese bridge construction, or on Ruskin’s aesthetic theories; we can find engineers to check the math, but how many bridge builders know China or Ruskin?
We’ll sometimes ask you to stretch a bit, knowing that parts of a paper might be well outside your field but nonetheless desiring for your opinion.
Mistakes happen. If you can’t make heads or tails of the paper, we might have been confused. (Oops! Wrong Barthelme!) Let us know; we’ll reassign it.
First, insist on clear, accurate, and complete statement of the facts. Where in political writing, for example, we allow some latitude for the author’s stance and bias, here we expect statements of fact to be scrupulous.
Impact is important. Does this result matter? Is it already known? Does this new knowledge suggest new avenues of research?
Next, scholarship matters. Precedents, related work, and alternative views should be identified, cited, and fairly appraised. Many students wrongly believe the point of citing related work is to argue for the superiority and originality of their own. This is a mistake to be gently but firmly corrected.
Good writing helps. As a discipline, we adopt a very generous attitude to writing – too generous in my opinion, but pay no attention to me. Many of our papers are written by people with a tenuous grasp of English prose, and others by people to whom English is a new language. We try to accommodate everyone and to avert our eyes when we must, but some papers may be incomprehensible. If a paper is impossible to understand, say so.
Try especially to note two kinds of defect:
Our model reader is highly motivated; she wants to build that bridge and she wants it not to fall down. In consequence, we put up with passive construction and predictable organization. If you see a way to improve a paper, do suggest it. And if you encounter a paper that seems particularly well written, or that has an especially confident or amiable voice, do mention that.
We’d also like your thoughts about the best way to present this work at the conference. Formats include:
Resist the temptation simply to allot more time to better work. A brilliant and important result may require very little time to present effectively. In other cases, a useful though minor discovery may benefit from having sufficient time to explain its novel methodology and to explore its potential consequences.
Reviews do not respect persons, but presentation mode may. When it comes to science and scholarship, the most senior researcher is held to precisely the same standard as the least-known student. We don’t attempt double-blind reviews but we have a strong tradition of not caring whether a paper is written by famous or influential persons. In recommending a presentation mode, however, it can be useful to know that Prof. Anson is an engaging speaker, or that Dr. Banks writes with an unusually amiable and confident voice.
Note that, unlike many conferences, presentation mode at Web Science does not affect publication mode. We can give a 30 minute slot to a short paper that needs the time, and we can allow a 7-minute pecha kucha to summarize a full-length paper with appendices and figures.
Conflicts of interest include anyone who works at your organization, current and former graduate students, former graduate advisors, and current or former romances. F. Scott Fitzgerald has a conflict with Zelda, with his editor Maxwell Perkins, and with his protégée Edmund Wilson. He doesn’t have a conflict with William Faulkner even though they both published in the Saturday Evening Post, or with Dorothy Parker, even though they often drank together. When in doubt, ask.
Reviews are anonymous, and are typically shared with the authors of the paper. Private remarks may be addressed to the committee, and these will not be shared.
Papers for review are confidential. Don’t cite them, don’t discuss them, don’t use their results in your own research. You may consult with colleagues if you need their expertise, but they also are to be bound be these constraints.
It is helpful to begin reviews with a sentence or two that summarizes the main contribution of the paper. This helps, among other things, to ensure against clerical errors.
Please submit your reviews on time.
After reviews are submitted, we may ask for additional discussion to resolve disagreements and discrepancies. Each review is read with care and contrasting opinions are weighed by the chairs. Quite often, we’ll ask reviewers to reexamine their opinion in light of contrasting views in order to establish consensus or to avoid being swayed by enthusiasm or placing undue emphasis on a detail. Your participation in these discussions may prove especially significant.