We needn't be malevolent grumps in 2012

[cross posted from my CACM blog]

A few months back, Bertrand Meyer wrote about the nastiness problem in computer science, questioning whether we as reviewers are “malevolent grumps”. Judging by the user comments on the page, this hit a nerve with readers who were the victims of such grumpiness!  Jeanette Wing then followed up on this with some numbers from NSF grant rejections which did indeed indicate that computer scientists are hyper-critical. Much as I enjoy the colourful phrasing, I feel that a field full of malevolent grumps is not something we should simply accept. In fact, even if there are only a few grumps out there, it’s in all our interests to civilise them.

So what can computer scientists do to reduce the nastiness problem when reviewing? Reviewers, authors, programme committee members, conference chairs and journal editors can all do their bit by simply refusing to tolerate discourtesy. Let’s embrace the “no asshole” rule: we no longer ignore bad behaviour. As reviewers, we can aim to be polite (yet stringent) ourselves but also to point out to co-reviewers if we find their impoliteness unacceptable. As authors, we don’t have to accept a rude review and just lie down to lick our wounds. We can (politely!) raise the issue of rudeness with the programme chair or editor so it is less likely to occur in the future. As editors, chairs and programme committee members we can include the issue of courtesy in the reviewing guidelines and be firm about requesting reviewers to moderate their tone if we notice inappropriate remarks.

One of the first steps is to separate intellectual rigour from discourtesy. It is possible to be critical without being rude or dismissive.  We can maintain standards in the field without resorting to ill-natured comments. (Believe it or not, it is also possible to ask genuine questions at a conference without seeking to show off one’s own intellectual chops, but that is another matter). The purpose of reviewing, in my view, is to help an author improve their work, not to crush them under the weight of your own cleverness. It’s not the author’s fault that you had a bad day, or that some other reviewer just rejected your own paper. 

Of course, there are some pockets of good reviewing practice within the field which we can draw on. I am sure there are many, but I have chosen CHI because I have been writing for it recently. The CHI conference is one of the biggest well respected annual human computer interaction conferences. Last year there were 2000 attendees from 38 countries. This year there were 1577 paper submissions with a 23% acceptance rate.  This was the first year I submitted papers to it, and I have been impressed by the quality of the reviews in terms of their fairness, constructiveness and level of detail. They contained greater insight and intellectual oomph than the reviews I had from a high impact journal recently. For one of my CHI submissions, the reviewers did not agree with the paper on some points – it is on a controversial topic- but they still offered suggestions for how to resolve these issues rather than simply rejecting the paper. Was I just lucky in the reviewers I was allocated? Possibly, but the CHI reviewing process has some interesting features built in to maintain review quality*.

1. In the guidelines for reviewers, courtesy is explicitly mentioned: “please be polite to authors. Even if you rate a paper poorly, you can critique it in a positive voice. As part of polite reviewing practice, you should always state what is good about a paper first, followed by your criticisms. If possible, you should offer suggestions for improvement along with your criticism.” 

2. Authors can select both the sub-committee and the contribution type for a paper, which maximises the chance that the paper will end up with reviewers with appropriate expertise , and that the reviewers will use criteria appropriate to the paper when assessing its suitability (e.g. not insisting on empirical evidence for a theoretical contribution).
3.  The reviewing process is thorough and has several opportunities for unfairness or discourtesy to be weeded out. Each paper is blind reviewed by three or more experts, and then an associate chair writes a meta-review to summarise the assessment of the paper, and what action (if any) should be taken to improve it. In this way, individual grumpiness is moderated. A variant of this good practice from other conferences is when reviewers of the same paper can see each other’s reviews (once they have submitted their own), thus introducing peer pressure not to be awful.
4.  Authors have a right to reply by writing a rebuttal of the review. The rebuttal is taken into account along with a revised meta-review (and potentially revised individual reviews) at a two day committee meeting when final accept/reject decisions are made.
5. All submitting authors are surveyed about their opinions of the reviewing process – yet another chance to raise issues about unfairness or discourtesy which have not been addressed in a rebuttal.
6. This point is more about the nature of the conference itself, rather than the reviewing procedures. Because CHI is so interdisciplinary, participants have a wide range of backgrounds from art and design to hard core engineering. They are therefore exposed to – and may in fact seek out- different perspectives which may make them open to different paradigms as reviewers. Could colleagues from the arts and social sciences be having a civilising influence on the grumpy computer scientists?

Ts This is a fairly heavy weight process, but if conference organisers adopted even just one more of the practices from points 1-5, or if journal editors added a courtesy clause to their review instructions the world would be a slightly better place. Make it your New Year's resolution to stop tolerating malevolvent grumps in 2012!

 

*Thanks to Tom Erickson – the person who runs the CHI author survey – for kindly raising some of these points.