Peer review reprised
In a previous issue of this Update, I offered some personal thoughts on the review process as it pertains to many, if not all, scholarly journals. In summary, I confronted the issue of colleagues who enjoy the fruits of others’ talents as reviewers, but scrupulously avoid the chore when asked by editors. In this note, I’d like to follow-up on some of the issues that were raised by readers. I was frankly surprised by the speed and volume of the correspondence and by the complexity of some of the issues raised in the messages that came both to me and to the publisher direct. I was also a little startled by the unanimity of the response, as the responses were generally constructive, albeit with the exception of one individual who cryptically observed that it was time for him to quit being an editor [a comment I took to relate to the alleged crassness of my insights rather than his impending demise; if not, I hope he is well]. I can break the responses into three categories, namely the existence of academic celebrities and a consequent tragedy of the commons; the misunderstandings that some possess about the purposes of review; and the need for more instruction and reward in the process. I’ll address these in turn.
I found the most detailed comments in a recent editorial, and while it appeared before my most recent musings, it did refer to previous versions of my statements. In it, Ben Martin and colleagues at Research Policy went further than I had done to systematize our concerns [Martin et al. 2009]. First, they suggest that there is a caste system evolving—some academics are simply “too posh to peer review”. And they see this as a systemic problem that constitutes “a new form of the ‘tragedy of the commons’” [p.696].
Successful review is best learned early in academic apprenticeship, alongside writing, presenting, lecturing and researching.
Is the issue quite this formal? There is no question that academic celebrities do exist and that they are as hard to reach as their counterparts in other fields; I have one Editorial Board member who has never responded to any of my requests, for instance [which raises, of course, the question of who gets to sit on such Boards, which may be a pertinent topic for a later discussion!]. Yet I detect a belief that there is more to this than one’s salary. One respondent observed that “nowhere do I find as much arrogance and negligence as in the UK—to a point where this steers my selection of referees”. Doubtless, British colleagues would respond that they are innocent victims of a post-Fordist and neo-Liberal regime in which their time is not their own, and I leave readers to make their own judgments. I would have to add though that there are always some broader cultural issues at work, and here I am thinking enviously of those many European academics who appear to just pack up their offices when summer appears, and don’t even read their email until September rolls around. I am sure their response would be to point in turn to those studies that show that American workers don’t even take the vacation that they accrue, but here’s the irony—most American faculty members aren’t even paid over the summer, so they undertake their communal obligations entirely ‘out-of-pocket’.
Stepping back a little from these generalizations, my comments about the narrowness of many academics seemed to strike a chord. One editor parodied a reviewer thus: "Oooh! You're asking me to review a paper on the use of relative clauses in Colombian Spanish. But my research was on the use of relative clauses in Puerto Rican Spanish! So, no, I can't possibly do that." While we should not encourage each other to express an opinion on everything, this false modesty is self-serving and may even miss the point of review in many instances. The intention is usually not to assess the ability of the manuscript to speak to the seven experts in one tiny subfield, but rather to contribute to a field or even a discipline as a whole. In consequence, the reaction of the relative clause expert to something beyond her immediate zone of comfort may be quite indicative of the paper’s value to its general readership. Another commentator took this insight further, pointing out that some reviewers seem literally unaware of the purpose of their task, turning in copious notes and rewriting the bad grammar [as if they were dealing with a thesis] but never actually providing an assessment of the potential value of the paper.
The intention is usually not to asses the ability of the manuscript to speak to the seven experts in one tiny sub-field, but rather to contribute to a filed or even a discipline as a whole.
This comment points to the reality that peer review—like personnel review—is a skill that tends to be learned the hard way, through receipt. Over time, we internalize the kinds of reviews that we like and dismiss those with which we don’t agree. That may not, though, be the most efficient way to learn how to perform a review that is thoughtful, balanced, and of some use to both the individual and the initiator of the request. What this suggests is that successful review is a skill best learned early on in the academic apprenticeship, alongside the other fundamental skills, including writing, presenting, lecturing and, of course, researching.
This makes the point once more that the review is not a minor part of the academic process. One concrete response to my column that fully caught my imagination was the lengthy piece by an Italian colleague, who argues that there should be more formal recognition of the reviewer—alongside the author—on the grounds that both take complex skills and a good deal of time. Her commentary, which too identifies a tragedy of the commons, follows mine.
By calling for even more measurement and more nuance in the review process, I am sure that yet more editors will throw up their hands and vent their frustrations with scholarly publication. I have some sympathy with their position; every day, it seems that the EES system itself becomes more and more complex. Yet the alternatives are much, much worse. Some of us may remember the journal editor in Amis’ Lucky Jim, who sat on manuscripts for years, rarely answered correspondence, and ultimately plagiarized submissions as his own. There was a good reason that we embarked down this path in the first place.
Arizona State University
Martin B. et al. 2009 ‘Editorial: EES and the continuing evolution of Research Policy’ Research Policy 38 695-99.
To cite this article, please use: Andrew Kirby, “Peer review reprised”, Elsevier Editors’ Update, Issue 28, November 2009.