Bocking, Stephen. (2005). “Chapter 4: Gatekeeper or Helpful Counsel? Practices and Perceptions in Academic Peer Review.” In Anne Meneley and Donna J. Young (Eds.), Auto-Ethnographies: The Anthropology of Academic Practices (67-79). Toronto, Ontario: Broadview Press.
In his article on the academic practice of peer review author Stephen Bocking dispels any preconceived notions one may have regarding this process: namely that the brightest and the best do not always get published. Drawing upon his previous experience as editor for the Journal of Canadian Studies, Bocking provides several reasons as to why this is the case. Elaborating on the endemic shortcomings of peer review, which can result in inconsistent outcomes for aspiring authors, he also addresses many of the criticisms lobbied against the peer review process while optimistically providing an alternative for forging ahead within the system.
In the publish-or-perish world of academia peer review has become the preeminent system for legitimizing scholars. The intention, or belief, is that peer review will illuminate the best research and, subsequently, funding can be allocated to the authors of this research. However, lack of consensus regarding what qualifies as appropriate “methods, evidence and forms of reasoning”[i] within any given discipline tends to emerge as a seemingly arbitrary and subjective review process. Furthermore, as peer review has increasingly become the main source for determining who receives academic jobs and funding the process– which was widely adopted by academia in the 1950’s with the intention of fostering community– has become somewhat antagonistic.
Bocking spends several pages describing the process of peer review and its historical development, using the space to expand upon several of the problems which have emerged within the practice.[ii] Reviews play a large role in informing an editor’s decision whether or not to publish. As it was mentioned in our class discussion, the prestige of the reviewer can be very persuasive in ultimately determining whether or not a work will be published. Even here, however, Bocking notes that this is not the sole deciding factor. In some cases, the relation of the work to the theme of the publication, the topic of the research, or even the regional distribution of authors may also determine which articles, no matter how good, ultimately make the final cut.
The question of anonymity for both authors and reviewers is another important issue raised in this work. Maintaining reviewer anonymity may allow for a more forthcoming commentary on the work, however, it may also allow reviewers to hide behind their veil of mystery. Perhaps much like the internet individuals may be emboldened by their anonymity and make statements which they would not normally make were their identity to be known, as they ultimately do not have to be accountable for them. As such, some have argued that reviewers should stand behind their responses and therefore be willing to put their name to them. In contrast, however, anonymizing an author may help to prevent any forms of bias which may emerge by reviewers based on name recognition. Certainly, there is ample evidence to suggest that removing authors’ names, and their institution, may help to prevent a perceived prestige bias in the review process.[iii]
Mid-way through his work Bocking addresses his titular question, “gatekeeper or helpful counsel?”[iv] As a former editor he establishes that essentially two categories of reviewers exist: the ‘gatekeeper’ and the ‘helpful counsel.’ The ‘gatekeeper’ sees it as his or her duty to parse out flaws in the work, and subsequently provide two general types of review. The first is a terse affirmative or negative commentary regarding the quality of the paper and whether or not it should be published. The second type of ‘gatekeeper’ provides a negative and densely methodical criticism of the paper as a whole, which may even be aggressive towards the author. Bocking notes that the one benefit to this form of critique is that, while negative, these reviews can be invaluable in pointing out errors in the research.
Before turning to discuss ‘helpful counsel’ reviewers Bocking reflects on whether the practice is “more trouble than it is worth.”[v] Reviewers are not remunerated for their assistance in the peer review practice. It is a way for scholars to give back to the process and promote the independence of the academic discipline. A criticism on this point is that it encourages the isolation and separation of academia from society-at-large, making it less able to discern research which may be of interest to the general public. This last point emerged in our class discussions as it was mentioned by some that this process of peer review tends to further contribute to the ‘ivory tower’ of academia where academic literature is produced solely for academics. While there was general agreement on this point it was also countered that peer-review publication is not the sole means of disseminating ideas and information; authors are able to submit to popular journals, newspapers, maintain blogs, etc.
Another problem which Bocking reflects upon in the peer review process is that it rests on the assumption that reviewers are motivated by benevolent or philanthropic agendas. In some cases, the reviewers may have more nefarious goals in mind, such as, to delay the publications of their competitors or even to plagiarize another’s work.
While peer review can help to validate quality research it cannot always detect fraudulent works or research funded from external sources which may have an embedded agenda. As well, reviewers also tend to shy away from innovative research and works which deviate from accepted disciplinary methods and theories. Finally, as reviews vary markedly and most works tend to generate mixed responses authors are led to believe that the process is arbitrary and subjective.
At the end of the chapter Bocking addresses the ‘helpful counsel’ reviewer. This type of reviewer seeks to foster a cooperative and collaborative environment. The process is more akin to a community where the review of the paper is not the end goal but rather a part of the writing process, and reviewers comments are intended to improve the quality of the work. This idea draws upon the founding notions of the peer review practice where it linked the academic community, shared ideas, and promoted academic growth. ‘Helpful counsel’ appears to be envisioned as more of an aspiratory role for the reviewer rather than a comment on the current practices of reviewers.
A cursory review of academic literature and the internet yields numerous articles lamenting the peer review process. The general consensus is that the process is indeed flawed, but there is no sign that it is going away. It has been famously “compared with democracy: a system full of problems but the least worst we have.”[vi] Of the current problems which plague the peer review process the most important may well be its consensus building nature. This may seem counterintuitive given the fact that disciplines have been oft criticized for their lack of agreement regarding appropriate methods, concepts and theories. However, Bocking and others comment on its ability to entrench ideas and maintain the status quo.[vii] As such, it is generally dismissive of new and innovative research which does not conform with perceived institutional rigours or ideas. Francis Bacon referred to this as the “confirmatory bias.”[viii] In academia it represents the tendency to accept or reject ideas based on the degree in which they accord with our own beliefs. This bias in the peer review practice can result in a relative degree of epistemic stagnation.
Bocking’s work captures multiple facets of the the peer review process. Despite the vast array of topics which the author covers the idea was brought forth in our class discussions that ultimately this work addresses knowledge and collaboration. Peer review can determine what we know and ultimately what ideas are perpetuated or given credence. As such, Bocking’s distinction between gatekeepers and helpful counsel were seen by many of us to be a question of ‘who decides?’ Certainly, Bocking seems to have a preference for the more collaboratory and encouraging role of the helpful counsel.
Bocking’s visionary thinking regarding the role that `helpful counsel’ can play in the review process is commendable. Seeking to foster community through peer review may reduce the separating-the-wheat-from-the-chaff way of thinking which currently pervades the academic practice. A more cooperative and collaborative process may overcome confirmatory bias and encourage more innovation within academic disciplines. Additionally, this may also reduce the burden which the disciplinary lack of consensus may play in the process. A study performed by Dunbar found that a substantial part of the scientific process of discovery occurred while collaborating and discussing with peers, a process he termed “distributed reasoning.”[ix] Simply put, ideas do not occur in a vacuum. Notions of an academic toiling away in solitude before their ‘eureka’ moment is a relative fiction. As academics we can use our respective areas of expertise to help others formalize and codify their own ideas, as they too can help us. Viewing one’s role as ‘helpful counsel’ in the peer review process, rather than ‘gate keeper’, may ultimately help to address many of the systemic problems currently hindering the peer review process.
[i] Bocking, 2005, 70.
[ii] For a more detailed explanation of the peer review process see, Bocking 68-70.
[iii] One study, undertaken by Peters and Ceci, resubmitted already published articles under new authors, titles and academic institutions. The study found that an overwhelming majority of the articles were rejected and only a few journals recognized that the works had already been published. For more on this study see, Peters, D., & Ceci, S. (1982 ). Peer-review Practices of Psychological Journals: the Fate of Submitted Articles, Submitted Again. Behavioural and Brain Sciences, 5, 187– 255.
[iv] Bocking, 72-79.
[v] Bocking, 75
[vi] Smith, Richard. (2006). Peer Review: a Flawed Process at the Heart of Science and Journals. Journal of the Royal Society of Medicine, 99: 178.
[vii] Bocking, 76. See also, Mahoney, Michael J.(1977). Publication Prejudices: An Experimental Study of Confirmatory Bias in the Peer Review System. Cognitive Therapy and Research, 1(2), 161 – 175, and also, Lee, Carole, & Schunn, Christian D. (2011). Social Biases and Solutions for Procedural Objectivity. Hypatia, 26(2), 352 – 373.
[viii] Mahoney, 162
[ix] Dunbar, K. (1999). How Scientists Build Models: In vivo Science as a Window on the Scientiﬁc Mind. In C. Magnani, N. J. Nersessian, and P. Thagard (Eds.), Model-based Reasoning in Scientiﬁc Discovery (pp. 85–99). New York: Kluwer Academic/Plenum; 92.
Dunbar, K. (1999). How Scientists Build Models: In vivo Science as a Window on the Scientiﬁc Mind. In C. Magnani, N. J. Nersessian, and P. Thagard (Eds.), Model-based Reasoning in Scientiﬁc Discovery (pp. 85–99). New York: Kluwer Academic/Plenum.
Lee, Carole, & Schunn, Christian D. (2011). Social Biases and Solutions for Procedural Objectivity. Hypatia, 26(2), 352 – 373.
Mahoney, Michael J.(1977). Publication Prejudices: An Experimental Study of Confirmatory Bias in the Peer Review System. Cognitive Therapy and Research, 1(2), 161 – 175.
Peters, D., & Ceci, S. ( 1982 ). Peer-review Practices of Psychological Journals: the Fate of Submitted Articles, Submitted Again. Behavioural and Brain Sciences, 5, 187– 255.
Smith, Richard. (2006). Peer Review: a Flawed Process at the Heart of Science and Journals. Journal of the Royal Society of Medicine, 99, 178 – 182.