What is peer review?
The process of external peer review is widely accepted as the major safeguard against the publication of poor quality or fabricated papers. Any paper submitted to a properly run journal should be read critically by at least one person selected by the editors because of their particular expertise in the subject matter of the paper. For the better journals there will probably be at least two referees with special interest in the area(s) covered by the paper and in many cases the paper will also be scrutinised by someone whose expertise is in statistics. The referees will try to ensure that the work is put into proper perspective by the way in which previous findings are properly acknowledged, summarised and related to the current work. The referees will consider whether the study has been soundly designed, whether suitable methods and statistical analyses have been appropriately used and satisfactorily described. They will also look critically at how the findings are interpreted and if there are alternative ways of interpreting the findings. They may consider the importance and wider significance of the findings. These referees should then submit independent reports of their deliberations and make recommendations as to whether the paper should be accepted, rejected or accepted subject to certain changes or conditions. The responsible editor will then synthesise these reports and make a decision about publication based upon them and upon their assessment of the originality, interest and importance of the work. Authors generally receive anonymised copies of the referees’ criticisms and comments.
What is not covered here?
The title of this post implies that it is the first in a series about some of the ways in which the scientific literature is protected against publication of fraudulent or poor quality data. A number of issues like those listed below are not discussed here and I aspire to discussing some of them in future posts.
- The professionalism and integrity of research scientists. Although every community has its rotten apples most research scientists want to produce and publish high quality and honestly generated data. Does the current “pressure to publish” tempt scientists to cut corners and try to publish data that is of low quality, dishonestly manipulated or even in extreme cases fabricated to fit their hypothesis?
- Are the penalties sufficient to deter those considering committing research fraud? Some notorious fraudsters have built lucrative careers upon fabricated data which in some extreme cases has caused harm or death to patients. Very few of them have been subject to criminal prosecution and only a handful worldwide have received custodial sentences. Should research fraud be a specific criminal offence? Are medical researchers who knowingly publish false clinical trial or safety data not morally guilty of manslaughter if extra patient deaths occur as a result of their dishonesty?
- Post publication peer review. As an alternative to anonymous pre-publication peer review, some publishers are experimenting with post publication review. In one model of this process, submitted papers are published online after light editorial screening and then criticisms and comments of the article are submitted and published online with the referee identified.
- Vigilant and actively involved co-authors are an important part of the peer review process. In the past, it was common practice for senior academics to have their names added to papers e.g. as a courtesy for providing a suitable research environment; so-called gift authorship. By signing their acceptance of authorship these co-authors are seemingly acting as guarantors for the paper. Their, often eminent, names on the author list tend to re-assure editors and peer-reviewers that several experts are attesting to the quality and veracity of the work. Gift authorship is now widely seen as a form of scientific misconduct. Professor Geoffrey Chamberlain resigned all of his positions, including president of the Royal College of Obstetricians and Gynaecologists, for accepting gift authorship on a faked paper by disgraced obstetrician Malcolm Pearce. Pearce claimed to have successfully re-located an ectopic pregnancy. Richard Smith (slide 15), a past editor of the BMJ, commented upon this case – “a terrible end to a distinguished career”….. “his crime was gift authorship, which was normal at the beginning of his career, scandalous by the end”.
- Is enough done to protect or even reward whistle-blowers who expose dishonest scientists? Analysis of past cases of notorious research frauds suggests that whistle-blowers have been the most common way in which they have been exposed. Some of these whistle blowers have suffered for their attempts to expose fraudulent colleagues. For example Marilyn Harvey who first tried to expose Ranjit Chandra who faked clinical trials of vitamin supplements)
- The role of training in improving the peer review process. Peer review is a largely unpaid and unacknowledged activity for which there is generally no training. Would some form of training improve the quality of peer review? Should science students be made more aware of scientific fraud by making it a specific curriculum topic in their courses?
Today’s students are tomorrow’s peer reviewers and editors.
A brief history of peer review
According to Ray Spier (2002) the modern process of peer review of scientific reports by outside experts can be traced back to the 1750s when the Royal Society in London took over editorial responsibility for its journal Philosophical Transactions of the Royal Society. Up until that point decisions about what to publish in the journal had been the responsibility of the editor and anyone he cared to consult. In 1752 a procedure was adopted whereby manuscripts submitted for publication were considered by a select group of members with suitable expertise. Spier suggests that a similar system had been used by the Royal Society in Edinburgh as early as 1731. Spier also describes a much earlier example of peer review from 9th century Syria whereby a physician’s detailed notes about the treatment of a patient were scrutinized and adjudicated upon by a local council of physicians when the patient had died or recovered. Physicians whose treatments were not regarded as up to standard could be sued by a patient judged to have been maltreated.
Despite this long history, peer review by outside experts did not become the norm for scientific and medical journals until the middle of the 20th century. Prior to this, decisions about which articles to publish were largely made in-house by the editor and editorial board. According to Spier, the journal Science and the Journal of the American Medical Association did not use outside reviewers until after 1940. In the publisher’s history, of Nature (the probably the world’s premier science journal) it is said that one of the first jobs of new editor John Maddox in 1967 was to introduce a formal system of peer review to tackle a backlog of 2,300 unpublished manuscripts. This means that Watson and Crick’s paper in Nature in which they suggested the double helical structure of DNA (Nobel prize-winners in 1962) was not formally peer reviewed. The lack of an efficient and formal peer review system meant that in 1937, Nature sent back Krebs (later Professor Sir Hans Krebs) paper on the citric acid cycle because the editor “has sufficient letters to fill the correspondence columns of Nature for seven or eight weeks” and suggests that Krebs might want to consider sending it to another journal or wait a until the backlog had cleared. In 1953 Krebs was awarded the Nobel Prize for this work!
According to Daniel Kennefick writing in 2005, perhaps only one of Albert Einstein’s more than 300 papers published between 1901 and 1955 may have been peer reviewed. This was entitled Do Gravitational Waves Exist? and was submitted in June 1936 to the Physical Review who had just introduced a formal peer review process. The critical comments of the reviewer were sent to Einstein who indignantly withdrew the paper from consideration rather than address the comments of the reviewer. The paper was published in another journal the following year and Kennefick suggests that the final published version had been edited in a way that answered some of the criticisms made by the reviewer. A number of factors made external peer review necessary and hastened its almost universal adoption by scientific and medical journals:
- The high volume of papers submitted to journals made it difficult for small editorial teams to cope with the load (as suggested in the Nature example above)
- The range of expertise required to assess papers on a variety specialised topics may have been beyond that of a small editorial team
- The introduction of the photocopier in 1959 facilitated the process of making multiple copies of papers for distribution to reviewers in different locations.
Peer reviewers do occasionally reject important work
A 2009 blog post entitled Three myths about scientific peer review by Michael Nielsen lists further examples of major scientific work that was initially rejected by peer reviewers:
- George Zweig’s discovery of quarks
- Solomon Berson and Rosalyn Yalow’s work on the development of radioimmunoassay (Yalow Nobel prize 1977- Berson died 1972)
- Stephen Wiesener’s work on quantum cryptography which was not formally published until a decade later.
Most immunologists will also have heard of the rejection by Science of the 1955 citation classic paper by Bruce Glick, Timothy Chang and R George Jaap, indicating that lymphocytes derived from the bursa of Fabricus (hence B cells) in chickens played a key role in antibody production. It was eventually published in Poultry Science, a journal that most immunologists and medical scientists would consider relatively obscure.
Peer review can lead to the initial rejection of work that later proves to be ground breaking work of major significance and of course we only know about work that that is eventually published. How many authors of important rejected papers have abandoned their work and their attempt to publish it? Are peer reviewers reluctant to accept, or fail to fully understand, work that contradicts current mainstream opinion? Do peer reviewers ever block or slow down the publication of work that they disagree with or maybe work that might pre-empt their own work in progress. Are they liable to overlook flaws in work that supports their own viewpoint? There will be examples of cases where a positive answer to each of these questions would apply but it is generally assumed/hoped that these are isolated aberrations in a basically sound process.
Limitations of peer review
Peer review will be undertaken with varying degrees of skill and diligence by referees. Conscientious editors are hoping to receive a set of searching and unbiased expert opinions about the strengths, weaknesses and importance of the paper and its suitability for publication in the journal. However, whatever their degree of skill and application, referees cannot easily judge the skill and care taken by the author team in using the techniques described. They will also generally have to accept that the authors’ descriptions of the design, methods and statistical analyses are honest and accurate. They will assume that the data that is presented and discussed has really been competently generated by the authors using the methods described. They will probably not actively consider the possibility that the author has deliberately tried to deceive them by falsely describing the procedures used or by submitting data that has been unacceptably manipulated, or even completely fabricated, unless some glaring anomaly forces them to consider this possibility. A design flaw, incorrect analysis or misinterpretation of data in an honestly described study is more likely to be spotted by a referee than a carelessly or incompetently executed study or even blatant fabrication of superficially plausible data. Many co-authors and referees when later confronted by evidence of blatant fraud that, in retrospect, looks quite obvious make comments along the lines of:
“I never even considered the possibility that s/he was making it all up”.
A 2012 paper in Perspectives on Psychological Science by Wolfgang Stroebe and his colleagues found that very few of a list of over 40 major known fraudsters were unmasked, during the peer review of one of their papers. Of course, this could be because when referees and editors suspect research fraud they simply reject the paper leaving the fraudulent author’s reputation intact and allowing them to continue trying to pollute the scientific record with their falsified or fabricated data and maybe to publish the rejected paper elsewhere. In a 2005 article by Caroline White in the BMJ, Professor Paul McKeigue suggested that suspected fraudster Dr Ram B Singh was actually using the detailed criticism of his rejected manuscripts in editorial and referees’ reports as a sort of tutorial on how to make the paper more convincing and believable when it was re-submitted to another journal.
Has proper peer review actually taken place?
It is clear from several of the fraud case studies on this blog that even peer review by some of the world’s best journals has failed to find fabricated data which in hindsight looks to be seriously and obviously flawed (e.g. Ranjit Chandra and Andrew Wakefield both published faked papers in The Lancet and Haruko Obokata and Jatinder Ahluwalia in Nature. There are also some occasions where little or no peer review has actually taken place even though the journal nominally peer reviews the papers it publishes.
One of the big innovations in scientific publishing in recent years has been the birth and growth of so-called open-access journals. These journals generate their revenue by charging authors publication fees rather than the traditional method of charging readers and libraries subscription fees or fees to access papers. Access to papers published in open-access journals is free to all potential readers from the first day after publication; a very positive consequence. However, this model, where the author rather than the reader becomes the customer, clearly has the potential for abuse by publishers or editors who may be tempted to accept poor quality papers just so that they can collect the publication fees. John Bohannon published a report in 2013 in the prestigious journal Science in which he attempted to put this accusation to the test. He generated 304 essentially similar versions of a spoof paper with serious flaws that should have been spotted by any person with even just basic scientific skills and knowledge in the appropriate area. These flaws should have led to rejection of the paper. The series of papers were about the ability of a chemical extracted from a variety of lichens to inhibit the growth of cultured tumour cells. He submitted these papers under a variety of fictitious African sounding names from a variety of fictitious institutions that were purportedly based in African cities. Of the 304 submissions of this obviously flawed paper from non-existent authors and institutions, 157 were accepted and only 98 were rejected by the time Bohannon’s article went to press. In some cases the editors immediately rejected the paper because of its obvious flaws but in many other cases, the paper seems to have been accepted without any evidence of peer review and some were accepted despite the serious deliberate flaws being highlighted during peer review.
Several of the journals that accepted one of these spoof papers were published by major international scientific publishers. The peer review system is being by-passed or downgraded by some publishers or editors in order to maximise their immediate income at the expense of the integrity of the scientific record?
In 2009 the The Open Information Science Journal accepted a spoof paper by Philip Davis and Kent Anderson. which was actually complete nonsense generated using a computer programme. The authors gave the institutional address of the mythical authors as The Center for Research in Applied Phrenology or CRAP! The paper was entitled Deconstructing Access Points and contains the following sentences which indicate the quality of what was submitted:
“We describe a novel heuristic for the extensive unification of web browsers and rasterization, which we call Trifiling Thamyn.”
“To accomplish this ambition for unstable models, we constructed new metamorphic algorithms”
Four months after the article was submitted, the authors received a statement saying that the paper had been peer reviewed and accepted for publication and requesting a publication fee of $800; at which point the authors withdrew the paper. Shortly after this hoax became public, the editor-in-chief of the journal resigned.
This spoof paper was generated by a computer program called SCIgen and this has been used by others to test the peer review process of other journals. In 2014 more than 120 papers published in supposedly peer reviewed conference proceedings were identified by a French computer scientist Cyril Labbe as having also been generated by this computer program. The publishers said immediately after being informed of the position by Labbe that they were removing these papers.
Peer review scams
I know from past experience, that it is often a time-consuming task for editors to identify suitable referees willing to undertake the unpaid task of reviewing a paper and to ensure that the referees’ reports are speedily produced. For some very specialist papers it can be difficult for editors to identify a referee with suitable background and experience. Nowadays most papers are submitted electronically and the refereeing process usually conducted via e-mail or through an online portal. To facilitate the process of identifying suitable referees, some journals now ask authors to provide the names and contact details of people who would be suitable referees for their paper. This process can be abused by unscrupulous authors if journal editors are not sufficiently vigilant. There have been several cases where authors have created fake e-mail addresses for real or phantom referees using one of the free e-mail providers like g-mail or Yahoo so that the paper ends up being sent to its author for review and criticism.
In 2014, sixteen papers co-authored by Pakistani economics professor Khalid Zaman were retracted from journals published by Elsevier for just such a peer review scam where his manuscripts were sent to fake e-mail addresses of real or imaginary academics which actually belonged to him. An editor became suspicious because all of the referees with non-institutional e-mail addresses gave very quick and positive reviews of Zaman’s work. When challenged by the editor, Zaman admitted that he had written the reviews of his own papers.
In July 2014, SAGE announced that it was retracting 60 articles from its Journal of Vibration and Control because of the existence of a large peer review and citation ring who were refereeing and citing each other’s papers at a strikingly high rate. The central character in the operation of this ring was Peter Chen a Taiwanese academic who co-authored almost all of these papers. During an investigation by a team of SAGE employees, 130 suspicious e-mail accounts were identified. Chen resigned from his academic position and is said to have taken sole responsibility for the operation of this ring. He admitted to adding the name of Taiwan’s education minister to several of these retracted papers without his permission. The minister himself also denied playing any active role in this process nevertheless he resigned his ministerial position:
“To uphold his own reputation and avoid unnecessary disturbance of the work of the education ministry”.
In November 2014, Cat Ferguson, Adam Marcus and Ivan Oransky discuss several peer review scams in a Nature article. These include the case of South Korean medicinal plant researcher Hyun-In Moon. Moon admitted to the editor of the Journal of Enzyme Inhibition and Medicinal Chemistry in 2012 that he had written many of the reviews of his papers himself. The reviews themselves were generally favourable but unremarkable. The editor’s suspicions were raised because of the speed with which reviews were submitted, often within 24 hours of the paper being sent. Ferguson and colleagues suggest a few signs that should raise suspicions:
- Authors list people to exclude as potential reviewers that include almost everyone in their specialist field
- Authors recommend referees who cannot be found in an online search
- Authors provide e-mail addresses for potential referees from free e-mail providers rather than institutional addresses
- Very rapid return of very positive reviews
- Reviewers are unanimous in their highly positive opinions.
Fraudsters acting as journal editors
Some notorious fraudsters have been editors of the journal where they publish many of their papers. It is clear that they have been able to control (avoid) independent peer review of their own papers.
Sir Cyril Burt was the founding editor of what is now the Journal of Mathematical and Statistical Psychology. Over a 16 year period, he published 66 papers in this journal under his own name i.e. over half of his published output of scientific papers. He also published various articles and other contributions under a variety of assumed names. He seemed to have effectively controlled what was published in this journal over this period and could publish anything he wanted.
For over 20 years (1981-2003), Ranjit Chandra was founding editor-in-chief of the Elsevier journal Nutrition Research. Chandra has 65 publications listed for this journal including 27 research articles, 26 editorials and 4 reviews. Two papers published in 2002 have come under particular scrutiny and were recently (2016) retracted. One of these papers is published under his own name and the other published under the name AL Jain who has not been traced and the paper is widely believed to have been written by Chandra and published under this pseudonym. Both of these papers support his earlier findings published in the Lancet that his patented supplement improves immune function in older people (now retracted). They provided support for his earlier findings at a time when the integrity and veracity of his research was being publicly questioned. Both papers were accepted for publication within 24 hours of submission which suggests that no independent peer review of either paper took place.
Suspected fraudulent author, Ram B Singh has been editor in chief of two open access journals, the World Heart Journal and the now discontinued Open Nutraceutical Journal. In the period 2008-2014, about three quarters of his 150 publications over this period were published in these two journals. The Open Nutraceutical Journal was published by Bentham Open and The World Heart Journal by Nova Publishers whose Wikipedia entry includes the following statement:
“Nova has been criticized for not always evaluating authors through the academic peer review process and for republishing, at high prices, old public domain book chapters and freely-accessible government reports as if they were new. These criticisms prompted librarian Jeffrey Beall to write that in his opinion Nova Science Publishers was at the “bottom-tier” of publishers.”
In the above discussion there are several quite shocking examples of lax or non-existent peer review and flagrant disregard of editorial responsibility. The last example is almost too difficult to believe. An Australian computer scientist Peter Vamplew was so annoyed by receiving unsolicited e-mails from predatory open access journals that he sent one of them, the International Journal of Advanced Computer Technology a spoof paper comprised entirely of the 7 words “get me off your f***ing mailing list”. A diagram from this spoof paper is reproduced below. The journal accepted the paper subject to payment of a processing fee which Vamplew declined to pay.
This paper was originally constructed by David Mazieres and Eddie Kohler of New York University and UCLA and the full version is available online.