Journals with high proportion of suspicious articles flagged by science integrity startups

Journals with high proportion of suspicious articles flagged by science integrity startups

Which academic publishers and journals have been most affected by fraudulent or questionable research — and which have done the least to clean up their portfolios? A technology start-up founded to help publishers identify potentially problematic articles has some answers and has shared its initial findings with Nature.

Knowledge integrity website Argos, launched in September by Scitility, a technology company based in Sparks, Nevada, assigns risk ratings to articles based on authors' publication dates and whether the article heavily references research that has already been retracted. An article classified as “high risk” may have multiple authors whose other studies were retracted due to misconduct. A high score doesn't prove that an item is low quality, but it does indicate that it might be worth reviewing.

Argos is one of a growing number of research integrity checking tools that look for red flags in articles. These include this Paper mill alarm, developed by Clear Skies, and Signals by Research Signals, both from London. Because developers of such software sell their manuscript screening tools to publishers, they are generally cautious about naming affected journals. However, Argos, which offers free accounts to individuals and wider access for integrity auditors and journalists, is the first to provide public insight.

“We wanted to develop a technology that could detect hidden patterns and bring transparency to the industry,” says Erik de Boer, co-founder of Scitility, based in Roosendaal, Netherlands.

As of early October, Argos had flagged more than 40,000 high-risk and 180,000 medium-risk items. It has also indexed more than 50,000 retracted articles.

Risk assessment of publishers

Argos' analysis shows that Hindawi, a now-defunct subsidiary of London publisher Wiley, has the highest number and highest proportion of articles already withdrawn (see "Publishers at Risk"). That's not surprising because Wiley has retracted over 10,000 articles published by Hindawi in the last two years, in response to concerns raised by editors and reviewers; this represents more than 4% of the brand's total portfolio over the last decade. One of its journals, Evidence-based Complementary and Alternative Medicine, has retracted 741 articles, accounting for more than 7% of its output.

Argos risk assessments mark more than a thousand remaining Hindawi articles — another 0.65% — as continuing to be “high risk”. That suggests that while Wiley has done a lot to clean up its portfolio, the problem may not be completely solved yet. The publisher told Nature that it welcomes Argos and similar tools and is working to resolve the problems with Hindawi.

Other publishers appear to need to do a lot more investigation, as the number of retractions is small relative to the number of high-risk articles flagged by Argos (publishers may have already investigated some of these articles and decided no action is necessary).

Amsterdam-based publisher Elsevier has about 5,000 retractions but over 11,400 high-risk articles, according to Nature's analysis of Argos — although all of these together account for just over 0.2% of the publisher's output over the last decade. Publisher MDPI has withdrawn 311 articles but has more than 3,000 high-risk articles — about 0.24% of its output. Springer Nature has more than 6,000 retractions and more than 6,000 high-risk articles; about 0.3% of its production. (Nature's news team is independent of its publisher.)

In responses to queries, all publishing divisions identified as the largest providers of high-risk articles say they work hard on research integrity, use technology to review submitted articles, and that their retractions demonstrate their commitment to cleaning up problematic content.

Springer Nature reported that it launched two tools in June that have since helped detect hundreds of fake manuscript submissions; several publishers raised their collaboration in one common integrity center which offers software that can flag suspicious items. Jisuk Kang, a publishing manager at MDPI in Basel, Switzerland, says products like Argos give broad indications of potential problems, but notes that the publisher cannot verify the accuracy or reliability of the numbers on the website. She adds that the largest publishers and magazines will inevitably have a higher number of high-risk articles, so share of production is a better metric.

The publishing brands with the highest proportions of high-risk articles in their portfolios, according to Argos data, are Impact Journals (0.82%), Spandidos (0.77%) and Ivyspring (0.67%). Impact Journals tells Nature that although its journals have had problems in the past, they have now improved their integrity. The publisher states that it has appeared in the journal Oncotarget in the last two years “0% irregularities” due to the introduction of image verification tools such as Image Twin, which have only become available in recent years. Portland Press, which has 0.41% high-risk articles in its portfolio, says it has taken correct measures to improve rigorous reviews.

Magazine risk assessments

Argos also offers numbers for individual magazines. Not surprisingly, Hindawi titles stand out in both the number and proportion of papers retracted, while other journals have many of the papers identified by Argos as high risk (see “Journals at Risk”). By volume, Springer Nature's mega-journal Scientific Reports leads with 450 high-risk articles and 231 retractions, which together represent about 0.3% of its output. On October 16, a group of reviewers wrote one open letter to Springer Nature, which raised concerns about problematic articles in the journal.

In response, Chris Graf, head of research integrity at Springer Nature, said the journal is investigating every issue raised. He adds that the proportion of content highlighted is relatively low compared to its size.

Journals with particularly large discrepancies between the number of retracted papers and potentially suspect articles include MDPI's Sustainability Journal (20 retractions and 312 high-risk articles; 0.4% of production) and Elsevier's Materials Today Proceedings (28 retractions and 308 high-risk articles; 0.8% of production). Elsevier's Biomedicine & Pharmacotherapy has the highest proportion of high-risk items — 1.61% of its production.

“The volume of fraudulent materials is increasing significantly, supported by systematic manipulations such as ‘papermills’ that produce fraudulent content for commercial purposes, and AI-generated content,” says an Elsevier spokesperson, adding that in response “we are increasing our investment in human oversight, expertise and technology.”

Open data

Argos developers emphasize that the site is based on open data collected by others. Sources include the website Retraction Watch, which maintains a database of retracted articles — free through an agreement with the nonprofit CrossRef — that includes the reasons for a retraction so that tools that review author data can focus on retractions that mention misconduct. The analysis is also based on Records of articles that heavily reference retracted works, which were compiled by Guillaume Cabanac, a computer scientist at the University of Toulouse in France.

Although Argos also tracks analysts who Networks of authors with a history of misconduct Focus, other integrity checking tools also flag articles based on suspicious content, such as close textual similarities to fake works or "disturbed phrases," a term coined by Cabanac when authors make strange choice of words to avoid activating plagiarism detection systems.

“Both approaches have merit, but identifying networks of researchers engaging in misconduct may be more valuable,” says James Butcher, a former Nature and Lancet publisher who now runs the consultancy Journalology in Liverpool, UK. “This is because AI-assisted writing tools could be used to help fraudsters avoid obvious textual clues,” he adds. Butcher adds that many major publishers have developed or purchased their own integrity tools to check for various red flags in manuscripts.

One of the trickiest problems for integrity tools that rely primarily on author retraction records is correctly distinguishing between authors with similar names — a problem that could skew Argos' numbers. “The problem of author discrimination is the biggest problem the industry has,” says Adam Day, founder of Clear Skies.

De Boer, who previously worked at Springer Nature, says anyone can create an account to use Argos for free, but Scitility plans to sell a version of the tool to major publishers and institutions that could integrate it directly into their manuscript screening workflows.

Butcher praises the transparency of the Argos team. “There needs to be more visibility for magazines and publishers who take shortcuts and fail to exercise appropriate care in the work they publish and monetize,” he says.