Writing for Research Fortnight, Jorge Quintanilla from the School of Physics and Astronomy, explains the view that scientists are safe from the disruptive threat of AI is dangerously complacent – and we need to stop judging researchers by their publications or risk a tsunami of fakes.
Scientists might think that their jobs are safe: after all, our task is to extract information from the natural world, not to create it. Generative AI, the argument runs, might have some niche applications in scientific work, or aid menial tasks such as retouching the draft of a paper, but it does not go to the core of the scientific process.
In the creative industries, generative AI tools such as ChatGPT or Midjourney threaten disruption not because what they produce is better than human work, but because they can produce it at scale. This is relevant to science, too—once you realise that there are fraudsters in the midst of the scientific community.
Infinite data
In 2002, Julia Hsu and Lynn Loo, then working at Bell Labs in the US, noticed two identical plots in separate papers co-authored by Jan Hendrik Schön. This observation lifted the lid on one of the most notorious frauds in the history of physics.
Over several years, Schön had fooled his collaborators and the scientific community at large into believing that he had achieved a series of breakthroughs that would revolutionise electronics. But he couldn’t resist the temptation to produce ever-more apparent discoveries at an ever-faster rate. In the end, he got sloppy, using the same data for more than one paper.
A modern-day fraudster armed with generative AI wouldn’t need to re-use old data. They could produce infinite amounts of seemingly genuine data, tailored to each new fraudulent paper, non-stop.
Tipping point
Scientific journals are not taking the threat from generative AI lightly. Nature requires authors to detail the use of AI tools, and has banned listing an AI as a co-author.
This is welcome, but unscrupulous authors will simply not own up. Generative AI will also make some ways of detecting fraudulent papers, such as requesting full data sets, of little use.
There is an argument that incorrect or made-up science gets detected by the normal operation of the scientific process. But this overlooks generative AI’s ability to scale fraud. A single, inexperienced individual can use the technology to produce an entire, plausible paper, including data, bibliography and text, in minutes. And that makes all the difference.
We do not have a predictive, quantitative theory of the scientific process. But it is reasonable to assume that, like any complex, self-correcting process, there is an error threshold beyond which it breaks down.
One feature of better understood self-correcting systems, from self-replicating biological molecules to error-corrected quantum algorithms, is that they don’t cross this threshold gradually, but suffer something akin to a phase transition, like the sudden freezing of water.
As the error rate rises, self-correcting systems keep working, but slow down. When their threshold is reached, failure is sudden: cells cannot replicate their DNA; quantum processors lose coherence; or the scientific process is no longer capable of the cumulative advancement of knowledge.
Old problem, new danger
Preventing this nightmare scenario starts with acknowledging the problem. We should consider that, even though the overwhelming majority of scientists are honest, swathes of the scientific literature are potentially fraudulent—and expect this fraction to get bigger really fast, potentially beyond a critical threshold.
Most importantly, we must tackle the root cause of the problem: an excessive reliance on papers as the end product of science. An individual or institution’s contribution to science should not be equated with the numbers of papers they have published, the journals they published in, or the citations their papers received. It is not even about how well-written or interesting those papers seem.
The real contribution is the discoveries and insights that resulted from their research. Until this is recognised, the totemisation of the scientific paper will leave science as vulnerable to generative AI as script writing or graphic art.
The problem has been recognised for some time; there is a growing effort to judge science not on its form, but on its underlying contents. But it is difficult to come up with alternative ways to quantify and assess scientific research. Generative AI is making this challenge much more urgent.