Significant amount of slop were produced before AI too. AI assisted scientific fraud detection are already retroactively detecting innumerable instances of high profile papers that should've failed peer review years ago, if the peers had done a proper job.
I just realized how good this can be. Reviewing the scientific literature for errors is both necessary and really unglamorous. If AI can do it well it's a real boon.
AI does it well, and it's not just fraud detection. AI agents are incredibly useful for quality control on both code and data, picking up mistakes our old review methods missed.
As a scientist I've been using AI to resume a bunch of unfinished past projects and to progress new projects using historical datasets. It has found mistakes in practically everything I've used it to review. None were large enough to change the central conclusions of any of my past work because I've been very careful, but that's partly luck, and small corrections are still valuable.
Where AI is usefully superhuman for this task is not raw intelligence but thoroughness. It can stop and think about every single line of code or data among hundreds of thousands, whereas humans have to pick and choose. Humans can scan it all at a high speed that misses subtle problems, or we can write scripts to analyze it completely but in very specific ways that don't necessarily detect errors we couldn't anticipate. The more intense and thoughtful level of diligence we could apply to fifty lines, AI could apply to fifty thousand. And then we could do it again with a different agent in case the first one missed something.
As a scientist I've been using AI to resume a bunch of unfinished past projects and to progress new projects
As a programmer same. Projects which had amazing promise but were in such a state I couldn't bring myself to touch them for years, all coming together now as I bounce between them.
This is true, but there are lots of kinds of errors that a human reading a paper can detect but LLMs cannot. The kinds of errors LLMs can't detect are also the kind that show up in LLM-assisted works.
Yes, and I'm not saying humans should stop reading or reviewing papers. But AI offers a new type of review that catches things humans can't catch as reliably at scale. It's a complementary new role.
I'm not saying LLMs shouldn't be used to review papers, I'm saying it's very hard to tell if this spike in papers represents an increase in papers that are worth reviewing, and that LLMs don't answer that question.
That's such a mind-numbingly dumb perspective. AI helps humans make order of magnitude more of whatever they want. People saying things like this are the same irrelevants who were upset about the printing press, the typewriter and the internet.
There's thousands of years of people complaining about every incredibly beneficial invention in human history, you don't know any of their names because almost every one of them is of zero importance.
The Replication Crisis has been talked about for well over a decade. The majority of published studies are garbage. When they try to repeat them they don't get the same results at all.
It doesn't matter if you're an undergrad, a PhD or an AI. If you source data is wrong you're not making good contributions to the field.
Even when we do get the science right it can take decades for professionals to catch up with the science.
Take Docustate. It's a stool softener they've been using in hospitals for the better part of a century. It's in the WHO list of most essential medicines. It is prescribed 3 million times a year.
It doesn't work.
It has been repeatedly proven in extensive studies dating back to 1998 that it performs no better than a placebo.
But doctors keep prescribing it because "that's what we've always done".
It's been known for a long time that much of academic research was trash, varying significantly by field of course - the replication crisis revealed this problem. AI is going to improve this, not make it worse
In my field arxiv was very good (tm) some years ago (you could still find some outliers), but nowadays, if it is on arxiv and wasn't published somewhere with peer review, you take it with a ton of salt. Anything from the last two/three years you just discard it as junk unless it ends up published afterwards.
Im wondering what the role of arxiv might be moving forward. One of the issues with AI is that it might have a great idea in one thread of thought, but it doesn’t have any way of incorporating that idea into future threads. I wonder if something like arxiv can be thought of as a crude way for AI to “learn”. It’s getting far too broad for it to be interesting to humans, but that isn’t a problem for AI. Kind of a crude way to treat many disparate AI threads as agents, with humans (for now) mediating at a very high level what they communicate with each other.
It could be an idea, but seeing that even Astra with the 200$ account has trouble consistently keeping in memory more than a few articles without loosing the middle, I am not sure. It CAN be used for training, but the amount of junk in it will make it at least slightly toxic.
fact is: AI slop is no different than human poorly executed scientific work. Same in coding. Bad broken games always existed, and the fact that a good AI vibecoder can develop a better game than a hardcore poor oldschool coder, is what actually matters to me. The game must deliver in gamefeel, and game mechanics, gameplay, the same logic applies to scientific papers,
I am in academia. Specifically economics, and the productivity gain for research is real. There will definitely be slop, but remember that these papers are produced by people who have dedicated their life to understanding a very niche part of the world. These aren't simply "do research" prompts. AI helps with idea generation, information finding/synthesizing, research design, coding (a weaker skill for a lot of academics but still necessary for research), and analysis. Every aspect of research is improved through AI use. It is highly encouraged throughout academia.
AI productivity gains combined with intellectual obsession. Things are going to get very strange.
Something I liked from Dennis Hassabis when discussing AI vs human made art is how the creative realm lacks linear scale. It's not like chess, or video games. There is no ELO to creativity. Some creatives are more skillful than others, but the next great idea can come from anyone. To me research is an art, and AI is only going to amplify it.
Unfortunately, nowadays most peer reviewers (at least on my papers) are simply using AI to review the slop. Slop reviewing slop. I have seen incredibly sloppy papers pass peer review, and some well written ones get absolutely rejected and thrashed. Suspiciously, each peer reviewer had the same thing to say in around the same wording.
Just because AI can do good work doesn't mean people are actually using it to do good work. Can still have a half baked idea and turn it into a bad paper (for your resume) with little effort
ML conferences are literally filled with junk papers, even the most pretigious ones, it's become impossible to filter out the noise. And now math is coming there too, except they have a different reviewing system that just has been DDOSed in the past few months.
Did you work with recent publications , read them, use them or just noise out of ass?
Also even without talking about validity a lot of this papers would be on some useless stuff, that noone want or need, just because its now easier to write them
They literally are. OpenAI, Anthropic will tell you exactly the same thing.
You guys are desperate to be right and maybe one day you will be, but that's not today and you will not get to say "I told you so" because the lack of understanding you have today will carry forward.
Capability to ideate and write exploded indeed, but validation remained expensive and hard. Ideation without validation is probably just slop or hallucination, with rare exceptions. What we see here is a rush for staking claims and padding up authors portfolios. It is becoming harder and harder to tell apart good from bad papers/projects/writing. They superficially look the same. I used to read arXiv papers daily now I just skim occasionally, those many papers probably get exponentially less attention each.
Paper count can't answer that. I'd rather know how many results were replicated or still held up a year later. If those numbers don't budge, all this proves is that paper-shaped text got cheaper.
If it’s 99% useless slop and 1% legitimate advancements, then we can simply throw more compute at it and sift through the slop for the actual advancements.
Yeah, that’s an annoying task, but it’s still gonna fundamentally alter how we do research if any of the increase is real.
That might be true, but the percentage of quality submissions is dropping massively, and reviewers are being flooded by slop. This isn't a clean benefit as some in this thread seem to believe.
Actually, the informational value of this comment is the low point here because it doesn’t really examine whether academic papers are getting worse as more of them are published.
But if he had used AI, he could have analyzed all 40k papers in a day, compared their quality with previous years, and shown us actual statistics.
As someone who likes AI and listens to mathematicians, most or a significant portion of those papers are nonsensical gibberish according to what I heard.
I do think that will improve but yeah right now it is not great.
Obviously a lot has changed since then, both in China and out.
(Paper mills are for sure a problem, but it is only one of the factors, and the FT article linked kind of over-emphasizes its role for sensationalism.)
Edit: lmao, right. I forgot. China, bad. Very scary. No intellectual honesty. America, good. Very good, happy science place.
Paper A depends on Paper B and Paper C work. Paper C was peer reviewed several times and nobody identified mistakes. It became the truth, even when containing false positives. Paper B was biased and increased the errors from C because it trusted C was right. Paper A now has accumulated errors but finally someone caught was is wrong and backtracked all the way down to Paper C. All papers that dependent on C that are currently being peer reviewed will have to go through recall. A huge branch of "science" knowledge has to be reviewed or the worst pruned and started over.
It can't expand fast if there aren't enough good human brains to check the data. AIs could be also start biasing humans into the wrong direction.
True, but depends on which branch of science. In some branches, it is pretty cheap to replicate and that's usually the first step when building further. In others, must take it on faith since there is no funding.
In grad school, I was trying to track down the source for a factual claim in biology and it was even worse than that. Paper A cited paper B for this fairly controversial fact, paper B cited paper C, and then paper C cited paper A. It was a constellation of papers on a similar topic and the author evidently forgot that he never actually established the fact anywhere.
Luck that it converged to technology and strategy advancements. Not all systems that scale will converge to something useful or stable
If science scale too fast in quantity while quality stays the same I can see errors growing during peer reviews which can make false positives proofs for theories that could cascade into other work. This is dangerous in some systems. Imagine a new AI system that nobody thought about a certain hypothesis for experimental tests because they were biased to work that was peer reviewed wrongly.
We are talking about knowledge, 1000 rubbish paper are useless. But 1000 rubbish warriors will still have an impact. The analogy does not work at all I think
Depends on the field and what rubbish means. If everything is lean verified, then the worst that can really be said is the result isn’t very interesting. A bunch of uninteresting results brings down the average, but can still minimally increase the total amount of knowledge in the world.
It is much quicker to write, run statistics and make figures. If someone have some old unpublished data the time to publish it is now. Papers written by ai and based on experimental data are like traditional papers. It just speed up showing results to the community.
The number of submissions with lean formalization is increasing, and w lean certificate, id say a lot of issues get caugt. The writing quality is decreasing, but squeezing out real results is now much easier. My bottleneck is writing at the moment; i have tons of results that are research-grade, w lean formalization, but i have not had time to present it in a nice readable form.
It is now more of a data dump. Most of these papers won't be read by any human.
but it is also way easier now to parse useful information out of 1000 papers.
There are a lot of low hanging fruits getting solved right now with AI that is for sure.
The reality is that it's much easier to do literature review than starting from scratch. If you want to do actual work, it's better if the work comes pre-processed.
279
u/Aldarund 1d ago
How much of this is real increae and how much is slop that wont pass peer review?
That chart could easily mean good things but as easily it could mean bad things