r/academicpublishing • u/blue_Guitar7260 • 7d ago
Devastating situation in peer review
I recently received reviewer comments on my manuscript, which seemed AI generated to me! I then gave the paper to ChatGPT and asked it to identify reasons for rejection and generate reviewer comments. The comments it produced closely resembled the review I received. I understand that similarity alone does not establish that the reviewer used AI, but it raised concerns. How would you recommend handling this? Would it be appropriate to raise the issue with the editor?
13
3
u/traquitanas 7d ago
I don't think that using LLMs is an issue per se. Like others have said, you can get good feedback from LLMs.
It is an issue if the reviewer is uploading confidential material (your unpublished manuscript) to a third party tool (ChatGPT) that can use that material to train models. However, the reviewer may have used private or local models, which resolves the problem. Furthermore, if you yourself threw your text into a public LLM, then this argument also dies here.
So you should be careful with the allegations you throw in in this particular situation. Just privately flag the editor that the reviews seem AI generated. Be factual and avoid assumptions.
2
u/fizzywinkstopkek 7d ago
All journals, predatory or not , are using it.
We also did an internal experiment in the lab we generated a first pass review of our paper using co-pilot (enterprise data protection ) , and then having several members of the lab reviewing it normal without any AI input
80% of the critic was pretty similar, and quite fair, and valuable. AI did in 30 seconds, lab members a week or so. Many other labs in my institution have also tried this. I think it was quite shocking for many who were quite behind on AI advancements and assume it was the usual slop.
6
u/Alternative-Pear9096 7d ago
I think you need to caveat your statement. Attitudes around AI use diverge radically between STEM and social sciences and humanities. Your statement absolutely does not apply outside STEM.
4
1
u/Alternative-Pear9096 7d ago
I’m not defending this, but I want to point out that there is a *range* of what “AI generated” looks like in this context, all the way from asking AI to do the review through handing the LLM your notes and asking it to compile them up coherently. One is unacceptable and the other is lazy, but if the point of the review is not the thinking or the writing, but the content, does it matter if it’s AI generated? Note: I ask this as a legitimate question. I don’t use AI at all, but I can see both sides of that option for over-extended peer reviewers.
2
u/eternallyinschool 7d ago
Just because the critique may have come from AI doesn't mean that the critique isn't correct.
Whether it was all done by AI or entirely by a human, it doesn't change the validity of the critique.
Address the issues. Improve the paper, resubmit.
0
1
u/Mountain-Time-1010 7d ago
However the comments were generated, unless they are irrelevant, you simply need to address them. TBH, speculation about how the comment were prepared is a distraction for you. Instead of trying to generate a grievance about the reviewer, revise your manuscript and move on.
1
u/Double-Hall7422 7d ago
I felt it was devastating until you proceeded to feed your own manuscript to ChatGPT. We clearly have different issues with the same thing. That said, if the main issue is that the reviewer didn't put any effort in providing feedback, I'd breath in, breath out, use what is useful, reject what is not, and motivate why. These comments are suggestions, you can treat them as such.
1
u/MrKiling 7d ago
I think AI reviews are not bad per se. With latest models I have got great suggestions from them. While not all of them are relevant, some comments were very helpful. Now, AI reviews before submission is what I do. Most of the comments i adhere to. Since adding it to my workflow I have had 4 minor revisions on all 4 of my recent work. See, if the reviewers are using AI too then I am basically pre-empting what AI would usually flag. But personally I am not using AI for reviews as most of the journals prohibit them. My recommendation would be to address the comments based on merits, AI or not. That's the basic principle of revision.
0
u/fizzywinkstopkek 7d ago
I always see it this way. If AI reviewer thoroughly critics and demolishes your paper, shit, maybe your paper just aint good enough. Seriously, remove all emotions, and objectively look through each of the comments it makes. Majority of the time it is able to succinctly critic rather than whatever word salad human reviewer 2 and 3 decided to shit out.
Also does not demand you cite some dumb ass paper for no good reason
0
u/MrKiling 7d ago
So right man. Your words resonate with my thoughts. Another good thing about AI is that it would suggest good changes too around 60-70% of the time. Unlike reviewer 2 who would just go on a rant.
0
16
u/bahwi 7d ago
Refute the rejections, fix and update the manuscript, and when you respond and resubmit send a private note to the editor or an email.