And those students then using AI to write assignments, so they never learn to research, evaluate data, & then write academic papers for themselves, and therefore have to rely on AI to write professional articles throughout their career... I've realised AI is nothing more than a pyramid scheme.

Yes, exactly.

This gets at a key problem posed by wide LLM adoption. It used to be, you had to put some *work* into plausible-sounding BS. Even AI hallucinations tend to *look right*. No “this looks like a lazy fabrication” tells. It’s not necessarily more errors than human, it’s *much better disguised* errors.

Given that I recently (for the first time) had to check a box to confirm I had not used AI in my review (for a journal I’ve reviewed for in the past, most recently in June), this may already be showing up, hence the new box.

What consequences do they get for falsifying references? It should be a professional punishment, like if an engineered were found to have faked the calculations for a building's safety. Banned from submitting to this and associated journals for a year or more, to start with.

In a book submitted to a publisher. This is a problem that goes well beyond students looking for ways to cut corners when writing a paper.

This is how LLMs work- the title of the work and the webpage URL are completely fabricated. If they did, like, 4 more seconds of validating they would be able to find the work they were looking for; the content, reference article, and even the website it based its response on are all real.

Happened to me as well. Half of the references in a submission to a top journal in my field were clearly hallucinated. The content was however plausible, even if somewhat incremental. Immediate reject.

I'd think publishers could (and should) develop detection methods for this sort of thing. It can't be all that hard to scan the bibliography and flag situations where a title can't be matched to an author. I wouldn't expect the editors to see it via manual checking. It's up to the publishers.

Since Google Scholar searches return citations, won't fabricated citations eventually turn up in them?

From a journal editor: I’m at the coalface of it here, and you wouldn’t believe some of what we see when editing. Proper academics who don’t seem to realise that they swim in what is actually a very small pond. 1/2

This is what I worry about.

AI is disrupting academic research as well (and fast)

This is insane! Generative AI goes rogue and makes things up. Plausible things, but still untrue.

Submitting an AI article with fake references to a peer reviewed journal should be a friable offence

Am reading a review copy and it is 🔥

If you're using AI, honestly, why are you in the game of scholarship? It's a hard life! Why choose it at all if you're just going to give the most interesting part of it to the machine?

I just had an issue with the index for a forthcoming book, which the publisher arranged and came back horribly mangled. I had never come across so many basic errors before.

This is not the first time I’ve heard of this and similar dodgy uses. I keep being told by AI truthers that there are good uses of AI but this certainly ain’t it.

Am reading a review copy and it is 🔥

“As for “existential terror,” there is simply no precedent in American history for the way Trump has turned the full force of the federal government against elite universities.”

“Written with passion, precision, honesty and courage. Essential reading on how American myopia, bias and hypocrisy set the stage for Gaza” — David D. Kirkpatrick “The world needs this book, now more than ever.’ — Fida Jiryis Pre-order America’s Middle East www.hurstpublishers.com/book/america...

A major setback in the First Word War

Message from a friend who’s a journal editor: ‘I’m at the coalface of it here, and you wouldn’t believe some of what we see when editing. Proper academics who don’t seem to realise that they swim in what is actually a very small pond. 1/2

Not just the students, Folks!!!!

All of these fake AI referenced articles should be immediately rejected and authors should not be given any opportunity to re-submit to the journal

There's been a small but clear presence of AI articles coming into @mwcjournal.bsky.social I really would like there to be a fieldwide blacklist of authors caught submitting them.

when the AI bubble finally collapses and everyone who spent a year smugly hyping it suddenly has to come crawling back with their tail between their legs, i just want to say i am not going to be the bigger person about it. i am going to be meaner than you can even believe

That’s actually what first raised a red flag as I read through the piece, it was a reference to a co-authored article by two academics whose work is so different I thought I had missed their joint article and I tried to track it down for my own reference.

It's even more fun when LLMs are generous enough to invent fictitious co-authors of the article you never wrote.

OMG so creepy! Same thing here. A chapter references an article that I allegedly published in Sociological Review 68 (3) in 2020 - except I didn't. The title sounds like I could have written it, and it also in part like another publication of mine, which is however on a different topic & not 2020.

Cambridge University Press just asked me to sign a document either agreeing or refusing to allow my recent Element to be used for AI training, with promise of some unspecified future revenues if I agree. Hell no.

Same happened to me. I reported it to the journal editor who is taking action, and our uni library services who are following up. we pay huge amounts for access to high quality journals - the journal publishers need to take action

That being said, the problem will only massively amplify from now on if we stick to our current academic practices (I hope to write something more detailed on this in a couple of weeks)

Reviewed a paper for a very good political theory journal last week, parts of it were clearly written with AI. A friend received reviewer reports from another one a couple of days ago. One report, clearly written with AI. I think editors also must do a much better job of screening articles/reviews.

This is something that I do with my students every semester now. I have them generate an essay and then I ask them to make the AI add citations to peer reviewed works in Chicago format. I then do a search for the citations and show them that the citations are completely bogus.

Given the immense scarcity of qualified reviewers, this is harming all real scholars. All journals should update their 'author instructions' to state that if it can be shown that a submission is generated with AI, the author will be blacklisted for future submissions to that journal.

The scale of the rabbit hole we are plunging into is very unsettling

Weirdly enough, I had a similar experience this summer. 2/3 of the references were made up. I didn’t provide a substantive review, just flagged these references. The piece got rejected. Based on dinner conversations with colleagues, it’s becoming very common.

Second case I see in about a month - this time peer review at least caught it.

Another example. Danyel Reiche is the purported author of ‘Qatar’s World Cup and the Challenge of Sports-Washing’ (Third World Quarterly, 44/2, 2023, 245-64) No such article exists (I checked the library catalog and with Danyel too). But if I google it, I get an AI summary of its probable content!

We are going to need a new option on Peer Review Forms: Accept Accept with Minor Changes Revise and Resubmit Reject Reject with Extreme Prejudice

Not only a problem in academia

There seem to be 2 types of reference fabrication - publications with the title listed correctly but attributed to the wrong author, and those which do not exist when I check with library catalogs.

I'm peer reviewing a manuscript for a reputable publisher and came across fake references which must be AI-generated - including to an article attributed to me which sounds fascinating but which I never wrote.

‘Grok can make mistakes, verify its outputs.’

Kristian Coates Ulrichsen unpacks #SaudiArabia’s changing approach to #Syria following the fall of the Assad regime, including the kingdom’s recent announcement of a $6 billion investment in the country. @kristianulrichsen.bsky.social

Opening the 15th Gulf Research Meeting in Cambridge with a timely and very relevant panel discussion

Very glad you liked it 🙏

Also includes Saudi international & Al Hilal opinions inc fans, Engulfed author @jamespiotr.bsky.social “everyone has their price”, analysis of Trump/US geo-political relations, @kristianulrichsen.bsky.social on Middle East instability impact & much more pod.fo/e/2f6765