By this point, most chatbot users have accepted the possibility that artificial intelligence (AI) tools will hallucinate in almost every scenario. Despite the efforts of AI content detectors, fact-checkers, and increasingly sophisticated large language models (LLMs), no developers have found a solution for this yet.
Also: Implementing AI? Check MIT's free database for the risks
Meanwhile, the consequences of misinformation are only getting higher: People are using generative AI (gen AI) tools like ChatGPT to create fake research.
A recent study published in the Harvard Kennedy School's Misinformation Review found 139 papers on Google Scholar, a search engine for scholarly literature, that appear to be AI-generated. The researchers found most of the "questionable" papers in non-indexed (unverified) journals, though 19 of them were found in indexed journals and established publications. Another 19 appeared in university databases, apparently written by students.
Even more concerning is the content of the papers. 57% of the fake studies covered topics like health, computational tech, and the environment — areas the researchers note are relevant to and could influence policy development.
Also: The best AI image generators of 2024: Tested and reviewed
After analyzing the papers, the researchers identified them as likely AI-generated due to their inclusion of "at least one of two common phrases returned by conversational agents that use large language models (LLM) like OpenAI's ChatGPT." The team then used Google Search to find where the papers could be accessed, locating multiple copies of them across databases, archives, and repositories and on social media.
"The public release of ChatGPT in 2022, together with the way Google Scholar works, has increased the likelihood of lay people (e.g., media, politicians, patients, students) coming across questionable (or even entirely GPT-fabricated) papers and other problematic research findings," the study explains.
Also: The data suggests gen AI boosts software productivity – for these developers
The researchers behind the study noted that theirs is not the first list of academic papers suspected to be AI-generated and that papers are "constantly being added" to these.
So what risks do these fake studies pose being on the internet?
Also: How do AI checkers actually work?
While propaganda and slapdash or falsified studies aren't new, gen AI makes this content exponentially easier to create. "The abundance of fabricated 'studies' seeping into all areas of the research infrastructure threatens to overwhelm the scholarly communication system and jeopardize the integrity of the scientific record," the researchers explain in their findings. They went on to note that it's worrisome that someone could "deceitfully" create "convincingly scientific-looking content" using AI and optimize it to rank on popular search engines like Google Scholar.
Back in April, 404 Media found similar evidence of entirely AI-fabricated books and other material on Google Books and Google Scholar by searching for the phrase "As of my last knowledge update," which is commonly found in ChatGPT responses due to its previously limited dataset. Now that the free version of ChatGPT has web browsing and can access live information, markers like this may be less frequent or disappear altogether, making AI-generated texts harder to spot.
While Google Scholar does have a majority of quality literature, it "lacks the transparency and adherence to standards that usually characterize citation databases," the study explains. The researchers note that, like Google Search, Scholar uses automated crawlers, meaning "the inclusion criteria are based on primarily technical standards, allowing any individual author — with or without scientific affiliation — to upload papers." Users also can't filter results for parameters like material type, publication status, or whether they've been peer-reviewed.
Also: I tested 7 AI content detectors – they're getting dramatically better at identifying plagiarism
Google Scholar is easily accessible — and very popular. According to SimilarWeb, the search engine had over 111 million visits last month, putting it just over academic databases like ResearchGate.net. With so many users flocking to Scholar, likely based on brand trust from all the other Google products they use daily, the odds of them citing false studies are only getting higher.
The most potent difference between AI chatbot hallucinations and entirely falsified studies is context. If users querying ChatGPT know to expect some untrue information, they can take ChatGPT's responses with a grain of salt and double-check its claims. But if AI-generated text is presented as vetted academic research conducted by humans and platformed by a popular source database, users have little reason or means to verify what they're reading is real.