New Tool Can Tell If Something Is AI-Written With 99% Accuracy

Raw Text

Breaking

2 hours ago

Tropical Storm Nigel Likely To Become Hurricane Tonight—Will Approach Major Hurricane Status This Week

3 hours ago

Indiana AG Sues State’s Largest Health System Over Doctor Who Discussed Abortion For 10-Year-Old Rape Victim

4 hours ago

Billy Miller—Soap Opera Star—Dies At 43: Here Are The Biggest Celebrity Deaths Of 2023

5 hours ago

'Barbie' Just Surpassed 'The Avengers' In U.S. Earnings—Here Are The 10 Movies Still Beating It

7 hours ago

Horror Movies Lead Box Office As ‘Haunting In Venice’ And ‘Nun 2’ Nearly Tie

7 hours ago

Drew Barrymore Backtracks: Talk Show Paused Until Hollywood Strikes End After Union Criticism

8 hours ago

Trump Calls For Government Shutdown If GOP Can’t Get An ‘Appropriate Deal’ As Deadline Looms

8 hours ago

Russell Brand’s Agents Drop Star Amid Sexual Assault Claims—And Say They Were ‘Horribly Misled’

+1 day ago

California Sues 5 Oil Giants: Here Are The Biggest Numbers Behind The Explosive Lawsuit

+1 day ago

Lee Slams Canada And Maine In Rare Northeast Post-Tropical Cyclone (Photos)

+1 day ago

Russell Brand Accused Of Rape And Sexual Assault By Four Women, Report Says

+1 day ago

Texas Senate Acquits AG Ken Paxton After Impeachment In House

+1 day ago

Shortage Of Weight-Loss Drugs Like Wegovy And Ozempic Persist—And Could For ‘Some Years’

+1 day ago

Maui Fire Fatalities Drop To 97: Here’s Why The Death Toll Has Been So ‘Fluid’

+1 day ago

Kansas Will No Longer Recognize Transgender Identities On Birth Certificates

+1 day ago

Prosecutors Seek Gag Order For Trump In Election Interference Case

+1 day ago

Ashton Kutcher Resigns From Anti-Child Sex Abuse Firm After Danny Masterson Support Caused Backlash

+1 day ago

It’s Fall Foliage Time (Almost): Here’s When—And Where—Colors Will Peak In The U.S. This Year

+1 day ago

Trump Could Have Unleashed Violence If He Knew About Search Warrant For His Tweets, DOJ Argued

+1 day ago

Stocks Suffer Worst Day Since August As Big Tech Staggers

+1 day ago

Pentagon Interviewing New Witnesses Of 2021 Kabul Airport Attack—Though Investigation Is Officially Closed

Edit Story

Forbes

Innovation

Breaking

Arianna Johnson Forbes Staff

I cover the latest trends in science, tech and healthcare.

Following

01:06pm EDT

|

Share to Facebook

Share to Twitter

Share to Linkedin

Topline

A new study found an AI detector developed by the University of Kansas can detect AI-generated content in academic papers with a 99% accuracy rate, one of the only detectors on the market specifically geared toward academic writing.

Genuine human error? Perhaps, but at least not artificial intelligence.

Key Facts

According to a report published Wednesday in the journal Cell Reports Physical Science, researchers created a tool that can prove AI detection in academic papers with 99% accuracy.

The team of researchers selected 64 perspectives (a type of article) and used them to make 128 articles using ChatGPT, which was then used to train the AI detector.

The model had a 100% accuracy rate of identifying human-created articles from AI-generated ones, and a 92% accuracy rate for identifying specific paragraphs within the text.

Tangent

According to a survey by BestColleges.com, 89% of college students have admitted to using ChatGPT to help with assignments, while 34% of educators believe the software should be banned, though 66% support students having access.

Crucial Quote

“Right now, there are some pretty glaring problems with AI writing," lead author Heather Desaire said in a statement. "One of the biggest problems is that it assembles text from many sources and there isn't any kind of accuracy check—it's kind of like the game Two Truths and a Lie."

Contra

AI detectors have not proven to be 100% accurate. A University of California at Davis student alleges she was falsely accused by her university of cheating with AI. After uploading a paper for one of her classes, she received an email from her professor claiming a portion was flagged in the program Turnitin as being AI-generated. Her case was immediately forwarded to the Office of Student Support and Judicial Affairs, which handles discipline for academic misconduct. The student pleaded her case and ultimately won, using time stamps to prove she wrote the paper. This wasn’t the only time educators falsely labeled assignments as AI-generated. A Texas A&M commerce professor attempted to flunk over half of his senior class after using ChatGPT to test whether the students used the chatbot to write their papers. He copied and pasted the papers into ChatGPT and asked if it wrote them, to which it replied yes. This caused their diplomas to be withheld by the university, though the professor offered the opportunity to redo the assignment. However, the university confirmed to Insider no students were failed or barred from graduating.

Key Background

OpenAI’s chatbot, ChatGPT, was opened to the public in November 2022, and in less than a week surpassed the million-users mark, with people using it for things like creating code and writing essays. The AI’s intelligence led to several schools either indefinitely or temporarily banning the software, including New York Public Schools, Seattle Public Schools and the Los Angeles Unified School District.

Tangent 2

There are already many existing programs and services that promise to identify AI-written content.

TurnitIn released an AI detection tool for papers. Before, it only had the capability to check for plagiarism. The feature has been added to its similarity report and shows an overall percentage of the amount of work AI software generated within the paper. The company claims its AI detection tool is 98% accurate at spotting AI-written work. Its detection model is trained to detect content from GPT-3 and GPT-3.5 language models, including ChatGPT. However, it claims that because GPT-4’s writing characteristics are similar to earlier models, it can detect content from this version “most of the time.”

Copyleaks claims to have an AI detection accuracy rate of 99%. Its software can detect AI-generated text across several models, including GPT-4 and earlier versions, and content created with Jasper AI. It also says it can detect AI content in multiple languages, including Spanish, Russian, French, Dutch and German.

Winston AI launched in February, and claims it performs this task with 99% accuracy. It only supports detection in English and French, though the company is looking to expand this soon to Spanish and German. It can detect content made using ChatGPT, Bard, Bing Chat, GPT-4 and other text generation tools.

OpenAI’s Classifier launched in January to distinguish between AI-written and human-written text. Though these are the same makers behind ChatGPT, the tool isn’t very accurate. It has a success rate of around 26% and incorrectly labels human work as AI work 9% of the time. However, OpenAI claims the accuracy increases as the length of the text increases as well. It’s “very unreliable” on texts with 1,000 characters or less, and OpenAI only recommends using the software on documents written in English. The company also warns against using Classifier as the primary decision making tool and suggests using it to complement other methods of detection.

AI Writing Check was developed by Quill and CommonLit to help teachers check for AI-created work in assignments. Its developers predict its accuracy is between 80% and 90%. It only allows detection for text up to 400 words at a time and for anything longer, users must break it down into sections. The detection software was created by OpenAI and is able to identify language syntactical patterns within text that aren’t quite humanlike.

Further Reading

ChatGPT In Schools: Here’s Where It’s Banned—And How It Could Potentially Help Students (Forbes)

Here’s What To Know About OpenAI’s ChatGPT—What It’s Disrupting And How To Use It (Forbes)

Follow me on

Twitter

LinkedIn

Send me a secure  tip

Arianna Johnson

Editorial Standards

Print

Reprints & Permissions

Single Line Text

Breaking. 2 hours ago. Tropical Storm Nigel Likely To Become Hurricane Tonight—Will Approach Major Hurricane Status This Week. 3 hours ago. Indiana AG Sues State’s Largest Health System Over Doctor Who Discussed Abortion For 10-Year-Old Rape Victim. 4 hours ago. Billy Miller—Soap Opera Star—Dies At 43: Here Are The Biggest Celebrity Deaths Of 2023. 5 hours ago. 'Barbie' Just Surpassed 'The Avengers' In U.S. Earnings—Here Are The 10 Movies Still Beating It. 7 hours ago. Horror Movies Lead Box Office As ‘Haunting In Venice’ And ‘Nun 2’ Nearly Tie. 7 hours ago. Drew Barrymore Backtracks: Talk Show Paused Until Hollywood Strikes End After Union Criticism. 8 hours ago. Trump Calls For Government Shutdown If GOP Can’t Get An ‘Appropriate Deal’ As Deadline Looms. 8 hours ago. Russell Brand’s Agents Drop Star Amid Sexual Assault Claims—And Say They Were ‘Horribly Misled’ +1 day ago. California Sues 5 Oil Giants: Here Are The Biggest Numbers Behind The Explosive Lawsuit. +1 day ago. Lee Slams Canada And Maine In Rare Northeast Post-Tropical Cyclone (Photos) +1 day ago. Russell Brand Accused Of Rape And Sexual Assault By Four Women, Report Says. +1 day ago. Texas Senate Acquits AG Ken Paxton After Impeachment In House. +1 day ago. Shortage Of Weight-Loss Drugs Like Wegovy And Ozempic Persist—And Could For ‘Some Years’ +1 day ago. Maui Fire Fatalities Drop To 97: Here’s Why The Death Toll Has Been So ‘Fluid’ +1 day ago. Kansas Will No Longer Recognize Transgender Identities On Birth Certificates. +1 day ago. Prosecutors Seek Gag Order For Trump In Election Interference Case. +1 day ago. Ashton Kutcher Resigns From Anti-Child Sex Abuse Firm After Danny Masterson Support Caused Backlash. +1 day ago. It’s Fall Foliage Time (Almost): Here’s When—And Where—Colors Will Peak In The U.S. This Year. +1 day ago. Trump Could Have Unleashed Violence If He Knew About Search Warrant For His Tweets, DOJ Argued. +1 day ago. Stocks Suffer Worst Day Since August As Big Tech Staggers. +1 day ago. Pentagon Interviewing New Witnesses Of 2021 Kabul Airport Attack—Though Investigation Is Officially Closed. Edit Story. Forbes. Innovation. Breaking. Arianna Johnson Forbes Staff. I cover the latest trends in science, tech and healthcare. Following. 01:06pm EDT. |. Share to Facebook. Share to Twitter. Share to Linkedin. Topline. A new study found an AI detector developed by the University of Kansas can detect AI-generated content in academic papers with a 99% accuracy rate, one of the only detectors on the market specifically geared toward academic writing. Genuine human error? Perhaps, but at least not artificial intelligence. Key Facts. According to a report published Wednesday in the journal Cell Reports Physical Science, researchers created a tool that can prove AI detection in academic papers with 99% accuracy. The team of researchers selected 64 perspectives (a type of article) and used them to make 128 articles using ChatGPT, which was then used to train the AI detector. The model had a 100% accuracy rate of identifying human-created articles from AI-generated ones, and a 92% accuracy rate for identifying specific paragraphs within the text. Tangent. According to a survey by BestColleges.com, 89% of college students have admitted to using ChatGPT to help with assignments, while 34% of educators believe the software should be banned, though 66% support students having access. Crucial Quote. “Right now, there are some pretty glaring problems with AI writing," lead author Heather Desaire said in a statement. "One of the biggest problems is that it assembles text from many sources and there isn't any kind of accuracy check—it's kind of like the game Two Truths and a Lie." Contra. AI detectors have not proven to be 100% accurate. A University of California at Davis student alleges she was falsely accused by her university of cheating with AI. After uploading a paper for one of her classes, she received an email from her professor claiming a portion was flagged in the program Turnitin as being AI-generated. Her case was immediately forwarded to the Office of Student Support and Judicial Affairs, which handles discipline for academic misconduct. The student pleaded her case and ultimately won, using time stamps to prove she wrote the paper. This wasn’t the only time educators falsely labeled assignments as AI-generated. A Texas A&M commerce professor attempted to flunk over half of his senior class after using ChatGPT to test whether the students used the chatbot to write their papers. He copied and pasted the papers into ChatGPT and asked if it wrote them, to which it replied yes. This caused their diplomas to be withheld by the university, though the professor offered the opportunity to redo the assignment. However, the university confirmed to Insider no students were failed or barred from graduating. Key Background. OpenAI’s chatbot, ChatGPT, was opened to the public in November 2022, and in less than a week surpassed the million-users mark, with people using it for things like creating code and writing essays. The AI’s intelligence led to several schools either indefinitely or temporarily banning the software, including New York Public Schools, Seattle Public Schools and the Los Angeles Unified School District. Tangent 2. There are already many existing programs and services that promise to identify AI-written content. TurnitIn released an AI detection tool for papers. Before, it only had the capability to check for plagiarism. The feature has been added to its similarity report and shows an overall percentage of the amount of work AI software generated within the paper. The company claims its AI detection tool is 98% accurate at spotting AI-written work. Its detection model is trained to detect content from GPT-3 and GPT-3.5 language models, including ChatGPT. However, it claims that because GPT-4’s writing characteristics are similar to earlier models, it can detect content from this version “most of the time.” Copyleaks claims to have an AI detection accuracy rate of 99%. Its software can detect AI-generated text across several models, including GPT-4 and earlier versions, and content created with Jasper AI. It also says it can detect AI content in multiple languages, including Spanish, Russian, French, Dutch and German. Winston AI launched in February, and claims it performs this task with 99% accuracy. It only supports detection in English and French, though the company is looking to expand this soon to Spanish and German. It can detect content made using ChatGPT, Bard, Bing Chat, GPT-4 and other text generation tools. OpenAI’s Classifier launched in January to distinguish between AI-written and human-written text. Though these are the same makers behind ChatGPT, the tool isn’t very accurate. It has a success rate of around 26% and incorrectly labels human work as AI work 9% of the time. However, OpenAI claims the accuracy increases as the length of the text increases as well. It’s “very unreliable” on texts with 1,000 characters or less, and OpenAI only recommends using the software on documents written in English. The company also warns against using Classifier as the primary decision making tool and suggests using it to complement other methods of detection. AI Writing Check was developed by Quill and CommonLit to help teachers check for AI-created work in assignments. Its developers predict its accuracy is between 80% and 90%. It only allows detection for text up to 400 words at a time and for anything longer, users must break it down into sections. The detection software was created by OpenAI and is able to identify language syntactical patterns within text that aren’t quite humanlike. Further Reading. ChatGPT In Schools: Here’s Where It’s Banned—And How It Could Potentially Help Students (Forbes) Here’s What To Know About OpenAI’s ChatGPT—What It’s Disrupting And How To Use It (Forbes) Follow me on. Twitter. LinkedIn. Send me a secure  tip. Arianna Johnson. Editorial Standards. Print. Reprints & Permissions.