OpenAI, the maker of the popular generative language model ChatGPT and Dall-E, has announced a new tool which will help detect AI-written content. It can “distinguish between text written by a human and text written by AIs from a variety of providers,” says the company.
“We’ve trained a classifier to distinguish between text written by a human and text written by AIs from a variety of providers,” reads the press release.
Also Read | ChatGPT is getting a paid version for those who don’t want to wait to talk to AI chatbot
Dubbed as Classifier, the new tool is easy to use. You should have an OpenAI account to use this tool. Simply paste the text and it will tell you if the text is very unlikely, unlikely, unclear if it is, possibly, or likely AI-generated. The tool is publicly available.
OpenAI has created Classifier using a dataset of pairs of human-written text and AI-written text on the same topic. The company collected this dataset from a variety of sources including human-written ones and the ones generated on prompts submitted to InstructGPT.
“We divided each text into a prompt and a response. On these prompts we generated responses from a variety of different language models trained by us and other organizations. For our web app, we adjust the confidence threshold to keep the false positive rate low; in other words, we only mark text as likely AI-written if the classifier is very confident.”
Also Read | ChatGPT to soon replace software engineers? OpenAI reportedly training AI to code at scale
The maker of Classifier warns that the classifier is not fully reliable and that it should not be used as a “primary decision-making tool” to determine the source of the content. During the evaluation of Classifier, it found that the classifier correctly identified “26% of AI-written text (true positives) as “likely AI-written,” while incorrectly labeling human-written text as AI-written 9% of the time (false positives).” The company says that the tool’s reliability improves as the length of the input text increases.
Classifier, as per the company, is unreliable for content under 1,000 characters. Sometimes, even the longer texts can be incorrectly labelled by the classifier. The tool comes with several other limitations like sometimes the human-written text can be labelled as AI-written or determining the source for predictable texts.
For the unawares, ChatGPT is a chatbot created by OpenAI and sees investments from tech giant Microsoft. It is an AI-based chatbot that is trained to conduct human-like conversations over complex and all kinds of topics.
Also Read | ChatGPT can “destroy” Google in two years, says Gmail creator
