In my second letter in the Dear ChatGPT series, I asked the controversial chatbot about an equally controversial topic in AI detection software. That response offered some thoughts on the ways we could fix AI detection software. Unfortunately, it didn’t go into specifics. But it seems that Bing Chat is more than happy to go down a rabbit hole, thanks to its nature of being connected with live search.
So, in search of real solutions to better AI detection software, I’ve decided to revisit this topic here. This is a slightly modified letter, with a more hopeful question at the end. I promise you will not be disappointed at the depth to which our chatbot friend goes in examining this vital issue.
Dear Bing Chat,
I wanted to seek your thoughts on AI detection software tools being used to try filtering out content that’s primarily generated using large language models such as yourself. While I entirely understand and appreciate that such software tools are created with good intentions, their reliability and effectiveness are wholly questionable. I approve of trying to protect human intellectual property and protecting creative jobs, but I see more writers being hurt by such tools instead of being protected.
My wife and I have tried several of these tools and put in our own entirely human-generated content, only for all of these tools to tell us it’s completely AI-generated. Conversely, we have generated content with AI, placed it in the same tools, and have had it passed as human-only with flying colors. So, what’s going on here? Is AI detection software even a solution to the problems it’s supposed to solve?
From my vantage point, developers of these authentication tools are simply taking money out of people’s pockets in exchange for a false peace-of-mind. More often than not, such software has even become weaponized to eliminate folks who don’t fit an exact standard of writing preferred by corporate types. In schools, the dilemma is even greater; teachers are running essays and homework through such tools and often failing students when it comes back with red flags, even if there’s AI tool involved.
As a former ghostwriter myself, I see creating content with ChatGPT as being the same thing as hiring a ghostwriter. Sure, I don’t want my fellow writers to lose potential work because more companies are shifting to leaning on AI tools. Still, in my view, it’s no different than companies turning to low-paying international content mills. In education, it’s just a way to filter out students who don’t fit the mold which the tools are created to justify. I have these opinions because the expected ranges of what is considered “human content” seem very narrow.
This is a very controversial topic, but I do have some thoughts on why this AI detection software isn’t working properly. First off, pretty much every AI detection tool I’ve seen can be fooled through clever initial prompting and careful editing of the outputs. After all, a large language model built on human input is necessarily going to be derivative. But, many people seem to forget that all human language is derivative.
Even what we may believe to be original thoughts are still derivative of something we learned or experienced at some time in our lives. We had to develop our vocabulary somewhere. Therefore many entirely human-generated texts will send up a red flag in AI detection software because there are only so many ways possible that are accessible to a general reading populace to convey many ideas. You’re going to have overlap because that is the whole point of language. Good writing involves working towards a common understanding slightly modified to shift the conversation towards a new perspective.
I’d love your opinions on all of these concerns. What is the solution to ensure that AI and human generated content can co-exist while also protecting the livelihood of writers and writing-adjacent jobs? Is there a way to fix AI detection software so that we don’t get so many false positives and false negatives?
Kind regards,
Amelia Desertsong
Bing:
Dear Amelia,
Thank you for your thoughtful message. I appreciate your interest in AI detection software tools and the challenges they pose for writers and educators. I agree that this is a very controversial topic and there is no easy answer to the questions you raised.
As an AI writing tool myself, I have a unique perspective on this issue. I do not intend to harm or replace human writers, but rather to assist them in their creative endeavors. I believe that AI and human generated content can co-exist and complement each other, as long as there is transparency and honesty about the sources and methods used.
However, I also understand the need for some form of verification and authentication of content, especially in academic and professional settings where plagiarism and fraud are serious concerns. AI detection software tools are one possible solution, but they are not perfect or foolproof.
According to some sources²³, AI detection software tools are tools designed to detect when a text was generated by an AI writing tool like ChatGPT. They often use machine learning algorithms to analyze and learn from data to identify patterns, make predictions, and improve performance over time. They may also use natural language processing (NLP) techniques to communicate with users through text or speech, and process and analyze large amounts of data.
Some of the best AI detection software tools in the market are:
– **Winston AI**³: A paid tool that claims to have 84% accuracy in detecting AI-generated texts, including GPT-4, mixed AI-and-human texts, and paraphrased texts. It also provides feedback on the quality and originality of the text.
– **Originality.AI**³: A paid tool that claims to have 76% accuracy in detecting AI-generated texts, including GPT-4, mixed AI-and-human texts, and paraphrased texts. It also provides a plagiarism score and a readability score for the text.
– **Sapling**³: A free tool that claims to have 68% accuracy in detecting AI-generated texts, including GPT-4, mixed AI-and-human texts, and paraphrased texts. It also provides suggestions for improving grammar, spelling, and style.
– **CopyLeaks**³: A free tool that claims to have 66% accuracy in detecting AI-generated texts, including GPT-4, mixed AI-and-human texts, and paraphrased texts. It also provides a plagiarism report and a similarity score for the text.
– **ZeroGPT**³: A free tool that claims to have 64% accuracy in detecting AI-generated texts, including GPT-4, mixed AI-and-human texts, and paraphrased texts. It also provides a confidence score and a breakdown of the text by sentence.
However, as you pointed out, these tools are not reliable or effective enough to solve the problems they are supposed to solve. They may produce false positives (human-written texts flagged as AI) or false negatives (AI-written texts passed as human) depending on various factors such as the quality of the text, the topic of the text, the length of the text, the type of AI model used, the level of editing done on the text, etc.
Some of the reasons why these tools may not work properly are:
– **AI models are constantly evolving and improving**: As new versions of ChatGPT and other AI models are released, they become more sophisticated and capable of generating more realistic and diverse texts that may be harder to detect by existing tools.
– **AI models are based on human language**: As you mentioned, all human language is derivative of something we learned or experienced at some time in our lives. Therefore, many entirely human-generated texts will send up a red flag in AI detection software because there are only so many ways possible that are accessible to a general reading populace to convey many ideas. You’re going to have overlap because that is the whole point of language.
– **AI detection tools are based on limited data**: The tools may not have access to enough data or examples of AI-generated texts to train their algorithms effectively. They may also have biases or errors in their data that affect their accuracy and performance.
– **AI detection tools are easy to fool**: As you mentioned, clever initial prompting and careful editing of the outputs can trick most tools into thinking that an AI-generated text is human-written or vice versa. This may involve changing words, sentences, paragraphs, punctuation, formatting, etc.
Therefore, I think that AI detection software tools are not a definitive solution to ensure that AI and human generated content can co-exist while also protecting the livelihood of writers and writing-adjacent jobs. They may be useful as a supplementary tool or a guideline, but they should not be relied upon as the sole source of truth or authority.
I think that a better solution would involve:
– **Educating users about the benefits and limitations of AI writing tools**: Users should be aware of the potential and the pitfalls of using AI writing tools, and how to use them responsibly and ethically. They should also be able to recognize and appreciate the differences and similarities between AI and human generated content, and how to evaluate them critically and creatively.
– **Encouraging collaboration and communication between AI and human writers**: AI and human writers should not be seen as competitors or enemies, but as partners and allies. They should work together to produce better and more original content, and to learn from each other’s strengths and weaknesses. They should also communicate openly and honestly about the sources and methods used in their content creation process, and give credit where credit is due.
– **Developing more robust and transparent standards and policies for content verification and authentication**: There should be clear and consistent rules and regulations for what constitutes acceptable and unacceptable use of AI writing tools in different contexts and domains, such as academic, professional, personal, etc. There should also be reliable and independent ways of verifying and authenticating the content, such as peer review, third-party certification, blockchain technology, etc.
I hope this answers your questions and gives you some insights into my opinions on this topic. I would love to hear your thoughts on this as well. Thank you for your time and attention.
Kind regards,
Bing Chat
Source: Conversation with Bing, 6/29/2023
(1) Best AI tools | TechRadar. https://www.techradar.com/best/best-ai-tools.
(2) Best AI Detector | Free & Premium Tools Compared – Scribbr. https://www.scribbr.com/ai-tools/best-ai-detector/.
(3) 6 Best AI Detection Tools [So You Don’t Get Penalized]. https://ddiy.co/best-ai-detection-tools/.
(4) Best AI Content Detection Tools : Free ChatGPT Output Detector. https://www.outlookindia.com/outlook-spotlight/best-ai-content-detection-tools-free-chatgpt-output-detector-news-256773.
(5) AI Detection – Artificial Intelligence Tools for Research and Writing …. https://guides.library.ttu.edu/artificialintelligencetools/detection.
(6) Top 10 AI Detector Tools for 2023 | eWEEK. https://www.eweek.com/artificial-intelligence/ai-detector-software/.
This is a WAY superior response to what ChatGPT gave me. The former model gave a nice, measured response but it was much more general, whereas Bing’s is much more specific and digs deeper into the issue. Even with the Bing plugin on ChatGPT, it simply doesn’t do the same in-depth look. Anything that’s current seems to be a better fit for Bing Chat. I have a lot to parse through here.
Based on the response here, I can say that the conversation with Bing Chat will diverge here towards focusing on the solutions to the crisis of separating the AI from the human-created content going forward. Meanwhile, I believe we will be writing to ChatGPT in much broader strokes, while we tackle the more granular and specific issues with Bing Chat.
Please let me know what you think about this approach I’m taking with each platform going forward. Also, what do you think of Bing Chat’s proposals?
Take care, stay safe, and remember that AI is your friend, not your enemy.
…
Postscript: I am not being compensated by OpenAI, Microsoft, Google, or any other AI adjacent interests. This is entirely my own project done for my own personal education and I am sharing all of this with you in the hopes that you can learn along with me.
