OpenAI Shutting Down Its Detection Tools Means Nearly Nothing
Plus, "The Score" releases a new episode. Plus, I was busy last week.
Issue 228
To join the 3,409 smart people who subscribe to “The Cheat Sheet,” enter your e-mail address below. New Issues every Tuesday and Thursday. It’s free:
If you enjoy “The Cheat Sheet,” please consider joining the 14 amazing people who are chipping in a few bucks a month via Patreon. Or joining the 14 outstanding citizens who are now paid subscribers. Thank you!
OpenAI/ChatGPT Shut Down Detection Tool
As you likely know by now, OpenAI, the maker of ChatGPT, shut down its AI detection tool last week. Several media outlets reported the news.
In reports, OpenAI said it was shuttering its detector due to low accuracy rates. If you read “The Cheat Sheet,” the low accuracy rates of OpenAI’s detector are not news. Every second it was available, it was the worst classifier. Now, GPTZero has that honor, though not for long as it’s going away too (see Issue 217).
Now, I am sure it’s a coincidence that the makers of both detection systems have started befriending cheating companies - OpenAI with Chegg and GPTZero with Course Hero. And it should also be obvious that, in the case of OpenAI, there is no real business case for making their product easy to find.
All that aside, the amazing part about the sunsetting of OpenAI’s classifier is how so many people wanted to use it as a proxy for the entire endeavor of identifying text that’s similar to AI-created text. So many people drew the illogical conclusion that, because OpenAI’s detector was awful and closing, all AI detection was awful. That’s like saying that Mercedes must be terrible because Yugo closed its doors.
Here is one example, from Times Higher Ed:
Toby Walsh, Laureate fellow and Scientia professor of artificial intelligence at UNSW Sydney, tweeted: “If the company that builds these chatbots give up on detecting chatbots (with all their inside information on weights, guardrails…) then there’s probably no hope for outsiders like Turnitin to detect real v fake text reliably.”
Ugh. No.
There are good AI classifiers on the market. The one OpenAI built was never one of them. I cannot say why exactly, but it’s true. And we said so, early and often. Maybe finding AI was never really in their core business model. But whatever the reason, its demise has zero to do with how well other systems work or whether they should be used.
“The Score” Podcast Releases New Episode
“The Score,” the podcast focused on academic integrity hosted by education reporter Kathryn Baron, released a new episode this week with an interview of Pete Van Dyke, the Certification Security Program Manager at Amazon Web Services. According to the show, that’s “the office responsible for minimizing cheating among people taking professional certification exams.”
As usual, it’s worth a listen.
When asked what people do to try to cheat on their exams, Van Dyke said:
It's not unusual for someone to try and have a third person, a third party individual in the room with them to help with the exam and indicate which items, which answers. And we've seen evidence in the past of people using things like recording devices built into eyeglass frames or even using earbud-type communicators so that someone can communicate with them what the correct answer is for items.
Adding:
One of the funnier instances that we've had of exam misbehavior, we had a candidate that actually fell asleep during his exam. His head was leaned over and he was snoring very loudly for about a 10 to 15 minute period, yet his exam continued to move because the exam proxy tester didn't realize that the candidate was sleeping and he was just moving forward as had been planned.
Funny, and true. The level of exam fraud, even and especially in professional testing, is enormous.
I don’t want to spoil the whole thing so I’ll share just one more bit. When asked about the size of the proxy test-taking market, Van Dyke says:
industry-wide, we estimate that this is a multi-hundred million dollar a year business. It's not unusual for a proxy tester to charge as much as $1,200, above and beyond the cost of an exam for someone to have an exam taken for them.
Taking tests for other people - in professional tests that confer credentials - is a multi-hundred million-dollar-a-year business. Just proxy test-taking. Just in professional testing. I think he’s right (see Issue 223).
If you have time, go listen.
Missing Me on Thursday
You may have noticed, there was no Issue of “The Cheat Sheet” last Thursday. Judging by the lack of public outcry, you didn’t.
I did not get an Issue out because I was at InstructureCON 23 in Denver - the Lollapalooza of LMS conferences. At least I think it is. I’ve never been to Lollapalooza.
I was pleasantly surprised to see at least four presentations on academic integrity - a few with lines to get in and well over capacity. People were turned away. I know because I was. Twice. That’s a great sign.
Also, Thursday morning, I was honored to join three professors on a panel regarding academic integrity - in a presentation to the Academy of Legal Studies in Business.
All that is to say that I wanted to get an Issue out last Thursday, I just couldn’t. Sorry about that.