New York Times Says AI "Cheating Fears" Were "Overblown." I Have Thoughts.

Plus, Turnitin says that in just seven months it's found nearly 4 million papers that were at least 80% AI. Plus, more companies selling AI detection bypass services.

Issue 261

To join the 3,710 smart people who subscribe to “The Cheat Sheet,” enter your e-mail address below. New Issues every Tuesday and Thursday. It’s free:

If you enjoy “The Cheat Sheet,” please consider joining the 15 amazing people who are chipping in a few bucks via Patreon. Or joining the 28 outstanding citizens who are now paid subscribers. Thank you!

---

NYT Says “Cheating Fears Over Chatbots Were Overblown”

It’s not often that the NYT gets its coverage wrong, but they did here. No question. Not egregiously wrong, as so many do so often. But still, pretty wrong.

The story is based on data from Stanford University, on a recurring survey of high school students. So, just high school.

Here’s the patch of Earth on which the story is built:

According to new research from Stanford University, the popularization of A.I. chatbots has not boosted overall cheating rates in schools. In surveys this year of more than 40 U.S. high schools, some 60 to 70 percent of students said they had recently engaged in cheating — about the same percent as in previous years, Stanford education researchers said.

So, OK.

I’ll begin with a tiny personal note. Every time — every single time — I share information from a school that shows that cases of misconduct have increased, someone says that an increase in cases does not necessarily mean an increase in cheating. Every time.

You know who you are.

And, as I say every time, that’s true.

Likewise, there’s no direct correlation between survey data and actual cheating. Both case numbers and survey responses are good data. I like knowing how many cases are being filed and what people say about their cheating habits. But neither is a complete stand-in for actual cheating.

Moreover, of the two, self-reported survey data is known to be inaccurate due to its chronic underestimations — people don’t like to admit doing things they know are wrong. According to research from 2021, actual cheating behavior may be two-and-a-half times what self-report surveys show (see Issue 52).

So, when the NYT uses a survey to determine “overall cheating rates in schools,” they are simply off base. No such conclusion can be drawn from that data. So, that’s a miss.

Also, setting aside that this data is from just 40 high schools, please tell me you did not skip right past the “some 60 to 70 percent of students said they had recently engaged in cheating” line. Again, self-reported survey data undercounts it.

In the article, one of the Stanford researchers, Denise Pope, dismissed the “panic” over cheating with AI and told The Times:

“we’re just not seeing the change in the data.”

Let me tell you why that may be.

If your data shows 60-70% of students admit to cheating “recently,” and we know that survey data is a low marker of actual misconduct — you may already be showing a real cheating rate at or near 100%. If the research on actual cheating is 2.5x what self-surveys show, the folks at Stanford have bumped their heads on the ceiling. There is simply no room in the data to move upwards.

Let me try this another way — using AI to cheat will not make high school students more truthful about cheating.

Even so, the NYT does not share with readers that self-reported cheating rates on a survey are not the same as cheating rates or that self-reporting is reliably known to underestimate actual behavior. Like I said, a miss.

Before moving ahead, let me add here that the NYT also did not interview, or at least did not quote, an actual outside expert on cheating. That feels like a miss as well.

The NYT story then shifts to another survey, this one from Pew Research, showing that many teens don’t know much about ChatGPT:

Many teens know little about ChatGPT, Pew found. And most say they have never used it for schoolwork.

Fair. The NYT graphic of the Pew data is:

Many Teens Have Never Heard of ChatGPT

That’s even though the results say that more than two-thirds of teens say they have heard of ChatGPT. The numbers are 67% to 32%, have heard of it. That’s two-to-one. But the NYT went with “many have never heard of ChatGPT.” Fair. But odd. And intentional. It’s more accurate to say that a strong majority (67%) of teens have heard of ChatGPT instead of reaching for the inverse with “many.”

The NYT also writes:

Pew also asked teens whether they had ever used ChatGPT to help with their schoolwork. Only a small minority — 13 percent — said they had.

We’re sharing self-report survey data again, without disclosing known under-counting. Even so, the 13% finding is out of sync with other survey data on the use of generative AI tools to do schoolwork. Many others put rates in the 30% range and above. Still, good to know Pew’s number.

That Pew number is somewhat contradicted however by the Stanford data shared in the NYT:

This year, the Stanford researchers added survey questions that specifically asked high school students about their use of A.I. chatbots. This fall, 12 to 28 percent of students at four East Coast and West Coast high schools said they had used an A.I. tool or digital device — such as ChatGPT or a smartphone — within the last month as an unauthorized aid during a school test, assignment or homework.

Among the high school students who said they had used an A.I. chatbot, about 55 to 77 percent said they had used it to generate an idea for a paper, project or assignment; about 19 to 49 percent said they had used it to edit or complete a portion of a paper; and about 9 to 16 percent said they had used it to write all of a paper or other assignment, the Stanford researchers found.

Again — four schools? In the New York Times? Or maybe it’s eight. Google tells me there were 23,519 public high schools in 2020-21 — just public. But lets draw conclusions from self-reported surveys from eight of them. I mean, seriously now.

Still, even that data is not great and I’m not sure it makes the case the NYT wants it to. That “12 to 28 percent of students” said they’d used an AI device “as an unauthorized aid during a school test, assignment or homework” feels like it ought to trigger alarm bells. And that’s “within the last month.”

Splitting the difference, that’s 20% of high school students admitting to using AI in an unauthorized way on schoolwork or tests — in the past month. One in five. Admitting it.

How exactly was that concern about AI cheating overblown again?

From the NYT:

The findings could help shift discussions about chatbots in schools to focus less on cheating fears and more on helping students learn to understand, use and think critically about new A.I. tools, the researchers said.

I confess that I have no idea what we’re talking about anymore.

Let me repeat. In their own data, one in every five high school students admitted to using AI devices to cheat, or at least in a way that was “unauthorized.” In the last month. But we should “focus less on cheating fears.”

I feel like I am going insane.

I will give credit to the NYT for adding this:

Cheating has long been rampant in schools. In surveys of more than 70,000 high school students between 2002 and 2015, 64 percent said they had cheated on a test. And 58 percent said they had plagiarized.

The source was the ICAI, the International Center for Academic Integrity. And yes, rampant. And yes, 64% admit to cheating on a test. Self-reported. But we should focus less on cheating fears. Good advice.

The NYT also gets credit for saying plainly that, despite all that other nonsense:

This does not mean that students are not trying to pass off chatbot-generated texts as their own schoolwork.

It does not mean that at all.

The paper of record also shares:

Christine Meade, an Advanced Placement history teacher at a high school in Vallejo, Calif., said chatbot cheating was widespread among 12th graders last spring. She even caught a few using the A.I. chatbots on their smartwatches during school tests.

“Chatbot cheating was widespread.” Among AP students.

Meanwhile, Sandford researchers say that 60-70% of high school students admit to cheating. And that one in five admit to unauthorized use of an AI device on academic work in the past month.

How is that not the headline?

Instead, the NYT thinks we should uncork the campaign and chill the f- out because we don’t think the massive levels of cheating have gotten worse. Which is very likely not true. Even so, celebrate everyone! It’s just two-thirds of high school students who admit to cheating. Dodged a bullet there.

Turnitin Has Reviewed 142 Million Papers for AI Similarity, Nearly 4 Million Papers Had 80% or More AI-likely Content

In this post on LinkedIn from a week or so ago, Turnitin says it has scanned or reviewed more than 142 million papers for AI fingerprints. That’s since April, when Turnitin’s AI similarity checker went live. And it’s only through November 9.

That feels immense.

More noteworthy to me however, is that the company also said that “nearly 4 million” of those papers were detected to have at least 80% of their content similar to what AI would create. Four million. In seven months. That’s 19,000 each and every day. Counting summer academic breaks and weekends.

And nearly 10 million, Turnitin said, had AI-likely content of at least 20%. That’s 7% of all scanned papers.

Naturally, four million papers with 80% or more of probable AI content does not necessarily mean that four million papers were violations of academic integrity policies. There are some good reasons a teacher or program may want students to use and submit AI-created documents.

But it’s reasonable to assume that most of those — where at least 80% of the paper’s content is probably composed by AI — are not authorized or appropriate. And there are, according to Turnitin, already literarily millions of them.

I also think it’s worth mentioning that for all the gyration and hyperventilation over AI detectors being flawed and unreliable and needing to be turned off to allow cheating and save humanity, 142 million papers running through Turnitin’s system is a ton. Fair to say, I think, that the teachers and schools that prefer to have no information about AI use are in the deep minority.

Finally, credit to Turnitin for releasing some numbers. I say every time, to get our heads and arms and policies around this, we need information. In this tiny space, too much of it is guarded far too zealously.

Cheaters Still Bragging About Ability to Fool AI Detectors. But This One is Special.

A company calling itself StealthGPT sent out a press release recently boasting that it was:

the only AI tool to consistently bypass Turnitin's advanced AI detection systems

But why bother? I thought AI detection systems did not work.

I know, I’m obnoxious.

And speaking of obnoxious, the headline of their press release is:

StealthGPT Triumphs Over Turnitin: A New Era in Academic Integrity and AI Technology

The actual first line of it is:

In a landmark development that is set to redefine the landscape of AI’s use in academia

Obnoxious.

What they’re pitching, of course, is software that will pass Turnitin’s detectors, which is clear when they write:

The results are clear: StealthGPT's advanced 'humanizer' technology and AI bypasser mechanism enabled it to produce content with a 0% detection rate on Turnitin, while all the other competitors are being detected at a rate of 100%.

In other words, they’re trying to be the best cheating enabler out there:

This highlights the limitations of standard AI bypassers and underscores the sophistication and effectiveness of StealthGPT's technology

The company CEO, Jozef Gherman said:

“Our software goes through rigorous testing to ensure we’re 1-step ahead of the AI detectors currently on the market”

Its company name is literally stealth. To help you remember, it’s steal - with a th at the end.

But what surpasses obnoxious and lands firmly in the land of bizarre is how StealthGPT tries to position itself as the ethical solution in academic work. No, really. For fun, try to untangle this haphazard pile of words and concepts:

StealthGPT's superiority in evading AI detection algorithms like Turnitin's not only establishes it as a leader in undetectable AI writing tools but also showcases its potential as a transformative aid in academic writing. Its advanced capabilities set a new benchmark for undetectable AI, offering peace of mind for those seeking to use AI writing tools ethically and effectively within the academic domain.

And:

As we navigate the new age of academic integrity, it is imperative to consider the ethical use of AI tools like StealthGPT. The tool's sophistication in mimicking human writing styles, while maintaining content authenticity, presents an opportunity to enhance learning and research without compromising academic integrity.

By being really good at helping you avoid AI detection, we’re an opportunity to “enhance learning and research without compromising academic integrity.” George Orwell could do no better.

At the end, the company asks people to:

Join the Revolution in AI-Assisted Writing

And

Embrace this technology not just as a tool, but as a partner in your academic and creative endeavors.

Cheating is easy. And for a few extra bucks, you can pay pirates to cover your tracks. Ethically, of course. Because honest academic work always needs an alibi.

2023 Best of The Year

It’s that time again when we will wrap up and share some of the best and worst in academic integrity for 2023. If you’re unfamiliar, here the “Best Of” Issues from 2021 and from 2022.

So, if you have suggestions, nominations, or ideas for what should make our Best and Worst lists, please let me know. A reply e-mail reaches me.