GPT Passes Harvard Assignments, Gets Good Grades
Plus, 500 KPMG accounts cheated exams in Netherlands. Plus, more Massachusetts police cadets fired for exam cheating.
Issue 225
To join the 3,381 smart people who subscribe to “The Cheat Sheet,” enter your e-mail address below. New Issues every Tuesday and Thursday. It’s free:
If you enjoy “The Cheat Sheet,” please consider joining the 14 amazing people who are chipping in a few bucks a month via Patreon. Or joining the 14 outstanding citizens who are now paid subscribers. Thank you!
GPT Passes Assignments at Harvard
Maya Bodnick over at the newsletter Slow Boring (which incidentally is one of the few newsletters to which I subscribe), has a great piece out today about ChatGPT and GPT-4. It’s a pretty important read.
The skinny is - she enlisted several Harvard professors to grade freshman-level written assignments created by GPT-4, telling them the work may or may not be created by AI. The grades AI earned at Harvard:
A. A. A. A-. B. B-. C. Pass.
She writes:
Not only can ChatGPT-4 pass a typical social science and humanities-focused freshman year at Harvard, but it can get pretty good grades. As shown in the report card above, ChatGPT-4 got mostly As and Bs and one C; these grades averaged out to a 3.34 GPA.
The article continues:
Several of the professors and TAs were impressed with ChatGPT-4’s prose: “It is beautifully written!” “Well written and well articulated paper.” “Clear and vividly written.” “The writer's voice comes through very clearly." But this wasn’t universal; my Conflict Resolution TA criticized ChatGPT-4’s flowery writing style: “I might urge you to simplify your writing — it feels as though you’re overdoing it with your use of adjectives and metaphors.”
Compared to their feedback on style, the professors and TAs were more modestly positive about the content of the essays. My American Presidency TA gave ChatGPT-4’s paper an A based on his assessment that “the paper does a very good job of hitting each requirement,” while my Microeconomics TA awarded an A in part because he liked the essay’s “impressive… attention to detail.”
Seriously, go read the piece. She put the GPT-created work in links and identified the courses and assignments. So, well worth a look.
Bodnick continues:
I think we can extrapolate from ChatGPT-4’s overall solid performance that AI-generated essays can probably get passing grades in liberal arts classes at most universities around the country.
I think it probably can. Further, I bet it already is.
She continues that, for cheating, GPT-4 is a major step forward and “made cheating on take-home essays easier than ever.” She says that in the past, students had to fear being caught by plagiarism detectors but now:
students don’t have to worry nearly as much about getting caught using ChatGPT. AI detectors are still very flawed and have not been widely rolled out by U.S. colleges and universities.
This part is not true.
AI detection regimes have limits, and most are well documented. But if students use GPT to submit work unedited, as Bodnick did, the chances are quite good that current detectors would flag it. And detection systems are nearly ubiquitous at American schools. Several systems are already fully embedded into LMS software, as well as in existing grading and review platforms.
And though I cannot understand why even one would do it, only a handful of schools have “opted-out” of using AI detectors. I call it the “we can’t see cheating if we close our eyes” approach.
Nonetheless, the piece continues:
ChatGPT has made cheating so simple — and for now, so hard to catch — that I expect many students will use it when writing essays. Currently about 60% of college students admit to cheating in some form, and last year 30% used ChatGPT for schoolwork. That was only in the first year of the model’s launch to the public. As it improves and develops a reputation for high-quality writing, this usage will increase.
And:
Next year, if college students are willing to use ChatGPT-4, they should be able to get passing grades on all of their essays with almost no work.
Next year? But point taken.
Unfortunately, she repeats a few of the flawed or already contradicted studies about AI detectors, including citing that OpenAI’s own detector “is only accurate 26% of the time.” It is. But it’s awful and not reflective of any other classifier. And so on.
She also says that professors and schools will likely have to revert to in-person or in-LMS writing to deter cheating. That’s probably right.
And importantly, she also writes that:
My initial reaction to the rise of AI was that teachers should embrace it, much like they did with the internet 20–25 years ago.
And that:
If educators can’t embrace AI, they need to effectively prevent its usage.
Exactly. The genie is loose. Preventing misuse is the only option that schools and teachers have.
This reality also highlights the frivolousness of telling teachers to simply “write better assignments” or whatever. That’s no answer - not now, not in the future. Fortunately, there are solutions. Changing the assignment is not necessarily among them.
Exam Cheating Among Accountants Spreads to the Netherlands
The trend of major accounting firms being sanctioned for systemic exam cheating rolls on (see Issue 171).
This time, news from The Netherlands is that KPMG has issues. According to the reporting:
At least 500 workers at KPMG in the Netherlands have cheated during the compulsory exams which accountants are required to take
Five hundred.
“A handful” were fired.
We see again that these major, planned exam cheating incidents are discovered not by the accountants and auditors themselves. Or by regulators. But by an examinee:
The investigation was sparked after one employee reported to senior management that others were sharing the answers to mandatory tests between them.
You have to wonder how any company allows so many employees to cheat for so long. Maybe - just maybe - it was pretty clear that no one at KPMG, or at the other implicated firms, cared about these exams. And when leaders don’t care, when they don’t protect the exams or seek out or punish cheating, it spreads. It becomes normalized, accepted, expected.
More Massachusetts Police Cadets Busted for Cheating on Exams
The police cheating scandal in Massachusetts (see Issue 220) also rolls on, according to local reporting.
This time, the news is that in the town of Chicopee, population 55,000:
Three student recruit officers were fired following an investigation into possible exam cheating.
The Chief issued the obligatory statement:
We value honesty, accountability, and truthfulness from our recruits and believe the actions taken were swift and appropriate. We strive to provide the best trained officers for our community and actions like this will not be tolerated.
That’s good. Maybe, once again, secure the exams in the first place. Just an idea.