Remote Testing, Proctoring of Law School Entrance Exams Crashes
Plus, International quick bites. Plus, The Markup messes up.
Issue 233
To join the 3,509 smart people who subscribe to “The Cheat Sheet,” enter your e-mail address below. New Issues every Tuesday and Thursday. It’s free:
If you enjoy “The Cheat Sheet,” please consider joining the 14 amazing people who are chipping in a few bucks a month via Patreon. Or joining the 18 outstanding citizens who are now paid subscribers. Thank you!
Remote Test Delivery, Proctoring of LSAT Goes Off the Rails
Earlier this month, there were some pretty big problems with the remote delivery and security of the LSAT, the go-to law school admissions assessment.
The issues this time seem to be capacity and execution of the online exam delivery itself. Long delays, lack of support, and proctoring inconsistency made examinees “furious” and led to “deep frustration,” according to the coverage.
In fairness, the issues aren’t with the LSAT itself, but with the LSAT’s remote exam and proctoring provider, Prometric, which was brought in just this past year.
From the coverage:
This was Prometric’s first administration of the LSAT since the law school council decided in April to reinstate in-person testing alongside the remote option it had exclusively offered since the COVID-19 pandemic. The council switched from its previous proctoring company, ProctorU (now known as Meazure Learning), to Prometric because the latter was better positioned to oversee both in-person and online testing [an LSAT spokesperson said]
It continued:
Despite what [the spokesperson] characterized as “many months” working with Prometric conducting “large-scale tests” to “make sure their systems and procedures were all ready” for the August administration of the LSAT, “a mix of staffing and systems issues” combined to “create much bigger problems” that “didn’t surface during our planning.”
Yikes.
There is no other way to describe that except - ‘not good.’
Two quick points.
It may be that Prometric and the LSAT underestimated the proportion of test-takers who preferred to test remotely when given that option. Selecting a remote exam security provider with a core competency around in-person exams leads to that assumption.
One takeaway is that, if you offer a choice between in-person testing and supervised remote options, many test-takers will choose remote. I suspect that the LSAT folks knew this at some point because, before the most recent exam, Reuters reported that:
It turns out people like taking the Law School Admission Test in the comfort of their own home.
More than half — 61% — of the 19,463 aspiring lawyers registered for this month’s LSAT have opted to complete it remotely, while 39% wanted to take it in-person at test centers
That should not surprise anyone. And it means they should have been ready.
The other takeaway is that delivering and securing remote exams is not easy. I’m not saying it’s NASA-level complicated, but I am saying it’s not as simple as turning on Zoom. Not everyone can do it and not everyone can do it well. Delivering in-person assessments and delivering remote exams are as similar as cougars and Krugerrands. They may sound alike, but they have nearly nothing in common.
Because of who takes the LSAT, I expect the consequences of this particular snafu to linger. However, I do not expect the preparation and execution issues at the Law School Admissions Council, which administers the LSAT, to last nearly as long. I think they’ll fix it one way or another.
They kind of have to, right?
The Markup Really Messes Up
I don’t know the publication, The Markup. And I am always reticent to post and share bad coverage of academic integrity because I feel like, oftentimes, getting attention is its only ambition.
Occasionally though, I feel it necessary to highlight the bad reporting in service of reminding people that a ton of what’s written - and what people may read - is just ill-informed, slap-dash nonsense.
With that, here’s this piece from a few weeks ago in The Markup.
The headline is:
AI Detection Tools Falsely Accuse International Students of Cheating
And that ought to tell you right away that whomever wrote that has zero idea what they’re talking about.
First, and for the seven hundredth time, AI detectors don’t make cheating accusations. They factually do not. Not ever. In fact, they don’t even really detect AI - but that’s probably too pedantic.
Moreover, the study that The Markup wants to talk about doesn’t address “International Students.” It addresses students for whom English is not their first language. US-born, non-native English speakers are included. “International Students” from Australia are not.
I do not understand why this is so hard to understand.
But with three errors in the nine-word headline, we are off. Even so, I do not have a strong enough stomach to go line by line. I don’t have the time for it either. At some point, you have to stop picking through the garbage and just describe it as garbage.
As alluded to above, the article wants to talk about the study by Stanford professors that aimed to test AI similarity checkers on the written work of non-native English speakers. I wrote about this paper, and the many problems with it, in Issue 216. I beg you to go read it.
Importantly, The Markup spends most of the time talking about and dismissing an AI-detection tool that the study never even tested.
Even so, and to summarize, the study itself is kind of a disaster. It has misidentified citations and, for its test sample, used 91 papers it got from a Chinese test-prep company. No, really. All it says is that they were “human-authored.” We’re not told how it is they know that. The paper also used outdated detection regimes to test newer AI models.
I still can’t understand how the paper came to be published. Although I do understand why publications such as The Markup are so eager to keep repeating it.
Further, to make its case that AI checkers are bad, The Markup writes:
OpenAI shut down its AI detector at the end of July because of low accuracy, and Quill.org and CommonLit did the same with their AI Writing Check, saying generative AI tools are too sophisticated for detection.
Along with GTPZero, which is also getting out of the AI-detection business, those three - OpenAI, Quill.org/CommonLit and GTPZero - are three of the detectors that the Stanford study tested. They only tested seven.
The Markup doesn’t mention that. Or any of the other glaring flaws with the research.
Anyway, the story has one silver lining - an entry for Academic Integrity Quote of the Year.”
From the story:
In a note to faculty at the end of June, the [University of Pittsburgh] teaching center said it didn’t support the use of any AI detectors, citing the fact that false positives “carry the risk of loss of student trust, confidence and motivation, bad publicity, and potential legal sanctions.”
While the experience of international students wasn’t at the center of their decision-making, John Radziłowicz, interim director of teaching support at the University of Pittsburgh, said his team tested a handful of available AI detectors and decided false positives were too common to justify their use. He knows faculty are overwhelmed with the idea of students using AI to cheat, but said he has been encouraging them to focus on the potential benefits of AI instead.
“We think that the focus on cheating and plagiarism is a little exaggerated and hyperbolic,” Radziłowicz said. In his view, using AI detectors as a countermeasure creates too much potential to do harm.
Amazing.
The University of Pittsburgh thinks “false positives” risk “bad publicity.” Wait until they see how bad the press is for massive cheating. And potential legal sanctions - they’re worried of being sued. Not fairness and integrity, legal challenges.
And faculty are “overwhelmed with the idea of students using AI to cheat.” They’re not overwhelmed by the cheating, but by the idea of cheating. But because they are, the University of Pittsburgh says they should focus on the benefits. Wow. Ignore the “idea” of cheating, look on the sunny side. Your roof is leaking like a waterfall but, chin up - you don’t have to shower! Nice.
And the focus on cheating and plagiarism is hyperbolic. Could not leave that out.
If academia is going to even start to address the aggressive cancer of academic misconduct, there’s plenty of work to be done close to home. Unbelievable.
International Quick Bites
In Egypt, ministers announced that two or three students had been caught using cell phones to record exam questions and answers during a recent high-level, college entrance assessment. Videos were posted to social media, which is how the misconduct was discovered.
The press report says cheating in Egypt is a criminal offense and punishable by “a prison term of between two and seven years, along with fines ranging from EGP 100,000-200,000 for those involved in printing, publishing, broadcasting, or promoting exam questions, answers, or evaluation systems with the intention of cheating or disrupting the examination system.” That’s $3,300 to $6,600. Plus, you know, the years in prison.
In India, police say about 40 people have been arrested for cheating on the exams to become police officers. Some of the arrested used, “SIM cards embedded in devices resembling debit cards and earbuds” to cheat. The report says that when police arrived to arrest two suspects, they were attacked by a mob of about 30 people.
Also in India, this post on the platform formerly known as Twitter is pretty priceless. Allegedly, a seventh-grade student tried to use ChatGPT to do his homework and was caught. His uncle shared it. The post, and especially this image, is worth your time:
Authorities in the Philippines have expelled two police cadets for cheating on exams. The police academy “stressed that the dismissal of the cadets should serve as a reminder to other students that the institution will not tolerate any form of dishonesty.” Imagine that.
