Stunning Investigator Report on Comprised NYPD Exam
Plus, good tips on spotting AI content without an AI detector. Plus, 300.
Issue 300
Subscribe below to join 3,858 other smart people who get “The Cheat Sheet.” New Issues every Tuesday and Thursday.
If you enjoy “The Cheat Sheet,” please consider joining the 18 amazing people who are chipping in a few bucks via Patreon. Or joining the 39 outstanding citizens who are now paid subscribers. Paid subscriptions start at $8 a month. Thank you!
Cheating on NYC Police Sergeant’s Exam
According to local press coverage in NYC, a new report found the city’s 2022 exam for police promotions “was riddled with problems,” which led to some egregious cheating.
What surprised me about the test errors in NYC was how amateur, obvious, and inexcusable they were. I’ve been covering cheating for years now and I’m not genuinely shocked too often. But here, I am.
Consulting
The coverage says:
The 2022 test was technically overseen by the Department of Citywide Administrative Services, which hired two private companies to develop and run the exam — Morris and McDaniel Inc. and PSI Services LLC, the report said.
Morris and McDaniel, their website says, is a test consultancy that runs or consultants on tests for, among other things:
OIL, GAS, AND PETROCHEMICAL
NUCLEAR ENERGY
FINANCIAL
MEDICAL
ENTRY-LEVEL LAW ENFORCEMENT
ENTRY-LEVEL FIRE
Given what happened with this NYPD exam, I am terrified that this same company has its hands in tests for nuclear energy and medical anything whatsoever.
The other company, PSI, says it plays some role in exams for architects, emergency nurses, and critical care nurses. They also work with a company called onlinedegree.com — which says it all, in my view.
The coverage does not say, but if the City of New York paid these companies any more than subway fare and box of Cracker Jacks, they were fleeced. I’m upset because there are many good people in academic integrity who, even drunk and half asleep, would have known to avoid the errors in this police promotion exam. I just don’t get it.
Ringers
Here are some of the things that those companies and police administrators allowed to happen, according to the new report, in the 2022 exam:
a retired captain who runs a promotion test school for cops was improperly allowed to take that exam and 18 others dating back to 2002
A person literally selling test coaching and could not possibly be seeking a promotion to sergeant, took the test 18 times. Eight. Teen. Times.
Test authorities screened applicants for test eligibility after the exam. No, really:
allowing ineligible candidates to sit for exams plainly poses a risk to exam integrity.
[The] policy to conduct screening for candidate eligibility post-exam is not sufficient to address that risk, because ineligible test takers could sit for the exam solely for facilitating the cheating of others. [Investigators] did not substantiate rumors that ineligible test takers were paid by various exam preparatory programs to memorize questions and answers, but eligibility screening for exam applicants would nonetheless be worthwhile, because it would reduce the risks of exam content compromise.
Wow.
Repeat
Further, the test was given twice a day for two days, and the report says:
Ninety-five of the 100 questions were the same on day 1 and day 2 of the test
Come on. What are we doing?
But actually, the official report says:
The Exam content was the same for the two test sessions on Day 1 and remained the same for the two test sessions on Day 2.
The same.
The report does not say, but I’d bet you my lunch money that our “retired captain who runs a promotion test school for cops” took the exam early on the first day. That captain, the paper reported, charges $800 for test preparation and services.
And because of the inexcusable error of running essentially the same test across two different days:
Most of the questions in the sergeants exam given on Aug. 3 and 4, 2022 were passed on in screenshots from cops in internet chat groups, and 35 questions and answers out of the 100 total questions were distributed verbatim to more than 1,200 test takers
Most of the questions were “passed on.”
Phones
That questions and answers were “passed on in screenshots” means that test-takers had cell phones in the exam. Come on.
In fact, the report says that:
Exam incidents reports from [test authorities] indicated dozens of instances where candidates’ cellphones were involved.
And that it:
identified several candidates who violated the cell phone policy by keeping the cell phones powered on. No action was taken to disqualify these candidates or nullify their scores
“No action was taken.”
Compromised
And we continue:
Soon after the test, about 80 officers filed cheating allegations, saying the test was identical on both days, that officers who took the test on day 2 got answers from cops who took it on day 1, and that some officers were able to use their cell phones immediately beforehand.
This is rinky-dink.
The investigation:
determined that test questions in all subject areas of the Exam had been compromised
Worse:
[investigators] only considered compromised any question where the answer was provided in the social media posts or text messages and could be clearly linked to the corresponding question. As such, [the] review concluded that 35 questions and answers were compromised across all exam versions.
In other words, if someone passed along a question only, or just a topic that would be on the exam, the inquiry did not count that as compromised. Still, there were 35 compromised questions where the question was shown with the answer on multiple versions of the test. The entire test is 100 questions.
Also from the report:
Therefore, it appears that with respect to a little over 10% of the exam (11 questions out of a total of 100 questions) not only were the questions compromised by information sharing, but a higher percentage of candidates gave correct answers to those questions in subsequent test-taking sessions than in the first test-taking session.
I’ll get to why this “higher percentage of candidates gave correct answers to those questions in subsequent test-taking sessions than in the first test-taking session” standard is wrong in just a minute.
Errors in Analysis
While I have an absolute zero level of confidence in the way this test was designed and administered, the investigation and report is suspect too. Despite all of the above, the official inquiry found:
These findings echo [the previous] conclusion that PSI analysis showed “no widespread cheating or if there was widespread cheating it was ineffective.”
Remember, 35 of the 100 questions — with correct answers — were shared with some 1,200 examinees beforehand. I guess I do not understand what “widespread cheating” is.
The idea that the cheating was “ineffective” rests on the fact that:
on a macro level, the number of candidates obtaining a passing score exhibited a general negative trend from one session to the next. [Investigators] found that 28.20% of candidates passed the first session, 18.52% passed the second session, 4% of candidates passed the third session, and 12.51% of candidates passed the fourth session.
Expanding that idea:
analysis found that the compromised questions had little to no impact on the Exam’s pass/fail rate
That’s very convenient logic. But it’s misleading and surfaces the answer to the wrong question.
Based on the passage rates by session, the first session was the most successful. This makes sense. Examinees who are prepared, confident, and eager usually want to get it done right away. Examinees who may have put off preparation or may be generally less invested, are likely inclined to wait. Therefore, comparing passage rates to the best-prepared and most motivated examinees is flawed. To isolate cheating, anomalies need to be determined in comparison to the averages, not the exceptional.
Moreover, this is the wrong question. The issue is not whether a greater percentage of people passed an exam after it was compromised — the question is whether individual examinees received unearned passing marks by misconduct.
The report never asks this question, though it would not have been too hard to establish. Their own, somewhat silly, standard of a compromised question produced 35 such examples. Eliminate those for subsequent sessions and re-score the test. Further, isolate examinees who answered compromised questions correctly and passed, but would not have passed with those questions eliminated.
But they did not do that.
It’s exceptionally unlikely that not a single examinee passed without being aided by compromised questions. But investigators did not check. Or at least they did not report that they did. That is, as the kids used to say, suspect.
And finally
It appears that not a single examinee who viewed or may have used compromised questions was sanctioned. Their scores counted.
Based on the report:
seven officers were disciplined for disclosing exam materials and received vacation day penalties ranging from three to 30 vacation days loss.
Yes, only examinees who shared test materials were sanctioned. And the penalty was a loss of vacation days.
So, I have to ask — what exactly is the risk in trying to cheat on the NYP sergeant’s exam?
Even if there is an investigation into cheating, it seems as though no one who cheated on the test will be targeted, and, by application, no penalties applied. Even if you blow up the integrity and fairness of the entire exam, you lose vacation days. You may, it seems, still pass the exam and be promoted.
I’m speechless.
But I also want to share that, according to the report:
In order to obtain a passing grade, 70 percent of the 100 Exam questions must be answered correctly.
The investigation found 35 fully compromised questions. Half of the score an examinee needed to pass was given to anyone who wanted it. Half. And here we are — the NYPD is treating the test as a valid measure of competency.
Words
Even though this entire thing is a sham, investigators repeated the idea that cheating was “ineffective” and wrote:
the discovery of police officers, sharing and possessing exam materials is a clear indication of a serious breach of public trust and undermines the integrity of the agency
And also that:
The repercussions for cheating have reaching effects, particularly when the cheating involves law enforcement officers who are a foundational element of our criminal justice process. Regardless of its efficacy, cheating, undermines credibility in the testing process and questions the abilities of those managing our police force.
Whatever.
According to the report, 1,730 people “passed” the exam and are now, presumably, NYP police sergeants, born of a process that was “a serious breach of public trust and undermines the integrity of the agency” and that “questions the abilities of those managing our police force.”
But what is going to be done about it?
Don’t be silly. You know that answer.
One More Line on the NYPD Exam
It did not fit in with what I wrote above, but I am sharing this one line from the NYPD investigation report:
[Investigators] found that a low proctor-candidate ratio, not only benefits the test takers by providing guidance and better addressing their concerns, but also reduces opportunities to cheat
You don’t say.
Ways to Spot AI Content Without Using AI Detection
A helpful article over at Tech and Learning by Erik Ofgang, a teacher at Western Connecticut State University, has some tips on how teachers can spot likely AI writing.
Despite that Ofgang previously wrote about that terrible AI detection study out of Stanford (see Issue 216), this piece is fair and good.
He writes, for example:
I first noticed an AI paper submitted last summer, and in the months since I’ve come to expect to see several per assignment, at least in 100-level classes.
And:
it’s important to remember that suspected AI use isn’t immediate grounds for disciplinary action. These cases should be used as conversation starters with students and even – forgive the cliché – as a teachable moment to explain the problems with using AI-generated work.
I agree. Suspected AI use — even and especially from an automated AI detector — is never grounds for action by itself. No credible source has said otherwise, as far as I know.
The tips he provides are good. And they are worth a review, especially by anyone who assigns, grades, or supervises academic writing.
That’s 300
This is the 300th Issue of The Cheat Sheet. Please do try to contain your enthusiasm.
It’s actually like Issue 308 or something as I forgot to number some and have had three or four Special Editions. But whatever.
Most importantly, thank you for reading and sharing. And caring.