New Prediction Tool Can Spot 95% of Cheated Exams

Plus, more on that NBER paper. Plus, Inside Higher Ed is having an unusual chat about academic integrity.

Issue 46

More on Test Proctoring from NBER Paper

Issue 45 of “The Cheat Sheet” had a snippet on the recent research paper on the quality of online classes.

First, I should have mentioned the names of the researchers who did the work. They are: Duha Tore Altindag, Elif S. Filiz and Erdal Tekin, professors and researchers at Auburn, Southern Mississippi and American Universities, respectively.

Their paper also has two nuggets on academic integrity and cheating that I didn’t get to in the last issue.

One is that it says that just 62 instructors in their sample of 1,086 signed up to use remote test proctoring. That’s a pretty small sample size from which to draw conclusions but, even so, the news to me is that just 62 out of 1,086 teachers used remote test proctoring to prevent or detect misconduct. That’s 5.7%.

It’s a good thing the research paper does not identify the school because that’s scandalous. And while I understand some misgivings or conflicts about using test proctoring, it’s flat malpractice to give an unproctored exam online. And I am sure that not all of the 1,086 teachers gave exams or assignments online, but if even just half did - if the denominator is 543 instead of 1,086 - that’s still only 11.4% of teachers using test security for online assessments. That’s incomprehensible.

The second point is that I asked one of the report’s authors, Altindag of Auburn, about the findings related to test proctoring - that students in online classes that used test proctoring were more likely to withdraw, less likely to get a passing grade and whether that was linked to cheating specifically. He said,

Yes, it can be harder to cheat with a proctor. But using a proctoring service could be a signal of totally something else. For example, it could be the case that instructors who use online proctoring may also be strict ones. Perhaps their standards are higher, and so on. So we cannot really say what exactly it is.

Not that it matters, but I agree. As established by other research, using a proctor is a strong signal that likely correlates to other aspects of classroom management and student conduct.

A New Tool Can Help Spot Test Cheating

A research paper from Firuz Kamalov and David Santandreu Calonge of Canadian University Dubai and Hana Sulieman, American University of Sharjah finds that complex AI and what they describe as, “a recurrent neural network and an outlier detection method” can spot cheating on tests by predicting expected results based on previous assessments.

The paper is hella complicated. Honestly, I did not get much of the complicated stuff. Here’s an example:

The benchmark methods are implemented in the popular scikit-learn machine learning library. We used the default parameter settings for the models with exception of the contamination parameter which was set to 0.09. Note that in the sklearn implementation, the default value of the contamination parameter is set to 0.1, i.e., 10% of the points are deemed outliers.

Cool. Let’s do that.

But this I did get:

The results of the experiments reveal that the proposed algorithm may be an effective tool in identifying cases of cheating on the final exam based on prior scores.

And:

The results of the numerical experiments reveal that the proposed algorithm achieves high levels of accuracy. It can identify almost all cases of cheating while avoiding falsely labeling a normal score.

The system catches 95% of actual cheating while false positives are less than 5%, the team says. If that’s true, layered in with other cheating detection systems, a passive system such as this could be quite useful in detecting misconduct.

Inside Higher Ed is Talking Academic Integrity and It’s Odd

Inside Higher Ed (IHE) is hosting a webcast “lively discussion” on academic integrity next week. And while visible conversations about academic integrity are always - usually - good, this one is odd.

First, having the discussion are two IHE editors and, as far as I can tell, just them. No professors, no researchers, no experts in academic integrity - just two editors. With just these editors, you have to wonder what level of conversation will be possible. At least I do.

It’s also odd because those editors are at IHE, which just took sponsorship money from Course Hero, one of the largest cheating companies (see Issue 40). So, I also wonder where the editors will land on cheating - with Course Hero, the cheating profiteer, or with Honorlock, the anti-cheating proctoring company that’s sponsoring this discussion. This is the Montagues and the Capulets, the Jets and Sharks. IHE has taken money from both to promote their events.

I mean, just this morning IHE sent another promotional e-mail from Course Hero:

But, oddest of all is the event’s messaging, which says, “How to Shift From Fighting Cheating to Promoting Academic Integrity.” As if “fighting cheating” is somehow not part of promoting academic integrity. Like that if you’re against cheating you’re not promoting integrity.

In fact, most of the event’s messages seem to be about not detecting cheating, not having consequences for misconduct - educating instead of enforcing. That’s fine. It’s a message Course Hero and other cheating companies are sure to endorse. But why Honorlock, who’s in the business of detecting and policing cheating, is sponsoring an event about getting away from it - well, that’s odd.

In the next “The Cheat Sheet” - I may yet get to a comment from Florida International University about that CBS story, if they respond. And we’ll look at misconduct numbers from the University of Virginia.

Side Note: I wrote a piece in Forbes on the NBER paper mentioned previously and above.

Side Note: We looked at other new tools to detect cheating in Issue 13 and Issue 20. Some enterprising computer science student is going to put these together with the one above and make millions.

To subscribe and share, the links are below: