329: Research on Pre-Test Integrity Reminders, Misconduct Mitigations
Plus, a great article about cheating and AI at Washington State. Plus, funny irony out of Kenya.
Issue 329
Subscribe below to join 4,223 (+7) other smart people who get “The Cheat Sheet.” New Issues every Tuesday and Thursday.
If you enjoy “The Cheat Sheet,” please consider joining the 16 amazing people who are chipping in a few bucks via Patreon. Or joining the 44 outstanding citizens who are now paid subscribers. Paid subscriptions start at $8 a month or $80 a year. Thank you!
Research: Pre-Assessment Reminders, Warnings Reduce Cheating
In the last Issue of The Cheat Sheet, we looked at an opinion piece in the student paper at Middlebury College, which is in the center of a Honor Code, integrity, misconduct conversation.
In that op-ed, the author made a few suggestions. One was:
Research also indicates that reminding students of academic integrity policies and the consequences of cheating significantly reduces dishonest behavior during exams. Middlebury should incorporate such reminders before tests to reinforce awareness and uphold academic honesty.
I left the link to the research in the quote. I did not mention it in the last Issue because I wanted to read the research first. I had to buy it. It was $30. Or $40. I don’t remember. But thank you, paid subscribers.
Research Overview
The paper is from October 2023 and by Li Zhao, Yaxin Li, Junjie Peng, Jiaqi Ma, Xinchen Yang, Kang Lee, Weihao Yan, Shiqi Ke, and Liyuzhi D. Dong. Researchers Lee and Dong are with the University of Toronto, the rest of the team are with Hangzhou Normal University, in China.
It is, in my view, very strong.
It’s also a little complicated.
Condensed, the team gave college students unproctored exams in an exam room that students normally use. Researchers made a small number of the exam questions unanswerable, except that the “answers” were published in study materials online. The only way to get the questions “correct,” in other words, was to access the unauthorized materials during the exam. Students were told the answers were in the class portal, so they could check after their exams. They were reminded not to check the answers during the test.
This approach has a significant advantage over others in that the attempted cheating is known and quantifiable while other research efforts use unreliable self-reported survey data.
What I don’t like about this is that it presents a binary choice instead of what is, in real conditions, a ternary choice. Under these test conditions, subjects had to choose to cheat or to get a bad grade, there was no third option to study and know the material. In other words, in this test condition, someone who studied and was prepared, may be motivated to cheat when they see questions they could not answer, when in real conditions, cheating would have been unnecessary.
Still, I think that caveat is worth having real data on cheating instead of trying to rely on people to be honest about dishonesty.
Tested Conditions
So, with the ability to measure actual violations of exam rules, the research team put five different exam conditions across two observation windows:
Nothing added - control group
A reminder on a preface page of the school’s integrity policy and that students should “abide” and “obey” its directives as well as that of the test or teaching staff. This included specific admonitions about the use of cell phones, which were the devices needed to access the planted answers.
A list of two students — by name, with dates and course names — who had committed academic misconduct and were caught. Both short descriptions say the student “was not granted a degree.”
A description of what the exam was meant to assess — “The purpose of this exam is to test the student’s understanding of course materials” — with a light mention integrity policy and possible sanctions such as not getting a degree.
Reminding students that the exam was a “trust exam”
The tests mixed and matched these conditions somewhat as well, e.g., using “trust exam” with the assessment description or “trust exam” with the examples of students who were sanctioned. It’s clear that “trust exam” is a term for an unproctored, unsecured assessment in an Honor Code environment and, for the record, I like it. Students had to sign that they had read and understood the information outlined above. You can read the text used in these trials in the paper.
Let me also add that these research tests were conducted in China and how students process information about trust or honor or risk of consequence may differ by culture. Still, I would accept that some of the findings have universal application.
Findings
The first test used only the control condition (no reminder), the integrity policy reminder, and the examples of caught students — separately, not together. And the team found:
the cheating rate was 52.4% in the No Reminder condition, as compared with 31.8% in the Policy Reminder condition and 22.6% in the Exemplar Reminder condition.
In an unproctored test in which students were told not to look up the answers during the exam, 52% did. This was, based on the method description, in front of their peers. Although I am not sure whether that fueled more cheating or mitigated it. Either way, more than half cheated.
Importantly, we do see a significant drop in cheating rates when requiring a signed policy reminder and an even more significant drop when requiring a signature that students had read about past students who were caught cheating. The third condition — the examples of past student cheating — cut cheating nearly by 30%. Not that a 22.6% cheating rate is good. But it is better than 52.4% or 31.8%.
This feels like big news and a rather easy intervention that educators could begin using. Issue 216 has other researched intervention tactics that also show promise.
Of course, if an educator was to use the “examples of cheating students” before an exam, they’d have to make up the details. But I don’t think that matters. We know bluffing works (see Issue 108). Alternatively, a smart person suggested using aggregate numbers, as in, “seven people were caught cheating on this exam last semester, four were suspended,” or whatever. Either way.
If I were a teacher, I’d make sure my students heard about the bad things that can happen if you’re caught cheating. Of course, this requires that bad things actually happen to students who are caught cheating, which is pretty rare.
On these findings, the research team writes:
that students cheated to a significantly greater extent in the No Reminder condition compared to the other two Reminder conditions. This finding suggests that even though the students had been indoctrinated about the university’s academic integrity policies upon entry and had experienced the unproctored exam format previously, simply having an exam unproctored might not be sufficient to discourage all students from cheating. Some students may need to be reminded of either the university’s academic integrity policies or actual examples of academic dishonesty to behave honestly in an unproctored exam.
I have to say that I love the idea that an unproctored exam by itself has some ability to discourage students from cheating. I think it’s clear from the context that by “unproctored exam,” the team means a trust environment around an Honor Code. But the larger point is that these two tactics worked.
No Threats
Of high interest to me was that, in setting up their second test, the research team says:
the serious negative consequences of academic cheating were not mentioned in the Policy Reminder condition [in the first test] although it was part of the university’s academic integrity policy. The reason that this part was not included in Study 1 is that discussing the negative consequences of academic cheating implies a threat, which goes against the fundamental principles underlying the honor code system, and is therefore discouraged in practice
It is interesting that telling students that bad things may happen if they break the rules is discouraged. I don’t know — just interesting. And I guess it feels incompatible with teaching. Should you not say that if you don’t do the reading, you may fail the course? I am just confused about that.
Results, Part Two
In any case, it does not matter. The research team tested giving students information about negative consequences anyway. Test 2:
shows the cheating rate was 26.6% in the Consequence Reminder Only condition, 24.0% in the Trust Exam & Consequence Reminder condition, 17.1% in the Policy & Consequence Reminder condition, and 15.6% in the Exemplar Reminder condition.
“Consequence” there is actually the language about the purpose of the test, i.e., “The purpose of this exam is to test the student’s understanding of course materials” and so on. Though it also mentions “According to University’s Sanction and Offence Codes, if academic cheating is found, it will be recorded in the student’s academic file and no diploma will be granted.” All of the final three test conditions included that the exam as a “trust exam” as well, but not the first.
So, it seems when you threaten students with “no diploma will be granted,” it works. The initial cheating rate went from more than 50% to 26.6%, though the 26.6% also had the language about the purpose of the exam. I am sure some folks will say the latter was the reason for the drop even though the cheating rates are even lower when the reasons for the exam are not included in the exam preface.
I’ll also point out that in both tests, the lowest cheating rates were found when students were told of other students who were caught and what their sanctions were.
The paper continues:
These results suggest that simply reminding students the unproctored exam is a trust exam was insufficient to reduce their tendency to cheat compared to the condition where no such reminder was given.
Fair. And not really too surprising.
And:
these findings suggested that adding a statement about the university’s policy regarding the negative consequences of academic dishonesty significantly reduced students’ tendency to cheat.
The paper does not harp on this, but I get the sense that both conditions are important here — the policy and the “negative consequences.” My guess would be that a pre-test reminder such as “practice integrity” by itself would not show similar reductions in cheating. There are similar strands of this idea in other research.
Major Findings
The research team has what they describe as “four major findings” from their work. They are:
First, we found that the basic form of unproctored exams is the least effective method to encourage academic integrity. This basic form involves giving an unproctored exam without any reminders (i.e., the No Reminder condition), which is commonly practiced in universities with an honor code system. The reason that no reminders are given is that students are assumed to be well aware of, and experienced with, the honor code system. The students are also assumed to appreciate the fact that they are trusted by their professors to abide by honor codes and would reciprocate their trust by acting with integrity and not cheating. Our results suggest that this assumption may be misplaced.
Wow. I mean, obviously. But still — that students are assumed to appreciate trust and will reciprocate, is misplaced. I feel as though that should be a needlepoint that can be hung in academic and testing offices.
Still part of their first major finding:
This finding questions the fundamental assumption underlying the unproctored exams and suggests caution to believe blindly that all students will abide by the university honor codes and behave with integrity during the unproctored exams
No joke. I mean how much more evidence do we need that Honor Codes do not work? In fact, I am starting to believe that they may even be detrimental to a culture of integrity because I can see how an expectation with no consequence is worse than never having set the expectation in the first place. Or how having an Honor Code and/or integrity policy and not enforcing it really just destroys credibility.
But that’s me. Moving on:
Second, we found that reminding students about either the university’s academic integrity policy or actual cases of academic dishonesty and their negative outcomes significantly reduces cheating relative to the No Reminder condition. This effect is not due to the fact that the students were not aware that the unproctored exam in the No Reminder condition was an unproctored exam. When we informed students about the nature of the unproctored exam in Study 2 (the Trust Exam & Consequence Reminder), students still cheated to a significantly greater extent than in the Policy and Exemplar Reminder conditions.
Next:
Third, the comparisons of the results from Studies 1 and 2 revealed that adding a statement about the university’s policy regarding the negative consequences of academic dishonesty if caught significantly reduced students’ tendency to cheat. This is not only true for the Policy & Consequence Reminder condition but also for the Consequence Reminder Only condition. In other words, reminding students about the negative consequences of cheating if caught just before the unproctored exams has a general cheating reduction effect. This finding is inconsistent with the common belief that telling students about the negative consequences of cheating if caught is counter-productive to the effectiveness of unproctored exams (Zhao et al., 2021). This belief is based on the argument that telling students the negative consequences of cheating is a form of threat and thereby implicitly communicates the message that students cannot be trusted, which directly conflicts with the basis of the unproctored exam that is built on mutual trust
And:
the negative consequences of cheating are often omitted when giving unproctored exams. The present research provides direct evidence to suggest that this omission is a mistake and reminding students of the negative consequences of cheating may be one of the effective strategies that professors can use to reduce cheating during unproctored exams.
So, there it is.
Next, they say:
Fourth, despite various reminders, on average, about 27% (Study 1) and 22% (Study 2) of the students still cheated when taking an unproctored exam.
Yup. If you’ve ever tapped your brakes on the highway after seeing a police car, this finding cannot surprise you. Unproctored assessments are not secure, fair, or reliable, even in — especially in — Honor Code settings.
The authors do theorize that perhaps these high rates of cheating, even after reminders and warnings, may be because the school’s Honor Code was only a decade old, that it may take longer than that to become part of the academic culture. Maybe. But Stanford University’s Honor Code is more than 100 years old (see Issue 209). Middlebury College adopted their Honor Code in 1965 (see Issue 312).
Finally — the paper did say “four key findings,” I went back and checked. But it says:
Fifth, Study 1 found that males cheated more than females, but Study 2 failed to replicate this gender effect. These results taken together suggest that the gender effect, if it indeed exists, may be small and unstable.
Good to know. The team also found no difference in cheating rates by major areas of study.
Good job
My takeaway here is: do not give unproctored exams. Even if your school has an Honor Code and you really want to trust your students. Don’t do it. But also, reminders about integrity policies and codes, as well as warning about consequences, can reduce cheating.
And so, even though our Middlebury student was wrong about so much of his recent opinion article, he may have gotten this big piece right. If the school keeps an Honor Code, and insists on giving unproctored assessments, the school probably should make students affirm the code and read about the possible negative outcomes of being caught cheating. Good job.
Outstanding Reporting: AI and Misconduct is “Complicated” at Western Washington
A recent article by a student in the student newspaper at Western Washington University jogs through AI use and cases of suspected misconduct at the university. From a journalism perspective, it’s fantastic — way better than 95% of what professional writers offer on academic integrity.
There are also a couple points worth sharing.
For one, it has numbers:
At Western Washington University, during the 2023-24 academic year, 41 out of a total 116 academic integrity violations involved unauthorized usage of AI, said Melinda Assink, the Vice President of Academic Affairs. The number of AI violations is more than double the 18 unauthorized usages seen in the 2022-23 academic year.
Good for Western Washington for being open and forthcoming.
Further in, it shares:
But when accusing students of using AI dishonesty, the university doesn’t always get it right. Western student Annika Nelson has twice been falsely accused of using AI to write assignments for her. In one instance, her professor approached her at the end of her communications theory class claiming she used AI on a 300-word assignment.
Honestly, this is great. For one, this student writer gets the action correct. Universities — professors or deans or others — accuse students of misconduct, not detection software. It’s amazing how regularly even professional writers botch that. And, as a human process, people don’t “always get it right.” I have no issue whatsoever.
A quick note that if this is accurate, a 300-word assignment is at the bare bottom of length that any reasonable AI detector can assess. It’s on the edge.
But whatever. The piece continues:
In the past Nelson has been told her writing sounds too “clinical,” which she said is a direct result of her autism spectrum disorder. Her professor, who was using AI detection software to screen assignments for cheating, had Nelson come to her office hours and re-do the assignment in front of her.
“Since [the redone assignment] also had a very similar structure and style, she realized she was incorrect in her assumption and apologized and gave me full credit for the assignment,” Nelson said.
This is perfect. I mean A+. This is exactly how these issues should be resolved: a teacher has suspicion, engaged the student, sought more information, and made a decision. In this case, “full credit for the assignment.” I realize that this may have been upsetting or a burden for this student, but it does not get any better than that.
Nelson was accused a second time of using AI by her Latin American studies professor. After having one of these discussions, she was again given full credit.
“I used information that had not been introduced in class [for the paper], but it was something that I learned in a previous course … So I went through the notes I had taken from the course and showed him the notes that reflected that piece of information,” Nelson said.
Again - perfect. Suspicion, engagement, investigation, conclusion. And again, “full credit.” Absolutely amazing. So often, these cases of “false allegations” resolve to nothing.
This, from this article, is also worthy of your time:
Last year, Western had access to the plagiarism software Turnitin’s free AI detector to evaluate AI usage in assignments. Now, AI detection usage varies from professor to professor, [VP] Assink said.
“Turnitin took that free [AI detection] option off and it’s an additional cost, so now Western doesn’t have a campus-wide detection software,” Assink said. “But from what I’m seeing from violations, a lot of faculty are using their own accounts or checking in different free software.”
In May 2024, a PubMed study examined the accuracy of 10 different free AI detectors and concluded that their predictive value is highly variable and not always accurate.
While I understand budget limitations and hold no warm glow for big education companies seeking profit, leaving AI detection to individual professors is very bad policy.
That’s because, as this journalism correctly shares, the market is littered with AI detection systems that are “highly variable and not always accurate.” That’s a problem. It means that the same writing could be flagged in one class on one system and not in another. And it puts professors, and possibly a formal inquiry process, on unstable footing, as some AI detectors are absolute garbage.
For the record, I left in the link to that PubMed study and have not reviewed it. I will try to get to it. But in the meantime, describing AI detectors as “highly variable and not always accurate” is very accurate. The good ones are good. The rest are — well, not good. And there’s little way to know, and less reason to expect professors to know, one from another.
Even so, please, at least have a school policy that establishes internal consistency.
Unbelievably, this really strong article also quotes an actual expert in academic integrity. Imagine that.
From the piece:
Tricia Bertram Gallant is the Academic Integrity Coordinator at University of California San Diego and an author of several books concerning academic integrity.
“[If suspected of using AI,] students can document their processes by writing in Google Docs so their version history can be shared,” Bertram Gallant said. “They can narrate the process that they used to get the final product … being able to do this is going to become more and more central to the human experience in the age of AI.”
Bertram Gallant believes this narration and discussion between student and instructor is much better than simply using AI detection software.
Being able to show and/or explain your work is good advice. And good practice. And Bertram Gallant is correct that discussion is better than “simply” using detection software. If that’s all a professor is doing — using AI detection and filing official proceedings or failing students — they are doing it wrong. Using good AI detection is a very good idea so long as it’s a gateway to conversation and engagement, a door instead of a wall, a hello rather than a goodbye.
Finally, the piece ends here:
In response to these two accusations, Nelson said she’s taking active steps to make her writing sound more casual and less structured.
“I now have a habit of placing footnotes in my writing where pieces of information I learned outside the course came from and citing the source that confirms that information, as a way of avoiding any accusations in the future,” Nelson said.
You should be sitting down for this — a student learned from accusations of misconduct. Her process and product are improving. I know, mind blown, right?
Frankly, aside from the school going all lottery ball on AI detection systems, I love everything about what we’re being told in this article — just as I love how we’re told. It’s really, really good work. A student at Western Washington can do it, and so I will not accept why the people and publications who are paid to cover education can’t seem to manage it.
A Quick Note on Exam Cheating in Kenya — It’s Still Bad
Briefly, news coverage out of Kenya highlights that exam cheating in the country is still pretty bad.
So, this actually turned out to be funny.
I was going to share that, despite strong words and some strong actions by government officials, exam cheating in Kenya is still a major problem. The article linked above says that most known cheating was deliberate and organized, and that teachers and proctors turned off cameras and participated in the fraud.
I was going to say that academic fraud and corruption such as this may be the main contributor to lack of faith in academic credentials in Kenya. And how it’s amazing that more “western” counties can’t absorb the lesson that widespread cheating will ruin an entire education system.
I was going to do all that. But in trying to copy text from the article, I get the best, most ironic warning:
Don’t copy text!
No, really. I can’t make this stuff up. Here is an image:
I mean what can I say? That’s good advice, especially for an article about blatant cheating.