Special Edition - Performance, Proctoring and Poly Pomona
A breezy 3,100 words on some really bad stuff.
Issue SE3
To join the 3,185 smart people who subscribe to “The Cheat Sheet,” enter your e-mail address below. It’s free:
If you enjoy “The Cheat Sheet,” please consider joining the 13 amazing people who are chipping in a few bucks a month via Patreon. Or joining the seven outstanding citizens who are now paid subscribers! I don’t know how that works, to be honest. But I thank you!
Proctoring, Bad Reporting and Bad Policy at Cal Poly Pomona
Incorrect Reporting
A few weeks ago, I came across this really bad reporting from Cal Matters, which describes itself as:
a nonpartisan and nonprofit news organization bringing Californians stories that probe, explain and explore solutions to quality of life issues while holding our leaders accountable
I wrote about it already in Issue 192, so parts here may be repetitive and I apologize. I’ll try to be brief. Buzz over to 192 if you want more.
It’s terrible reporting because it links the Ohio court decision regarding remote “room scans” (see Issue SE2 and Issue 188 ) and California colleges, which have zero to do with one another. The headline is:
California colleges still use remote proctoring despite court decision
As you may know, the court decision was not about remote proctoring. The incident which triggered the case was about a pre-exam room scan and used Zoom, not a proctoring company.
Worse, the Cal Matters “summary” of the article is:
A federal judge last year found that the room scans performed by some remote proctoring software are unconstitutional. But some California colleges continue to use the software, which is designed to prevent cheating in online exams.
No - “scans performed by some remote proctoring software” were not declared unconstitutional. It was actually just one scan and it was Zoom, which is not remote proctoring software at all. That California colleges continue to use completely unrelated proctoring software has nothing to do with a court case with no legal impact in California or anywhere else. That summary is just nonsense.
And They Know It
What’s more, Cal Matters knows these things are not related - or at least someone there does. Buried near the end of the article is this:
Cleveland State is currently appealing the federal court’s decision, which focused specifically on room scanning and did not address whether e-proctoring as a whole is permissible.
They’re not appealing anymore, but the last part is true - this case did not consider the legality of remote exam proctoring. So, the headline that California Colleges “still use remote proctoring despite court decision” is just bizarre. Their own story says the court decision has nothing to do with remote proctoring.
The article also reports that:
The ruling bars Cleveland State from requiring [the student] to take tests with room scans; [the attorney] is filing a motion to extend that ban to the rest of the university.
That was mostly right - the ruling was limited to just one student. So, for the fourth time, this ruling has zero to do with schools in California. So, I cannot figure out why Cal Matters is even writing this article about schools in California.
But I can. The article is performance pouting.
Performance Art as Journalism
As I noted in the previous dissection of this article, if you’re really well versed in academic integrity and remote exam security conversations, the big tell that this is grievance performance instead of journalism is the use of the term “e-proctoring.” Only people who want to debate and litigate the need for exam fairness and security use the term. It’s a slip of the keyboard that reveals everything.
And, sure enough, the article helpfully links to and repeats the very groups that continue to make disproven or unproven charges about remote exam proctoring, calling it “racist” and an “invasion of privacy” and so on.
Cal Poly Pomona
Nonetheless, the Cal Matters story continues through several California schools, reporting that most are hit and miss, with remote proctoring available to professors or programs as they see fit. It does not, of course, mention that such a patchwork policy is part of what got Cleveland State University into trouble in the Ohio case. If the school simply required remote exam proctoring rather than making it optional, the case probably would have gone differently.
But, in any case, this passage from Cal Matters stuck out:
Unlike other California college campuses, Cal Poly Pomona has a strict policy against the use of e-proctoring “due to the concerns about privacy and equity,” a university spokesperson said.
“E-proctoring” again.
In fact, for fun, count how many times that shows up between CPP and Cal Matters.
But because Cal Matters is so obviously not doing actual journalism, I thought there was a better than average chance that this nugget about Poly Pomona having a “strict policy against” proctoring was wrong. So, I asked.
Poly Pomona Says
The response from the school seemed pretty clear, though stunning:
E-proctoring has not been used at Cal Poly Pomona since the Fall 2020 semester started.
Again, “E-proctoring.”
Saying that it had a pilot with Respondus, the school said:
In Summer 2020, the national conversation about e-proctoring occurred, and again the Provost made the decision that Cal Poly Pomona would NOT continue with Respondus due to the concerns about privacy and equity.
They don’t say what those “privacy and equity” concerns are. And I’d suggest that few things are less equitable than cheating. And I’m not sure why taking an exam in private is expected, or ever a good idea.
Still, it does appear that Cal Poly Pomona does not secure its online exams with proctoring, by policy. Which is jaw dropping. That any school would administer online exams with no one watching, as a matter of declaration, is indefensible. It’s intentional educational malpractice.
Somewhat stunned, I asked if the school had any assessment security features whatsoever. Poly Pomona’s media relations folks were very helpful and cooperative and returned this, in its entirety:
The following answer regarding online exam and assessment integrity is from Victoria Bhavsar, director of the Center for the Advancement of Faculty Excellence:
“Online exams and assessments at CPP are NOT entirely unsecured. Faculty can use a variety of features in the Canvas LMS to safeguard academic integrity, such as: password required to take the exam, question pools so that every student gets a different batch of questions, timed exams to reduce opportunity for looking up answers, shuffling answer order for multiple choice questions, limited availability windows, filtering IP addresses, and setting so that students cannot see their answers after clicking through. Note that MANY of these compromise learning, so it’s always a balance.
“It’s a LOT more effective to cultivate positive, strong relationships with students and to offer them a well-designed course that actually helps them to succeed than it is to engage in a technological arms race.
“E-proctoring cannot keep exams entirely secure either; a person who is determined to circumvent the system can very likely do so.”
E-proctoring again. What’s that, like seven?
Anyway, that is why this is a Special Edition - I had to get into it.
Now It’s Time for a Breakdown
The director names several techniques and tactics and tools that she believes help secure online exams at CPP. Some of these are a joke. Others are marginally effective, though not nearly sufficient. Even together, they in no way meet even the most milquetoast meaning of exam security or fairness.
Let’s go through them.
Requiring a password to take an exam. That one has to be a joke. Right?
How many examples of impersonation do we need in order to know that, if you’re paying someone to take your courses or exams, you give them your password. Exam cheats share passwords.
A year ago, CBS interviewed a paid exam cheater in Africa and showed him logged in to the LMS at Florida International University, where he was taking classes and tests for students (see Issue 47). He had the passwords. This is like first-day-of-academic-integrity stuff.
Moreover, what’s the alternative - not using a password to access the school’s LMS? How does that work? Seriously, come on.
Question pools. We’ve covered this one ad nauseum. But we will do it again.
Once a question is used in an online test one time, it’s compromised. Period. Especially if the exam is unsupervised or unproctored.
Easy access to test bank questions and answers is why Chegg and Course Hero and others are in business. And for most test bank questions, those cheating providers aren’t even necessary as Google is more than helpful enough. Put another way, if you are using test bank questions in unsupervised exams, you are being cheated.
Just last year, two professors studied this. And though they are not the only ones to do it - and not the only ones to reach the same conclusion - their findings were clear (see Issue 105). From past coverage of their paper:
Finding answers to homework, quiz, or test questions using these websites is easy. For example, a question taken from the text bank of a major publisher was plugged into Chegg’s search function, and the answer (which even contained the test bank’s unique identification number) was promptly displayed.
And:
[Using Course Hero] the answer to a random test bank question was available on demand
Or see Issue 71, in which other academic professionals specifically advise avoiding test bank questions because they are so easy to cheat. One says, for example, that to raise integrity standards professors should:
Avoid test bank questions. If you Google some of them, they’re almost all compromised on Chegg and some of the other sites.
Or, if you prefer, see Special Edition 1 where a professor, using a test bank pool for low-stakes weekly unproctored quizzes discovers “a volcano of cheating.” In this case, the students simply created a group chat and shared the answers.
I could go on. But using a question bank or question pool is by no means an exam security feature. In most cases, it makes cheating easier and more likely.
Poly Pomona also says professors can use timed exams as a security feature.
This can work to mitigate cheating, but timing is also incredibly easy to work around. That is especially true if the timed exam has a pause or stop-and-resume feature or if the timed exam extends over a longer test window - a one hour limit but 24 hours in which to finish the test, for example.
Cheating on time-limited exams is common when there’s a pause feature or a “finish later” feature. Students simply access the test, copy or photograph the questions then go research the answers. If an unproctored test period extends over many hours or days, students share answers. Some volunteer or are paid to take the test immediately and will sell or share the test questions and answers with those taking the exam later.
And a timed test feature, of course, won’t stop exam impersonation or even in-test collaboration. Limiting a test to just 30-minutes will not, for example, stop a student from taking the test with five classmates.
In other words, limiting the time on tests is good but, again, insufficient to significantly mitigate misconduct.
CPP says they can shuffle answer order on multiple choice tests to reduce cheating. Again, this is fine but it’s like treating a gunshot with Bactine. It won’t hurt anything. But it won’t turn the trick either.
Shuffling answer order won’t stop Chegg. Or Google. It won’t stop exam impersonation. It won’t stop collaboration.
In other words, switching answer order is like locking the door at a jewelry store. It’s a good idea but if you really want to protect your valuables, you have to do more than that. It’s really a bare minimum standard of care.
Poly Pomona also lists “limited availably window” as a “safeguard.”
I confess I’m not sure, but that sounds like the window in which someone would take an exam. They don’t elaborate but, if you jump up you’ll note that this safeguard is entirely dependent on how long the window is - two hours in which to take an exam with a 60-minute limit is probably somewhat effective. Twenty-four hours is an invitation to cheat.
But again - what’s the alternative? Not closing off the time to take an exam? Like forever? How’s that work? I’m just not sure setting a deadline by which an exam must be completed counts as cheating prevention.
“Filtering IP addresses” is also mentioned. I am also not confident what this means specifically, but presume it means that professors or others can examine the IP address from which an online exam was accessed and taken.
This is good. And, in my view, should be mandatory - not just for odd locations but for matching locations. A solid policy would require that every student who scored an “A” or “B” on an exam would have their IP address reviewed. The policy should be public. It’s not clear whether CPP has such a policy - it’s a safe bet they do not. Even so, of their entire list of security measures, this is the only possible intervention that can fluster exam impersonation.
Now, it’s possible to hide or fake an IP address. Any good proxy cheater will know how to do this. And checking IP addresses won’t stop Chegg or Google or sharing answers or questions or even in-person impersonation such as paying a prepared classmate to take your exam on your laptop. Still, it’s good to have.
Finally, Poly Pomona says it’s possible to set up an exam so students can’t review their answers after they submit them. I think they mean questions and answers - the inability to go backwards in a test. Limiting only the answers leaves plenty of cheating on the table.
But like time limits, this is a good feature to deter some misconduct. It can, for example, limit the ability to view a test, pause it and go look up - or pay for - answers. It won’t do anything to limit proxy test taking or using Chegg or Course Hero in real time. The companies promise on-demand answers in minutes with no more than a snapshot and text message. Once again - good, but weak and overwhelmingly insufficient.
Most importantly, these imagined or inadequate security features only work if professors or programs use them. Given where CPP is on exam proctoring - flatly refusing to use a tool that’s been proven to reduce cheating - I’m skeptical many educators at CPP are leveraging them at all.
Philosophy, not Integrity
I’m further skeptical that many of these tools and tactics are actually being used because of the last line from Poly Pomona in their list of theoretical protections:
Note that MANY of these compromise learning, so it’s always a balance.
They do?
Randomizing question order or checking IP addresses compromises learning? How’s that? I’d love someone to explain to me how requiring a password to take an online exam compromises learning. That feels made up.
Further, I do not get the “balance” between some “compromise” in learning from cheating prevention tools and cheating itself. Follow me closely - cheating compromises learning. Creating or allowing an environment in which cheating goes undetected and unaddressed compromises learning. It’s the antithesis of learning. You cannot protect learning by allowing cheating.
I feel as though that’s important so let me type it again - cheating compromises learning.
Clearly, this anti-security stance at CPP is philosophy and not pedagogy or logic.
That’s obvious because the statement from CPP continued:
“It’s a LOT more effective to cultivate positive, strong relationships with students and to offer them a well-designed course that actually helps them to succeed than it is to engage in a technological arms race.
“E-proctoring cannot keep exams entirely secure either; a person who is determined to circumvent the system can very likely do so.”
Well, sure.
“Positive, strong relationships with students” and good courses are great. But we are talking about online courses at a regional public university. Moreover, I wonder whether the sizes of these online classes and who’s teaching them adequately lend themselves to such relationship building. I think we all know those answers.
Most importantly, there’s no evidence that I’ve seen that says that these relationships are “more effective” at preventing or detecting or addressing misconduct than any other approach. I mean, maybe, under a given circumstance. But to say that student relationships are not just more effective but “a LOT” more effective is speculative at best, wishful thinking at the midlevel and downright fantasy at the worst.
And, not for nothing, I am deeply doubtful that a permissive and blind policy related to cheating fosters good relationships.
In fact, we know it won’t. Researchers have asked and students have answered. A laissez-faire approach to cheating is deeply destructive to student relationships because it demonstrates that schools do not care enough to value and protect the actual work of honest students.
I also highlight the word “either” in the statement above. It’s a concession. CPP knows that the tools they may possibly use for online exams aren’t “entirely secure.” True, they are not. In fact, even if all of CPP’s tools are used in unison, they are still way less effective than even the most basic exam proctoring or supervision. In other words, they know full well their security features don’t provide much security.
CPP is happy to brag about the coverage of their fig leaves while refusing to actually put clothes on.
The numbers
When I made my follow up to Poly Pomona, I asked for their numbers of academic misconduct cases. Here’s what they sent me:
20-21: 387
21-22: 368
22-present: 189
Enrollment at CPP is about 30,000 students. Taking the highest number, the 387, that’s 1.3% of students, assuming no one was caught twice.
I don’t know what it should be, but 1% is absurd and simply not credible. But it is entirely expected that you don’t see much cheating if you close your eyes.
You may also note that the numbers declined since the school decided it was not going to supervise online assessments. It’s supposition, but such a low evidence of academic misconduct is exceptionally likely linked to a lack of any effort to find it.
At Long Last - In Conclusion
Cal Poly Pomona has gone all-in on not securing its online programs and assessments and the grades and degrees they compose. It’s a choice and there’s simply no other way to see it.
Knowing what I know, and what 30 years of academic integrity research makes clear, cheating at CPP is widespread, fueled in no small measure by the school’s refusal to put common safeguards in place. Absence of consequence creates abundance.
Waves of research show that one of the most impactful things that can be done to reduce misconduct is altering the risk/reward calculation. At CPP there is no risk. Literally no one is watching.
A level up, CPP is telling their students, faculty, stakeholders and their community that they could move to protect their credentials and degrees and their academic reputation, but they’d rather not. They’d rather not know. Ignorance may be bliss, but it’s still ignorance.
Eventually, the market will sort this out. Hiring managers and graduate school admissions officers learn which degrees matter and which do not. In the meantime, schools will continue to make public choices about what they value. Or in the case of Cal Poly Pomona, what they do not.