Using Innovative Data Forensics to Improve Test Security

These days, candidates might take some interesting measures to commit misconduct. In the first installment of our two-part blog series on ways to improve test security, we talked about traditional test security measures such as using a proctor, having a seat map, and communicating a strong and clear exam policy to your test-takers. In this blog, we will discuss the latest innovative data forensics techniques which work to solve misconduct and test security issues "behind the scenes."

There are various ways candidates can commit misconduct, from test-takers receiving pirated test content before the test to copying answers directly from another test-taker while taking the exam. We have ways to address these situations with item and option scrambling, however additional security measures can (and should) be taken to mitigate cheating. It's important to explore other options and we are implementing more innovative security initiatives including ongoing operational measures and data forensics. 

Data Forensics

A successful test security program employs early detection methods for identifying cheating and other fraudulent activity when it occurs. Detected activity can include item harvesting (an attempt to collect a large number of test questions for future distribution) and proxy testing (when the test is completed by someone other than the test taker). These and other cheating activities can diminish the integrity of a certification program, so identifying suspected test fraud and taking action as early as possible is a crucial component of test security.

Forensic analysis involves scientific tests to detect the occurrence of a criminal act. Locard’s principle on forensic evidence is summarized by the mantra “Every contact leaves a trace.” While generally discussed in terms of physical evidence left by contact points of people at the scene of a crime, we can likewise state that when people interact with tests and test questions in abnormal and fraudulent ways they are likely to leave traces of their acts, occurring as irregularities in their answer choices, score patterns, and response times.

Data forensics in the psychometric context employs statistical trends and analytic techniques to detect these traces of fraudulent behavior that may indicate cheating by one or more test takers. PSI’s data forensics program provides ongoing surveillance of test items, test takers, schools, and test sites. Special attention is given to unusually high pass rates, abnormal item scores, and irregular response patterns.  Data visualization techniques are incorporated to graphically and clearly show any anomalies consistent with cheating trends and is also used to assist with early detection.

After testing occurs, our data forensics team completes analysis of candidate responses for clients testing around the globe. This analysis can be completed daily, monthly, or quarterly depending on the size of candidate volume. A test site, school, or candidate associated with anomalous findings may trigger further in-depth analysis and an investigation. Investigative measures may incorporate a review of electronic records including video or photos from the test administration as well as "drop-in" inspections or a "secret shopper” for exams administered at a test site. Upon confirmatory evidence of fraudulent activity, corrective action will be taken. Results of the investigation are summarized in executive-level reports. 

A Flexible and Research-supported Approach to Test Security

Using data forensics is a more effective way to identify anomalous testing behaviors, as statistics are sensitive to different patterns of interest. It’s relatively easy to interpret data with tables and graphs, and fewer assumptions can be made and fewer data is required for analysis. PSI’s data forensics approach is supported by research studies and investigation evidence and is a flexible approach to mitigate cheating.