Evaluating the pros and cons of potential solutions for online testing can feel overwhelming, particularly for those who are doing so for the first time.
In my previous blog: A Guide To Going Multi-Modal With Testing – Part I, I explained what multi-modal testing is, along with several pros and cons of different testing modes, and I outlined the growing body of research available in support of a multi-modal program in order to address different needs. However, much of this research was completed before COVID-19, and we are now living under quite different circumstances.
Due to the large-scale move to online testing in recent months, we now have a wealth of new data to consider. This blog summarizes the insights this data provides as a guide to support your decision-making with multi-modal testing in 2020 and beyond.
The Picture Before COVID-19
Before COVID-19, multi-modal testing was primarily used by IT credentialing organizations running large global programs in competitive environments. Convenience of testing has always been essential for these test-takers who are far more likely to select a program in which testing will not involve a lengthy drive.
However, while providing candidates with options for accessible testing was an important factor for credentialing organizations prior to COVID-19, most multi-modal programs tested no more than 10-15% of their candidates through online proctoring, with most candidates still opting to take their exam at a test center. Not surprisingly, this picture has changed dramatically since March 2020.
The Picture Since COVID-19
Many more credentialing organizations have now migrated to flexible multi-modal testing across many sectors: IT, Healthcare, Finance, Construction, Legal, Admissions, and more. At PSI, we have migrated over 40 organizations to multi-modal testing since March 2020, with even more organizations in various stages of implementation. Among organizations that have already transitioned, as many as 71% of test-takers chose to take their exam via online proctoring.
Interestingly, even those credentialing bodies that were deemed as essential businesses during the COVID-related lockdowns – and, therefore, were able to continue testing in test centers – saw a nearly identical uptake of online proctoring, with 68% of these candidates testing remotely. In IT credentialing programs, as many as 80% of candidates used online proctoring, compared to 67% in healthcare and 49% in construction, which indicates that industry may impact the decision of the testing mode selected.
Candidate Satisfaction Trends
The increased volume of test takers using online proctoring has allowed us to measure the impact on candidate experience compared to onsite proctoring to a greater extent than ever before. Results are encouraging, with candidate satisfaction rated highly for both the PSI online proctoring and test center modalities. As many as 99% of test center candidates and 95% of online-proctored candidates rate their experience as Excellent or Good.
Satisfaction scores show that in some sectors, candidates rate their satisfaction with online proctoring even higher than did their peers who tested in test centers. The IT sector is a good example of this, where candidates are likely more accustomed to online proctoring. We believe this indicates that as online proctoring becomes more commonplace, candidate satisfaction trends will improve even further for this modality.
Early Research Findings
A closer look into test-taker performance also supports multi-modal testing. Among test-takers between January and June 2020, when comparing those testing via online proctoring and those in test centers, both mean scores and pass rates were comparable across exams. Equally, candidate experience ratings were positive, unrelated to modality, and had no impact on test performance. See more candidate satisfaction data regarding multi-modal testing options and the impact testing modality has on candidate performance.
Security and Test Fraud
Statistical surveillance is an essential component to ensure the security of high-stakes multi-modal testing. Data forensics show comparability across the multiple metrics that PSI employs to monitor and investigate suspected test fraud. For example, response times across different modes can show potential item harvesting. Unexpected response patterns, such as getting difficult items correct and easy items wrong, can also indicate malpractice. In our sample, there were no flags to indicate an increase in malpractice in either test centers or online proctoring.
Comparability Across Modes
Due to PSI’s experience with a wide array of testing programs in different sectors, often involving large numbers of candidates, we are in a unique position to examine meta trends. Our early research findings detailed in this report are encouraging. We feel the information provided could prove beneficial to organizations considering the move to multi-modal testing and could draw from these results to begin their own research.
Moving to Multi-Modal
If you are considering the move to multi-modal testing, our advice would be to begin as early as possible. None of us know what the future holds but, regardless of the broader context, by taking the first steps now, you will have more time to test, implement, and communicate with existing and future candidates.
We’d always recommend speaking to other organizations who have made the transition. And of course, it’s important to find the right partner who has experience and can quickly deliver. At PSI, we understand that the change to multi-modal testing is a big decision for any organization, and making the move isn’t as easy as flipping a switch. But our data shows that with a little time, forethought, and preparation, you can have complete confidence in the comparability, security, and candidate experience of mixed-mode testing.