If you are developing assessments that measure the knowledge, skills, abilities, traits, or attitudes of your test takers, you will likely come into contact with a psychometrician at some point. The application of psychometric principles is vital at all stages of the testing process: test design, test development, and evaluation.
Drawing on the knowledge of our team of incredibly talented psychometricians, we’ve pulled together a guide, Effective Testing with Psychometric Principles, to help you understand how to better apply psychometric principles to your assessments. We hope that the guide demonstrates how our psychometricians can support you at every stage of the process and to help ensure your tests are fair, valid, and legally defensible.
Effective Test Design
The landscape of learning is changing, and we need to constantly adapt how we assess learning to reflect this. However, there have historically been differences between the learning sciences and psychometrics when it comes to assessment, underpinned by a divergence in how we view validity. Now, advances in computing power and Artificial Intelligence have the potential to bridge the gap. We explore the differences between the two paradigms in our guide.
We also address some of the most pressing questions we are faced with during the test design process. How many items do we need to create a test? And how often do we need to create more? The answer is…it depends (more on this later). Of course, there will always be a need for more items, but how many more? To help identify the right answer for your program, we recommend looking at item usage, the rate of change to test content, and the potential for pool-based approaches like linear on-the-fly testing (LOFT).
Covid-19 has caused many of us to adjust our work processes. In our guide, we look at one example relevant to test design: the job analysis, often cited as the foundation of a certification program. We consider how the changing external environment has affected the performance of a job analysis, and share steps that you can take to run efficient and productive job analysis meetings.
Find out more about psychometric principles for effective test design.
High Quality Test Development
Moving on to the test development stage, we look at how enforcing a policy on referencing items is an excellent method to ensure test content is accurate and up-to-date, as well as making results more defensible. We share our top recommendations for item referencing, including an approved reference list, a policy for referencing items and more.
Diving even deeper, we explore the concept of human cognition as it relates to certification exams. The concept of an item’s cognitive level, or thought process required to arrive at the correct answer, is an integral piece to developing high-quality exams. We explore how when and how use of cognitive levels can support the test development process.
Recruiting subject matter experts (SMEs) is vital to ensuring the content-validity of your assessments. But who are your SMEs? What characteristics lend themselves to a quality SME? And how many SMEs should there be? Our guide provides steps you can take to answer these questions, from defining your target population, to recruiting and assembling your committee, and finally putting them to work.
Find out more about psychometric principles for high-quality test development.
Rigorous Test Evaluation
When it comes to test evaluation, we always aim to apply our statistical knowledge and expertise to the real-life questions and decisions our clients are facing. That said, the solution isn’t always cut-and-dried, so it may not be uncommon to hear “it depends” from your psychometrician.
In our guide, we explore what psychometricians and non-psychometricians alike can do to get beyond “it depends” and formulate a more detailed response to questions related to test evaluation.
We also look at scaled scoring for your certification exam – a process that is simpler than you might think – and how scaled scoring may be able to help to establish consistency of score reporting and better management of score interpretation.
Lastly, we explore some of the most common questions when it comes to test evaluation, such as the right frequency for statistical analysis and reporting.
Find out more about psychometric principles for rigorous test evaluation.
Come and Test Us
Many of our clients tell us that our talented psychometricians are a big reason why they love working with us. What’s more, the psychometricians on our team aren’t just brilliant, they’re also accessible. We don’t hide our technical experts away. It’s part of the PSI culture.
Our team of quality psychometricians are heavily involved with client projects from day one, providing knowledgeable guidance and advice to our clients on a daily basis. Come and test our psychometricians.
