As a consultant and project manager, it’s my job to recommend the right tool for my clients’ hiring needs. I also get to answer their questions about why these tools are the best fit. And without exception, there is one question that I get asked more often than any other: Is this assessment validated?
This is a fun question to answer. The psychologist in me wants to delve into the theoretical and research-based implications of that question, while the consultant in me wants to keep things as straightforward as possible. While I’ve done both in the past, I think the core of the question is best addressed when we look at it from a scientific and a practical perspective.
So, when you ask about validity, you’re not actually asking a “yes or no” question, because validity is a collection of various factors and evidence. Here are some of the key ways that this question can be interpreted. Fair warning: I’m going to geek out a little here.
Does the assessment hit on all of the areas related to the job?
One way that a test can be valid is in terms of what it includes – and what it doesn’t. For example, the MCAT is intended to measure your likelihood of success in medical school. It measures problem solving, critical thinking, and knowledge of medical science concepts, because those core areas were determined to be the best predictors of medical school success. If the MCAT also tested applicants on areas that weren’t shown to be related to medical school performance (for example, tap dancing), then this test would no longer be a valid measure of its target (medical school performance). Additionally, if the MCAT tested on some areas that were important (problem solving and critical thinking) but didn’t include other areas that we know are crucial (knowledge of medical science concepts), then its validity would also be compromised.
Are the items in the assessment necessary and appropriate?
Let’s use the MCAT example again. We may agree that those core areas of problem solving, critical thinking, and knowledge of medical science concepts are a comprehensive and accurate representation of how to measure medical school success. But now we need to ensure that the actual questions within each core area are also appropriate. For example, if the section on medical science concepts only asks about head injuries, then we haven’t measured the core concept very well. Head injuries are only one area within the larger concept of medical science knowledge. In order to have valid content, each section of an assessment needs to be comprised of appropriate questions that capture the scope of the general measure.
How do you know that someone’s score on the test is a good measure of their future job performance?
This is what most of my clients mean when they ask about validation. This question boils down to: do you have proof that this test works? One way to answer this is to review the previous types of validity that we’ve discussed: does the test have a clear and comprehensive definition of what its measuring, and have the items been developed soundly? If so, then you’re in good shape already. But to determine if the test is measuring the intended outcome (in our MCAT example, medical school performance) then you’re asking about a statistical relationship. These relationships are determined by comparing scores on an assessment with on-the-job performance. Most established assessments will have aggregated evidence that their test is related to future job performance; in other words, that the test “works” in general. In order to get more specific, the test scores for your own applicants would need to be compared to their on-the-job performance. For example, we know the general relationship between MCAT scores and medical school performance is strong, and we can therefore assume that the relationship will be stable across medical schools. But, we would need to do more research to determine the exact relationship between MCAT scores and performance at any one particular school.
While general validation evidence is usually sufficient to support the use of an assessment, your organization can dig in deeper and explore those relationships on a more customized level. Don’t be afraid to ask your consultant about these big picture issues – that's why we're here!