I’ve seen some funny quips about questions, answers, and the nature of truth in measurement. For example: “A man with two watches never really knows what time it is.” Or this one: “Ask two psychometricians a question, and you’ll get at least three opinions.” These quips are funny because they are true. A common response from psychometricians is also true, but some may not find it funny anymore! I’m sure it’s happened to you: you ask a psychometrician a simple question, and the answer is “it depends.”
You: “What’s the ideal sample size for equating?”
Psychometrician: “It depends.”
You: “What measurement model is best?”
Psychometrician: “It depends.”
You: “Is the Bookmark Method of standard setting useful?”
Psychometrician: “It depends.”
While in conversation with one of my colleagues, she broached the subject with me and suggested we put a moratorium on “it depends.” Surprisingly, she was speaking as a psychometrician about “we” as a community of psychometricians, showing that even some psychometricians have grown a little weary of this response! The problem with “it depends,” though, is that it is true. The answers to the questions above DO depend. Because when faced with questions related to assessment or measurement, lots of factors need to be considered: technical limitations, business limitations, availability of resources, time, goals, risks, benefits…the list can go on and on.
Getting past "it depends"
If we are going to get past “It depends” as the first answer to so many questions, both psychometricians and non-psychometricians can play a role. For psychometricians, answering with "it depends" is too easy and may seem to others like we’re trying to avoid the real discussion - or even worse, that we're being lazy with our answers. So, as a psychometrician, try to not even say “It depends” until you’re absolutely ready to talk about what it depends on!
Non-psychometricians should beat us to the punch. Try to ask questions in a particular context. This will drastically reduce the temptation for us psychometricians to say “It depends.” Below are suggestions for better ways to phrase the above questions:
- “Given our program’s history, how many candidates do we need to have confidence in our equating results?”
- “What measurement model makes the most sense for our program?”
- “Do you think the Bookmark Method would be useful to set the standard after our next job analysis?”
Asking a specific question in a particular context, you’ll drastically reduce the number of factors that could affect the problem at hand, therefore reducing the likelihood of an “It depends” answer and helping us (your psychometrician) formulate a more detailed and useful response. It can be useful to know what other programs might do, but it’s even more useful to understand what solution paths are more or less promising for you.
In those inevitable situations where you still get “it depends” as an answer, try to be patient with your psychometrician. Perhaps simply nod encouragingly, and the rest of the answer will unroll with a collection of factors and points of view to consider. And remember, psychometricians often resort to this overused phrase because one size typically does not fit all and there are few absolutes when it comes to measurement.
Daniel H. Breidenbach, PhD is the Director of Psychometrics, Certification, and has been a psychometrician with PSI for more than 10 years. He works with a wide variety of certification clients, offering assistance with program direction as well as providing technical guidance for examination programs. He is involved in all aspects of the examination development cycle – from job analysis through standard setting, and he particularly enjoys helping certification professionals think through psychometric issues as they maintain credentialing programs.
