What's the Right Frequency for Statistical Analysis and Reporting?

I remember my first job in the testing industry very well. I had two great mentors – the two principals of the small consulting group I worked for who shaped my entire career with these simple words: What do you want to know?

Picture a PhD student who exclusively reads academic journal articles, co-teaches the advanced statistics course, lives for learning new statistical techniques, and is totally terrible at small talk. You can imagine his first few interactions with clients and prospects; they were riddled with overly academic lines of questioning and awkward jumps from topic to topic: Is your candidate score distribution sufficiently far from normal that non-parametric analyses are necessary? Are there any factors in your eligibility requirements that would lead to range restriction? Have you considered correcting that correlation for attenuation?

Then my mentors intervened, and they explained that clients don’t talk like that and that clients certainly don’t think about their programs like that either. Instead, the first question should always be: What do you want to know?

From there, relations with those clients improved and rapport was established – and most importantly, the conversations necessary to understand how to best support those clients were underway. All that statistical knowledge was finally being applied to real-life questions and decisions.

Since then, one of the most frequent questions clients ask me is: How big does our sample size need to be?

An older and wiser version of that PhD student now eschews any discussion of statistical research (at least at first) and looks to understand what type of analysis is needed to support the client’s decision-making. Since those formative early years, I’ve added my own second question to the mix: What do you want to be able to do with that information?

While there is much to say about different measurement models and such, the real determinant is going to be the decision that you want to make with the results of an analysis. My answer will need to be different whether we are looking to evaluate exam items, identify an outlier among standard setting judges, or conduct a feasibility analysis for a possible new credential.

Another version of that same question that I hear often is: How often should we conduct analysis?

This is one of my favorite questions because it lets me know that a client is thinking practically about their responsibilities to their certification program. The answer is simple: Only as frequently as meaningful data becomes available. In other words, there is not much utility in conducting analyses monthly if we only obtain data from 5-10 candidates each month. Instead, quarterly or twice-yearly analyses make more sense because the amount of additional data becomes substantive enough for interpretation.

There is another piece to that answer: Only as frequently as you are prepared to act on the results. Even when data is available for meaningful analysis, sometimes it is not practical to make decisions at that same pace. For example, even for examinations with hundreds of examinees per month, it may not be feasible to try to develop new forms or review pretest items every month. In other words, conducting analyses more frequently than you are prepared to do something about it is a wasted effort.

Related: Getting Beyond "It Depends:": How to Get the Best Response from Your Psychometrician 

One of the most important actions taken with results of an analysis is to report it to stakeholders. Some of those stakeholders include candidates, educators, or accrediting bodies like the NCCA (who require annual reporting that is based on either a calendar year or an equivalent period of time). Facilitating the awareness of their stakeholders and allowing those stakeholder processes to be performed are essential to a certifying body’s mission.

A lot has changed since I first started working in testing, but the principles that guide my approach to consulting and psychometric services have not changed. It always comes back to: What do you want to know and what do you want to do with that knowledge?