<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=353110511707231&amp;ev=PageView&amp;noscript=1">

How to Predict On-the-Job Success in Your Residency Matching Process

May 24, 2018

improve-residency-matching-processWhile the process of selecting medical residents and fellows is unique, it’s still a selection/hiring process intended to identify the candidates most likely to succeed. Decades of tradition and, particularly with regard to residency programs, a rather formal matching process, have kept organizations from thinking about this selection process in the correct light. Not surprisingly, there is little data supporting the efficacy of the current process.

After nearly a decade of helping hospitals and health systems to more successfully identify the right candidates for senior leadership roles, manager roles, nursing roles, and other front-line staff positions, these same organizations started asking about a more deliberate approach to hiring physicians and physician leaders. Now we are getting inquiries about a better way to select medical residents and fellows.

Leading organizations in other industries have been using these methods to identify attributes and the data that are most predictive of success on the job to build a selection system to objectively, deliberately and consistently evaluate these criteria for decades. In fact, we adopted the term “evidence-based hiring” to describe these methods and processes. The same concepts apply to physicians, residents and fellows.

An Example of Evidence-Based Hiring

A large health system invested millions of dollars in a program to train high potential employees in a year-long leadership intensive. After three years, they found that many participants did not complete the program and many who did, didn’t perform well in leadership roles after the program. How were they choosing participants? A simple nomination process where leaders from various departments and business units picked someone they thought might do well.

Looking at the data from three years of the program, we were able to identify traits, experiences, and skills that were common in those who succeeded and those who failed. For example:

  • Had the person worked in multi-disciplinary, cross -departmental initiatives?

  • Had they handled formal presentations to director-level or above leadership teams?

  • What manager level responsibilities had they handled?

  • How did they perform on a formal evaluation of leadership traits?

From this data, we were able to build an admissions process of sorts. Program attrition was cut by 60%. Promotability of program graduates improved by 50%.

The context of choosing medical residents and fellows is unique but these are roles that have clear performance standards and once we identify what predicts success, we can improve the process of predicting success. We were recently asked by a leading academic medical center to improve their candidate screening process for much sought-after fellowships.

The Traditional Approach Doesn’t Seem to Work

A traditional approach to resident and fellow selection evaluates:

  • The application

  • Performance during an in-person interview

  • Performance on standardized tests

  • Medical school transcripts

  • Letters of recommendation from faculty

  • Indications of commitment to the specialty based on scholarly and research endeavors

There’s scant research supporting the idea that any one of these criteria or any specific combination is actually predictive of success. For instance, program directors often report that the in-person interview is the most important part of the process. Program directors – and even candidates – routinely report that their gut instinct during the interview is highly persuasive in their decision-making process.

We know that interviews in any field are notoriously NOT predictive of on-the-job performance. This is particularly true when the interview is unstructured and conducted by someone not trained or highly skilled at interviewing. The best interviewers are experienced and use a deliberate, structured approach, including utilizing behaviorally-based questions. These interviews are twice as effective as an unstructured interview, but even the best interview is not as predictive as other methods.

And note that few physicians are skilled interviewers. In fact, they are probably more prone than others to trust their instincts. Even when people are faced with data showing that their interviewing skills are not predictive, they tend to ignore that more predictive data in favor of their instincts! 

Related: Traditional Job Interviews are Useless

Physician leaders with no interview training are probably wasting their time. Research bears this out. From a recent article in the Journal of Graduate Medical Education:

The resident selection interview also has been criticized for its “dubious value,”32 due to the lack of a standardized approach and low interrater reliability.27 During the interview, there is a strong potential for a “halo effect,” in which interviewers' prior knowledge about an applicant's academic record (grades, test scores) affect the outcome. 

Even very specific physical skills may not be predictive. Another study looked at fine-motor dexterity, executive functioning, processing speed, and attention to see whether they’d be predictive of on the job performance of anesthesiologists. They weren’t. 

Interestingly, that study indicated that specific personality traits may be MORE predictive of performance.

Fellow Behavioral Traits – One Example

We were recently asked to assist with a program to choose five surgical fellows from a pool of 35 candidates. Their interview process? The candidate has three interviews, each lasting only 10 minutes. The best interviewer in the world cannot be predictive in 10 minutes (no matter how many articles you see about the “Single most effective interviewing question used by Google Execs”).

As part of our recent fellowship selection process, we inserted a physician-specific personality assessment into the selection process. First, we gave more structure and objectivity to the interview by identifying specific behavioral competencies, incorporating physician-specific structured behavioral interviewing questions, and applying consistent scoring. We then built a coherent method of rating all selection criteria into a global candidate score. Part of this score is the candidate’s performance on Select Assessment for Physicians, which measures specific personality traits that have shown to be predictive of performance.

Related: A Case Study on Why You Need a Physician-Specific Hiring Process

In order to calibrate the assessment to this fellowship program, we surveyed Chief, Associate, and Assistant Attending Physicians about what attribute they valued most in their fellows. Rated most important were attributes such as: Working Collaboratively, Positive Impact, Compassion, and Accountability. Rated lower: Adaptability, Conflict Management, and Managing Change.

What we found interesting: Organizations looking to hire physicians today often rate these latter three attributes as being quite important. Our theory is that fellows are functioning in a more structured environment for a shorter period of time. The organization’s expectations of them are clear and are not likely to change during the fellowship. They are expected, however, to be accountable and collaborative.

As these fellows look for positions with a hospital or health system, though, it’s important for the hospital or health system to realize that adaptability, managing change, and conflict management are often challenges for new physicians. To learn more, download our whitepaper:


Bryan Warren Bryan Warren is the President of J3 Personica, a consulting, assessment, training, and coaching firm, and a guest blogger for PSI. Bryan is an expert in progressive talent strategies, with a particular focus on leader and physician selection and development.