In an extensive study of 19 different employee selection methods, reference checking came in a dismal 13th in predicting the success of a new hire. There is also evidence that candidates from privileged backgrounds perpetuate that privilege in the reference-checking process. Apparently, most reference-checking practices aren’t much better than gathering advice from strangers on the internet (according to the internet, this is a bad idea).

To get more value from our reference checks, we started asking different questions. Our goal was to improve the ratio of gathering facts vs. opinions. We call this “competency-driven reference checking” and it naturally follows our Competency-Driven Interviewing process. The difference was striking. All we changed was the reference questions themselves (because we already had a cleverly drafted reference release form, a protocol for asking for who we wanted to speak with, and we were already independently verifying the identity of the person giving the reference).

What We Changed

We stopped asking for opinions about general concepts. Instead of asking questions like this:

  • How would you characterize their ability to create and implement a ___ strategy?
  • Can you elaborate on their skills working with the communications team?
  • How were their relationships with colleagues and direct reports?

We focused our questions on the specific impact or effect of someone’s work:

  • Describe a process improvement they implemented. What was the impact on the organization?
  • Please share a few examples of their work in the strategic planning process, and its impact on your business objectives.
  • What steps did they take to promote a culture of inclusivity and collaboration across teams?

Although the language changes were quite small, they led to big changes in the conversation. The questions about impact led the reference to think about and share more information about specific outcomes. Rather than thinking generically about how to best characterize the situation, the reference just shared facts. We achieved our goal of gathering more facts and fewer opinions.

Why We Changed Our Reference Checking Process

In making this change, we were influenced by Marcus Buckingham’s groundbreaking research on the unreliability of performance evaluations. Here’s an excerpt of his thinking from Harvard Business Review:

“Over the last fifteen years a significant body of research has demonstrated that each of us is a disturbingly unreliable rater of other people’s performance. The effect that ruins our ability to rate others has a name: the Idiosyncratic Rater Effect, which tells us that my rating of you on a quality such as “potential” is driven not by who you are, but instead by my own idiosyncrasies—how I define “potential,” how much of it I think I have, how tough a rater I usually am. This effect is resilient — no amount of training seems able to lessen it. And it is large — on average, 61% of my rating of you is a reflection of me.”

Idiosyncratic rater bias only partially explains why reference checking has proven to be a poor predictor of success in hiring. A candidate’s former boss is also unfamiliar with the context of your hiring. That’s why we stopped asking strangers for their opinion about unfamiliar situations.

In making the change, we have found that someone’s former manager can almost always describe the impact of their work and the results they achieved. In contrast, their opinion about the candidate generally revealed more about the manager than it did about the candidate. In fact, it might be the reason the employee chose to leave that organization.

How Competency-Driven Reference Checking Works

In our experience, the person giving the reference will cheerfully offer 10 to 15 minutes of their time to volunteer information. Beyond that, most people will begin to suffer from “reference fatigue” and rush you off the phone. So it’s important to gather key information quickly. (The exceptions are retirees and the occasional overly enthusiastic reference. But again, the amount of time they offer is more a reflection of the person giving the reference than the candidate.)

General questions invite lengthy stories. The reference is not sure of what you know and often feels compelled to offer quite a bit of context to justify their opinion. “You see, our company started in my basement in 2002…” By the time the reference has finished answering their first question, half your time is gone, and you might only learn that they have strong opinions about a few things. Generic questions force the reference-giver to guess at what you want to know, so you mostly hear statements they could make about anyone, “Everyone liked working with Jamal.”

To avoid “reference fatigue,” we ask specific and substantive questions about impact right away and then continue asking about impact with each additional question. Specific questions invite shorter and more fact-based answers. It’s easier for the reference to be truthful, and their answers reveal quite a bit about the candidate’s personal accomplishments. Ideally, this information corroborates what you already learned in the interview and is consistent with what the candidate claimed credit for. This validation should help you become more certain in your own mind (or conversely, it might raise doubts about what they took credit for).

When you begin the reference conversation with specific and factual questions, the reference-giver assumes you already understand their context and spends less time trying to educate you and more time verifying what you want to know. Yes, references might occasionally spin out a few carefully worded sentences to try to help candidates look good, but almost no one will outright lie about the results someone achieved.

Because in-person interviewing is not going to be possible for quite some time, some people suggest that employers should do more reference checking to compensate. I disagree. Get as much value as you can from reference checking, but put more of your attention on the factors that really predict success in hiring.