A Tale of Two Tests: Effective Interview Testing

A few months ago, I received an email from a greater Toronto company asking me to pass on news about job openings for Linux professionals to the members of PegaSoft. I had heard from a couple of people that this company had poor hiring practices, so I replied to the email and asked the recruiter to confirm that PegaSoft members would receive fair and quality interviews. The recruiter never responded. I never passed on the job news.

In "The Tyranny of the Label" (Lone Coder, January 2006), I discussed how many common hiring practices don't work because they rely too much on short-cuts. In my unfinished online book, "The Big Online Book of Making a Linux Startup or Mendacity and the IT Company", I go further, pointing out that most companies don't even screen for honest, reliable people. Sometimes they hire "destroying angels", unethical people who pass the interview but who destroy departments or complete companies. And in "IT Hiring: We Don't Want the Best" (Lone Coder, June 2005), I discussed how automated job application systems, skill ranking and years of experience do a poor job of producing the best candidates.

This month I'm going to look closer a job interview testing. Conducting a job interview is a difficult task and there are no guarantees that an interviewer will identify the perfect candidate. That doesn't mean it's hopeless. Like most risky activities, being smart about how you review candidates can reduce the odds of making a poor choice.

The most common method used to screen candidates is written or verbal testing. Google has been in the media a lot lately, talking about how they are recruiting the best people (For example, Google Hires Former NASA Astronaut, Wired.) Michael Kaplan at CNN reported that Google relies on "brainteasing" questions ("Want a job at Google? Try these brainteasers first", CNN Money).

This summer I had interviews with a few of companies, including Google. While a Google interviewer asked me not to discuss the specific interview questions on my column, I'd like to draw an analogy of the experience using the old TV game show, "You Bet Your Life." (Wikipedia).

This year I was interviewed by a Canadian company as well, one hiring Linux people. I was given a written test where I had to write solutions for four problems in web programming, system administration and database design. The interviewer returned in 20 minutes and the conversation went something like this.

Two very different interview tests asking very different questions. Both approaches claim to identify excellent candidates. Are they equal, or which is the better solution?

Brainteaser questions fall into three categories.

The first is "trivia questions". These are questions about obscure, obsolete or arcane knowledge, such as questions about rarely used features. You may think that asking trivia questions is a good way of finding experts because only an expert would know a useless fact. But it doesn't work that way. First, this approach is like a blind man with a shotgun. Computer languages today such as C++, Java or PHP are simply too large, too complex and evolve too quickly for any one person to master or experience all the features. Trivia questions test what the candidates experienced recently, not what they've done in the past or can do in the future. Second, trivia questions test book knowledge. They don't show whether or not the candidate can apply the facts. A candidate with a good memory may have simply read a book before the interview. Trivia questions are not an effective way of testing.

The second type is "canned questions". These questions, like university assignments, begin with words like "suppose the universe is a lot simpler than what it is...how would you solve the following problem". You might think that such questions demonstrate intelligence, creativity or abstract reasoning skills. However, as the old university saying goes, "life is a special case": there are no simple answers in the real world. There are always limits to time or resources or the need to optimize for criteria like time or space. Canned questions don't test the ability to set priorities. Also, any answers are affected randomly by similar real-life problems that a candidate may have recently worked on. Canned questions are so unrealistic and so non-specific that they really don't tell you anything about the problem-solving skills or fit of a candidate.

The Google interviews were very short and used almost exclusively these two types of questions. Both trivia questions and canned questions are the kind of questions asked on university exams...and TV game shows. That's why I used the "You Bet Your Life" analogy.

A third category is "I.Q. questions". These are abstract reasoning questions such as determining what doesn't belong in a set or picking the next number in a sequence. I've had interview tests that ask one or two of these kind of questions. I.Q. (intelligence quotient) testing has had a long and controversial history since they were first used decades ago to screen people for U. S. citizenship. First, the interpretation of these kind of questions is a matter of debate. What exactly do they measure? Second, it requires a large set of questions to get a meaningful (read: statistically significant) result. One or two questions during an interview test don't identify the applicant as intelligent. And, third, these kind of tests are hard to apply: it's not clear if a high I.Q. score is a significant attribute for a superior programmer. For example, is I.Q. a more important factor than ethics, short-term memory retention or typing speed? Even criminals can have high I.Q. scores.

Google has been bragging in the media about the fine candidates that they are hiring. However, it was clear to me that, if my experience was typical, any good candidates hired by Google are merely a fluke. They simply didn't know how to ask the right questions to find the people they want. Rather than been novel or original, the Google T.V. game show approach is the way most companies I've interviewed for conduct job interview tests. It's rare to have someone actually think about the consequences of the questions they are asking. So, rather than using an exceptional approach to hire exceptional people, Google is simply like most other companies--ineffective at hiring the best candidates. Hiring so many people means the odd time they do get a good candidate though their system, they make a big deal of it in the media. That says more about their failure than their success.

When a company gives an interview, they are also giving an impression of themselves to the candidate. The impression of the Canadian company was one of care, professionalism and an aim for long-term success. They asked real-word questions and interpreted the results accordingly. In fact, the company had been in business for several decades.

Looking in Robert X. Cringely's PBS archive, I found this recent comment that he made about Google: "Google is an arrogant and geeky company with leaders who have isolated themselves to the extent that they may no longer be in touch with reality. So much success so quick may have convinced them they are smarter than they actually are." ("Is Google on Crack?", I,Cringely). Google's actions in the news left me with a similar impression ("Google: Lawful Good or Chaotic Neutral", Lone Coder February 2006). I had hoped that the Google interviews would have proved me wrong and make me more optimistic about their claims of excellence. But that didn't turn out to be the case. Google may think they're smart, but I'm still waiting for the proof. My father used to say "Those that think they're smart are really annoying to those of us who are."

Which company would I rather work for? Although one might be tempted to have the most prominent Linux company on their resume, one has to assess the cost. Do you want to work for a boss that learned everything he or she knows by watching television shows?

Neither would I.

Google may be a big name today, but they might implode tomorrow. That name on your resume may not be as important as you thought. In fact, it may even be a negative.

When the Google recruiter first approached me, she asked me for a list of other people who were suitable to work for her company. Not being a fool, I waited until I had gone through the interview process before responding. When it was over, I considered the people I know: some of the smartest, most intelligent, most creative people in Linux. Polite and professional. Nope, I decided. Based on my experience, I didn't know anybody who was a good fit. Not one.

So if you're interviewing people and plan to include interview tests, ask yourself what the questions you are asking really tell you about your applicants. Make sure you aren't asking game show questions that do more to boost your ego than to select good candidates. These kind of questions may keep the best people from making it to your "immunity round".

September 15, 2007