I was part of a team that revised the interview process at my last company (for positions like my own, not all hiring). One thing that came up was that most everyone's "favorite" questions helped reveal a strength if answered well, but didn't reveal much more than the lack of that particular area if they didn't answer it well. It was easy to go through an interview of such questions and end up with no real idea of the candidate was good or bad - they just hadn't happened to hit any of the particular strengths.
Our goal was to make it so that
- After an interview we had confidence in our evaluations
- We had clear criteria to minimize impact of unconscious bias
- Separate interviews would generate similar results for the same basic skill (this is subjective but still a goal)
Some tactics we tried:
- Present a coding question with a business case. This ended up being much the same: candidates familiar with the business area could show strength here as they asked the right questions and had a larger context of understanding, but people who didn't but produced acceptable code showed only that they couldn't take in an unfamiliar context in 5-10 minutes (an entirely reasonable "lack"). This was fine if we expected devs to be familiar with our business case, but often it caused familiarity/lack to cover for/highlight lack of coding comfort.
- Present a list of topics to have someone pick from to discuss. The idea was that if we wanted them to know, say, ONE of a list of seven topics, we'd let them pick their strongest and discuss it, with the assumption that if they can show mastery of a concept they demonstrate the ability to handle (or learn) the others. In practice this never worked. People always went for the first topic (we tried scrambling the order - they went for whatever was first) even if they later showed they understood a different topic better. We didn't run this a huge number of times, but the first several tries didn't work out. I was very disappointed in this result, as I had high hopes for this approach.
- Gave them "broken" code and asked them to fix it (testing to see if they could understand the intention of the code, as well as handle basic debugging). I really like the "understand the intention of the code" part of this, but the "debug and fix" turned out to be highly subject to whether they had run into this class of problem before, which returns to the original problem of "favorite questions".
Ultimately I left before the effort was mostly successful. The changes WERE a dramatic improvement over the pre-effort interviews - if nothing else, all questions were vetted by a small group in advance to weed out the ones that were very misrepresentative or out of line for difficulty. However the key difficulties of getting a grasp on a nuanced set of skills in a short time period were never truly resolved.
Some takeaways I walked away with (all subjective):
- Start with a quick, easy coding question (when you get to coding questions). 5 mins on something trivial can give them a confidence boost that will save you more than 5 minutes later, both in getting them out of a mental block of recrimination and in them showing you their best self in a situation that otherwise encourages less than that. You can even tell the candidate this in advance - So long as it is quick they shouldn't judge, and it helps establish to them what to expect from YOU with a more involved question (such as how nit-picky you are with syntax, if you emote your opinion, etc)
- Vet your questions in advance. You want four things: (1) Does doing well show a skill or skills we desire? (2) Does doing poorly on this question truly speak generically to their capability? (3) Do I have ways to provide hints or direction to someone blocked that doesn't render the question useless? (4) Do I have places to go, both more and less advanced, if this question is going easily/poorly?
- I personally stress that I don't expect perfect answers. If (when) they screw something up, I say things like "Great, this gives us a chance to look at your debugging, which is something we do all the time" - this should be reassuring, and has the benefit of being completely true.
Interviewing is still something we (humans) are learning how to do well, and even if I know how _I'd_ like to be interviewed, that isn't true for everyone.
Our goal was to make it so that
- After an interview we had confidence in our evaluations
- We had clear criteria to minimize impact of unconscious bias
- Separate interviews would generate similar results for the same basic skill (this is subjective but still a goal)
Some tactics we tried:
- Present a coding question with a business case. This ended up being much the same: candidates familiar with the business area could show strength here as they asked the right questions and had a larger context of understanding, but people who didn't but produced acceptable code showed only that they couldn't take in an unfamiliar context in 5-10 minutes (an entirely reasonable "lack"). This was fine if we expected devs to be familiar with our business case, but often it caused familiarity/lack to cover for/highlight lack of coding comfort.
- Present a list of topics to have someone pick from to discuss. The idea was that if we wanted them to know, say, ONE of a list of seven topics, we'd let them pick their strongest and discuss it, with the assumption that if they can show mastery of a concept they demonstrate the ability to handle (or learn) the others. In practice this never worked. People always went for the first topic (we tried scrambling the order - they went for whatever was first) even if they later showed they understood a different topic better. We didn't run this a huge number of times, but the first several tries didn't work out. I was very disappointed in this result, as I had high hopes for this approach.
- Gave them "broken" code and asked them to fix it (testing to see if they could understand the intention of the code, as well as handle basic debugging). I really like the "understand the intention of the code" part of this, but the "debug and fix" turned out to be highly subject to whether they had run into this class of problem before, which returns to the original problem of "favorite questions".
Ultimately I left before the effort was mostly successful. The changes WERE a dramatic improvement over the pre-effort interviews - if nothing else, all questions were vetted by a small group in advance to weed out the ones that were very misrepresentative or out of line for difficulty. However the key difficulties of getting a grasp on a nuanced set of skills in a short time period were never truly resolved.
Some takeaways I walked away with (all subjective):
- Start with a quick, easy coding question (when you get to coding questions). 5 mins on something trivial can give them a confidence boost that will save you more than 5 minutes later, both in getting them out of a mental block of recrimination and in them showing you their best self in a situation that otherwise encourages less than that. You can even tell the candidate this in advance - So long as it is quick they shouldn't judge, and it helps establish to them what to expect from YOU with a more involved question (such as how nit-picky you are with syntax, if you emote your opinion, etc)
- Vet your questions in advance. You want four things: (1) Does doing well show a skill or skills we desire? (2) Does doing poorly on this question truly speak generically to their capability? (3) Do I have ways to provide hints or direction to someone blocked that doesn't render the question useless? (4) Do I have places to go, both more and less advanced, if this question is going easily/poorly?
- I personally stress that I don't expect perfect answers. If (when) they screw something up, I say things like "Great, this gives us a chance to look at your debugging, which is something we do all the time" - this should be reassuring, and has the benefit of being completely true.
Interviewing is still something we (humans) are learning how to do well, and even if I know how _I'd_ like to be interviewed, that isn't true for everyone.