Last time, I discussed the importance of aligning skill sets for the role you’re looking for to your company values. Once they are established, the next step is to design the interview questions and train interviewers to conduct them. In this post, I will share with you three types of questions I have found to provide strong signal for behavioral interviews.
1. Ask open-ended questions
Open-ended questions are deceptively difficult to ask right. Candidates are incentivized to present their best selves based on what they assume the company is looking for. One may substitute “assertive” for “aggressive” or “attention to detail” for “micromanagement”. It is therefore important to learn both how to ask the question and how to evaluate the responses.
In First, Break All the Rules: What the World’s Greatest Managers Do Differently, the author explains:
Ask open-ended questions that offer many potential directions, and do not telegraph the “right” direction — questions such as “How closely do you think people should be supervised?” or “What do you enjoy most about selling?”
Not asking leading questions is critical. Here is an actual interview exchange with an executive candidate with 20+ years of experience:
Me: “What are your thoughts on diversity?”
Candidate 1: “I don’t think diversity is necessary or critical to a technical team’s success.”
Had I asked “how do you think we should improve diversity?” we would never have found out that Candidate 1 did not care about diversity, which would indicate a mismatch with our core values. Based on research from Gallup, the direction the candidate spontaneously takes to an open-ended question will be most predictive of her future behaviors.
The above exchange may be an extreme example because there are few who would argue against diversity (at least openly) in Silicon Valley today. Here’s another example with an executive candidate towards the end of an interview.
Me: “What do you look for in a new company?”
Candidate 2: “I look at the tech stack and the business model.”
Me: “Anything else that comes to mind?”
Candidate 2: “Growth opportunity and compensation.”
At the beginning of the interview, we explicitly explained that this role is to help grow our team and establish culture, yet Candidate 2 did not mention anything related to culture in his response. Unsurprisingly, when we asked him later on what the key elements of a successful culture are, he stumbled without coming up with anything concrete.
In summary, ask open-ended questions to gauge the candidate’s values. Make sure they are not leading questions, such as “how important do you think transparency is?” Finally, observe which values surface first: those are typically how that person will make decisions down the line.
2. Look for specific examples (or ask them explicitly)
Strong candidates typically respond to open-ended questions with specific examples. For those that do not, there is a more direct way to engage them: “Tell me about a time when you …”
Some candidates, especially experienced ones, rattle on about the theory of a particular value. In my experience, junior interviewers are often in awe by the eloquence of the candidate’s response. However, unless the role you’re hiring for primarily concerns itself with how to respond to interview questions eloquently, this is not a good indicator of future success. Here’s an example:
Me: “Tell me about a time where you had a disagreement with a report.”
Candidate 3: “Oh I think having healthy disagreements within the team is extremely important. I always ensure that there’s an open dialogue. I have an open-office policy where anyone can come to me and flag any concern. Another reason this is important is because …”
Contrast that with Candidate 4’s response:
“A month ago, I proposed that we break down a particular feature into three parts. However, one of my teammates suggested that we should break it down another way and reorder the work because of X, Y, and Z …”
For Candidate 3, a lot of interviewers would probe further with “Can you give me a specific example?” a few times until they get a satisfactory answer. They then evaluate based on the smoothness of the final response. This is a dangerous approach: if the candidate requires multiple prods to come up with a specific example, it means this particular behavior or value is not a recurring part of her life. It is more likely that Candidate 4 has a management style that encourages his team to challenge him, whereas Candidate 3 is either always right, or her team never challenges her (which is why it is so hard for her to find an example).
To recap, ideally the candidate provides specific examples of the attributes you are looking for when you ask open-ended questions. If not, “Tell me about a time when …” is a perfectly good way to look for clues from past behavior. However, make sure you evaluate whether a specific example comes to mind immediately — if the specifics come after multiple prompts, no matter the quality of the detail, it is likely not something the candidate consistently does or thinks about.
3. Simulate real-life scenarios
This is my favorite type of interview. While a seasoned veteran can preemptively prepare for open-ended questions or specific examples, it is very hard to prepare for specific mock scenarios. In a mock scenario, you present an unstructured but realistic problem the person will run into and ask the person to solve it during the interview. The goal is to evaluate how the candidate breaks down the problem and which areas she focuses on. This gives us insight into how she would approach the problem if we were to hire her.
Here’s an example of how it would go if you were hiring for a head of engineering in a fast-growing startup and have determined org-building as a key skill to evaluate:
Me: “Now we’re going to do a mock scenario. Imagine that I am the CTO and you are hired as the head of one of our engineering groups. I will give you some context and a prompt and we’ll work together towards a solution. Feel free to ask me any clarifying questions while we do this.”
Candidate 5: “Cool, let’s do it.”
Me: “OK, so our company has 55 people. About 30 of them are engineers, and your group has 15. In light of our growing business, we want to grow the engineering team. For your group, we anticipate the need to grow to 22 people. This is day 1: what do you do?”
Sometimes the candidate goes into hypotheticals, such as “If you want more senior engineers, do X.” I typically try to bring the conversation back to the actual problem, “Let’s remember this is a mock, so if you want more information feel free to ask me.”
Different candidates take this in different directions. Some ask for the org chart and divide the needs based on seniority and function while others look at the metrics and the recruiting funnel to find ways to improve the process. Either is fine; the important part is to break down the problem in a logical manner and justify it.
The key to designing a mock scenario is to draw from a real problem. For example, we once identified interdisciplinary collaboration skills as a key attribute for a product manager role. I had a software engineer and a client support associate tag-team for the mock. During the interview, the client support associate presented a client feature request and the deadlines, while the software engineer explained that it could not be done or prioritized. The product manager would then have to step in and try to resolve this conflict in a way that is acceptable to all parties.
To summarize, look for a set of real-life problems you want the person in the role to solve and map it to the values and skills you’re looking for. Then, design a scenario and play it out with the candidate. Avoid hypotheticals and treat the situation as if the candidate is now an employee working on this real problem. Finally, evaluate how the candidate reacts initially as well as the approach to the problem.
I hope my experiences can help you better design interview questions to evaluate soft skills, enabling you to identify stronger candidates. If you’d like to discuss more about recruiting, feel free to leave a comment or reach out via twitter.
If you enjoyed this article, check out the other articles in this series:
- Part 1: How to Assess Soft Skills Effectively & Consistently
- Part 3: 3 Steps to Assess Your Prospective Employer’s Culture
For more musings on tech culture, organization building, and management, follow me on Twitter @kenk616.
This story is published in The Startup, Medium’s largest entrepreneurship publication followed by +431,678 people.
Subscribe to receive our top stories here.
