At work I have been involved in hiring processes and execution across multiple levels. This document keeps a few notes coming from my experience that might be helpful for others. In case you are starting hiring processes at your own workplace, I would recommend starting with the section on hiring from The Manager's Handbook which covers a lot of best practices.
I have been primarily responsible for hiring people in the Machine Learning (ML) function at Skit. There are many things we do differently here, including many interesting technical rounds. I will try to document these later as and when I get free.
1. TODO General Structure
During the interview process for ML roles, we evaluate a few traits that are important for us1.
2.1. Learning Attitude
Our field evolves and technologies get outdated rapidly. A good candidate is always learning. You will find people who try to offset this by their experience, specially in senior roles. But that should be kept independent from this.
There are two aspects here, ability and intention. Ability can be noted in their history or via rounds like paper reading2. Intention can be covered via a few specific questions like the ones below:
- How do you learn? Going deep in these tells whether they have spent time in learning new things on their own. Follow up with questions like how did they pick up the last thing they learnt and what would they like to learn if given a free month or year.
- What new learning would you apply on a past project?
- What non-work entities do you study or are interested in studying? While not absolutely essential, being a philomath is a heavy plus and an answer to this question helps in establishing that.
- Which concept was the most difficult to study or grasp? Identifying where people struggle tells a lot about their capabilities and persistence. Follow up with asking about what they still don't understand and are learning. Life long learners always have a list somewhere.
2.2. Love for Craft
As a member in our ML team, the person will have to produce artifacts of some form. For Researchers, this could be papers, for Engineers this could be programs and systems. People who love their craft are fastidious about the quality of their produce and are self-motivated to improve them.
A good first start to evaluate this love is to look at the candidate's portfolio. Depending on the work, it could be on GitHub, Google Scholar, personal webpages, etc. This is sufficient most of the time. Else, and additionally, follow the way of criticism. Ask them to criticize their or someone else's output. Here are a few examples conversations that you could have with the candidate for this:
- This repository looks like it was hacked around quickly under certain deadlines. What would you do if you have more time to work on this?
- How is the future release plan looking like for this project?
- The arguments in final section don't match the once used in abstract, what do you think about that?
- Introduction section has inconsistencies, and it seems like the core idea has changed a lot over the process of writing. What would you do to make this cleaner?
- Here is a blog post written by one of our interns (let the intern also join the call), could you critique the writing and flow of arguments?
- Here is an old repository from our work, let's run through it and see what all would you change if you have to revive and start maintaining this.
2.3. Ownership and Agency
We want people who can own the problems they work on, and do whatever is necessary to get them solved. This maps to different actions based on the levels, teams, and skills. A common pattern is of where someone goes beyond their core expertise to ship something. There are more patterns that cut different borders, but in all the cases, the person takes ownership of solving a problem and takes initiatives beyond what was planned, in a relatively self-reliant way.
Few questions that could help in assessing this part:
- Have you built something that's not yet delivered the value because of something outside your control? This tells how they perceive work and their own agency over it.
- How is your scope connected to top level company or team goals? To solve a problem in ways that's different from a handed-over-plan, you need to understand how it's connected to other pieces around it. Knowing the goal hierarchy helps here.
- How has your scope changed over the past few months or years? People who take ownership well naturally grow faster.
- Can you tell about a situation where you had to do something outside your scope to solve a problem?
2.4. Ambitiousness and Vision
We want candidates who have something in front of them that they can see and have ambitions to get there. People with ambitions understand their goals deeply and don't only have generic statements like "I want to understand how human consciousness works".
To understand depth, you might find the 'admiration scale' handy. In short, you ask them what one (or few) work(s) they admire the most in their field, what would they like to do in near future as the next milestone, and where they currently lie. People who have more veracious ambitions will be able to describe many milestones to reach the final aim, which itself will be highly ambitious.
Additionally, depending on the field and the role level you might want to understand their views on the larger problems in their domain. For example in AI, it's insightful to ask questions like 'What constitutes intelligence according to you?' or 'How do you see path to general intelligence from here?' etc. This will tell you a lot about the depth of their thought process.
2.5. Ability to create and organize Knowledge
Working on a complex product with dynamic team structure, it's important to understand the enormous accidental complexity3 that comes from poor information organization. Almost all true failures that we have ever had can be traced to this. Our systems have performed poorly, we had delays, expectation setting was wrong, etc. all because of implicit information not being explicated and communicated clearly. This is basically the problem of knowledge creation.
We want candidates who can create and organize knowledge. This translates, safely, to "we need good and articulate writers". As Zissner says:
Clear thinking becomes clear writing; one can't exist without the other.
We find people who are fastidious about representing knowledge in various forms. This can be seen in their blogs, project documentations, ticket descriptions, questions, answers, and arguments.
For people who have worked with other people and teams in industries, it's instructive to ask how information flows and knowledge gets created in their environment, and what part they play in it. As usual, people who love doing this also have well-formed views on things like organizing wikis, promoting communication etc. They would also know common health signs for a workplace's knowledge repository and asking about that is also helpful.
3. TODO Technical Rounds
Few other questions I like to ask:
- If you absolutely have to hire a complementary person to help with this work, what would that role look like? I have found this to be a better way of asking about their weaknesses. Going for the description of the role also gives strong information about their level of self-awareness, something that's essential for leadership roles.
The list has evolved from the time we started building the ML team to now.
Here we ask the candidate to pick a research paper from a bunch, read up, and then have a discussion on that. This tells a lot about how deeply they can pick up new concepts.
A lot has been written about accidental and essential complexity in programming, so I would assume familiarity with these phrases.