I've thought about this lately. In order to do that, you need to know where people typically stumble, and then create a rubric around that. Here are some things I'd look for:
- Ability to clearly define requirements up front (the equivalent mistake in coding interviews is to start by coding, rather than asking questions and understanding the problem + solution 100% before writing a single line of code). This might be the majority of the interview.
- Ability to anticipate where the LLM will make mistakes. See if they use perplexity/context7 for example. Relying solely on the LLM's training data is a mistake.
- A familiarity with how to parallelize work and when that's useful vs not. Do they understand how to use something like worktrees, multiple repos, or docker to split up the work?
- Uses tests (including end-to-end and visual testing)
- Can they actually deliver a working feature/product within a reasonable amount of time?
- Is the final result looking like AI slop, or is it actually performant, maintainable (by both humans and new context windows), well-designed, and follows best practices?
- Are they able to work effectively within a large codebase? (this depends on what stage you're in; if you're a larger company, this is important, but if you're a startup, you probably want the 0->1 type of interview)
- What sort of tools are they using? I'd give more weight if someone was using Claude Code, because that's just the best tool for the job. And if they're just doing the trendy thing like using Claude Agents, I'd subtract points.
- How efficient did they use the AI? Did they just churn through tokens? Did they use the right model given the task complexity?
- Ability to clearly define requirements up front (the equivalent mistake in coding interviews is to start by coding, rather than asking questions and understanding the problem + solution 100% before writing a single line of code). This might be the majority of the interview.
- Ability to anticipate where the LLM will make mistakes. See if they use perplexity/context7 for example. Relying solely on the LLM's training data is a mistake.
- A familiarity with how to parallelize work and when that's useful vs not. Do they understand how to use something like worktrees, multiple repos, or docker to split up the work?
- Uses tests (including end-to-end and visual testing)
- Can they actually deliver a working feature/product within a reasonable amount of time?
- Is the final result looking like AI slop, or is it actually performant, maintainable (by both humans and new context windows), well-designed, and follows best practices?
- Are they able to work effectively within a large codebase? (this depends on what stage you're in; if you're a larger company, this is important, but if you're a startup, you probably want the 0->1 type of interview)
- What sort of tools are they using? I'd give more weight if someone was using Claude Code, because that's just the best tool for the job. And if they're just doing the trendy thing like using Claude Agents, I'd subtract points.
- How efficient did they use the AI? Did they just churn through tokens? Did they use the right model given the task complexity?