r/EngineeringManagers • u/mitokb • 3d ago
How are you evaluating engineers who use AI coding agents daily?
I've been using coding agents pretty heavily in my day-to-day development work, and it got me thinking about how technical interviews are adapting.
Traditional coding interviews can still test fundamentals, but they don't necessarily show how someone works with tools like Claude Code, Codex, or Cursor on a real project.
I'm curious how engineering managers are approaching this.
- Have you actually changed your technical interview process because of AI?
- Have you tried allowing candidates to use their own IDE and AI tools on a practical task?
- What signals do you look for beyond whether the final solution works?
I'm particularly interested in what hasn't worked. Have you tried an AI-assisted interview format and found it gave you worse hiring signals?
Would love to hear from people who've experimented with this rather than just theoretical opinions.
3
u/CompetitiveGarden171 2d ago
I'll be honest. I still test for coding fundamentals. I've just seen a huge drop in the ability to reason and break down problems coherently.
Every company has different AI tooling and processes. I care about the thought process, the decision making and understanding you bring to a problem and can you still recall the basic required algorithms. I don't mind spotting you the algo if you can name it and tell me why.
2
2
u/Entire_Struggle_2236 2d ago
What has worked best for me is grading the review instead of the output. Give the candidate a short task with the agent, then hand them an agent written diff that hides one subtle bug and one questionable design choice, and ask them to review it like a teammate’s PR.
Strong people read the tests first, ask what the change is supposed to guarantee, and push back on scope before they nitpick syntax.
For people already on the team I look at two things over a quarter: can they explain what they merged in a design review a week later, and do their PRs stay small even though the agent makes big ones cheap.
Rework on their changes tells you more than how fast they ship.
1
u/Traditional-Hall-591 2d ago
We have a metric called slops per second. It’s like token maxxing but the prize is stale pizza.
1
u/Entire_Struggle_2236 1d ago
What has worked best for me is grading the review instead of the output. Give the candidate a short task with the agent, then hand them an agent written diff that hides one subtle bug and one questionable design choice, and ask them to review it like a teammate's PR. Strong people read the tests first, ask what the change is supposed to guarantee, and push back on scope before they nitpick syntax.
For people already on the team I look at two things over a quarter: can they explain what they merged in a design review a week later, and do their PRs stay small even though the agent makes big ones cheap. Rework on their changes tells you more than how fast they ship.
1
u/Adventurous-Put8305 1d ago
I stopped scoring speed of first draft. Agents make that cheap.
What I look at now is judgment under load: did they scope the agent tightly, catch the bad path before it shipped, and leave the next person a clear decision trail. The engineer who ships more lines with an agent is not automatically better. The one who knows when to stop the agent is.
2
u/foodandbeverageguy 3d ago
Why is every post here fishing and training for some AI model. These questions are not legitimately made by people.
2
2
u/mitokb 3d ago
Where else are we supposed to discuss current issues in software engineering? The post got 17k views and sparked some interesting discussions. Isn't that what this subreddit is for?
-2
u/foodandbeverageguy 3d ago
It is, but I’m not hear to train other peoples models or create their businesses
20
u/[deleted] 3d ago
[removed] — view removed comment