Hiring an AI agent developer? Ask about everything around the model.

You have a candidate on the call. The resume says agents, RAG, LLMOps, production. They talk fluently about the latest model release and they built a chatbot over a weekend that answers questions about PDFs. The question is whether they can keep a system like that standing for a year, with actual users, production data, and a model underneath that changes every few months. That is the hire you are actually making. If you are about to hire a GenAI developer, these are the areas I would walk through in the interview, not as trivia, but to hear whether the person has been there. I already wrote about filtering out the fake resumes before you get to this point. This is what comes after.

Start with the harness

The first thing I ask about is the harness paradigm. The term is still emerging and people slice it into different categories, but the general idea is a separation of concerns between the intelligence and the infrastructure around a large language model. An agent is a model plus a harness. The harness is everything you manage around the model: context engineering, tools, contracts, the permission model, the memory layer, routing, logging, audit, monitoring and observability, plus the data you keep beside it, like a knowledge base or a test dataset for validation. I’ve written more on this in learn agents by building the harness and harness as a service.

Then I ask the trap question. Say that next quarter we get a better, cheaper, faster model. Do we remove the harness? The answer is no. The harness doesn’t go away when the model improves. It gets maintained: you adjust the contracts, you revisit which concerns it handles, you move the liability around. Something still has to be sandboxed and gated from the full range of possible outcomes, because the point of the harness is to control generation so you get results in a reliable, close to deterministic way. A candidate who thinks a smarter model makes the harness obsolete has never been on the hook for what the model did.

Guardrails: where do you check?

Next is the guardrails approach and the security layers. I want to hear whether the candidate understands where inspection has to happen. Is it enough to inspect the prompt going into the model? Or do you need to check each intermediate step of generation that happens inside the system, every tool call, every retrieved chunk, every sub-agent output? Or do you need to evaluate the generated outputs combined, because each step can look fine on its own while the sum of them is something that damages your brand reputation or opens a security hole?

There isn’t one right answer for every system, and I don’t expect one. What I am listening for is that they see the question at all. People who have only built demos put a filter on the input and call it done. People who have run agents in production know the dangerous outcome is usually assembled from steps that each passed their own check.

Memory, and which shape of data

Then memory. There are different approaches, and I ask the candidate to tell me the right use case for each: when an ontology earns its place, when a plain knowledge base is enough, and when you want a graph-based representation of the data that you ground the model on. I don’t need the textbook definitions. I need to hear them reason about the tradeoff, what the model will actually be asked, how the data changes over time, and what it costs to keep each structure correct. If you want a concrete pattern to discuss, blackboard memory is one I use.

RAG, step by step

This is the longest part of the interview, because retrieval is where most production agents go wrong. I ask them to walk me through the typical workflow of a RAG system, and then I start pulling on threads. How would you implement permissions in our RAG, so a user never gets an answer built from a document they aren’t allowed to read? How would you evaluate each step of the pipeline? Do you instrument metrics at every stage of the architecture, or can you skip some? How do you handle ranking and scoring, retrieval, generation, resolution? How do you address latency?

Then the harder ones. How do you deal with data poisoning? How do you handle data that is skewed in one direction of the semantic space, so retrieval keeps pulling from the same corner? How do you find the cutoff where the system should say it can’t give a reliable, factual answer to this query, instead of generating something plausible? A lot of this is in RAG beyond the tutorial, and it is exactly the part the tutorial skips.

That last point matters more than it sounds. I ask how they would stop the system from drifting out of its knowledge domain and answering questions that have nothing to do with what it was built for. A model will happily help with anything. The system shouldn’t, and someone has to decide where the edge is and enforce it.

Evals, and what happens after launch

Last, and most important: verification, testing, evals, and how they run all of this in production. Does the candidate have actual production knowledge? How do they monitor the system once it is live, how do they notice it getting worse, and how do they adjust and evolve it over time as the data, the users and the model all change underneath them? I want specifics here. Which metrics, which traces, what dataset they validate against, what broke last time and how they found out.

A strong candidate answers the production question with a story, not a framework. They tell you about the eval set that stopped matching real traffic, or the regression that only showed up in one customer’s data, with details you didn’t ask for. That is the person you want.

If you are hiring for this role and want a second opinion, I also run interviews and technical validation of candidates for teams. Email me and we can set up the usual engagement for this kind of work, or see how I work on agentic AI projects.