Does a Chief AI Officer need to code? They need to have built something.

A Chief AI Officer does not need to write production code each day. But the person must have built and operated at least one AI system. Without this experience, the person cannot judge the decisions that cost the most money. These decisions are the model, the data to expose, the quality tests and the time to stop a project.

One fractional CTO said in a short video that the CAIO title makes him want to “hurl.” His reason: a CTO must know how to produce software, because the CTO must change raw materials into value. A CFO must know finance. A COO must know operations. The same rule applies to an AI executive.

I agree with the rule. This page explains which hands-on skills matter, which do not, and how to test a candidate.

Why is hands-on experience important for AI decisions?

Most AI decisions look simple in a presentation. They become difficult with real data. A leader without hands-on experience cannot see the difficulty in advance.

Here are five decisions that a CAIO makes. Each decision has a hidden problem.

A leader who built a system knows these problems. A leader who only read about them will learn them from incidents, at your cost.

Which technical skills does a CAIO need?

The CAIO does not need to be the best engineer in the company. The CAIO needs enough experience to ask the correct questions and to see a weak answer. Here is the minimum.

Built a working AI system. The person built a system that uses a model with real data. The system can be small. It must not be only a tutorial.

Operated the system in production. The person ran the system with real users for some months. The person saw the failures, the cost and the support requests.

Wrote or designed an eval. The person knows how to measure the quality of model output on a set of examples. Without evals, nobody can tell if a change made the system better.

Read a trace. The person can look at the log of one request: the prompt, the retrieved data, the tool calls and the answer. The person can find where it went wrong.

Estimated cost per task. The person can calculate the cost of one user action from the number of tokens and the model price.

Understood the data flow. The person can draw where data goes: from the user, to the model provider, to logs, to indexes. The person knows which data must not go where.

These skills do not need a computer science degree. They need hands-on time.

Which technical skills does a CAIO not need?

Some skills are not necessary for the role. Do not reject a good candidate because of them.

• Model training from scratch. Most companies use models from providers. Few companies train their own large models.
• Research publications. Research skills help in a research lab. They help less in a company that uses AI for operations.
• Deep knowledge of one framework. Frameworks change each year. The concepts stay the same.
• Daily coding. A CAIO in a large company spends most hours with people. That is correct.

What about the business skills?

Technical skills are necessary but not sufficient. The CAIO also needs these skills:

• Find the business problem. The person asks: “Where does the company lose the most hours or money?” before “Where can we use AI?”
• Stop projects. A large part of the job is to say no. No to a tool, no to a second pilot, no to a feature without an owner.
• Manage change. People must change how they work. This is often more difficult than the technology.
• Speak to the board. The person explains risk and return in business terms.

A strong candidate has both sets of skills. A person with only business skills will depend on vendors. A person with only technical skills will build correct systems that nobody uses.

How do I test a CAIO candidate?

Use these five tests. Each test takes less than one hour. Do them before you sign a contract, for a full-time hire or for a fractional hire.

Test 1: Ask for a system they built

Ask: “Show me an AI system that you built or led. What did it do? Who used it? Is it still in production?”

A good answer has details: the model, the data, the number of users, the cost and what broke. A weak answer talks about “AI transformation” and has no system.

Test 2: Ask for a failure

Ask: “What failed after the launch? What did you change?”

People who operated AI systems remember failures in detail. For example: the model invented a policy, the cost increased after a new feature, the retrieval found old documents. People who did not operate a system give general answers.

Test 3: Give them a trace

Give the candidate a log of one bad answer from your system or from a public example. Ask: “Where did this go wrong?”

A good candidate looks at the retrieved data, the prompt and the tool calls. A weak candidate says “the model hallucinated” and stops there.

Test 4: Ask for a cost estimate

Describe a simple feature, for example “summarize each support ticket.” Give the number of tickets per day and the average length. Ask: “What will this cost each month, and how can we make it cheaper?”

A good candidate calculates with tokens and model prices. The candidate also suggests a smaller model, caching or batch processing. A weak candidate cannot give a number.

Test 5: Ask what they would not build

Ask: “Here are three AI ideas from our team. Which one would you stop, and why?”

A good candidate stops at least one idea and gives a business reason. A weak candidate wants to build all three.

What are the warning signs?

Be careful when you see these signs:

• The candidate shows certificates and courses, but no system.
• The candidate uses many framework names but cannot explain an eval.
• All examples are from vendor presentations or case studies of other companies.
• The candidate recommends a platform before the candidate understands your data.
• The candidate gets a referral fee from a vendor and does not tell you.
• The candidate says that AI projects fail because of “culture” and never because of design.

One sign is not proof. Three signs together are a strong warning.

Is the problem the title?

Partly. The title “Chief AI Officer” is new. There is no standard path to it. Some people take the title after a short course. Some people take it after years of production work. The title does not tell you which person you have.

For this reason, do not hire by title. Hire by evidence. The evidence is a system, a failure story, a trace analysis and a cost estimate. If the candidate has the evidence, the title is not important.

What if my CAIO is already hired and has no hands-on experience?

This is common. You do not need to replace the person. You need to add the missing experience.

1. Pair the CAIO with a senior engineer or a fractional AI CTO who has production experience.
2. Give the technical person the authority over model choice, evals and architecture.
3. Give the CAIO the authority over policy, priorities and adoption.
4. Make one rule: no AI system goes into production without an eval suite and a cost estimate.
5. Ask the CAIO to build one small system with the technical person. This gives the CAIO real experience.

After six months, the CAIO will make better decisions. The decisions will come from real systems, not from presentations.

Does a fractional CAIO need more hands-on skill than a full-time CAIO?

Yes, usually. A full-time CAIO in a large company has a team. The team includes engineers and data scientists who do the hands-on work. The CAIO can depend on them for the technical details.

A fractional CAIO in a small company usually has no AI team. Sometimes the company has no engineers at all. The fractional CAIO must do more of the technical work, or must select and review the people who do it. A fractional CAIO without hands-on skills will depend on vendors for each technical decision. The vendor then has more control over your AI program than you do.

For this reason, ask a fractional candidate for more technical evidence than a full-time candidate. Ask for a repository, an eval or a trace analysis. The vetting guide explains how.

What does hands-on work look like in the first month?

A CAIO with hands-on experience does these things in the first month. Use the list to check the work of a new hire.

1. Reads the logs of the existing AI systems. The CAIO looks at real requests and real answers. The CAIO does not depend only on reports from the team.
2. Builds a small eval set. The CAIO collects 50 to 100 real examples with known good answers for one system. The team uses it to measure quality.
3. Calculates the real cost. The CAIO finds the cost of each AI system per task and per month. The CAIO compares it with the value.
4. Draws the data flow. The CAIO makes a diagram of where company data goes for each AI tool.
5. Tests one vendor claim. The CAIO takes one claim from a vendor and tests it with company data.

A CAIO without hands-on experience does different things in the first month. The CAIO meets stakeholders, writes a strategy document and asks vendors for presentations. These activities are not wrong. But they do not produce evidence. After a month, the company knows more about the CAIO’s opinions and nothing new about its systems.

Next step

Use the five tests in your next interview. If you need the hands-on technical owner, read how a fractional AI CTO engagement works. For a full selection process, read how to vet a fractional CAIO. For interview questions at the engineer level, read interview questions for an AI agent developer.