METAHEURISTIC
To hire a fractional Chief AI Officer, ask for evidence before you ask for a proposal. The evidence is a system that the person built or led, and the code or the architecture of that system. It also includes an eval and a story about a failure in production. A good candidate shows these things in the first call. A weak candidate shows slides.
The market for this role is new. In a review of about 200 short videos about fractional CAIOs, almost all sellers talked to the camera. Very few showed code, an eval or a dashboard. This does not mean that they cannot do the work. It means that you must ask.
I sell fractional AI CTO work. Apply this process to me too.
What do I need before I start the search?
Do these three things before you talk to any candidate. They take less than one day.
- Write the problem in one sentence. For example: “Our support team spends 40 hours each week on the same questions, and our chatbot pilot never went into production.”
- Write what must exist after 90 days. For example: “One AI system in production with an eval suite, a cost dashboard and an owner on our team.”
- Name one owner on your side. This person works with the fractional CAIO each week and will own the systems later.
Without these three items, each candidate will sell you a different problem.
Step 1: Ask for evidence
Send each candidate the same message before the first call:
Before our call, please send one example of an AI system that you built or led. Include what it did, who used it, and whether it is still in production. A link to code, an architecture diagram or a public write-up is useful.
Then look at what you get back.
Strong evidence
- A repository, an architecture diagram, a public write-up or a demo with real data
- Specific numbers: users, cost, quality
- A clear statement of what the candidate did personally
Weak evidence
- A slide deck, a list of certificates, logos of large companies
- General words: transformation, strategy, innovation
- No way to tell what the candidate did personally
Some candidates cannot share client code. That is normal. But they can describe the system in detail, draw the architecture or show an open-source project. A candidate with real experience always has something to show.
My own public evidence is the agentic_security project and the articles on this site.
Step 2: Ask the five technical questions
Ask these questions in the first call. Listen for specific answers.
- “What failed after the launch, and what did you change?” People who operated AI systems remember failures. Examples: the model invented a policy, the cost increased, the retrieval found old documents.
- “How did you measure quality?” A good answer describes an eval: a set of examples with known good answers, and a score.
- “What did the system cost per task?” A good answer gives a number and explains how the candidate reduced it.
- “Which data went to the model provider?” A good answer describes the data flow and the contract terms.
- “What did you decide not to build?” A good answer names a project that the candidate stopped, with a business reason.
The guide on hands-on skills for a CAIO explains why these questions matter.
Step 3: Ask the five business questions
The role is not only technical. Ask these questions too.
- “How do you find the first problem to solve?” A good answer starts with cost or time in the business, not with a tool.
- “How do you get people to use the system?” A good answer describes training, a measured pilot and an owner in the team.
- “Who do you report to, and what do you need from the CEO?” A good answer asks for authority and a budget.
- “How do you work with our CTO or our agency?” A good answer describes a clear division of decisions.
- “What do we keep when you leave?” A good answer lists documents, code, evals and a handoff plan.
Step 4: Check for conflicts of interest
Ask this question directly: “Do you get any money from a vendor that you can recommend to us?”
Money includes referral fees, commissions, partner discounts, resale margins and equity in a vendor. A conflict is not always a reason to reject a candidate. But the candidate must tell you, and must show the conflict in each recommendation.
Be careful when a candidate recommends a platform in the first call. At that point, the candidate does not know your data or your processes.
Step 5: Give a paid test
Before a long contract, give the candidate a small paid task. The task takes one to five days. Examples:
- Review one existing AI system. The candidate reads the logs, the prompts and the code, and writes a short list of problems and fixes.
- Estimate one use case. The candidate estimates the cost, the quality risk and the time for one AI system that you plan.
- Write a 90-day plan. The candidate writes a plan for your problem, with deliverables and dates.
Pay for the task. Unpaid tests get unpaid quality. Compare the results of two or three candidates on the same task.
Step 6: Speak to references
Ask for two references from clients where the candidate worked on a system in production. Ask the references these questions:
- What did the candidate deliver? Is it still in use?
- Who maintains the system now?
- What does the system cost to run each month?
- What was difficult in the work with the candidate?
- Would you hire the candidate again for the same work?
Make sure that the reference is a real person at a real company. Find the person on the company website or on LinkedIn. Call the person. Some sellers now use automated outreach and fake profiles. The guide to fake AI consultants explains how to check.
What are the red flags?
Stop the process when you see two or more of these signs:
- No system to show. Only slides, frameworks and certificates.
- A platform recommendation before the candidate saw your data.
- A scope that says “AI transformation” or “AI strategy” with no deliverables.
- A long minimum contract and a long notice period.
- The candidate wants to keep the code or the prompts.
- References that you cannot verify.
- The candidate promises a result before the candidate measured your current process.
- The candidate cannot explain an eval in simple words.
What must the contract include?
Keep the contract short. Include these items:
- Deliverables with dates. What exists after 30, 60 and 90 days.
- Authority. The decisions that the CAIO can make without approval.
- Hours and reply time. The monthly block and the reply time for urgent problems.
- Ownership. All code, prompts, evals, data and documents belong to your company.
- Access. The CAIO works in your accounts. You can remove the access in one step.
- Conflicts. The CAIO tells you about each vendor relationship.
- Checkpoint. A review after 30 days. Either side can stop after the review.
- Exit. A notice period and a list of handoff items. The exit plan guide lists them.
How do I run the first 30 days?
The selection does not stop at the signature. Use the first 30 days as a final test.
- Week 1. The CAIO meets the key people and gets access to the systems and the data.
- Week 2. The CAIO delivers an inventory of AI use and a list of the top three problems.
- Week 3. The CAIO delivers a plan for the first system, with an eval method and a cost estimate.
- Week 4. You review the work. If it is useful, continue. If it is not, stop.
The 90-day plan guide describes the next two months.
How long does the process take?
A good selection takes two to four weeks:
| Week | Activity |
|---|---|
| 1 | Write the problem and the 90-day result. Send the evidence request to five candidates. |
| 2 | First calls with three candidates. Technical and business questions. |
| 3 | Paid test with two candidates. Reference calls. |
| 4 | Contract with one candidate. Start. |
This is faster than a full-time executive search, which often takes three to six months.
How do I score the candidates?
Use one scorecard for all candidates. Give each line a score from 1 to 5. Write one sentence of evidence beside each score. Do not score from memory after the calls. Score during the call or right after it.
| Area | What a 5 looks like |
|---|---|
| Built systems | A system in production with users, cost and quality numbers |
| Failure knowledge | A specific failure, the cause and the fix |
| Evals | A clear description of how the candidate measured quality |
| Cost | A cost per task and a method to reduce it |
| Data and risk | A clear data flow and a list of agent actions that need approval |
| Business sense | Starts from your cost or time problem, not from a tool |
| Independence | States all vendor relationships without a prompt |
| Paid test | A useful result that your team can use without the candidate |
| References | Two real people who confirm a system in production |
Add the scores. A candidate below 30 out of 45 is a risk. A candidate with a 1 in “Built systems” or “Independence” is a risk, even with a high total.
How is this different from a full-time executive search?
A full-time search focuses on long-term fit: culture, leadership style and career plans. A fractional search focuses on short-term output. The question is: “What will this person deliver in 90 days?”
This has three effects on the process:
- The paid test is more important. It shows the output directly.
- The reference calls are more specific. Ask about deliverables, not about personality.
- The contract is shorter and easier to stop. A wrong fractional hire costs weeks. A wrong full-time hire costs a year.
Do not copy a full-time interview process. It takes too long and it measures the wrong things.
What if I cannot evaluate the technical answers?
Many CEOs and founders are not technical. That is normal. You have three options:
- Ask a technical friend or advisor to join the calls and review the evidence.
- Ask your CTO or senior engineer to do the technical questions and the paid test review.
- Focus on the evidence and the references. A system in production with a real reference is strong evidence, even if you cannot judge the code.
The guide for non-technical founders has more advice for this case.
Next step
Send the evidence request in Step 1 to your candidates this week. If you want to include me, read how my fractional AI CTO engagement works. If you are not sure which role you need, read the CAIO vs AI CTO comparison first.
