AI pilot to production checklist: the deliverables your vendor skipped.

Use this checklist before an AI pilot goes to all users. It has eight sections: outcome, quality, observability, access, incidents, data, cost and people. Each section has items that a vendor or an internal team often skips. Do not launch until each item has an owner and a status.

A pilot delivery from a vendor often contains three things: API keys, a prompt and a demo. A production system needs a different set of deliverables. One short video from 2026 put it simply: the vendor delivered a demo, and production needs monitoring, access control, an incident playbook and trained users. Those are not the same deliverable.

I use a version of this checklist in my own reviews. Change it for your systems.

How do I use this checklist?

1. Copy the checklist into a document or a ticket.
2. Give each item an owner and a status: done, in progress or not applicable.
3. Write the reason for each “not applicable.”
4. Review the list with the business owner and the technical owner together.
5. Launch only when all items are done or have an accepted reason.

The items are short. Some take an hour. Some take a week. Together, they usually take two to six weeks after a good pilot.

Section 1: Outcome

• One number is defined. The number that the system must change, for example hours per week or error rate.
• The baseline is measured. The value of the number before the system.
• The target is agreed. The value that counts as success, and the date to measure it.
• A business owner is named. One person in the department that uses the system.
• A stop condition is agreed. The result that makes you stop the system.

Without this section, nobody can say if the launch worked.

Section 2: Quality

• An eval set exists. 100 to 300 real examples with known good answers, or more for high risk tasks.
• The eval set includes hard cases. Exceptions, unusual inputs, inputs that must get a refusal.
• The current score is measured. And it is above the agreed minimum.
• The eval runs before each change. Prompt changes, model changes, data changes.
• Failures go into the eval set. Each real failure becomes a new test.
• A human review step exists where necessary. For high risk outputs, a person approves before the output goes out.

The LLM hallucinations guide explains how to handle wrong answers in production.

Section 3: Observability

• Each request has a trace. The input, the retrieved data, the prompt, the tool calls, the output and the cost.
• Traces are searchable. You can find all requests from one user or with one error.
• Alerts exist. For error rate, latency, cost and unusual volume.
• A dashboard exists. Usage, quality signals, cost per task and errors, by day.
• Somebody reads the dashboard. A named person, each week.
• Sensitive data in logs is controlled. Logs do not keep data longer than allowed.

The AI agent monitoring guide has details.

Section 4: Access control

• The system has its own credentials. Not the credentials of a developer or a vendor.
• The system reads only what it needs. No access to all documents or all records “for convenience.”
• Users see only their own data. The system checks user permissions before it retrieves data.
• Tool actions are limited. Each tool that the system can call has a clear limit.
• High risk actions need approval. For example, sending money, deleting records or sending email to customers.
• Prompt injection is tested. Inputs from users, documents or emails cannot change the system’s instructions.

The prompt injection guide explains the tests.

Section 5: Incident response

• A runbook exists. What to do when the system gives wrong answers, costs too much or stops.
• An on-call owner exists. A named person who gets the alerts.
• The system can be stopped quickly. A switch that turns it off and returns to the old process.
• The old process still works. People know how to do the task without the system.
• Users can report a problem. A simple button or channel.
• Customer-facing errors have a plan. Who contacts the customer, and what to say.

Section 6: Data

• The data sources are listed. Where each piece of input data comes from.
• The data is current. A process updates documents and records on a schedule.
• Old or wrong data is removed. Old policies, old prices and draft documents are not in the sources.
• The data flow is approved. Legal or the data protection owner knows which data goes to which vendor.
• The vendor contract covers the data. Retention, training use and location.
• The system was tested on a random sample of real data. Not only on a selected sample.

Section 7: Cost

• The cost per task is measured. Model cost, tool cost and infrastructure.
• The monthly cost at full volume is estimated. Cost per task times the real volume.
• A cheaper model was tested. On the eval set, against the current model.
• A budget limit exists. With an alert before the limit.
• The cost is compared with the value. The cost per task is lower than the value per task.

The LLM cost guide has methods to reduce the cost.

Section 8: People

• Users are trained. A short session and a one-page guide.
• Users know the limits. What the system can do, what it cannot do, and when to check its output.
• The system is inside the daily tool. Not in a separate window, where possible.
• Feedback is collected. Weekly, in the first month.
• Managers know the number. They see the result each month.
• A technical owner is named. A person who can change the system after the launch.

What does a vendor usually deliver, and what is missing?

Ask your vendor for each missing item before you sign the final payment. Some vendors deliver them for an extra fee. Some vendors cannot deliver them. In that case, your team or another person must add them.

What changes for an agent?

An agent is a system where the model selects and calls tools. It can do actions, not only give answers. Agents need extra items in the checklist.

• Each tool has a written purpose and limit. For example: “can read orders, cannot change them.”
• The agent has a step limit. It stops after a set number of tool calls.
• The agent has a cost limit per run. It stops when the run costs more than the limit.
• Loops are detected. The agent cannot call the same tool with the same input many times.
• Each action is logged. Who asked, which tool, which input, which result.
• Each agent has a named owner. The agent accountability guide explains how to assign one.
• Credentials are separate per agent. One leaked key does not open all systems.

The tool call cycles article explains how loops occur and how to stop them.

What changes for a customer-facing system?

A system that talks directly to customers has a higher risk. Its answers can become legal statements. In 2024, a Canadian tribunal held Air Canada responsible for wrong information from its chatbot about a fare policy. Add these items:

• The system answers only from approved sources. Policies, prices and terms come from a controlled list.
• The system refuses outside its scope. It hands over to a human for questions it cannot answer from the sources.
• A human handover exists. Customers can reach a person.
• Legal reviewed the scope. Especially for prices, refunds, contracts and health or financial advice.
• Answers are sampled each week. A person reads a random sample of real conversations.

In which order should I do the sections?

If time is short, use this order:

1. Outcome. Without it, you cannot measure success.
2. Quality. Without evals, you cannot change the system safely.
3. Incident response. Without a stop switch, a bad failure can last for hours.
4. Access control. Without limits, one bad input can do damage.
5. Observability. Without traces, you cannot find the cause of a failure.
6. Cost. Without limits, the bill can surprise you.
7. Data. Without clean data, the quality will drop over time.
8. People. Without training, people will not use it.

All eight sections are necessary. This order only tells you where to start.

How long does this take?

For a typical internal assistant or document system, after a good pilot:

Some of this work can occur at the same time. Two to six weeks is a realistic total.

What if my team cannot do this work?

Many teams can build a pilot but have never put an AI system into production. You have three options:

• Ask the vendor to deliver the missing items, with a price and a date.
• Hire an engineer with AI production experience.
• Bring in a fractional leader who owns the move to production and trains your team.

The guide on why AI pilots fail explains the causes behind each section. If the pilot came from an agency and the quality is uncertain, read how to rescue a failing AI project.

Next step

Copy the checklist and fill it in for your next launch. Count the open items. If more than ten items are open, plan two to six weeks of work before the launch. If you need help to close them, read how my fractional AI CTO engagement works.