Why AI pilots never reach production: the model works, the deployment does not.

AI pilots fail to reach production for organizational reasons more often than technical ones. The pilot had no measured goal, so nobody could say it worked. It had no owner in the business, so nobody pushed it forward. It sat beside the workflow instead of inside it, so people did not use it. And it skipped the production layers: tests, monitoring, access control and support.

In 2025, a report from the MIT NANDA initiative looked at generative AI pilots in companies. It said that about 95% of them gave no measurable effect on profit and loss. Business media repeated this number many times. The exact percentage is open to debate. The pattern is not. Most companies have more AI pilots than AI systems in production.

This page lists the common causes. For each cause, it gives a sign that you can check and a fix.

What is the difference between a pilot and production?

A pilot proves that the model can do the task. Production proves that the company can use the model to do the task, every day, at a known cost and a known quality. These are different problems. The second problem is larger.

Cause 1: No outcome was defined

Sign: Ask the pilot team: “What number will change if this works?” They cannot answer, or they say “productivity.”

The pilot started because AI was available, not because a business problem was measured. At the end, the team says that users liked it. Management asks for the return. Nobody knows. The pilot stops.

Fix: Before any build, write one number and measure it. For example: “The average time to answer a support ticket is 11 minutes.” After the launch, measure the same number. If you cannot write the number, do not start the pilot.

Cause 2: No owner in the business

Sign: The pilot belongs to IT, to an innovation team or to an outside vendor. Nobody in the department that uses it is responsible for it.

The department sees the pilot as an IT project. When the pilot needs changes in the process, nobody in the department has the authority or the interest to make them.

Fix: Name one owner in the department that uses the system. This person decides on the process changes, collects the feedback and reports the number. IT supports the owner, not the other way around.

Cause 3: The tool sits beside the workflow

Sign: Users must open a separate window, copy the data in, copy the answer out and paste it into their normal system.

Each extra step reduces use. After the first weeks, people go back to the old way. The pilot shows good results in the demo and low use in real life.

Fix: Put the AI inside the tool that people already use: the CRM, the ticket system, the email client or the document editor. If that is not possible yet, make the extra steps as few as possible. Measure the use each week.

Cause 4: No evals

Sign: Ask: “How do you know that the system is good enough?” The answer is: “We tested it, and it looked good.”

Without an eval set, nobody can measure the quality. Each prompt change can make the system better or worse, and nobody knows which. When a manager finds one bad answer, the pilot loses trust and stops.

Fix: Build an eval set before the launch. Collect 100 to 300 real examples with known good answers. Measure the score. Agree on a minimum score for the launch. Run the eval set before each change. A known error rate is much easier to accept than an unknown one.

Cause 5: The production layers are missing

Sign: The vendor or the team delivered an API key, a prompt and a demo. Nothing else.

A system in production needs more than a working model. A short video from 2026 listed five layers that pilots usually skip:

Fix: Add each layer before the launch. The pilot to production checklist lists the details.

Cause 6: The data is not ready

Sign: The pilot used a clean sample. Production data comes from old systems, scanned documents, spreadsheets and email.

The quality drops when the system meets real data. The team spends months on data cleanup that nobody planned.

Fix: Test with real data in the first week. Take a random sample from the real source, not a selected sample. If the data is poor, make data cleanup the first deliverable, or select a different first problem.

Cause 7: The cost was not estimated

Sign: The pilot used the largest model for all tasks. Nobody calculated the cost for all users.

When the pilot goes to all users, the monthly bill is much larger than expected. Management stops the project because of the cost.

Fix: Calculate the cost per task during the pilot. Multiply by the real volume. Test a smaller model on the eval set. Often a smaller model gives a sufficient score at a much lower cost. The LLM cost guide explains how.

Cause 8: The process was never written down

Sign: The AI does the task as described. The result is still wrong, because the real process has many unwritten rules.

People do the task with knowledge that is not in any document: exceptions, special customers, verbal rules from years ago. The AI does not know them. The undocumented workflow guide explains this cause in detail.

Fix: Write down the process before the build. Interview the people who do the work. Collect the exceptions. Put them into the eval set.

Cause 9: Nobody decided to stop or continue

Sign: The pilot runs for months. Some people use it. Nobody decides to launch it fully or to stop it.

Pilots without a decision date become permanent. They cost money and attention, and they block the next project.

Fix: Give each pilot a decision date when it starts. On that date, compare the number with the goal. Launch it, change it or stop it. A stopped pilot is a good result if you learned something.

Cause 10: The vendor owns the pilot

Sign: The pilot runs on the vendor’s platform, with the vendor’s prompts, in the vendor’s account. Your team cannot see the logs or change the configuration.

The vendor shows good results. But your team cannot check them, cannot fix problems and cannot move the system to another tool. When the vendor asks for a large contract to go to production, you have no information to judge it.

Fix: Ask for access to the logs, the prompts and the configuration from the first day of the pilot. Run your own eval set against the vendor’s system. If the vendor refuses, treat the pilot as a sales demo, not as a test.

A hypothetical example

A logistics company ran a pilot of an AI assistant that answered customer questions about shipments. The pilot ran for five months with 15 friendly customers. The feedback was good. Then management asked: “Should we launch it to all customers?” Nobody could answer.

A review found five of the causes above:

• No number was defined. The team had collected feedback, not data.
• The owner was the innovation team. The customer service manager had never used the system.
• The assistant ran in a separate chat window, not in the customer portal.
• There was no eval set. Nobody knew the error rate.
• The cost per conversation was unknown.

The company did not stop the pilot. It reset it for eight weeks. The customer service manager became the owner. The team built an eval set from 200 real customer questions, and the score showed an error rate of about one in eight. Most errors came from old shipment data in one system. The team fixed the data connection, moved the assistant into the portal and measured the number of calls to the service team. After the reset, management had real numbers to decide with.

How do I check my own pilots?

Make a list of all AI pilots in the company. For each one, answer these questions with yes or no.

A pilot with fewer than six “yes” answers will probably not reach production. Fix the missing items, or stop the pilot and start a smaller one.

Which pilots should I stop?

Stop a pilot when one of these conditions is true:

• Nobody can write the number that it must change.
• The cost per task is higher than the value per task, and no cheaper model is good enough.
• The data that it needs is not available, and nobody will make it available this year.
• The department that uses it does not want it.

Stopping a pilot is not a failure. It frees time and budget for a pilot that can work.

What does a good pilot look like?

A good pilot has these properties:

• Small. One team, one type of task.
• Measured. A number before and after.
• Owned. One person in the business.
• Real. Real data, real users, inside the real tool.
• Tested. An eval set from the first week.
• Time-limited. A decision date six to ten weeks after the start.

A good pilot is already half of a production system. When it ends, the only steps left are the remaining production layers and the launch to all users.

Who should own the move to production?

The move from pilot to production needs one person with technical skills and business access. In a large company, this is often an AI lead or an engineering manager. In a small company, it can be the CTO, a senior engineer or a fractional AI CTO.

This person must be able to say no. No to a launch without evals. No to a launch without an owner. No to a pilot without a number. The advisor vs operator guide explains why this person must have authority, not only give advice.

Next step

Use the ten questions on each of your pilots this week. If you have a pilot that is close but stuck, read the pilot to production checklist. If the pilot was built by an agency and you are not sure what you have, read how to rescue a failing AI project. If you need a person to own the move to production, read how my fractional AI CTO engagement works.