Rescue a failing AI project? Get the assets before you judge the code.

Do three things in this sequence. First, move every account, key and data set into the name of your company. Second, measure the system with a small set of real tasks that have known correct answers. Third, make a decision: fix the system, rebuild it or stop it.

Most teams start with the third step. They read the code, make an opinion and argue with the agency. That sequence is wrong. You cannot judge an AI system when you read its code. You can judge it only when you measure what it does with real inputs.

I take over AI projects as a fractional AI CTO. Read this page with that fact in mind. You can do all of the steps on this page without me.

This page has six parts:

1. How to know if the project is failing or only late.
2. How to get control of the assets.
3. How to measure the system.
4. How to decide between fix, rebuild and retire.
5. How to stop the agency contract safely.
6. How to prevent the same failure again.

Is my AI project failing, or only late?

A late project has a clear list of work that is not complete. Each item on the list has an owner and a date. The list becomes shorter each week. That project needs time, not a rescue.

A failing project has a different pattern. Look for these signs:

• The demo works, but nobody can say how often. The agency shows good examples. Nobody shows a number for the bad examples.
• Each fix breaks a different part. A prompt change for one customer makes the answers worse for a different customer. Nobody sees the problem until a user complains.
• The next phase is always the solution. Each phase ends with a proposal for a new phase. The proposal adds a new tool, a new model or a new framework.
• The cost of one task is not known. The monthly API bill increases. Nobody can tell you the cost of one user action.
• Only one engineer at the agency knows the system. When that person is not available, the work stops.
• The scope changed, but the test did not. The project started with one task. Now it has five tasks. Nobody wrote a new definition of “done”.
• Status reports describe activity, not results. The reports say “we improved the retrieval” or “we refined the prompts”. They do not say “the correct answers increased from 60% to 75%”.

One sign is normal in a difficult project. Three or more signs tell you that the project has no technical owner. The code is a symptom. The missing owner is the cause.

Why do AI projects fail in this way?

Normal software has a clear result. A button saves a record, or it does not save the record. A test can check this. An AI system gives a different result for each input. The result can be correct, partly correct or wrong. Thus, a team must measure the system with many inputs to know its quality.

Many agencies come from normal software work. They use the same process for AI. They build features, show a demo and go to the next feature. Nobody measures the quality of the full system. The problems increase in silence until users find them.

This is not always a problem of honesty or skill. It is a problem of process. The agency does not have a test, so the agency cannot see the failure. You do not have a test, so you cannot prove the failure.

What must I get from the agency first?

A normal software handover gives you the repository and the hosting account. An AI system needs more items. These items are often in the accounts of the agency, in a database table or on the laptop of one engineer. Without them, you can have all of the code and still not operate the system.

The AI project handover checklist gives the full list, with 37 items in seven groups. Use it as the request that you send to the agency.

When must I ask for the assets?

Ask for the assets in writing, with a date. Do this before you tell the agency that you are not satisfied. An agency that expects to lose a client often works more slowly. It is correct to ask for the assets at any time. A good agency gives them to you without a problem.

Write the request in a neutral way. For example: “We want to have full ownership of all project assets in our own accounts. Please transfer the items on the attached list by [date].” This request is normal. It does not tell the agency that you plan to stop the contract.

What does the contract say?

Read your contract for these three items:

1. Ownership of the work product. Who owns the code, the documents and the data?
2. Deliverables at the end. What must the agency give you when the contract stops?
3. Notice period. How many days of notice must you give?

Contracts often name “source code” but not prompts, eval data or fine-tuned models. If the contract does not name these items, ask a lawyer how the ownership clause applies to them. Do this before you start a discussion with the agency.

How do I test an AI system that I did not build?

Make a small eval set. An eval set is a list of real tasks with a known correct answer. Use it to measure the system before anyone changes it.

1. Collect 30 to 50 real inputs. Get them from the logs, from support tickets or from the people who will use the system. Do not let the agency select the inputs.
2. Include difficult inputs. Add the inputs that users complained about. Add inputs that the system must refuse. Add inputs at the limits of the scope.
3. Write the correct answer for each input. A person who knows the business must do this. It takes one or two days.
4. Operate the current system on each input. Record each output.
5. Mark each output as correct, wrong or unsafe. Unsafe means that the system did a thing that it must never do. Examples: it gave incorrect data to a customer, it ignored a limit or it did an action that needed a human.
6. Record the cost and the time for each task. Get the cost from the model provider dashboard or from the traces.

Now you have a number. Here is an example: “The system answers 62% of real tasks correctly. Each task costs 9 cents. Three answers were unsafe.” This is a fact. You can make a decision with a fact. You cannot make a decision with “the system feels unreliable”.

Why only 30 to 50 inputs?

Thirty to fifty inputs are sufficient to see large problems. If the system is correct on 90% of the tasks, you will see it. If the system is correct on 50% of the tasks, you will also see it. You do not need a perfect measurement to make the first decision.

Later, you can add more inputs. A production system needs more cases, especially for rare and dangerous inputs. But do not wait for a large eval set before you make the first decision.

How do I find the cause of each error?

For each wrong or unsafe output, write one short reason. Use these groups:

• Missing data. The system did not have the information that it needed.
• Wrong data. Search found the wrong document or an old version.
• Wrong instruction. The prompt told the model to do the wrong thing, or it did not tell the model about a rule.
• Model limit. The model had the correct data and instruction, but it made an error.
• Integration error. A tool call failed, a format was wrong or a timeout occurred.
• Out of scope. The user asked for something that the system must not do. The system did not refuse.

Count the errors in each group. This count tells you where the problem is. It also tells you which decision to make in the next step.

Keep the eval set

The eval set is the most valuable result of the rescue. It stays useful after you change the model, the prompts or the team. Each future change must pass the eval set before it goes to production.

Put the eval set in your repository. Write a script that operates it with one command. If somebody changes a prompt or a model, they run the script and compare the score.

Should I fix, rebuild or retire the project?

Use the eval results and one more question: does a user need this result?

When to fix

Fix the system when the errors are in a small number of groups. For example, most wrong answers come from one type of document. Or most wrong answers come from questions that need data that the system cannot see. A group of errors has one cause. You can fix one cause.

A fix has a clear plan. Select the largest group of errors. Change one thing. Operate the eval set. If the score increases and no new errors occur, keep the change. Then go to the next group.

When to rebuild

Rebuild the system when the errors have no pattern, or when you cannot operate the system without the agency. Also rebuild when the design has too many parts to test one at a time.

A rebuild is usually smaller than the first build. Now you know the actual tasks, the actual data and the correct answers. Often the new system is simpler. For example, one model call with good retrieval can replace a chain of five agents.

Keep these items from the old system:

• The eval set.
• The source data and the data clean-up scripts.
• The prompts that gave correct results.
• The list of errors and their causes.

Replace the rest. If the old code is also hard to maintain, read fix or rewrite a vibe-coded app. That page explains how to rewrite from a written specification.

When to retire

Retire the project when the eval shows that the task does not need AI, or when nobody uses the result. Some AI features can be replaced by a scheduled script, a search box or a form. To stop a project is a correct result. It costs less than another phase.

Ask these questions before you retire a project:

• Did real users use the feature in the last month?
• Does a user lose time or money if the feature stops?
• Can a simpler tool give 80% of the value?

If the answers are “no”, “no” and “yes”, retire the project.

A hypothetical example

A company paid an agency for a support assistant. The eval set shows 70% correct answers. Almost all wrong answers are about orders. The cause is “missing data”: the assistant cannot see the order system.

This is a fix. The team connects the order data and operates the eval set again. The score increases to 88%. The remaining errors are in two small groups. The team fixes them one at a time.

Now change one fact. The wrong answers are in all topics. The agency built a chain of five agents, and nobody can test one agent alone. In this case, a rebuild costs less. The team builds one retrieval step and one model call. Then it measures the new system with the same eval set.

How do I stop the agency contract without losing the system?

Do these steps in this sequence:

1. Get the assets. Use the handover checklist. Make sure that you can operate the system from your own accounts.
2. Operate the eval set on your copy. If your copy gives the same results as the agency copy, you have all of the items. If the results are different, an item is missing.
3. Change every key. After the transfer, make new API keys and passwords. Remove the access of the agency in one step.
4. Give notice. Obey the notice period in the contract. Ask for a short final call to answer open questions. Record the call.

Do not stop the payments before the transfer is complete. A dispute about money makes the handover slower. You need the handover more than the agency needs your last invoice.

Can I keep the agency for some of the work?

Yes. Sometimes the agency builds well, but nobody gives it a clear target. In that case, keep the agency and change the process. Put an owner on your side. Give the agency the eval set and a target score. Pay for results against the eval set, not for phases.

The independent consultant vs agency guide explains how to divide the roles. The team that builds must not also judge its own work.

Who must own the system after the rescue?

The rescue fails again if nobody owns the decisions. One person must own these four items:

• The eval set. This person adds new cases and approves changes to the score.
• The model choice. This person selects the model and plans the change when the provider retires it.
• The cost of each task. This person measures the cost and stops features that lose money.
• The rules for human approval. This person decides which actions the system can do alone and which actions need a human.

That person can be on your team, a new employee or a part-time technical owner. The title is not important. The ownership is important.

How do I prevent the same failure with the next vendor?

Before you sign with a new agency, read how to review an AI agency proposal. That page lists nine red flags. The most important red flag is a proposal with no acceptance test.

Put these items in every new contract:

• All accounts are in your name from the first day.
• The contract names prompts, eval data and fine-tuned models as your property.
• Each milestone has a target score on the eval set.
• The vendor reports the eval score and the cost of each task in each status report.

How long does a rescue take?

The first two parts are fast. The asset transfer takes one to two weeks if the agency helps. The eval set takes one to two weeks. After that, you have a decision based on facts.

The duration of the fix or the rebuild depends on the result of the eval. A fix can take days. A rebuild can take weeks. But you know the scope before you start, because the eval set defines “done”.

Next step

Send the agency the asset list today, with a date. Then make the eval set. You can do both steps before you decide anything.

Do you want a person to do the triage and own the decision? Read how my fractional AI CTO engagement operates and how I set rates.

Do you have a different problem? If the system is in production and it gives wrong answers, read how to stop a chatbot from hallucinating. If the problem is the bill, read why LLM API costs are so high.