AI project handover checklist. The items a normal handover forgets.

The handover is complete when your team can operate the system from your own accounts. The system must also give the same results that it gives for the agency. Each item below helps you get to that result.

Send this list to the agency in writing, with a date. Mark each item when you receive it. Then do the three tests at the end of this page.

Use this list for each type of handover:

• An agency at the end of a contract.
• A freelancer who stops the work.
• An engineer who leaves the company.
• An internal team that gives the system to a different team.

If the project is also failing, read how to rescue a failing AI project first. That page explains what to do after the handover.

Why does an AI handover need a special list?

A normal application has two main parts: the code and the data. If you have both, you can operate the application.

An AI system has more parts. Many of these parts are not in the code:

• The prompts are often in a database or an admin panel.
• The model is in the account of a provider. The provider can change or retire it.
• The search index depends on one specific embedding model.
• The tests are often in a spreadsheet, not in the repository.
• The quality of the system depends on settings that nobody wrote down.

If one of these parts is missing, the system can still start. But it gives different answers. Nobody sees the change until users complain. For this reason, the last test on this page compares the results of the two systems. A system that starts is not sufficient.

How to use this list

1. Copy the list into a shared document. Add a column for the owner and a column for the date.
2. Send it to the agency. Give a date for each group. Two weeks is usually sufficient.
3. Mark each item when you receive it. Write where you put it.
4. Write “does not exist” when an item does not exist. This is also information. For example, no eval data means that nobody tested the quality.
5. Do the three tests. Do them before you remove the access of the agency.

Accounts

Each account must be in the name of your company. Your company card must pay for it. A shared login is not a transfer. Ask the agency to transfer ownership. Then remove their access.

• Model provider accounts (OpenAI, Anthropic, Google, AWS Bedrock, Azure and others)
• Vector database or search service
• Tracing and observability (LangSmith, Langfuse, Helicone, Datadog and others)
• Hosting, cloud project and container registry
• Domain, DNS and email service
• Code repository, CI pipelines and their secrets
• Third-party tools that the agent calls (CRM, ticket system, payment, calendar)
• Usage limits and cost alerts on each paid account

Why this is important. If the model provider account is in the name of the agency, the agency controls your product. The agency can stop the product when it stops the payment. It can also see all customer data that goes through the model.

What to look for. Some agencies use one large account for many clients. In this case, the agency cannot transfer the account. Ask the agency to make new accounts in your name. Then ask it to move the configuration and test that the system operates.

Prompts and configuration

Teams often keep prompts outside the repository. Prompts can be in a database table, an admin panel, a prompt management tool or an environment variable.

• Every prompt and system message, with its full version history
• The location of each prompt at runtime, and how a change goes to production
• Few-shot examples and all templates that build the final prompt
• Tool and function definitions that the model can call
• Feature flags and settings that change the behavior of the model for each customer

Why this is important. A prompt is a part of the program. A small change in a prompt can change many answers. If you do not have the current prompt, you cannot make the system give the same answers.

What to look for. Ask the agency to show you how a prompt goes from a change to production. If the answer is “an engineer edits it in the admin panel”, nobody tested the change. Write this down. It is a risk that you must fix after the handover.

Models

“We use GPT” or “we use Claude” is not an answer. A different model version can give different answers to the same prompt.

• The exact model name and version for each call
• Temperature and the other generation settings for each call
• Fallback models, and the rule that changes to them
• Fine-tuned models: the model ID, the training data, the training script and the owner of the weights
• Retirement dates that the provider announced for each model in use

Why this is important. Providers retire models on their own schedule. When a model is retired, you must change to a new model. Without an eval set and the exact current version, you cannot know if the new model is better or worse.

What to look for. Some code uses a model name with no version, for example “the latest model”. In this case, the provider can change the model without a warning. Write down the version that the system uses today. Then fix the version in the code.

Retrieval and data

A search index operates only with the embedding model that built it. Without the script and the source documents, you cannot rebuild the index. You also cannot add new documents correctly.

• The embedding model name and version
• The script that divides documents into chunks, with its settings
• The script that builds or updates the index, and its schedule
• The list of source documents and the origin of each document
• Each reranker or filter that the system applies after the search
• A copy of the current index, or proof that you can rebuild it

Why this is important. Retrieval errors are a frequent cause of wrong answers. If the search finds the wrong document, the model gives a wrong answer. To fix this, you must be able to change the chunks, the index and the filters.

What to look for. Ask when the index was last updated. If the answer is “at the start of the project”, the system uses old data. Users can get answers from old documents.

Tests and evals

An eval is a test that you can do again. It compares the output of the system with examples that have a known correct answer. If the agency tested the system, this data exists.

• Eval data sets: inputs, expected outputs and the scoring rules
• The script that operates the evals, and the last results
• Normal unit and integration tests, and how to operate them
• Known failure cases that the agency found and did not fix

If no eval data exists, write this down. It tells you how the agency checked the quality.

Why this is important. The eval set is the only way to know if a change made the system better or worse. Without it, each change is a guess.

What to look for. Some agencies show a demo set of 5 to 10 good examples. A demo set is not an eval set. An eval set includes difficult inputs, inputs that the system must refuse and inputs that users complained about.

Logs and user data

Logs show what the system does with real users. They also contain customer data. Plan where the logs go and who can read them.

• Request and response logs, or traces, for a recent period
• User feedback data (thumbs up or down, corrections, escalations)
• The location of personal data, and the retention period
• Data processing agreements with each model provider and tool provider

Why this is important. Logs are the best source for a new eval set. They show real inputs, not inputs that the team expected. They also show the errors that users saw.

What to look for. Make sure that the transfer of logs obeys your privacy rules. Some logs contain names, emails or payment data. Ask your data protection officer or lawyer before you move them.

Documents and knowledge

• An architecture diagram that agrees with the current system
• A list of the decisions that the agency made, and the reason for each decision
• Runbooks: how to deploy, roll back, change keys and respond to an outage
• Monthly cost for each service for the last three months
• A recorded walkthrough call with the engineer who knows the system best

Why this is important. The code tells you what the system does. It does not tell you why. Without the reasons, your team can remove a part that prevents a problem.

What to look for. Ask for the walkthrough call early. Ask the engineer to show one full request from the user input to the answer. Ask what breaks most frequently. Record the call and keep it with the documents.

How do I confirm that the handover is complete?

Do these three tests before you remove the access of the agency.

Test 1: Rebuild from zero

A person on your side deploys the system into your own accounts. This person uses only the repository and the documents. Each question that this person must ask the agency is a missing item. Add it to the list.

Test 2: Compare the results

Operate the same eval set on the agency system and on your copy. The results must agree. If they do not agree, an item is different. Usually it is a prompt, a model version or the index.

If no eval set exists, make a small one first. Collect 30 inputs from the logs. Operate both systems on them. Compare the answers one by one.

Test 3: Change every key

Make new API keys and passwords for each account. Then make sure that the system still operates. If you cannot change a key, you do not control it. Find out where the key is and who owns it.

After these three tests, remove the access of the agency in one step. Then give notice as the contract tells you.

What if the agency does not give me an item?

Read the contract first. Look for these three items:

• The definition of the work product.
• The list of deliverables at the end.
• The notice period.

If the contract does not name prompts, eval data or fine-tuned models, ask a lawyer how the ownership clause applies to them.

Then decide if you need the item. Some items are useful but not necessary:

• Old prompt versions. Useful for history. You can operate with the current version.
• Logs. Useful for a new eval set. You can collect new logs in a few weeks.
• Eval data. You can write new eval data in a few days.

The account transfer is the one item that you must not skip. Without it, the agency can stop your product.

How do I prevent this problem next time?

Put these rules in each new contract with a vendor:

• All accounts are in your name from the first day. The vendor gets access, not ownership.
• Prompts, eval data, fine-tuned models and documents are your property. The contract names each item.
• All prompts are in your repository, with a version history.
• The vendor gives you a handover package each quarter, not only at the end.

Before you sign, read how to review an AI agency proposal. It lists nine red flags. Two of them are about accounts and ownership.

Next step

Copy this list into a document and send it today. Do you want help with the handover, or with the decision about the system after it? Read how my fractional AI CTO engagement operates.