Reviewing an AI agency proposal? Look for the test, not the tech stack.

A good AI proposal tells you how both sides will know that the system works. It names a set of real tasks, a target score and the cost of each task. A weak proposal tells you which frameworks the agency will use and how many phases there are.

Read the proposal for the test first. If you cannot find a test, the price and the timeline are not useful. You cannot know when the project is done. The agency also cannot know.

I review AI proposals for clients as a fractional AI CTO. I do not build for agencies. I do not get referral fees from agencies.

This page has five parts:

1. What a good proposal contains.
2. Nine red flags.
3. Green flags.
4. Questions to ask before you sign.
5. How to compare two or more proposals.

Why is an AI proposal different from a software proposal?

A normal software proposal describes features. Each feature has a clear result. A login page lets a user log in, or it does not. A test can check this. Thus, a list of features is a good definition of “done”.

An AI feature does not have one clear result. The same feature can be correct on 95% of the inputs or on 50% of the inputs. Both versions pass a demo. Only a measurement on many real inputs shows the difference.

For this reason, a list of AI features is not a definition of “done”. The proposal must also say how the agency will measure quality. It must also give a target. Without these items, the agency can deliver all of the features and you can still have a system that nobody can use.

What must a good AI proposal contain?

The task

A good task is narrow. It names one type of user, one job and one result that you can check. Here are examples:

• Narrow: “The assistant answers questions about order status and returns. It uses the order system and the returns policy. It sends all other questions to a human.”
• Too wide: “An AI assistant that improves the customer experience across all channels.”

A narrow task is easier to test, cheaper to build and faster to improve. If the first task operates well, you can add a second task later.

The test

The test is a set of real inputs with known correct answers. The proposal must say:

• Who collects the inputs. You or your team must supply them, not the agency.
• How many cases the test set has. Thirty to fifty cases are a good start.
• How the agency scores each output: correct, wrong or unsafe.
• Which score means “done”. For example: 85% correct and zero unsafe outputs.

The cost

The proposal must give two numbers. The first number is the build price. The second number is the cost to operate the system. The second number includes the model fees for each task at your expected volume.

If the agency does not give the second number, ask for it. A feature can be popular and still lose money each time that a user uses it. Read why LLM costs become too high for the math.

The owner

The proposal must say who owns each asset at the end. The list must include more than “source code”. It must name prompts, eval data, fine-tuned models, documents and accounts. The AI project handover checklist gives the full list.

Nine red flags in an AI agency proposal

1. No acceptance test

The proposal ends with “deployment” or “handover”. It does not end with a score on a test set. Without a test, the project is done when the budget is used.

Ask: “Which test must the system pass before we accept it?”

2. The technology comes before the problem

The first pages describe LangChain, a vector database, multi-agent orchestration and a fine-tuned model. The task comes later, or in one line. The tools must come from the task. The task must not come from the tools.

Ask: “Which part of the task needs each component?”

3. Multi-agent by default

A system of several agents that talk to each other is hard to test. It is also expensive to operate, because each agent adds model calls. Many tasks need one model call with good retrieval.

Ask: “Why is one model call not sufficient?” A good agency gives a specific answer, for example: “The task needs a plan of several steps, and each step uses a different tool.”

4. Fine-tuning in phase one

Fine-tuning needs a test set to show that it helped. If the proposal fine-tunes a model before it measures a simpler system, the agency sells work, not results. Fine-tuning also makes it harder to change to a newer model later.

Ask: “Which score does the system get without fine-tuning? How much must fine-tuning improve it?”

5. No cost per task

The proposal gives a build price but no cost for each user action. You will learn the real cost only after the launch, from the bill.

Ask: “What does one task cost in model fees at 1,000 tasks a day? At 10,000?”

6. Accounts in the name of the agency

The proposal says that the agency “manages” the model provider, the vector database or the hosting. This makes it hard to stop the contract. The agency can also see all customer data that goes through those accounts.

Ask: “Can all accounts be in our name from the first day, with access for your team?”

7. Prompts and eval data are not in the ownership clause

The contract says that you own “the source code”. Prompts are often in a database. Eval data is often in a spreadsheet. Neither is source code in a strict reading.

Ask: “Can the contract name prompts, eval data, fine-tuned models and documents as our property?”

8. Phase two is the answer to each risk

Each risk in the proposal has the same answer: “We will refine this in the next phase.” A project with this pattern often does not end.

Ask: “What happens if phase one does not reach the target score? Do we pay for phase two?”

9. No plan for wrong answers

The model will sometimes be wrong. The proposal must say what happens then. Options include a human review, a confidence threshold, a fallback answer or a refusal. If the proposal does not say, the plan is that users find the errors. Read how to set a handoff threshold for one method.

Ask: “What does the system do when it is not sure? Who sees the error?”

Green flags

These signs show that the agency knows how to deliver AI systems:

• The agency asks for 30 to 50 real examples before it gives a price. It wants to know how difficult the task is.
• The first milestone is small and measured. One task, one test set, one score.
• The proposal says what the agency will not build, and why. A good agency removes scope.
• The agency suggests a simpler option than your request. For example, a search box instead of a chatbot.
• The team that sells is the team that builds. Or you meet the builders before you sign.
• The proposal includes the operation cost. Not only the build price.
• The agency shows a test report from a previous project. With the client’s permission, or with the data removed.

Questions to ask before you sign

Send these eight questions in writing. Compare the answers from each agency.

1. Which tasks will the system do, and which tasks will it refuse?
2. Who writes the test set, and when? Which score means “done”?
3. What does one task cost in model fees at our expected volume?
4. Which model and version do you plan to use? What happens when the provider retires it?
5. Why do you need each component in the architecture? What is the simplest version that can pass the test?
6. In whose accounts will the system operate?
7. What do we receive at the end? Please name each item.
8. What happens if the system does not reach the target score at the end of phase one?

Clear answers are more important than impressive answers. If the agency cannot answer question 2, do not sign until it can.

How do I compare two or more proposals?

Do not compare the price first. Two proposals with different tests are not the same product.

Use this sequence:

1. Remove the proposals with no test. If the agency does not add a test after your question, remove it.
2. Compare the tasks. Make sure that each proposal solves the same task. If one proposal is wider, ask it for a price on the narrow task only.
3. Compare the targets. A proposal with an 85% target and a proposal with a 95% target are different products. Ask each agency which target is realistic, and why.
4. Compare the operation cost. A cheaper build can have a higher cost for each task. Calculate the total cost for the first year.
5. Compare the ownership terms. Remove the proposals that keep accounts or prompts.
6. Then compare the price.

Should I pay for a small paid test first?

Yes, if the project is large. Ask two agencies to build a small version of the task in one or two weeks. Give both agencies the same test set. Compare the scores, the cost for each task and the quality of the communication.

This costs a small part of the full budget. It shows you how each agency works before you commit to the full project.

Should a technical person review the proposal?

Yes, if nobody on your team has put an AI system into production. One review costs little compared with a phase of work that you cannot use.

The reviewer must obey two conditions:

• The agency must not pay the reviewer.
• The reviewer must not want the build work.

The second condition is hard to obey. Ask the reviewer directly. A reviewer who wants the build work has a reason to find problems in the proposals of other agencies.

The independent consultant vs agency guide explains how to divide the roles. The team that builds must not be the team that judges.

What must I put in the contract?

After you select an agency, put these items in the contract:

• The task and the test set. Attach them to the contract.
• The target score. For each milestone.
• Payment linked to results. A part of each payment depends on the target score.
• Ownership. Code, prompts, eval data, fine-tuned models, documents and accounts are yours.
• Accounts. All accounts are in your name from the first day.
• Reports. Each status report includes the eval score and the cost of each task.
• Handover. The agency gives you a handover package each quarter and at the end.
• Notice period. Short, so that both sides can stop without a dispute.

Next step

Mark each red flag in your proposal. Send the eight questions to the agency. Do you want an independent review? Read how my fractional AI CTO engagement operates.

Did the project already start, and does it go badly? Read how to rescue a failing AI project.