METAHEURISTICBook a call →
[Model layer]

Hire an LLM engineer.

The model layer: evals that mean something, prompts under version control, routing that cuts spend without cutting quality, and guardrails that hold up against someone trying to break them.

Book a 20-minute call → See the CV
-71%
Inference cost, one system
938
Stars on langcorn (LLMOps)

The model is the cheapest part of your system to change and the most expensive part to change blindly. An LLM engineer is the person who makes that change measurable: prompts under version control, an eval suite that runs in CI, a cost model per task, and a routing layer so no single vendor is load-bearing.

The work

Evals first

A golden set drawn from real traffic, pass criteria written by whoever owns the outcome, and a regression gate in CI. Once that exists, prompt changes stop being arguments and become diffs with numbers attached. Before it exists, every release is a vibe.

Prompts as code

Versioned, reviewed, diffed, and pinned to a model version. A prompt that silently changed behaviour when a provider shipped a point release is a real incident and it is entirely preventable.

Model routing and cost

Most workloads have a long tail of easy cases and a short head of hard ones. Routing them to different models - with a quality gate on the cheap path - is the single biggest cost lever in most systems, ahead of caching and well ahead of fine-tuning. Prompt caching, context pruning and batching follow.

Latency budgets

Time-to-first-token is a product decision, not an infrastructure accident. Streaming, speculative prefetch of likely tool calls, parallel retrieval, and an explicit budget per stage of the pipeline.

Guardrails

Input and output validation, schema-constrained generation, refusal and escalation behaviour, PII handling, and adversarial cases in the eval suite. Red-teaming with agentic_security, the open-source scanner I maintain.

LLMOps

Traces with prompt, model version, tokens, cost and latency on every call. Dashboards that answer “what changed” rather than “how many requests.” Canary releases and a rollback that is a config change.

What you get in a typical engagement

  • An eval harness your team can extend, running in CI, with a golden set drawn from your own traffic.
  • A cost and latency model per task, with the top three levers ranked by effort and impact.
  • A routing layer and provider abstraction, so switching models is a config change.
  • Guardrail coverage mapped against a written threat model.
  • Traces and dashboards, plus the runbook for reading them at 3am.

If the bottleneck is retrieval quality rather than the model, start at hire a RAG engineer. If the system takes multiple steps and calls tools, start at hire an agentic AI engineer.

Questions before you hire.

Should we fine-tune? +

Usually not first. In order: fix retrieval, fix the prompt, fix the task decomposition, try a bigger model, then consider fine-tuning a smaller one to hit a cost or latency target you have actually measured. Fine-tuning bought before an eval set exists is a way to make an unmeasured system unmeasurably different.

How do you build an eval set for something subjective? +

Start with 30 to 50 real cases from your logs, labelled by the person whose judgement you would defer to. Pairwise comparison beats absolute scoring for subjective work, and an LLM judge is only usable once it agrees with your human labels on a held-out slice. The eval set is the asset - the prompts are disposable.

How much can we realistically cut inference cost? +

It depends on how the system is built, and I will not quote a number before seeing traces. The levers are prompt caching, routing easy cases to smaller models, cutting context that adds no measurable quality, batching, and deleting retries that a budget should have prevented. One system on this site went from about $4,120 to about $1,180 a month at the same quality and latency.

Do you work with open-weight models? +

Yes - Llama, Qwen, Mistral and similar, self-hosted on vLLM or your provider of choice. Common for on-premise, data-residency and regulated cases, and increasingly for cost. The routing layer means the choice is not permanent.

Can you review a system you did not build? +

Yes. A one to two week review gives you a written teardown: eval gaps, cost model, latency budget, guardrail coverage, and a prioritised list of what to fix in what order.

Need an agentic AI engineer this quarter?

Contract, fractional, or embedded with your team. Start with a 20-minute call - you leave it with an honest read on whether your use case is buildable, whatever happens next.

Book a 20-minute call → Download CV (PDF)