5 questions to ask an AI engineering consultant BEFORE the contract

A CTO checklist: how to distinguish a production contractor from a seller of PoCs and presentations before signing.

Why this checklist exists

The AI engineering market is flooded with “everyone”: from a solo LinkedIn enthusiast to consultancies with 1,000 engineers who have been “doing AI since 2024.” 80% of them sell PoCs and presentations. The five questions below distinguish a contractor who works in production from one who will lead you into the failed 96%.

Question 1: “Name a case with a measurable production result and a limitation”

What we test. Whether the contractor has worked in production, not just PoCs.

A correct answer looks like this: “On project X we improved lead time N times. Limitation: we could not spread it to the 3rd team because of Y.”

Red flags:

  • An answer without numbers. “We greatly improved productivity” without a metric means do not believe it.
  • No limitations. If “everything went perfectly,” the person is either lying or has not worked in real production.
  • Only PoCs and demos. No cases with 6+ months of production support.

Question 2: “What do you not implement, and why?”

What we test. Whether the contractor has a conscious “no”, or implements everything indiscriminately.

Correct answer: “We do not implement autonomous agents without guardrails. We do not implement AI as a replacement for review on critical blocks. We do not recommend universal platforms. Reason: …”

Red flags:

  • “We implement everything for your needs.” No principled “no” means no engineering filter.
  • They do not know the risks of autonomous agents, the AI Act, or audit logs.

Question 3: “Which metrics do you promise, and how will you measure them?”

What we test. Whether the contractor works with lead time, MTTR, change-failure rate, or sells LoC and “productivity.”

Correct answer: “Lead time from commit to production, change-failure rate, MTTR. Measured from git/CI/incident tracker. Baseline is required in the first 2 weeks, then checkpoints every 30 days.”

Red flags:

  • KPIs on LoC or instrument adoption rate. That is not an output metric.
  • They promise a specific acceleration percentage on the scoping call, before seeing your data.
  • They cannot answer “where does the baseline come from.”

Question 4: “How do you work with governance and the AI Act?”

What we test. Whether the contractor knows what happens on August 2, 2026 and how to build policy-as-code.

Correct answer: “agents.md as policy-as-code, audit log for AI calls, classification by GPAI/limited risk, tracking the August 2, 2026 deadline for GPAI providers.”

Red flags:

  • “The AI Act does not apply to us,” if you are in the EU or your users are in the EU.
  • No practice around audit logs and documentation of AI outputs.
  • They do not distinguish AI Act risk categories.

Question 5: “How do you transfer ownership after yourself?”

What we test. Whether the contractor builds your capability or your dependence on them.

Correct answer: “By the end of the project you have an internal practice owner, documentation, runbooks, and internal skills/recipes. We can leave without production risk.”

Red flags:

  • “We stay on permanent support” is a lock-in signal, not enablement.
  • No internal owner by the end of the contract.
  • Documentation “on request,” not as part of deliverables.

How to use

  1. Ask the questions on the initial call.
  2. Do not look for “perfect” answers on all five. 4 out of 5 is a good result. 2 out of 5 is a reason to look for another contractor.
  3. Add one question for your industry: regulated industries, fintech, healthcare, public sector.
  4. Review the answers with your team after the call, not during it.

Keep reading

Offers 8 min

2-week AI-readiness discovery: what you get and how it works

A format for CTOs of 30-150 engineer companies: interviews, review/CI audit, baseline, governance gaps, roadmap, and a kill / rescue / rollout decision.