Why this checklist exists
The AI engineering market is flooded with “everyone”: from a solo LinkedIn enthusiast to consultancies with 1,000 engineers who have been “doing AI since 2024.” 80% of them sell PoCs and presentations. The five questions below distinguish a contractor who works in production from one who will lead you into the failed 96%.
Question 1: “Name a case with a measurable production result and a limitation”
What we test. Whether the contractor has worked in production, not just PoCs.
A correct answer looks like this: “On project X we improved lead time N times. Limitation: we could not spread it to the 3rd team because of Y.”
Red flags:
- An answer without numbers. “We greatly improved productivity” without a metric means do not believe it.
- No limitations. If “everything went perfectly,” the person is either lying or has not worked in real production.
- Only PoCs and demos. No cases with 6+ months of production support.
Question 2: “What do you not implement, and why?”
What we test. Whether the contractor has a conscious “no”, or implements everything indiscriminately.
Correct answer: “We do not implement autonomous agents without guardrails. We do not implement AI as a replacement for review on critical blocks. We do not recommend universal platforms. Reason: …”
Red flags:
- “We implement everything for your needs.” No principled “no” means no engineering filter.
- They do not know the risks of autonomous agents, the AI Act, or audit logs.
Question 3: “Which metrics do you promise, and how will you measure them?”
What we test. Whether the contractor works with lead time, MTTR, change-failure rate, or sells LoC and “productivity.”
Correct answer: “Lead time from commit to production, change-failure rate, MTTR. Measured from git/CI/incident tracker. Baseline is required in the first 2 weeks, then checkpoints every 30 days.”
Red flags:
- KPIs on LoC or instrument adoption rate. That is not an output metric.
- They promise a specific acceleration percentage on the scoping call, before seeing your data.
- They cannot answer “where does the baseline come from.”
Question 4: “How do you work with governance and the AI Act?”
What we test. Whether the contractor knows what happens on August 2, 2026 and how to build policy-as-code.
Correct answer: “agents.md as policy-as-code, audit log for AI calls, classification by GPAI/limited risk, tracking the August 2, 2026 deadline for GPAI providers.”
Red flags:
- “The AI Act does not apply to us,” if you are in the EU or your users are in the EU.
- No practice around audit logs and documentation of AI outputs.
- They do not distinguish AI Act risk categories.
Question 5: “How do you transfer ownership after yourself?”
What we test. Whether the contractor builds your capability or your dependence on them.
Correct answer: “By the end of the project you have an internal practice owner, documentation, runbooks, and internal skills/recipes. We can leave without production risk.”
Red flags:
- “We stay on permanent support” is a lock-in signal, not enablement.
- No internal owner by the end of the contract.
- Documentation “on request,” not as part of deliverables.
How to use
- Ask the questions on the initial call.
- Do not look for “perfect” answers on all five. 4 out of 5 is a good result. 2 out of 5 is a reason to look for another contractor.
- Add one question for your industry: regulated industries, fintech, healthcare, public sector.
- Review the answers with your team after the call, not during it.