How to Evaluate an Offshore AI Development Agency

6 Red Flags That Disqualify an AI Agency Immediately

Red flag 1: Case studies with no metrics. 'We built an AI chatbot for a retail company' with no mention of accuracy, latency, cost, or user adoption. Real AI work has numbers.

Red flag 2: Their AI team is their web dev team. Ask who specifically will work on your project. If the 'AI engineers' also build React apps and do DevOps, they are generalists wearing an AI hat.

Red flag 3: They can't name the models they use. 'We use the latest AI technology' is not an answer. GPT-4o, Claude 3.5 Sonnet, Llama 3.1 70B, Mistral Large real AI agencies make deliberate model choices with documented rationale.

Red flag 4: No eval framework. Ask 'How do you measure the quality of AI outputs before releasing to production?' If the answer doesn't include a systematic evaluation process with defined metrics, they are guessing.

Red flag 5: They've never dealt with a compliance requirement. Enterprise AI requires DPDP, SOC2, GDPR awareness. If they've never delivered an AI product under a compliance framework, they will create liability for you.

Red flag 6: Fixed-price contracts for AI work. AI development is inherently iterative model behaviour changes, evals reveal new failure modes, fine-tuning requires multiple cycles. Fixed price indicates they don't understand how AI development actually works.

The 12-Question Scorecard for AI Agency Evaluation

Score each 1-5. Anything below 40/60 is a disqualifier.

1. Show me 3 production AI systems you've shipped with measurable outcomes.
2. What models have you used in LLM production, with what selection criteria?
3. How do you measure hallucination rate in production RAG systems?
4. What is your eval framework? (Tool + methodology)
5. Describe a time an LLM behaved unexpectedly in production. How did you fix it?
6. How do you handle prompt injection attacks in your applications?
7. What does your AI inference cost monitoring look like?
8. How do you manage model versioning when foundation models update?
9. Have you worked under DPDP, SOC2, or GDPR compliance requirements?
10. What is your standard data processing agreement for AI data?
11. Who specifically will work on this project? (Names and GitHub profiles)
12. What happens if the AI component underperforms against agreed benchmarks?