Choose an AI data annotation vendor without guessing

A language catalogue does not establish reviewed production capacity for your task. Ask: “For which of our required languages, scripts and task types have you delivered comparable production work, and what reviewed team is available for this scope?” Check the evidence, the assessment method and any sourcing dependencies before accepting a coverage claim.

Test this by: "For [your target language], how many annotators have completed at least 100 hours of annotation work? Can you show me their quality scores?"

What you gain: Protection against the most common vendor failure: quality that looks strong in pilot and degrades at scale.

Ask whether the pilot team will continue into production, how additional annotators will qualify, and what evidence will show that quality is maintained as the team changes. Include the escalation and recalibration rules in the production plan.

Ask: "What percentage of your pilot annotators stayed on the program through the first three production months? What was the quality delta between pilot and month-three production batches?"

Why it matters: Without batch-level quality visibility, bad annotations reach your training pipeline before anyone notices.

Look for structured QA, not just "we check the work." A credible quality governance structure includes:

• Calibration sets built before production starts
• Inter-annotator agreement (IAA) scoring tracked per batch
• Multi-layer review (annotator, reviewer, QA auditor)
• Error trend analysis with recalibration triggers
• Per-annotator performance tracking that identifies weak performers early

Ask: "Show me a sample batch QA report from a recent production program. What IAA threshold triggers a recalibration cycle?"

What you gain: Clarity on your quality ceiling and language coverage floor before you commit to a production contract.

Where annotators come from shapes what a vendor can realistically deliver. Vendors who source from crowdsourcing platforms have breadth but limited control. Vendors who source from professional linguist networks have control but may lack rare-language access. Vendors who source from community networks (diaspora, academic, and professional communities) can often reach languages that marketplace-dependent vendors cannot.

Ask: "For rare or low-resource languages, where do you source annotators? Do you recruit directly or subcontract?"

AI training data often contains sensitive content: personally identifiable information, proprietary business data, or content requiring safety evaluation. The vendor's security posture must match the data sensitivity level. A mismatch here does not just create risk — it can halt a program entirely or trigger legal exposure.

Look for:

• ISO 27001 certification (or equivalent)
• NDAs signed by every individual annotator, not just the vendor entity
• Access controls scoped by project role
• GDPR-aligned data handling if working with EU data
• Encrypted data in transit and at rest

Ask: "Do your individual annotators sign NDAs, or just your company? How is project data segmented from other client work?"

What you gain: Predictable delivery cadences that let you plan model training iterations without waiting on late batches.

Production annotation programs need predictable delivery, not "we will try to finish by Friday." Look for:

• Structured batch delivery on a daily or weekly cadence
• Penalty-clause SLA readiness (not just target dates)
• Capacity planning that accounts for annotator attrition
• Escalation protocols when quality or timeline is at risk

Ask: "Do you offer penalty-clause SLAs? What happens when an annotator drops out mid-batch: how quickly do you backfill without quality disruption?"

Annotation vendors use different pricing structures, and the model you accept shapes both budget predictability and incentive alignment. Getting this wrong means either overpaying for throughput you do not need or creating incentives that push annotators to rush.

• Per-unit pricing (per word, per image, per audio minute): Predictable cost per item. Works well for high-volume, standardized tasks. Risk: vendors may rush to maximize throughput at the expense of quality.
• Per-hour pricing: Pays for annotator time regardless of output. Better for tasks where quality requires deliberation. Risk: no built-in efficiency incentive.
• Project-based pricing: Fixed cost for a defined scope. Predictable budget. Risk: scope creep disputes if requirements change.
• Retainer / managed service: Monthly commitment for a dedicated team. Best for ongoing programs with predictable volume. Offers the most control over annotator continuity.

Ask: "What pricing model do you recommend for our use case, and how does your model handle scope changes mid-project?"

What you gain: Verified, auditable process governance rather than unsubstantiated quality claims.

Certifications alone do not guarantee quality, but their absence is a signal. For AI data annotation work, the relevant standards are:

• ISO 9001 (quality management system)
• ISO 27001 (information security)
• ISO 17100 (translation services, relevant for vendors who also handle translation tasks)

A vendor holding all three has invested in auditable process governance across quality, security, and linguistic operations.

Ask: "Which ISO certifications do you hold? When were they last audited?"

Verify by requesting: a breakdown of who actually handles your data and how task complexity maps to annotator capability.

Strong vendors classify their workforce into defined tiers rather than treating all annotators as interchangeable. A credible tiering system typically includes:

• Core annotators: Assess language and task fit against a representative calibration set and the buyer-approved acceptance rules.
• Domain specialists: Verify relevant subject knowledge and assess performance on the actual task, including difficult examples and escalation decisions.
• Senior reviewers and adjudicators: Verify the expertise needed to resolve disagreements, maintain calibration examples and explain decisions.

For each role, agree the assessment method, pass criteria, review depth and reassessment triggers for this project before staffing is accepted.

When a vendor cannot explain their tiering criteria or how annotators move between tiers, you are relying on unstructured talent allocation. The risk: your safety-critical evaluation tasks get assigned to annotators qualified only for bulk labeling.

Ask: "How do you classify your annotators? What qualifies an annotator to handle domain-specific or safety-critical tasks versus standard labeling?"

Annotator attrition is inevitable in long-running programs. The question is not whether it happens, but how quickly the vendor recovers without quality disruption. A vendor with no replacement plan leaves you exposed the moment a key annotator drops out.

Questions to agree for each required language and role:

• What replacement time can the supplier commit to for this scope, and what evidence supports it?
• Which qualified backup people are available now, and what assessment or recalibration remains?
• How will the supplier notify the buyer if a replacement cannot meet the agreed time or quality requirement?

Record the agreed replacement conditions, backup coverage and exception owner in the project plan.

Ask: “What replacement conditions can you support for each required language and role, and which dependencies remain?”

What you gain: A ramp plan grounded in assessed people, review capacity and the project’s acceptance requirements.

Ask the supplier to show the proposed increase by language and role, the people already qualified and available, any recruitment or training still needed, and the evidence required before each increase.

• People qualified and available for each required language and role
• Recruitment, training and review-capacity dependencies
• Evidence and buyer-agreed acceptance requirements before each increase

Use a comparable delivery record or a representative pilot to test the assumptions.

Ask: “If our required throughput changes, what can you support, by when, with which dependencies and quality controls?”