Data Labeling on AWS, Azure, Google Cloud and OCI: The 2026 Solution Architect’s Guide

By Shrenik Parekh · Updated October 27, 2026 · 12 min read

TL;DR for busy architects

If you last evaluated cloud data labeling in 2023 or 2024, your assumptions are probably out of date. In 2026 the managed labeling landscape looks very different:

AWS: SageMaker Ground Truth Plus reached end of support on June 30, 2026, and Ground Truth and Mechanical Turk closed to new customers on July 30, 2026. Existing customers can keep using them, but AWS does not plan new features.
Azure: Microsoft’s retirement notice for Azure Machine Learning data labeling set September 30, 2026 as the date workloads are deleted and application data is lost. That date has passed.
Google Cloud: The Vertex AI Data Labeling Service was shut down on October 3, 2024. Google points users to console labeling and partner solutions in Google Cloud Marketplace.
Oracle OCI: OCI Data Labeling remains an active, SDK-supported service for text, image and document datasets, with a free allowance of annotated records.

The architectural takeaway: stop treating labeling as a feature you rent from a single cloud. Treat it as a portable platform capability (tool, workforce, quality controls, lineage and governance) that sits on top of whichever cloud storage, identity and audit stack you already run. The rest of this guide shows how.

1. What changed in 2026, and why it matters

For years the default answer was simple: pick the labeling service from your cloud. That answer has expired.

Cloud	Native managed labeling status (as of Oct 27, 2026)	What architects should do
AWS	Ground Truth: closed to new customers since July 30, 2026; Ground Truth Plus: end of support June 30, 2026; Mechanical Turk: closed to new customers	Existing pipelines can continue; plan an exit path for new initiatives
Azure	Azure ML data labeling retired September 30, 2026	Microsoft’s guidance is to move to third-party providers and copy labeled data back into Azure ML storage
Google Cloud	Vertex AI Data Labeling Service shut down October 3, 2024	Use console labeling for small jobs or Marketplace partners
Oracle OCI	OCI Data Labeling active	Good fit for OCI-centric, text/image/document workloads

Sources: AWS Service Availability Updates (June 30, 2026), Azure retirement notice, Vertex AI deprecations.

Why it matters: a labeled dataset is a long-lived asset. If the tool that produced it disappears, you can lose the labeling guidelines, annotator history and audit trail that make the dataset defensible. Portability of labels and their metadata is now a first-class requirement.

Related reading on this site: my guide to MLOps architecture patterns and designing cloud exit strategies.

2. Cloud-by-cloud comparison: price, convenience, security, availability

Because the native managed options are shrinking, compare four things per cloud: the labeling option, price model, convenience, and the security and global footprint of the surrounding stack.

Dimension	AWS	Azure	Google Cloud	Oracle OCI
Native labeling option	Ground Truth (legacy, existing customers only)	None after Sept 30, 2026	None first-party; partners via Marketplace	OCI Data Labeling (active)
Price model	Historically per-object (Mechanical Turk) or per-label (vendors, Plus); I could not confirm current public rates, so request a quote	The feature itself carried no extra charge; you paid compute and storage. Now retired	Partner-defined	First 1,000 annotated records free, then per-1,000 records; see the official pricing page
Convenience	Tight S3 and SageMaker integration, but no new features planned	Exports to MLTable, COCO and CSV were native; now needs a replacement	Console labeling plus partner tools	Datasets reference Object Storage; export to JSONL; SDK, Terraform and Ansible support
Security building blocks	IAM, KMS, VPC endpoints, CloudTrail	Entra ID, Key Vault, Private Link, Azure Monitor	IAM, Cloud KMS, VPC Service Controls, Cloud Audit Logs	IAM, Vault, Cloud Guard, Audit
Governance and catalog	Glue Data Catalog, Lake Formation	Microsoft Purview	Dataplex	OCI Data Catalog
Global availability	Verify the region list for each service you use	Verify	Verify	Oracle states services are region-specific by default; verify per region

Price note: public pricing for labeling changes frequently and is often quote-based. Treat any number in a blog post (including mine) as a starting point and confirm it on the vendor page and in the cloud pricing calculator before committing.

Availability note: labeling tools you self-host (Label Studio, CVAT and similar) are available in any region where you can run Kubernetes or VMs. That is a real advantage for data-residency requirements.

3. Reference architecture: the labeling platform in seven layers

A cloud-agnostic design lets you swap the tool without losing governance.

Ingestion and storage: raw data lands in object storage (S3, Blob/ADLS, Cloud Storage, OCI Object Storage) in a restricted zone, encrypted with customer-managed keys.
Labeling tool: a managed vendor or self-hosted platform, deployed in your VPC/VNet with private connectivity and SSO.
Workforce: internal subject-matter experts, vetted vendors, or a mix, with role-based access and task-level data minimization.
Quality engine: gold sets, consensus, adjudication and automated checks (section 5).
Metadata and lineage: every label carries who, when, which guideline version, which model pre-labeled it, and which dataset version it belongs to.
Observability: pipeline, workforce and data-quality metrics in one place (section 7).
Governance and audit: policy, approvals, retention, access reviews and evidence collection.

Component mapping by cloud

Layer	AWS	Azure	Google Cloud	OCI
Storage	S3	Blob / ADLS Gen2	Cloud Storage	Object Storage
Keys	KMS	Key Vault	Cloud KMS	Vault
Identity	IAM / IAM Identity Center	Entra ID	Cloud IAM	IAM / Identity Domains
Sensitive-data discovery	Macie	Purview / Defender	Sensitive Data Protection	Data Safe
Catalog and lineage	Glue Data Catalog, Lake Formation	Purview	Dataplex	Data Catalog
Logs and audit	CloudTrail, CloudWatch	Azure Monitor, Activity Log	Cloud Audit Logs, Cloud Monitoring	Audit, Logging, Monitoring
Workflow orchestration	Step Functions, MWAA	Data Factory, Logic Apps	Workflows, Composer	Data Integration, Functions
Posture management	Security Hub, Config	Defender for Cloud, Policy	Security Command Center	Cloud Guard, Security Zones

Tip for architects: confirm the exact product names and regional availability in each provider’s documentation before finalizing a design, as service names change often.

4. Labeling tools and technology options

With native services shrinking, most teams choose from three categories.

1. Open-source, self-hosted. Label Studio (multimodal) and CVAT (computer vision, video, 3D) run in your own VPC. Best when data residency, cost control or air-gapped operation matters. Trade-off: you operate it (upgrades, scaling, backups, SSO).

2. Commercial platforms and managed services. Enterprise editions of the tools above and a large vendor market (often available through cloud marketplaces) bundle workflow, QA dashboards, SSO and support. Some provide managed workforces. Trade-off: per-seat or per-label pricing and vendor lock-in risk. Microsoft’s own retirement guidance for Azure ML data labeling points customers toward third-party providers.

3. Cloud-native building blocks. For OCI-centric teams, OCI Data Labeling covers simple text, image and document needs. On other clouds, you can assemble a minimal tool from storage, identity and a web front end, though this is rarely worth the maintenance.

Technology accelerators

Model-assisted pre-labeling: a model proposes labels and humans correct them. Often the largest throughput gain.
Active learning: prioritize the examples the model is least sure about.
LLM-assisted labeling and review: use a language model to draft labels or flag likely errors, with humans as final authority for anything regulated or high-impact.
Label-error detection: statistical tools (for example, Cleanlab-style confident learning) surface suspect labels in an existing dataset.
Synthetic data: useful to cover rare classes, but track it separately from human-verified data.

5. Methodology and best practices

Write the labeling guideline first. Include definitions, positive and negative examples, edge cases, and a decision tree. Version it like code.
Run a pilot. Label 200 to 500 items with a small expert group. Fix ambiguous guidelines before scaling.
Build a gold set. Expert-labeled items hidden inside normal queues to measure annotator accuracy continuously.
Measure agreement. Use Cohen’s kappa for two annotators or Krippendorff’s alpha for many annotators and missing data. Low agreement usually means unclear guidelines, not bad annotators.
Use consensus plus adjudication. Multiple labels per item for hard tasks; a senior reviewer resolves disagreements.
Sample for QA. Review a statistically meaningful sample per annotator per batch rather than eyeballing.
Minimize data exposure. Mask or redact PII before tasks reach the workforce. Restrict which workforces can see regulated data.
Version datasets. Never overwrite labels; snapshot datasets and record the guideline and tool versions.
Close the loop with the model. Feed model errors back to the labeling queue.

6. Verification vs. validation of labels

These terms are often mixed up. Use them precisely:

Verification asks: did we label it the way we said we would? Checks conformance to guidelines: schema validity, class definitions, annotator agreement, gold-set accuracy, completeness, duplicate detection, and PII checks.
Validation asks: are the labels fit for the real-world purpose? Checks the dataset against the intended use: does a model trained on it perform well on a held-out, independently labeled test set from production-like data? Is the class balance representative? Are subgroups fairly represented?

Stage	Example checks	Typical owner
Verification (during labeling)	Schema validation, gold-set accuracy, inter-annotator agreement, spot-check audits	Labeling operations / QA
Validation (before release)	Held-out test performance, bias and subgroup analysis, drift versus production data, domain-expert sign-off	Data science and domain owner
Re-validation (after release)	Monitoring for data drift, periodic re-audit samples	MLOps

Release a dataset only when both pass, and record the evidence.

7. Applications, databases, observability, auditability and governance

Applications

Expose labeling through APIs and events rather than manual exports. Trigger labeling jobs from your application pipelines (new data arrives, uncertainty is high, a user corrects a prediction) and publish “dataset version ready” events that downstream training jobs subscribe to.

Databases

Store labels and metadata in a transactional store with a clear schema: item ID, label, annotator ID (pseudonymized), timestamps, guideline version, confidence, review status.
Keep raw data in object storage and reference it by URI; do not copy sensitive payloads into the labeling database.
Use immutable, append-only history for labels so every change is auditable.
Plan for export formats your tools and training stack understand (JSONL, COCO, CSV, Parquet).

Observability

Track, in one dashboard:

Pipeline: queue depth, task latency, throughput per hour, error rates.
Quality: gold-set accuracy, agreement, rejection and rework rate, drift of label distribution.
Workforce: active annotators, time per task, anomalies (suspiciously fast or uniform answers).
Cost: cost per accepted label and per dataset version.

Standardize telemetry with OpenTelemetry and capture dataset and job lineage with OpenLineage so traces connect labeling jobs to training runs and deployed models.

Auditability

An auditor should be able to answer: who labeled this item, under which guideline version, with what pre-label model, who reviewed it, and which model versions were trained on it? Make that answer a query, not a forensic project. Enable cloud audit logs for storage and identity, retain them according to policy, and export them to an immutable store.

Governance

Align with recognized frameworks: the NIST AI Risk Management Framework and ISO/IEC 42001 both emphasize data quality, documentation and accountability. Practical controls:

Data classification and approval before data enters labeling.
Role-based access and periodic access reviews for annotators and vendors.
Vendor due diligence (security certifications, subprocessors, data location).
Retention and deletion rules for raw data, labels and annotator records.
A dataset “card” documenting source, purpose, labeling method, known limitations and approvals.

8. OKRs for a data labeling program

The targets below are illustrative starting points; calibrate them to your task difficulty and risk level.

Objective 1: Deliver trustworthy labeled data faster

KR1: Reduce time from data request to first usable dataset from 6 weeks to 3 weeks.
KR2: Increase pre-labeling acceptance rate (labels accepted without edits) to 60% for mature tasks.
KR3: Cut cost per accepted label by 25% without lowering quality.

Objective 2: Keep label quality measurable and high

KR1: Inter-annotator agreement (kappa or alpha) at or above 0.80 on production tasks.
KR2: Gold-set accuracy at or above 95% for active annotators.
KR3: Rework rate below 5% after QA sampling.

Objective 3: Be audit-ready by default

KR1: 100% of datasets have lineage from source to model version.
KR2: 100% of datasets have a completed dataset card before release.
KR3: Zero PII exposure incidents to external workforces.

Objective 4: Stay portable

KR1: Complete a tested export-and-restore of labels plus metadata to a second tool each quarter.
KR2: No single vendor holds the only copy of labeled data or guidelines.

9. Decision guide for solution architects

If your situation is…	Consider…
Strict data residency or air-gapped data	Self-hosted Label Studio or CVAT in your own region and VPC
Heavy OCI footprint, text/image/document tasks	OCI Data Labeling, with Object Storage as the source
New AWS, Azure or GCP initiative	A cloud-agnostic platform or marketplace vendor integrated with native storage, IAM, KMS and audit logs
Regulated data (health, finance)	Internal or tightly vetted workforce, private networking, customer-managed keys, full audit trail
Existing Ground Truth or Azure ML labeling pipelines	Inventory datasets, export labels and metadata now, and plan the migration before the next audit or model refresh

Migration checklist: export labels and guidelines, preserve annotator and review history, map label schemas, re-run a gold-set comparison on the new tool, update pipeline references (for example, datasets referenced by ARN or pipeline stage), and keep the old exports in immutable storage.

Frequently asked questions

Is AWS SageMaker Ground Truth still available? Existing customers can continue using it, but it has been closed to new customers since July 30, 2026, and AWS says it does not plan new features. Ground Truth Plus reached end of support on June 30, 2026.

Is Azure Machine Learning data labeling still available? Microsoft set September 30, 2026 as the retirement date, after which workloads are deleted. Confirm your status in the Azure portal and migrate to a third-party provider.

Does Google Cloud offer a managed data labeling service? The Vertex AI Data Labeling Service was shut down in October 2024. Google’s deprecation notice recommends console labeling or partner solutions from Google Cloud Marketplace.

What is the cheapest option? For small text, image or document projects on Oracle, OCI Data Labeling’s free allowance is hard to beat. At scale, the dominant cost is human time, so pre-labeling, active learning and clear guidelines usually matter more than the tool license.

How do I measure label quality? Combine gold-set accuracy, inter-annotator agreement and QA sampling for verification, then validate with held-out, independently labeled data before release.

Key takeaways

Native managed labeling has contracted on AWS, Azure and Google Cloud; OCI Data Labeling is the one native option still actively offered of the four.
Design labeling as a portable platform with lineage, quality gates and audit evidence.
Separate verification (did we follow the process?) from validation (is it fit for purpose?).
Define OKRs for speed, quality, auditability and portability, and review them quarterly.

References and further reading

AWS, AWS Service Availability Updates, June 30, 2026.
Microsoft Azure Updates, Retirement: Azure Machine Learning – Data labeling.
Google Cloud, Vertex AI deprecations.
Oracle, OCI Data Labeling pricing.
CVAT, CVAT vs. Azure Machine Learning Data Labeling (vendor comparison; read with that in mind).
Label Studio and CVAT project sites.
NIST, AI Risk Management Framework.
ISO, ISO/IEC 42001 AI management systems.
OpenTelemetry and OpenLineage.

Prices, regions and service status change often. Verify against each provider’s official pages before making purchasing or architecture decisions.