Skip to main content
AWS provides a comprehensive set of services to build end-to-end machine learning (ML) data pipelines. These services cover the common lifecycle stages: ingestion, storage, preparation, labeling, feature engineering, and training/orchestration. This article reviews each stage, highlights the most commonly used AWS services, and provides guidance to choose the right tool for your ML workloads. Start-to-finish, a typical ML data pipeline includes:
  • Ingestion — bringing data into AWS from devices, applications, or other systems.
  • Storage — persisting raw and processed data in an appropriate store.
  • Preparation — cleaning, transforming, and enriching data for ML.
  • Labeling — annotating data for supervised learning.
  • Feature engineering — creating and storing features for training and inference.
  • Training and orchestration — running model training at scale and managing the ML workflow.
The image illustrates an ML Data Pipeline on AWS, outlining steps from data ingestion and storage to preparation, labeling, feature engineering, and training.
Service summary: mapping pipeline stages to AWS offerings Data ingestion on AWS Choose ingestion services based on latency/throughput needs and destination targets (S3, Redshift, Feature Store, etc.):
  • Amazon Kinesis (Data Streams and Data Firehose) — real-time streaming ingestion and delivery to targets such as Amazon S3, Redshift, or Elasticsearch.
  • AWS Glue — ETL movement and transformation between stores when building batch pipelines.
  • AWS Data Pipeline — orchestration for scheduled data movement and long-running or legacy workflows.
  • AWS IoT Core — secure ingestion of telemetry and messages from connected devices.
Consider throughput, ordering guarantees, and the downstream target when selecting among these options.
The image shows icons representing different AWS data ingestion services: AWS Glue, Amazon Kinesis, AWS Data Pipeline, and AWS IoT Core.
Storage: choose the right store for data type and access pattern
  • Amazon S3 — scalable object storage and the de facto data lake for raw and processed datasets, model artifacts, and feature export.
  • Amazon Redshift — columnar data warehouse for analytical queries and BI workloads at scale.
  • Amazon RDS — managed relational databases for structured OLTP workloads.
  • Amazon DynamoDB — fully managed NoSQL key-value and document store for low-latency, high-scale access patterns.
  • AWS Lake Formation — simplifies building, securing, and managing a centralized data lake on S3 with fine-grained access control.
These options cover structured, semi-structured, and unstructured data. Use Lake Formation to centralize governance and fine-grained permissions.
The image lists different data storage services: Amazon S3, Amazon Redshift, Amazon RDS, Amazon DynamoDB, and AWS Lake Formation, each with their respective icons.
Processing and transformation Prepare data for ML by cleaning, profiling, and transforming at the required scale:
  • AWS Glue — serverless ETL that discovers schemas, generates PySpark or Scala code, runs ETL jobs, and catalogs metadata in AWS Glue Data Catalog.
  • AWS Glue DataBrew — a no-code/low-code visual tool for data cleaning, profiling, and exploration; great for analysts and data scientists collaborating on data prep.
  • Amazon EMR — managed Hadoop/Spark cluster platform for large-scale distributed processing and custom Spark jobs where fine control and ecosystem tools are required.
Select Glue/DataBrew for serverless, fast development and EMR for heavy custom distributed processing or when you need advanced Spark/Hadoop control.
The image displays three AWS services for data processing and transformation: AWS Glue, AWS Glue DataBrew, and Amazon EMR, with a brief description of each.
Labeling, feature management, and orchestration
  • Amazon SageMaker Ground Truth — scalable labeling with human labelers, private workforces, and automated/ML-assisted labeling (active learning) to reduce cost and improve label quality.
  • Amazon SageMaker Feature Store — centralized feature repository with both offline (batch) and low-latency online stores; supports consistent feature retrieval for training and inference.
  • Amazon SageMaker Pipelines — orchestrate, automate, and version end-to-end ML workflows: data processing, training jobs, evaluation, and deployment. Pipelines integrates tightly with SageMaker components and other AWS services.
For training, use Amazon SageMaker training jobs (managed infrastructure) within SageMaker Pipelines to scale model training and track run metadata. Integrated flow example A common integrated flow looks like this:
  1. Ingest streaming events with Kinesis or batch files into Amazon S3.
  2. Catalog and transform data with AWS Glue (or run Spark jobs on EMR for large-scale processing).
  3. Use SageMaker Ground Truth to label a representative subset of data.
  4. Store engineered features in SageMaker Feature Store for reuse and consistent serving.
  5. Orchestrate data processing, training, and deployment with SageMaker Pipelines and run managed SageMaker training jobs.
Select services based on data velocity, scale, and required SLAs: use streaming services (Kinesis) for real-time pipelines, EMR for heavy custom distributed processing, and serverless Glue/DataBrew for most ETL and interactive cleaning tasks.
Links and references Use this guide as a starting point to design ML-ready data pipelines on AWS. Choose services according to latency, throughput, operational model (serverless vs. managed clusters), and governance needs.

Watch Video