> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Overview of AWS Data Services for ML
> Overview of AWS services for building end-to-end machine learning data pipelines including ingestion, storage, preparation, labeling, feature management, training, and orchestration.
AWS provides a comprehensive set of services to build end-to-end machine learning (ML) data pipelines. These services cover the common lifecycle stages: ingestion, storage, preparation, labeling, feature engineering, and training/orchestration. This article reviews each stage, highlights the most commonly used AWS services, and provides guidance to choose the right tool for your ML workloads.
Start-to-finish, a typical ML data pipeline includes:
* Ingestion — bringing data into AWS from devices, applications, or other systems.
* Storage — persisting raw and processed data in an appropriate store.
* Preparation — cleaning, transforming, and enriching data for ML.
* Labeling — annotating data for supervised learning.
* Feature engineering — creating and storing features for training and inference.
* Training and orchestration — running model training at scale and managing the ML workflow.
Service summary: mapping pipeline stages to AWS offerings
| Pipeline Stage | Purpose | Common AWS Services | When to choose |
| ---------------------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------: | ------------------------------------------------------------------------------------------------------------------------------------- |
| Ingestion | Collect streaming or batch data from sources | Amazon Kinesis (Data Streams, Firehose), AWS IoT Core, AWS Data Pipeline | Low latency or streaming => Kinesis; device telemetry => IoT Core; scheduled batch workflows or legacy orchestration => Data Pipeline |
| Storage | Persist raw and processed data | `Amazon S3`, Amazon Redshift, Amazon RDS, Amazon DynamoDB, AWS Lake Formation | Data lake & artifacts => `S3`; analytical warehousing => Redshift; transactional => RDS; low-latency key-value => DynamoDB |
| Preparation & Transformation | Clean, profile, and transform datasets for ML | AWS Glue, AWS Glue DataBrew, Amazon EMR | Serverless ETL & metadata catalog => Glue; interactive visual cleaning => DataBrew; large-scale Spark/Hadoop workloads => EMR |
| Labeling | Create ground-truth annotations for supervised learning | Amazon SageMaker Ground Truth | Human-in-the-loop and auto-labeling support for scalable labeling workflows |
| Feature Management | Store, serve, and share features across teams | Amazon SageMaker Feature Store | Ensure consistency between training and inference; provide low-latency online access |
| Training & Orchestration | Run training jobs and manage ML pipelines | Amazon SageMaker (Training Jobs, Pipelines) | Managed training infrastructure, model management, and CI/CD integration |
Data ingestion on AWS
Choose ingestion services based on latency/throughput needs and destination targets (S3, Redshift, Feature Store, etc.):
* Amazon Kinesis (Data Streams and Data Firehose) — real-time streaming ingestion and delivery to targets such as `Amazon S3`, Redshift, or Elasticsearch.
* AWS Glue — ETL movement and transformation between stores when building batch pipelines.
* AWS Data Pipeline — orchestration for scheduled data movement and long-running or legacy workflows.
* AWS IoT Core — secure ingestion of telemetry and messages from connected devices.
Consider throughput, ordering guarantees, and the downstream target when selecting among these options.
Storage: choose the right store for data type and access pattern
* `Amazon S3` — scalable object storage and the de facto data lake for raw and processed datasets, model artifacts, and feature export.
* Amazon Redshift — columnar data warehouse for analytical queries and BI workloads at scale.
* Amazon RDS — managed relational databases for structured OLTP workloads.
* Amazon DynamoDB — fully managed NoSQL key-value and document store for low-latency, high-scale access patterns.
* AWS Lake Formation — simplifies building, securing, and managing a centralized data lake on `S3` with fine-grained access control.
These options cover structured, semi-structured, and unstructured data. Use Lake Formation to centralize governance and fine-grained permissions.
Processing and transformation
Prepare data for ML by cleaning, profiling, and transforming at the required scale:
* AWS Glue — serverless ETL that discovers schemas, generates PySpark or Scala code, runs ETL jobs, and catalogs metadata in AWS Glue Data Catalog.
* AWS Glue DataBrew — a no-code/low-code visual tool for data cleaning, profiling, and exploration; great for analysts and data scientists collaborating on data prep.
* Amazon EMR — managed Hadoop/Spark cluster platform for large-scale distributed processing and custom Spark jobs where fine control and ecosystem tools are required.
Select Glue/DataBrew for serverless, fast development and EMR for heavy custom distributed processing or when you need advanced Spark/Hadoop control.
Labeling, feature management, and orchestration
* Amazon SageMaker Ground Truth — scalable labeling with human labelers, private workforces, and automated/ML-assisted labeling (active learning) to reduce cost and improve label quality.
* Amazon SageMaker Feature Store — centralized feature repository with both offline (batch) and low-latency online stores; supports consistent feature retrieval for training and inference.
* Amazon SageMaker Pipelines — orchestrate, automate, and version end-to-end ML workflows: data processing, training jobs, evaluation, and deployment. Pipelines integrates tightly with SageMaker components and other AWS services.
For training, use Amazon SageMaker training jobs (managed infrastructure) within SageMaker Pipelines to scale model training and track run metadata.
Integrated flow example
A common integrated flow looks like this:
1. Ingest streaming events with Kinesis or batch files into `Amazon S3`.
2. Catalog and transform data with AWS Glue (or run Spark jobs on EMR for large-scale processing).
3. Use SageMaker Ground Truth to label a representative subset of data.
4. Store engineered features in SageMaker Feature Store for reuse and consistent serving.
5. Orchestrate data processing, training, and deployment with SageMaker Pipelines and run managed SageMaker training jobs.
Select services based on data velocity, scale, and required SLAs: use streaming services (Kinesis) for real-time pipelines, EMR for heavy custom distributed processing, and serverless Glue/DataBrew for most ETL and interactive cleaning tasks.
Links and references
* [Amazon S3 — Object Storage](https://learn.kodekloud.com/user/courses/amazon-simple-storage-service-amazon-s3)
* [Amazon SageMaker — Managed ML Platform](https://learn.kodekloud.com/user/courses/aws-sagemaker)
* [AWS Glue — Serverless ETL](https://aws.amazon.com/glue/)
* [Amazon Kinesis — Real-time Data Streaming](https://aws.amazon.com/kinesis/)
* [Amazon EMR — Managed Hadoop/Spark](https://aws.amazon.com/emr/)
* [AWS Lake Formation — Data Lake Management](https://aws.amazon.com/lake-formation/)
* [Amazon RDS & DynamoDB — Managed Databases](https://learn.kodekloud.com/user/courses/introduction-to-aws-databases)
Use this guide as a starting point to design ML-ready data pipelines on AWS. Choose services according to latency, throughput, operational model (serverless vs. managed clusters), and governance needs.