> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Data Formats for Machine Learning
> Practical overview of data formats for machine learning pipelines with format recommendations for analytics, training, media, and AWS services
Machine learning pipelines are only as good as the data feeding them. Data can appear in many formats, and choosing the right one at each stage (ingest, storage, preprocessing, training, serving) directly affects training throughput, storage cost, query latency, and integration with analytics or ML services.
Selecting the optimal format reduces I/O overhead, improves compatibility with engines (Spark, Athena, Redshift), and lowers compute and storage bills—especially at scale.
Below is a concise, practical overview of commonly used ML data formats, grouped by data type and use case. Use this as a quick reference when designing pipelines for analytics, feature stores, distributed training, and model serving.
Match format to pipeline stage: use columnar compressed formats (Parquet/ORC) for analytics and feature stores, binary ML-native formats (TFRecord/RecordIO) for high-throughput training, and media-specific formats (JPEG/MP4/WAV) for raw multimedia inputs.
## Tabular data formats
Common choices for structured/tabular data, ranked by typical usage:
| Format | Best for | Notes |
| ------- | ------------------------------------------------------------: | ---------------------------------------------------------------------------- |
| CSV | Small datasets, quick inspection, spreadsheet interchange | Human-readable but lacks schema, inefficient for large-scale analytics |
| Parquet | Big data analytics, feature stores, query engines | Columnar, compressed, supports predicate pushdown (good for Athena/Redshift) |
| HDF5 | Large numerical arrays, hierarchical datasets, tensor storage | Excellent for large tensor datasets and model checkpoints in deep learning |
Example: reading these formats with Python:
```python theme={null}
# CSV
import pandas as pd
df = pd.read_csv("s3://bucket/data.csv")
# Parquet
df = pd.read_parquet("s3://bucket/data.parquet")
# HDF5
import h5py
with h5py.File("data.h5", "r") as f:
arr = f["/dataset"][...]
```
## Text and NLP formats
* TXT: Plain text for raw corpora. Universal and simple; requires preprocessing (tokenization, normalization).
* JSON / JSONL: Structured text for annotations and nested metadata. `JSONL` (one JSON object per line) is common for scalable ingestion and streaming.
* Recommended patterns:
* Use `JSONL` for document-level metadata and per-example labels.
* Use compressed archives (e.g., `.gz`) when transporting large text corpora.
## Binary ML-native formats
These formats optimize sequential reads, sharding, and integration with training frameworks:
| Format | Typical frameworks | Use case |
| ------------- | ---------------------------: | -------------------------------------------------------------------- |
| TFRecord | TensorFlow | Binary, protobuf-backed; good for sharding and fast streaming reads |
| RecordIO | MXNet, some legacy pipelines | Sequential binary records optimized for streaming |
| Pickle (.pkl) | Python ecosystem | Serializes Python objects and models; convenient for local artifacts |
Pickles can execute arbitrary code on load. Do not unpickle data from untrusted sources. Prefer language-agnostic formats (TFRecord, Parquet, JSON) for shared datasets.
## Image, audio, and video formats
Media formats depend on fidelity, compression needs, and downstream processing tools:
* Images:
* JPEG — lossy, widely used for image-model training where compression is acceptable.
* PNG — lossless, useful for images requiring pixel-perfect fidelity.
* Audio:
* WAV — uncompressed, high quality; larger storage footprint.
* MP3 — compressed, lower size; good for distribution.
* FLAC — lossless compressed audio; good for speech datasets.
* Video:
* MP4 (H.264) — widely supported, good compression/quality balance.
* AVI — less compressed, larger files; sometimes used when minimal codec transformations are desired.
Quick reference for preprocessing libraries:
* Images: PIL / OpenCV / torchvision
* Audio: librosa / torchaudio / pydub
* Video: ffmpeg / OpenCV
## AWS storage and analytics: format recommendations
AWS supports many formats; choose based on workload patterns (scan-heavy analytics vs. sequential training reads):
| AWS Service | Recommended formats | Rationale |
| ------------------------ | -------------------------------------------: | -------------------------------------------- |
| Amazon S3 | Parquet, TFRecord, CSV, images, audio, video | Object store for raw and processed artifacts |
| Amazon Athena / AWS Glue | Parquet, ORC, JSON, CSV | Querying and ETL for analytics |
| Amazon Redshift | Parquet, ORC (via Spectrum) | Columnar formats for fast analytic queries |
| AWS Lake Formation | Parquet, ORC, JSON | Centralized governance for data lakes on S3 |
* Amazon S3 is the foundational storage layer for many pipelines. Use lifecycle policies and compression to control cost.
* Consider partitioning and file sizing (e.g., Parquet row groups \~128 MB) to optimize query and read performance.
## Processing and streaming on AWS
Match processing tools to data format and latency requirements:
* AWS Glue: Serverless ETL; supports `CSV`, `JSON`, `Parquet`, `Avro`, and `ORC`. Good for schema discovery and format conversions.
* Use Glue jobs or Glue Studio for converting CSV/JSON to Parquet for analytics.
* Amazon EMR (Spark/Hadoop): Read/write Parquet, ORC, Avro, sequence files. Ideal for large-scale transformations and feature engineering.
* Amazon Kinesis: Ingest streaming events (often JSON or binary protobufs); used for near-real-time use cases.
* AWS Lambda: Receives JSON payloads (and base64-encoded binary data). Suitable for lightweight transformations and event-driven processing.
Best practices:
* For streaming ingestion, keep messages small and schema-stable (e.g., Avro/Protobuf).
* Convert raw event streams to columnar formats in downstream storage for efficient analytics.
## ML services and specialized formats
Common AWS ML services and their preferred input types:
* Amazon SageMaker: Supports CSV, Parquet, TFRecord, images, audio, and pickled artifacts for training and hosting. Use SageMaker processing jobs to convert formats at scale.
* Amazon Transcribe: Accepts WAV, MP3, FLAC, and MP4 for speech-to-text.
* Amazon Rekognition: Works with standard image and video formats (JPEG, PNG, MP4) for computer vision tasks.
When designing ML pipelines on AWS:
* Use columnar compressed formats (Parquet/ORC) for analytics and feature stores.
* Use binary ML-native formats (TFRecord/RecordIO) to maximize throughput for distributed training.
* Keep raw media in their native formats (JPEG/WAV/MP4) and extract features or convert to optimized formats only when necessary.
## Summary checklist
* Use Parquet/ORC for analytics and feature engineering—columnar layout reduces I/O and improves query performance.
* Use TFRecord/RecordIO for high-throughput, sharded training pipelines.
* Keep media in standard containers (JPEG/PNG, WAV/FLAC, MP4) and preprocess with specialized libraries.
* Avoid insecure serialization formats (untrusted pickles) when sharing datasets across teams or services.
* Optimize file sizing, partitioning, and compression to balance read performance and storage cost.
## Links and references
* [Amazon S3](https://aws.amazon.com/s3/)
* [Amazon Athena](https://aws.amazon.com/athena/)
* [Amazon Redshift](https://aws.amazon.com/redshift/)
* [AWS Glue](https://aws.amazon.com/glue/)
* [Amazon EMR](https://aws.amazon.com/emr/)
* [Amazon Kinesis](https://aws.amazon.com/kinesis/)
* [Amazon SageMaker](https://aws.amazon.com/sagemaker/)
* [Amazon Transcribe](https://aws.amazon.com/transcribe/)
* [Amazon Rekognition](https://aws.amazon.com/rekognition/)