Skip to main content
Machine learning pipelines are only as good as the data feeding them. Data can appear in many formats, and choosing the right one at each stage (ingest, storage, preprocessing, training, serving) directly affects training throughput, storage cost, query latency, and integration with analytics or ML services.
The image illustrates the importance of data format in machine learning, showing that machine learning relies on data, which is shown in a specific format.
Selecting the optimal format reduces I/O overhead, improves compatibility with engines (Spark, Athena, Redshift), and lowers compute and storage bills—especially at scale.
The image visually explains why data format matters, highlighting its impact on performance, stage efficiency, and compatibility. It includes icons for each aspect and emphasizes that choosing the right format improves speed, cost, and compatibility.
Below is a concise, practical overview of commonly used ML data formats, grouped by data type and use case. Use this as a quick reference when designing pipelines for analytics, feature stores, distributed training, and model serving.
Match format to pipeline stage: use columnar compressed formats (Parquet/ORC) for analytics and feature stores, binary ML-native formats (TFRecord/RecordIO) for high-throughput training, and media-specific formats (JPEG/MP4/WAV) for raw multimedia inputs.

​
Tabular data formats

Common choices for structured/tabular data, ranked by typical usage: Example: reading these formats with Python:
The image is a table comparing tabular data formats: CSV, Parquet, and HSF5, detailing their respective use cases and notes. It highlights CSV for structured data, Parquet for big data and cloud ML, and HSF5 for large numerical datasets.

​
Text and NLP formats

  • TXT: Plain text for raw corpora. Universal and simple; requires preprocessing (tokenization, normalization).
  • JSON / JSONL: Structured text for annotations and nested metadata. JSONL (one JSON object per line) is common for scalable ingestion and streaming.
  • Recommended patterns:
    • Use JSONL for document-level metadata and per-example labels.
    • Use compressed archives (e.g., .gz) when transporting large text corpora.

​
Binary ML-native formats

These formats optimize sequential reads, sharding, and integration with training frameworks:
Pickles can execute arbitrary code on load. Do not unpickle data from untrusted sources. Prefer language-agnostic formats (TFRecord, Parquet, JSON) for shared datasets.
The image is a table comparing binary formats, highlighting the use cases, key features, and common tools for TFRecord, RecordIO, and Pickle (.pkl).

​
Image, audio, and video formats

Media formats depend on fidelity, compression needs, and downstream processing tools:
  • Images:
    • JPEG — lossy, widely used for image-model training where compression is acceptable.
    • PNG — lossless, useful for images requiring pixel-perfect fidelity.
  • Audio:
    • WAV — uncompressed, high quality; larger storage footprint.
    • MP3 — compressed, lower size; good for distribution.
    • FLAC — lossless compressed audio; good for speech datasets.
  • Video:
    • MP4 (H.264) — widely supported, good compression/quality balance.
    • AVI — less compressed, larger files; sometimes used when minimal codec transformations are desired.
Quick reference for preprocessing libraries:
  • Images: PIL / OpenCV / torchvision
  • Audio: librosa / torchaudio / pydub
  • Video: ffmpeg / OpenCV
The image is a table summarizing different media formats (JPEG, PNG, WAV, MP3, MP4, AVI), their types, use cases, notes, and popular tools associated with each format.

​
AWS storage and analytics: format recommendations

AWS supports many formats; choose based on workload patterns (scan-heavy analytics vs. sequential training reads):
  • Amazon S3 is the foundational storage layer for many pipelines. Use lifecycle policies and compression to control cost.
  • Consider partitioning and file sizing (e.g., Parquet row groups ~128 MB) to optimize query and read performance.
The image is a table showing supported data formats for various AWS services like Amazon S3, Amazon Athena, Amazon Redshift, and AWS Lake Formation. Each service lists the specific formats it supports, such as CSV, JSON, Parquet, and ORC.

​
Processing and streaming on AWS

Match processing tools to data format and latency requirements:
  • AWS Glue: Serverless ETL; supports CSV, JSON, Parquet, Avro, and ORC. Good for schema discovery and format conversions.
    • Use Glue jobs or Glue Studio for converting CSV/JSON to Parquet for analytics.
  • Amazon EMR (Spark/Hadoop): Read/write Parquet, ORC, Avro, sequence files. Ideal for large-scale transformations and feature engineering.
  • Amazon Kinesis: Ingest streaming events (often JSON or binary protobufs); used for near-real-time use cases.
  • AWS Lambda: Receives JSON payloads (and base64-encoded binary data). Suitable for lightweight transformations and event-driven processing.
Best practices:
  • For streaming ingestion, keep messages small and schema-stable (e.g., Avro/Protobuf).
  • Convert raw event streams to columnar formats in downstream storage for efficient analytics.
The image is a table listing AWS services and their supported data formats for data processing and streaming, including AWS Glue, AWS EMR, Amazon Kinesis, and AWS Lambda with formats like CSV, JSON, and AVRO.

​
ML services and specialized formats

Common AWS ML services and their preferred input types:
  • Amazon SageMaker: Supports CSV, Parquet, TFRecord, images, audio, and pickled artifacts for training and hosting. Use SageMaker processing jobs to convert formats at scale.
  • Amazon Transcribe: Accepts WAV, MP3, FLAC, and MP4 for speech-to-text.
  • Amazon Rekognition: Works with standard image and video formats (JPEG, PNG, MP4) for computer vision tasks.
When designing ML pipelines on AWS:
  • Use columnar compressed formats (Parquet/ORC) for analytics and feature stores.
  • Use binary ML-native formats (TFRecord/RecordIO) to maximize throughput for distributed training.
  • Keep raw media in their native formats (JPEG/WAV/MP4) and extract features or convert to optimized formats only when necessary.

​
Summary checklist

  • Use Parquet/ORC for analytics and feature engineering—columnar layout reduces I/O and improves query performance.
  • Use TFRecord/RecordIO for high-throughput, sharded training pipelines.
  • Keep media in standard containers (JPEG/PNG, WAV/FLAC, MP4) and preprocess with specialized libraries.
  • Avoid insecure serialization formats (untrusted pickles) when sharing datasets across teams or services.
  • Optimize file sizing, partitioning, and compression to balance read performance and storage cost.

Watch Video