Small to Medium Datasets Data Transformation Pandas Polars

Hello and welcome back. In this lesson we cover practical, efficient techniques for transforming small-to-medium datasets. These approaches are ideal for tasks such as cleaning sales records for a mid-sized retailer, preparing survey results for analysis, or building preprocessing steps inside an ETL pipeline. We’ll focus on clear, repeatable steps you can apply in production MLOps workflows using pandas or Polars.

A slide titled "Data Transformation" with a funnel graphic on the left where folder icons enter the top and a database icon comes out the bottom. The design uses a dark blue background with teal outlines and a small "© Copyright KodeKloud" in the corner.

Two Python libraries are especially useful for these dataset sizes: pandas and Polars. Each has strengths:

pandas: ubiquitous, feature-rich, and integrates with the broader PyData ecosystem (plotting, I/O, modeling).
Polars: designed for speed and parallelism, often achieving faster queries and lower memory usage for equivalent operations.

A dark-blue infographic titled "Pandas vs Polaris" shows two circles — Pandas on the left ("known for flexibility and versatility") and Polaris on the right ("optimized for performance and efficiency") — with a "VS" icon between them. A small note below references "cleaning messy customer feedback data for insights" alongside a cleaning icon.

Pandas is great for breadth of functionality and ecosystem integration (plotting, modeling, I/O). Polars is worth considering when you need faster query performance or lower memory usage for the same operations.

How this fits into MLOps

These libraries are commonly used inside ETL and preprocessing stages. Many production pipelines operate on small-to-medium datasets that are best handled with pandas or Polars rather than distributed systems.
Use these tools for repeatable, testable transformations before model training, validation, or reporting.

A presentation slide titled "Data Cleaning With Pandas" that asks "How does this fill into MLOps?" and shows two outcomes: "ETL pipelines and data transformation" on the left and "Smaller datasets, easily transformed" on the right with directional arrows.

Common preprocessing sequence We recommend following a consistent sequence when preparing small-to-medium datasets. This helps keep pipelines predictable and easy to test:

Handle missing values
Remove duplicates
Standardize data types
Filter, sort, and perform row/column transformations
Aggregate for insights
Merge / join datasets

The rest of this document shows patterns and code examples (with pandas) for each step. The same conceptual steps apply in Polars — consult Polars docs for syntax differences. Key operation mappings (quick reference)

Operation	pandas	Purpose / Example
Missing values	`dropna`, `fillna`	Remove or impute missing cells
Duplicate removal	`drop_duplicates`	Ensure unique records per key
Numeric conversion	`pd.to_numeric(..., errors="coerce")`	Coerce invalid values to `NaN` then fill
Date conversion	`pd.to_datetime(..., errors="coerce")`	Normalize date formats
Filter / Sort	boolean indexing, `sort_values`	Select rows, order data
Aggregation	`groupby(...).agg(...)`	Summarize for reporting
Merge / Join	`pd.merge`	Combine tables on keys

Detailed patterns and examples

Missing values

Missing or malformed values can skew analyses and break downstream code. Prefer explicit assignment over using inplace=True to keep transformations clear and avoid deprecation surprises. Example patterns:

import pandas as pd

# Example dataframe
df = pd.DataFrame({
    "id": [1, 2, 3, 4],
    "age": ["25", None, "30", "27"],
    "sales": [100.0, 200.0, None, 150.0]
})

# Option 1: remove rows with any missing values
df_no_na = df.dropna()

# Option 2: fill missing values with defaults (carefully; choose column-level rules)
df_filled = df.fillna({"age": "0", "sales": 0.0})

For numeric conversion, coerce invalid strings to NaN and then decide whether to drop or impute:

df["age"] = pd.to_numeric(df["age"], errors="coerce")  # invalid -> NaN
df["age"] = df["age"].fillna(0).astype(int)            # example imputation

Avoid relying on inplace=True. Assigning the result back to a variable (df = df.dropna()) improves readability, reduces surprises, and is compatible with future pandas behavior.

Removing duplicates

Duplicate rows can distort counts and aggregates. Use drop_duplicates() and specify a subset of columns if you only want uniqueness on specific keys.

# Drop exact duplicate rows
df_unique = df.drop_duplicates()

# Drop duplicates based on specific columns (e.g., primary key)
df_unique_by_id = df.drop_duplicates(subset=["id"])

Standardizing data types

Consistent data types are crucial for correct math, joins, and model input. Common helpers are pd.to_numeric, pd.to_datetime, and astype.

# Convert sales to numeric (coerce errors), then fill or handle NaNs
df["sales"] = pd.to_numeric(df["sales"], errors="coerce").fillna(0.0)

# Convert a date-like column to datetime
# df["order_date"] = pd.to_datetime(df["order_date"], errors="coerce")

Filtering and sorting

Use boolean indexing to filter rows and sort_values to order results.

# Filter transactions for a product category
df_filtered = df[df["product_category"] == "Electronics"]

# Sort by date or sales
df_sorted = df.sort_values(by="sales", ascending=False)

Aggregation (groupby and agg)

Aggregate for reporting or feature engineering with groupby and agg. Reset the index after aggregation to keep a flat DataFrame.

# Group sales by region and compute total sales
df_grouped = df.groupby("region").agg({"sales": "sum"}).reset_index()

Merging and joining datasets

Use pd.merge to combine tables — this is conceptually equivalent to SQL joins.

# Example merge of customer and sales tables on 'id'
df_merged = pd.merge(customers_df, sales_df, on="id", how="inner")

Consolidated example pipeline

The following concise example demonstrates a complete flow: cleaning → conversion → transformation → aggregation → join. This is a practical pattern you can adapt to ETL jobs and MLOps preprocessing steps.

import pandas as pd

# Sample data
customers = pd.DataFrame({
    "id": [1, 2, 3],
    "name": ["Alice", "Bob", "Charlie"],
    "age": ["25", "thirty", None]
})

sales = pd.DataFrame({
    "id": [1, 2, 2, 3, 4],
    "region": ["North", "South", "South", "East", "West"],
    "sales": [100, 200, 150, None, 50]
})

# 1) Handle missing values — coerce invalid sales to NaN then fill
sales["sales"] = pd.to_numeric(sales["sales"], errors="coerce").fillna(0.0)

# 2) Remove duplicates (for example, duplicate transaction rows)
sales = sales.drop_duplicates()

# 3) Standardize data types (coerce ages, fill, and convert)
customers["age"] = pd.to_numeric(customers["age"], errors="coerce").fillna(0).astype(int)

# 4) Transformations: filter sales to known customers and sort by sales
sales = sales[sales["id"].isin(customers["id"])]
sales = sales.sort_values(by="sales", ascending=False)

# 5) Aggregate: total sales by region
sales_by_region = sales.groupby("region").agg({"sales": "sum"}).reset_index()

# 6) Merge: attach customer info to sales (left-join to preserve sales)
sales_with_customers = pd.merge(sales, customers, on="id", how="left")

print(sales_by_region)
print(sales_with_customers.head())

Typical transformation flow (conceptual)

Raw data ingestion (CSV, Parquet, DB, API)
Missing-value handling (drop or impute per-column)
Data type standardization (numeric, datetime, categorical)
Data transformation (filter, compute features, normalize)
Aggregation (groupby + agg) for summaries or feature creation
Merge/join to combine related datasets

References and further reading

pandas documentation: https://pandas.pydata.org/
Polars documentation: https://www.pola.rs/
MLOps and ETL best practices: consider idempotent transformations, unit tests for preprocessing logic, and small, testable pipeline steps.

With these patterns, you have a compact, practical toolkit to clean and transform small-to-medium datasets using pandas or Polars. Apply these steps in your ETL pipelines and MLOps workflows to produce reliable, production-ready datasets.

Introduction to MLOps

Data Collection and Preparation

Model Development and Training

Model Deployment and Serving

Automating Insurance Claim Reviews with MLflow and BentoML

Data Security and Governance

Sneak Peek into AWS SageMaker

Small to Medium Datasets Data Transformation Pandas Polars

Missing values

Removing duplicates

Standardizing data types

Filtering and sorting

Aggregation (groupby and agg)

Merging and joining datasets

Consolidated example pipeline

Typical transformation flow (conceptual)

Watch Video

​Missing values

​Removing duplicates

​Standardizing data types

​Filtering and sorting

​Aggregation (groupby and agg)

​Merging and joining datasets

​Consolidated example pipeline

​Typical transformation flow (conceptual)

Watch Video

Missing values

Removing duplicates

Standardizing data types

Filtering and sorting

Aggregation (groupby and agg)

Merging and joining datasets

Consolidated example pipeline

Typical transformation flow (conceptual)