> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Small to Medium Datasets Data Transformation Pandas Polars

> Practical techniques for cleaning and transforming small to medium datasets with pandas and Polars for ETL and MLOps, covering missing values, duplicates, type standardization, aggregation, and joins

Hello and welcome back.

In this lesson we cover practical, efficient techniques for transforming small-to-medium datasets. These approaches are ideal for tasks such as cleaning sales records for a mid-sized retailer, preparing survey results for analysis, or building preprocessing steps inside an ETL pipeline. We'll focus on clear, repeatable steps you can apply in production MLOps workflows using pandas or Polars.

![A slide titled "Data Transformation" with a funnel graphic on the left where folder icons enter the top and a database icon comes out the bottom. The design uses a dark blue background with teal outlines and a small "© Copyright KodeKloud" in the corner.](https://mintcdn.com/kodekloud-c4ac6d9a/fpyZwL72OkFO0HuG/images/Fundamentals-of-MLOps/Data-Collection-and-Preparation/Small-to-Medium-Datasets-Data-Transformation-Pandas-Polars/data-transformation-funnel-folders-database.jpg?fit=max\&auto=format\&n=fpyZwL72OkFO0HuG\&q=85\&s=32e662668a9b6c62c49df5d69ff8ca1e)

Two Python libraries are especially useful for these dataset sizes: pandas and Polars. Each has strengths:

* pandas: ubiquitous, feature-rich, and integrates with the broader PyData ecosystem (plotting, I/O, modeling).
* Polars: designed for speed and parallelism, often achieving faster queries and lower memory usage for equivalent operations.

![A dark-blue infographic titled "Pandas vs Polaris" shows two circles — Pandas on the left ("known for flexibility and versatility") and Polaris on the right ("optimized for performance and efficiency") — with a "VS" icon between them. A small note below references "cleaning messy customer feedback data for insights" alongside a cleaning icon.](https://mintcdn.com/kodekloud-c4ac6d9a/fpyZwL72OkFO0HuG/images/Fundamentals-of-MLOps/Data-Collection-and-Preparation/Small-to-Medium-Datasets-Data-Transformation-Pandas-Polars/pandas-vs-polaris-data-cleaning-infographic.jpg?fit=max\&auto=format\&n=fpyZwL72OkFO0HuG\&q=85\&s=a790e75ab9b8c1c51fa1814d190500ff)

Pandas is great for breadth of functionality and ecosystem integration (plotting, modeling, I/O). Polars is worth considering when you need faster query performance or lower memory usage for the same operations.

How this fits into MLOps

* These libraries are commonly used inside ETL and preprocessing stages. Many production pipelines operate on small-to-medium datasets that are best handled with pandas or Polars rather than distributed systems.
* Use these tools for repeatable, testable transformations before model training, validation, or reporting.

![A presentation slide titled "Data Cleaning With Pandas" that asks "How does this fill into MLOps?" and shows two outcomes: "ETL pipelines and data transformation" on the left and "Smaller datasets, easily transformed" on the right with directional arrows.](https://mintcdn.com/kodekloud-c4ac6d9a/fpyZwL72OkFO0HuG/images/Fundamentals-of-MLOps/Data-Collection-and-Preparation/Small-to-Medium-Datasets-Data-Transformation-Pandas-Polars/pandas-cleaning-mlops-etl-small-datasets.jpg?fit=max\&auto=format\&n=fpyZwL72OkFO0HuG\&q=85\&s=b2d771c59e993dab53032e2f680aeb93)

Common preprocessing sequence
We recommend following a consistent sequence when preparing small-to-medium datasets. This helps keep pipelines predictable and easy to test:

1. Handle missing values
2. Remove duplicates
3. Standardize data types
4. Filter, sort, and perform row/column transformations
5. Aggregate for insights
6. Merge / join datasets

The rest of this document shows patterns and code examples (with pandas) for each step. The same conceptual steps apply in Polars — consult Polars docs for syntax differences.

Key operation mappings (quick reference)

| Operation | pandas | Purpose / Example |
| -: | :-: | - |
| Missing values | `dropna`, `fillna` | Remove or impute missing cells |
| Duplicate removal | `drop_duplicates` | Ensure unique records per key |
| Numeric conversion | `pd.to_numeric(..., errors="coerce")` | Coerce invalid values to `NaN` then fill |
| Date conversion | `pd.to_datetime(..., errors="coerce")` | Normalize date formats |
| Filter / Sort | boolean indexing, `sort_values` | Select rows, order data |
| Aggregation | `groupby(...).agg(...)` | Summarize for reporting |
| Merge / Join | `pd.merge` | Combine tables on keys |

Detailed patterns and examples

## Missing values

Missing or malformed values can skew analyses and break downstream code. Prefer explicit assignment over using `inplace=True` to keep transformations clear and avoid deprecation surprises.

Example patterns:

```python theme={null}
import pandas as pd

# Example dataframe
df = pd.DataFrame({
    "id": [1, 2, 3, 4],
    "age": ["25", None, "30", "27"],
    "sales": [100.0, 200.0, None, 150.0]
})

# Option 1: remove rows with any missing values
df_no_na = df.dropna()

# Option 2: fill missing values with defaults (carefully; choose column-level rules)
df_filled = df.fillna({"age": "0", "sales": 0.0})
```

For numeric conversion, coerce invalid strings to `NaN` and then decide whether to drop or impute:

```python theme={null}
df["age"] = pd.to_numeric(df["age"], errors="coerce")  # invalid -> NaN
df["age"] = df["age"].fillna(0).astype(int)            # example imputation
```

Avoid relying on `inplace=True`. Assigning the result back to a variable (`df = df.dropna()`) improves readability, reduces surprises, and is compatible with future pandas behavior.

## Removing duplicates

Duplicate rows can distort counts and aggregates. Use `drop_duplicates()` and specify a `subset` of columns if you only want uniqueness on specific keys.

```python theme={null}
# Drop exact duplicate rows
df_unique = df.drop_duplicates()

# Drop duplicates based on specific columns (e.g., primary key)
df_unique_by_id = df.drop_duplicates(subset=["id"])
```

## Standardizing data types

Consistent data types are crucial for correct math, joins, and model input. Common helpers are `pd.to_numeric`, `pd.to_datetime`, and `astype`.

```python theme={null}
# Convert sales to numeric (coerce errors), then fill or handle NaNs
df["sales"] = pd.to_numeric(df["sales"], errors="coerce").fillna(0.0)

# Convert a date-like column to datetime
# df["order_date"] = pd.to_datetime(df["order_date"], errors="coerce")
```

## Filtering and sorting

Use boolean indexing to filter rows and `sort_values` to order results.

```python theme={null}
# Filter transactions for a product category
df_filtered = df[df["product_category"] == "Electronics"]

# Sort by date or sales
df_sorted = df.sort_values(by="sales", ascending=False)
```

## Aggregation (groupby and agg)

Aggregate for reporting or feature engineering with `groupby` and `agg`. Reset the index after aggregation to keep a flat DataFrame.

```python theme={null}
# Group sales by region and compute total sales
df_grouped = df.groupby("region").agg({"sales": "sum"}).reset_index()
```

## Merging and joining datasets

Use `pd.merge` to combine tables — this is conceptually equivalent to SQL joins.

```python theme={null}
# Example merge of customer and sales tables on 'id'
df_merged = pd.merge(customers_df, sales_df, on="id", how="inner")
```

## Consolidated example pipeline

The following concise example demonstrates a complete flow: cleaning → conversion → transformation → aggregation → join. This is a practical pattern you can adapt to ETL jobs and MLOps preprocessing steps.

```python theme={null}
import pandas as pd

# Sample data
customers = pd.DataFrame({
    "id": [1, 2, 3],
    "name": ["Alice", "Bob", "Charlie"],
    "age": ["25", "thirty", None]
})

sales = pd.DataFrame({
    "id": [1, 2, 2, 3, 4],
    "region": ["North", "South", "South", "East", "West"],
    "sales": [100, 200, 150, None, 50]
})

# 1) Handle missing values — coerce invalid sales to NaN then fill
sales["sales"] = pd.to_numeric(sales["sales"], errors="coerce").fillna(0.0)

# 2) Remove duplicates (for example, duplicate transaction rows)
sales = sales.drop_duplicates()

# 3) Standardize data types (coerce ages, fill, and convert)
customers["age"] = pd.to_numeric(customers["age"], errors="coerce").fillna(0).astype(int)

# 4) Transformations: filter sales to known customers and sort by sales
sales = sales[sales["id"].isin(customers["id"])]
sales = sales.sort_values(by="sales", ascending=False)

# 5) Aggregate: total sales by region
sales_by_region = sales.groupby("region").agg({"sales": "sum"}).reset_index()

# 6) Merge: attach customer info to sales (left-join to preserve sales)
sales_with_customers = pd.merge(sales, customers, on="id", how="left")

print(sales_by_region)
print(sales_with_customers.head())
```

## Typical transformation flow (conceptual)

* Raw data ingestion (CSV, Parquet, DB, API)
* Missing-value handling (drop or impute per-column)
* Data type standardization (numeric, datetime, categorical)
* Data transformation (filter, compute features, normalize)
* Aggregation (groupby + agg) for summaries or feature creation
* Merge/join to combine related datasets

References and further reading

* pandas documentation: [https://pandas.pydata.org/](https://pandas.pydata.org/)
* Polars documentation: [https://www.pola.rs/](https://www.pola.rs/)
* MLOps and ETL best practices: consider idempotent transformations, unit tests for preprocessing logic, and small, testable pipeline steps.

With these patterns, you have a compact, practical toolkit to clean and transform small-to-medium datasets using pandas or Polars. Apply these steps in your ETL pipelines and MLOps workflows to produce reliable, production-ready datasets.

> **[Watch Video](https://learn.kodekloud.com/user/courses/fundamentals-of-mlops/module/d72a3430-8b54-48d6-89ad-6a5f8b74f4ab/lesson/9dce8d6d-fed1-4e76-a7b8-f4bca2c5d482)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.