Ingesting Data - KodeKloud

Let’s say you’ve just joined a small coffee startup as a trainee data engineer. Roastflow sells beans online — multiple roasts, grind types, and bundles — through a Shopify store. They run paid ads on Instagram and TikTok, but no one is reliably tracking which campaigns actually drive purchases. It’s late summer, and leadership has given you until the end of the year to build a data pipeline that provides reliable monthly sales insight. Before you can answer those business questions, you need to bring data together cleanly, consistently, and without duplication — and that all starts with ingestion.

The image includes a diagram illustrating a data pipeline process with various elements like calendar months, monthly sales graph, and a section labeled "Clean." A person is also present, gesturing as if explaining the concept.

Ingestion is the first step in a data pipeline: capture, validate, and land raw data so downstream analytics can run reliably. By the end of this lesson you will be able to:

Differentiate between batch and streaming data ingestion.
Describe the three core principles of robust ingestion: idempotency, schema-awareness, and observability.
Ingest and clean order data using Jupyter, Python, Pandas, and Matplotlib.

The image shows a person wearing a "KodeKloud" shirt next to a presentation that includes a cartoon cat and text about differentiating between batch and streaming data ingestion.

First, distinguish how data is generated from how it is processed. Generation and processing are independent choices. At Roastflow, some sources are bursty: monthly order exports from Shopify, or ad-spend spreadsheets dropped in Google Drive by marketing. Other sources are continuous: website clicks, pageviews, and real‑time events.

The image illustrates various data metrics such as monthly orders, ad spend, and website clicks associated with a coffee shop graphic. A person is standing to the right wearing a KodeKloud shirt.

Even when generation is continuous, we can choose a processing strategy: collect and process in batches or process events as they arrive.

The image features a person wearing a "KodeKloud" T-shirt alongside illustrations labeled "Monthly Orders," "Ad Spend," "Website Click," and "Pageviews," depicting various digital marketing and online activities.

Two primary ingestion approaches:

Batch processing — accumulate data and process it in scheduled chunks (hourly, daily, weekly).
Stream processing — process individual events continuously, or in very small micro-batches.

Batch is often simpler and reliable for routine analytics (e.g., daily sales reports). Streaming suits near real-time needs (live dashboards, user tracking), but is typically more complex to build and operate.

In production, many “real-time” pipelines still use batching for efficiency — for example, accumulating events for one-minute micro-batches. Understanding the difference between how data is generated (burst vs continuous) and how it’s processed (batch vs stream) is key to designing the right pipeline for Roastflow. Analogy: batch = reservoir released on a schedule; stream = water flowing through a turbine and processed immediately. Streaming provides immediate insights but increases operational complexity; batch is easier to manage and scales well for reporting.

The image contrasts batch and streaming data processing, highlighting features like reliability versus instant updates, and routine analytics versus management difficulty. A person wearing a "KodeKloud" shirt is gesturing in front of the comparison.

As a data engineer you’ll work with both approaches depending on the use case. Core properties of a robust ingestion step A dependable ingestion process should meet three requirements:

Idempotent — safe to run multiple times without producing duplicates.
Schema-aware — validates the structure and basic quality of data.
Observable — logs enough metadata to trace, debug, and replay.

Let’s unpack each. Idempotency Idempotency ensures repeating the same ingestion with the same input does not change the final state. Practically this means:

No unintended duplicates.
Existing records are not corrupted by repeated runs.
Safe replays and retries without manual cleanup.

Idempotency is critical for reliable pipelines. Without it, retries (common after failures) can introduce duplicate revenue or user records, skewing analytics and costing money.

Schemas A schema defines expectations about data shape and basic quality checks. Apply validations at multiple levels:

Schema Level	Typical Checks
Column-level	Are expected columns present? Types correct? Required fields non-null?
Row-level	Business rules per row (quantities >= 0, timestamps valid, email formats)
Table-level	Primary key uniqueness, foreign key consistency across tables

The image shows a presentation slide on database schemas with levels (Column, Row, Table) and related questions, alongside a person wearing a KodeKloud shirt.

You can extend these checks with cross-table constraints and domain-specific rules, but column/row/table checks are the backbone of schema-aware ingestion.

The image features a section labeled "02 Schemas" with categories such as Column Level, Row Level, Table Level, Cross-Table Relationships, and Business-Specific Rules on the left, and a person standing on the right wearing a KodeKloud shirt.

Observability Observability means recording metadata about each ingestion action: file names, timestamps, row counts, success/failure status, and error details. Good logs let you audit, investigate, and replay data when necessary.

The image features a presentation slide on 'Observability' with a list related to files or processes, and a person standing to the right wearing a KodeKloud t-shirt.

Quiz — quick comprehension check Which two statements are true? A. Batch data is ingested continuously as soon as it’s generated.
B. Streaming data is ideal for real-time user tracking or live dashboards.
C. A good ingestion step should be schema-aware, observable, and idempotent.
D. Idempotency means applying the same operation multiple times produces the same result before storing the data. Correct answers: B and C.

B is true: streaming supports real-time use cases like live dashboards.
C is true: ingestion should avoid duplicates, validate structure, and log sufficiently.
A is false: continuous ingestion describes streaming, not batch.
D is misleading: idempotency ensures the same final state after repeated runs, not a description of pre-storage transformation.

Quick recap

Batch: process data in scheduled chunks (e.g., monthly exports, uploaded spreadsheets).
Stream: process data continuously (e.g., real-time click tracking).
Batch is often simpler and reliable for routine analytics; streaming gives immediate insights but adds complexity.
Ingestion must be idempotent, schema-aware, and observable.

The image describes the differences between batch processing and stream processing, with a person standing next to the text boxes on a black background.

The image features a person wearing a TekKloud t-shirt and two informational cards about ingestion, focusing on idempotency and schema-awareness. There is also an illustration of a person working on a laptop.

Practical demo: Jupyter + Pandas ingestion pattern You’ll build a simple but realistic ingestion in Jupyter to illustrate the concepts without being tied to specific cloud services. The micro-pipeline will:

Read messy order CSV exports.
Validate columns and basic row rules.
Deduplicate (idempotent behavior) based on a unique key.
Log ingestion metadata for observability.
Produce a cleaned dataset ready for analytics (e.g., top products by revenue).

Example Pandas pattern (conceptual):

import pandas as pd
from pathlib import Path
import json
import hashlib
from datetime import datetime

def ingest_file(path, existing_keys=set()):
    df = pd.read_csv(path)
    # Column-level check
    required_columns = {"order_id", "product_id", "quantity", "price", "order_ts"}
    missing = required_columns - set(df.columns)
    if missing:
        raise ValueError(f"Missing columns: {missing}")

    # Row-level check
    df = df[df["quantity"] >= 0]
    df = df.dropna(subset=["order_id", "product_id", "order_ts"])

    # Idempotent dedupe: keep rows not already seen
    df["unique_key"] = df["order_id"].astype(str) + "-" + df["product_id"].astype(str)
    df = df[~df["unique_key"].isin(existing_keys)]

    # Observability: write a small log for this ingestion
    log = {
        "file": str(path),
        "timestamp": datetime.utcnow().isoformat(),
        "rows_in": int(df.shape[0]),
        "rows_out": int(df.shape[0]),
        "status": "success"
    }
    Path("ingest_logs").mkdir(exist_ok=True)
    with open(Path("ingest_logs") / f"{Path(path).stem}.json", "w") as f:
        json.dump(log, f)

    return df

This pattern:

Validates schema basics before transforming.
Filters invalid rows.
Removes duplicates using a stable, unique key (idempotency).
Emits a minimal log for observability.

Use a stable unique key (e.g., order_id + product_id) for deduplication. In production, consider deterministic hashing and persistent storage of seen keys (or de-duplication via database constraints) for stronger guarantees.

Next steps and references

Start by cataloging your sources (Shopify exports, ad-spend sheets, website events) and classify each as batch or stream.
Design ingestion jobs with idempotency in mind: deterministic keys, safe upserts, or dedupe strategies.
Implement schema checks early and fail fast on critical errors so dirty data doesn’t propagate.
Capture minimal but consistent ingestion logs for every run to enable replay and debugging.

Watch Video