Passing Data Between Components

In this lesson we’ll cover reliable ways to pass data between components in a pipeline. Typical pipelines include stages such as: fetch data → process data → train model → evaluate model. Each stage must hand off the right information to the next — for example, fetch provides raw data to process, process provides prepared data to training, and training produces a model for evaluation.

A simple flowchart titled "Passing Data" with four boxes: Fetch Data → Process Data → Train Model → Evaluate Model. Arrows between the boxes are labeled "Unprocessed Data", "Processed Data", and "Model" to show the data flow.

There are two common approaches to pass data between components: parameters and artifacts. Use parameters for small scalar values and artifacts for file-based or large outputs. The following sections explain both methods with examples and best practices.

Overview: Parameters vs Artifacts

Mechanism	Typical Use Cases	Passing Semantics	When to Use
Parameters	Hyperparameters, thresholds, small numeric or string values	Passed by value (inline)	Use for scalars, flags, or small JSON-serializable objects
Artifacts	Datasets, trained models, reports, large JSON/CSV files	Passed by reference (object storage paths)	Use for large files, binary objects, and reproducible artifacts

How to pass data — Parameters

Parameters are the simplest method. They pass scalar values by value between components. Supported types commonly include integers, floats, strings, booleans, and small JSON-serializable objects. Typical parameter uses are hyperparameters, evaluation thresholds, or small metadata.

Limitations:

Size limits: parameters are not appropriate for large datasets or files.
Not suited for binary objects or large structured outputs — use artifacts in those cases.

Example — single scalar parameter Return the scalar from the producing component and consume it as an input in the next component. For a single return value you can access the value with .output on the producing task.

from kfp.dsl import component, pipeline

@component
def step1() -> int:
    stats = 400
    return stats

@component
def step2(stats: int) -> None:
    # Pretend training logic
    print(f"stats: {stats}")

@pipeline()
def ml_pipeline():
    s1 = step1()
    # Use the single scalar return via s1.output
    s2 = step2(stats=s1.output)

Example — multiple named outputs (NamedTuple) When a component needs to return multiple scalar values, use a NamedTuple (or other typed multi-output). The component returns values in the declared order, and the pipeline can read them from the producing task’s .outputs dictionary.

from typing import NamedTuple
from kfp.dsl import component, pipeline

Step1Outputs = NamedTuple("Step1Outputs", [("data", int), ("stats", int)])

@component
def step1() -> Step1Outputs:
    data = 20
    stats = 400
    return (data, stats)

@component
def step2(data: int, stats: int) -> None:
    # Pretend training logic
    print(f"data: {data}, stats: {stats}")

@pipeline()
def ml_pipeline():
    s1 = step1()
    # Use named outputs via s1.outputs["<name>"]
    s2 = step2(data=s1.outputs["data"], stats=s1.outputs["stats"])

Note: Single-return components expose .output while multi-output components expose .outputs["name"].

How to pass data — Artifacts

Artifacts are file-based outputs stored in object storage (for example, MinIO in many Kubeflow setups or Amazon S3). Artifacts are passed by reference: the producing component writes files to an artifact path and the consuming component reads from that path. This makes artifacts ideal for large datasets, trained models, evaluation results, and reproducible outputs.

A presentation slide titled "Passing Data – Artifacts" with two notes: "File-based output stored in object storage" and "Passed by reference, not by value." Below are MinIO and S3-style object storage icons and a KodeKloud copyright.

Common artifact types include Dataset, Model, Metrics, HTML, and a generic Artifact type:

A presentation slide titled "Passing Data – Artifacts" showing a dotted box labeled "Common Artifact Types" with five rounded boxes: Dataset, Model, Metrics, HTML, and Artifact (generic). A small "© Copyright KodeKloud" appears in the bottom corner.

Artifact usage pattern:

The producing component declares an Output[...] typed artifact (for example, Output[Dataset]) and writes files to the provided .path.
The consuming component declares an Input[...] typed artifact (for example, Input[Dataset]) and reads from .path.
The pipeline wires producer to consumer by passing producer_task.outputs["artifact_name"] into the consumer argument.

Example pipeline using artifacts This example demonstrates retrieving raw data, preprocessing it, training a model, and saving the trained model as an artifact. Components use Output and Input artifact typing and read/write files via the .path property.

from kfp.dsl import Input, Output, Dataset, Model, component, pipeline

@component(packages_to_install=["pandas"])
def retrieve_data(output_data: Output[Dataset]):
    import pandas as pd
    df = pd.DataFrame({
        "x": [1, 2, 3, 4, 5],
        "y": [2, 4, 6, 8, 10]
    })
    # Save CSV to the artifact path provided by the runtime
    df.to_csv(output_data.path, index=False)

@component(packages_to_install=["pandas"])
def preprocess_data(
    raw_data: Input[Dataset],
    processed_data: Output[Dataset]
):
    import pandas as pd

    # Read the CSV from the artifact path
    df = pd.read_csv(raw_data.path)

    # Simple "feature engineering"
    df["x_squared"] = df["x"] ** 2

    # Write processed CSV back to the artifact path
    df.to_csv(processed_data.path, index=False)

@component(packages_to_install=["pandas", "scikit-learn", "cloudpickle"])
def train_model(
    training_data: Input[Dataset],
    model: Output[Model]
):
    import pandas as pd
    from sklearn.linear_model import LinearRegression
    import cloudpickle

    # Read training data
    df = pd.read_csv(training_data.path)
    X = df[["x", "x_squared"]]
    y = df["y"]

    # Train a simple model
    reg = LinearRegression()
    reg.fit(X, y)

    # Save the trained model to the artifact path
    with open(model.path, "wb") as f:
        cloudpickle.dump(reg, f)

@component(packages_to_install=["cloudpickle"])
def evaluate_model(model: Input[Model]):
    import cloudpickle
    # Load model from artifact path and print a basic message
    with open(model.path, "rb") as f:
        reg = cloudpickle.load(f)
    print("Loaded model for evaluation:", reg)

@pipeline(name="dummy-artifact-pipeline")
def dummy_pipeline():
    data_task = retrieve_data()
    preprocess_task = preprocess_data(raw_data=data_task.outputs["output_data"])
    train_task = train_model(training_data=preprocess_task.outputs["processed_data"])
    evaluate_task = evaluate_model(model=train_task.outputs["model"])

Notes on wiring and names:

The output parameter name in the component signature (for example, output_data: Output[Dataset]) becomes the key in the producing task’s .outputs dictionary.
To read or write files for an artifact, use the provided .path attribute (for example, raw_data.path or model.path).
Artifacts are stored in the cluster’s object storage (e.g., MinIO), and consumers are given the path reference to access them.

Do not pass large datasets, binary files, or trained models as parameters. Parameters are size-limited and intended for scalars or small JSON-serializable content only.

Use parameters for small scalar values (hyperparameters, flags, small JSON). Use artifacts for large files, datasets, models, and results — artifacts are stored in object storage and passed by reference.

Quick Best-Practices

Use parameters for hyperparameters, simple thresholds, or small metadata.
Use artifacts for datasets, model files, images, or any outputs too large to inline.
Name artifact outputs clearly in component signatures to create predictable keys in .outputs.
Prefer artifact-based workflows for reproducibility and when working across clusters.

Course Introduction

Fundamentals of Kubeflow

Working With Kubeflow

KServe and Katib

Profiles and Multi-Tenancy

Bringing It All Together

Passing Data Between Components

Overview: Parameters vs Artifacts

How to pass data — Parameters

How to pass data — Artifacts

Quick Best-Practices

Links and references

Watch Video

​Overview: Parameters vs Artifacts

​How to pass data — Parameters

​How to pass data — Artifacts

​Quick Best-Practices

​Links and references

Watch Video

Overview: Parameters vs Artifacts

How to pass data — Parameters

How to pass data — Artifacts

Quick Best-Practices

Links and references