Workflow Orchestration DAGs Steps and Event Driven Automation

Controllers perform continuous reconciliation, but not every platform task is continuous. Some tasks need to run once or as a finite multi-step process — for example, a one-off data migration or an environment provisioning workflow. Those use cases go beyond the simple controller pattern, and that’s where workflow orchestration shines. What you’ll learn in this lesson:

Why shell scripts and naive CI pipelines fail for complex automation.
How to author Argo Workflows using Steps (list-of-lists) and DAG patterns.
How to create reusable WorkflowTemplates with parameters.
How to pass artifacts (files) between workflow steps.
How to trigger workflows from external events using Argo Events.

A real-world example: a platform team had a 50-line Bash script to provision a development environment. The script created a namespace, applied RBAC, deployed monitoring agents, configured Ingress, and set up DNS. It executed everything sequentially even though many steps were independent — and then it failed.

A horizontal flowchart titled "From Sequential Script to Resilient Workflow" showing a seven-step sequential deployment process (Create Namespace, Apply RBAC, Deploy Monitoring, Configure Ingress, Setup DNS, Sequential Wait, Failure). The diagram notes this "BEFORE (Sequential Bash)" workflow takes about 15 minutes and ends in failure with no retries.

Steps three and four in that script could have run in parallel, cutting runtime from ~15 minutes to ~8 minutes. When the final step failed because of a transient DNS timeout, the script exited with no retries. Engineers had to undo prior steps manually and start over — there was no resume capability. Why scripts and simple pipelines aren’t enough The diagram below highlights the recurring gaps in shell scripts, CI jobs, and cron-based automation.

A presentation slide titled "Scripts and Pipelines Hit Their Limits – What's Missing" listing four problems: No Dependency Graphs, No Parallel Execution, No Retry Logic, and No Observability, each with a short explanatory note. The items are shown as numbered orange gradient boxes with brief gray captions underneath.

Common limitations

Limitation	Impact
No dependency graph	Hard to express “run C only after A and B succeed”
No parallel execution	Independent tasks run sequentially and waste time
No retry logic	Transient failures abort the whole run
Poor observability	Limited visibility into which step failed or runtime metrics

For complex platform automation you need a workflow engine — not more shell scripts. How Argo Workflows helps Argo Workflows runs each workflow step as a Kubernetes pod, so every step benefits from Kubernetes features: resource limits, service accounts, RBAC, secrets, node selectors/affinity, and priority classes. Argo provides a focused set of capabilities for reliable platform automation:

DAGs and Steps: express dependencies and control parallelism or sequence.
Retries and timeouts: automatic retries, backoff, and per-step/workflow timeouts.
Artifacts: share files between steps using S3, GCS, MinIO, or other backends.
UI and CLI: visual workflow graphs, per-step logs and timings; submit and monitor from the CLI.

Workflows are Kubernetes custom resources (YAML) that you can store in Git — declarative and portable. Two ways to express execution order

Steps (list-of-lists)

The outer list represents sequence.
Each inner list contains tasks that run in parallel.
Best for mostly-linear pipelines with occasional parallelism.

Correct example steps layout (build → parallel tests → deploy):

steps:
  - - name: build
      template: build

  - - name: test-a
      template: test-a
    - name: test-b
      template: test-b

  - - name: deploy
      template: deploy

DAGs (Directed Acyclic Graphs)

Tasks declare explicit dependencies with dependencies.
Any task whose dependencies are satisfied will run immediately, providing maximal parallelism.
Use DAGs for complex dependency graphs.

Example DAG:

dag:
  tasks:
    - name: build
      template: build

    - name: test-a
      template: test-a
      dependencies: ["build"]

    - name: test-b
      template: test-b
      dependencies: ["build"]

    - name: deploy
      template: deploy
      dependencies: ["test-a", "test-b"]

Use Steps for simple, mostly-linear pipelines. Choose DAGs when you need complex dependencies and maximum parallelism.

Templates — reusable workflow building blocks Templates are the primary building block in Argo Workflows. They let you package a task once and reuse it across multiple workflows. Typical template types:

Template type	Purpose	When to use
Containers	Run a container image with args, resources, env	Standard unit of work (build, test, deploy)
Scripts	Run inline code inside a container image	Short, single-file tasks that don’t need a dedicated image
Resource	Create/patch/delete Kubernetes resources and optionally wait	Declarative infra steps (apply manifests, wait for readiness)
Suspend	Pause workflow for manual approval or external signal	Gate long-running or sensitive changes

A slide titled "Templates – Reusable Workflow Building Blocks" showing four numbered blocks: 01 Containers, 02 Script, 03 Resources, and 04 Suspend. Each block has a short caption underneath (e.g., "Run a container image", "Inline script in any language", "Create/Patch K8s resources", "Wait for manual approval").

WorkflowTemplates are cluster-scoped resources that let platform teams register reusable templates once and reference them from many Workflows — ideal for building a platform task library. Parameters make templates flexible: pass values at runtime or wire outputs from previous steps into later inputs. Example WorkflowTemplate (kaniko build) that accepts a repo parameter:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: build-and-push
spec:
  templates:
  - name: kaniko-build
    inputs:
      parameters:
      - name: repo
    container:
      image: gcr.io/kaniko-project/executor:latest
      args:
      - "--destination={{inputs.parameters.repo}}"

Artifacts — passing data between steps Because each step runs in its own pod and pods do not share a filesystem, Argo uses artifacts to move files between steps. A producing step declares an output artifact and writes a file to a known path; Argo uploads that file to the configured artifact repository. A consuming step declares the artifact as an input and Argo downloads it into the consumer pod. Producer and consumer example templates:

templates:
- name: produce
  container:
    image: alpine:3.18
    command: [sh, -c]
    args: ['echo "hello from producer" > /tmp/msg.txt']
  outputs:
    artifacts:
    - name: message
      path: /tmp/msg.txt

- name: consume
  inputs:
    artifacts:
    - name: message
      path: /tmp/msg.txt
  container:
    image: alpine:3.18
    command: [cat, /tmp/msg.txt]

A slide titled "Artifacts – Passing Data Between Steps" showing a simple flow where step A generates an artifact.tar file that is passed to and consumed by step B. The diagram uses an orange circle for A (Generate), a blue rectangle for artifact.tar, and a purple circle for B (Consume).

Storage backends are configured in the artifact repository ConfigMap in the Argo namespace. For local development or CI testing you can use MinIO; in production use S3, GCS, or another durable object store. Use artifacts for build outputs, test reports, generated configuration, and any files that must be transferred between steps. Argo Events — event-driven automation Argo Events extends Argo Workflows so workflows can be triggered by external events. The main components are:

EventSource: connects to an external source (Git, webhooks, S3, Kubernetes, etc.) and listens for raw events.
Sensor: filters, correlates, and transforms events; extracts useful data (for example: commit SHA or filename).
Trigger: the action started by the sensor, commonly submitting an Argo Workflow with parameters populated from the event.

A slide diagram titled "Event-Driven Automation With Argo Events" showing an architecture flow. It depicts an Event Source (Webhook, S3, K8s) → Sensor (Filter + Transform) → Trigger (Start Workflow).

A sensor can extract values from an event and pass them as workflow parameters. Common real-world triggers include:

Slide titled "Event-Driven Automation With Argo Events" showing a panel of real use cases. It lists examples: "Git push → Build pipeline", "File uploaded → Process data", and "CRD created → Provision infra."

Git push → build and test pipeline
File uploaded to storage → data processing workflow
CRD created in Kubernetes → provision infrastructure
Cron schedule → nightly cleanup or maintenance

Argo Events + Argo Workflows give you a fully event-driven automation platform: declare what should happen when an event occurs, and the system will run, retry on failures, store artifacts, and provide visual and log-based visibility. Quick comparisons

Aspect	Scripts / CI	Argo Workflows
Execution model	Sequential or limited parallelism	Steps (sequenced/paralleled) or DAG (dependency-based)
Retry/Backoff	Manual or CI-limited	Per-step retries and backoff policies
Artifacts	Ad-hoc copying	Built-in artifact passing via object store
Observability	Logs only	UI with graphs, per-step logs, timings
Triggering	Cron or manual	Event-driven via Argo Events, cron, or API

Summary

Use a workflow engine like Argo Workflows for complex, multi-step platform automation instead of long Bash scripts.
Choose Steps for simple, mostly-linear flows; choose DAGs when you need explicit dependencies and high parallelism.
Build reusable WorkflowTemplates and use parameters to make platform tasks flexible and composable.
Use artifacts to pass files between isolated pods; configure a durable object store (S3/GCS/MinIO) for artifact storage.
Extend automation with Argo Events to trigger workflows from webhooks, storage events, Kubernetes resources, or schedules.

Course Introduction

Platform Architecture and Infrastructure

GitOps and Continuous Delivery

Platform APIs and Self-Service Capabilities

Observability and Operations

Security and Policy Enforcement

Conclusion

Workflow Orchestration DAGs Steps and Event Driven Automation

Watch Video