> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Workflow Orchestration DAGs Steps and Event Driven Automation > Explains using Argo Workflows and Argo Events to replace brittle scripts with DAGs, steps, templates, artifacts, retries, and event-driven automation for Kubernetes platform tasks. Controllers perform continuous reconciliation, but not every platform task is continuous. Some tasks need to run once or as a finite multi-step process — for example, a one-off data migration or an environment provisioning workflow. Those use cases go beyond the simple controller pattern, and that’s where workflow orchestration shines. What you’ll learn in this lesson: * Why shell scripts and naive CI pipelines fail for complex automation. * How to author Argo Workflows using Steps (list-of-lists) and DAG patterns. * How to create reusable WorkflowTemplates with parameters. * How to pass artifacts (files) between workflow steps. * How to trigger workflows from external events using Argo Events. A real-world example: a platform team had a 50-line Bash script to provision a development environment. The script created a namespace, applied RBAC, deployed monitoring agents, configured Ingress, and set up DNS. It executed everything sequentially even though many steps were independent — and then it failed. ![A horizontal flowchart titled "From Sequential Script to Resilient Workflow" showing a seven-step sequential deployment process (Create Namespace, Apply RBAC, Deploy Monitoring, Configure Ingress, Setup DNS, Sequential Wait, Failure). The diagram notes this "BEFORE (Sequential Bash)" workflow takes about 15 minutes and ends in failure with no retries.](https://mintcdn.com/kodekloud-c4ac6d9a/jFwNGqGMK4QeJs-g/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Platform-APIs-and-Self-Service-Capabilities/Workflow-Orchestration-DAGs-Steps-and-Event-Driven-Automation/sequential-bash-deployment-failure.jpg?fit=max\&auto=format\&n=jFwNGqGMK4QeJs-g\&q=85\&s=457936f82d94d3f15f15200e8ed65b4a) Steps three and four in that script could have run in parallel, cutting runtime from \~15 minutes to \~8 minutes. When the final step failed because of a transient DNS timeout, the script exited with no retries. Engineers had to undo prior steps manually and start over — there was no resume capability. Why scripts and simple pipelines aren’t enough The diagram below highlights the recurring gaps in shell scripts, CI jobs, and cron-based automation. ![A presentation slide titled "Scripts and Pipelines Hit Their Limits – What's Missing" listing four problems: No Dependency Graphs, No Parallel Execution, No Retry Logic, and No Observability, each with a short explanatory note. The items are shown as numbered orange gradient boxes with brief gray captions underneath.](https://mintcdn.com/kodekloud-c4ac6d9a/jFwNGqGMK4QeJs-g/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Platform-APIs-and-Self-Service-Capabilities/Workflow-Orchestration-DAGs-Steps-and-Event-Driven-Automation/pipelines-missing-dependencies-parallel-retry-observability.jpg?fit=max\&auto=format\&n=jFwNGqGMK4QeJs-g\&q=85\&s=6363f17306119520a2ed34b401084180) Common limitations | Limitation | Impact | | - | - | | No dependency graph | Hard to express "run C only after A and B succeed" | | No parallel execution | Independent tasks run sequentially and waste time | | No retry logic | Transient failures abort the whole run | | Poor observability | Limited visibility into which step failed or runtime metrics | For complex platform automation you need a workflow engine — not more shell scripts. How Argo Workflows helps [Argo Workflows](https://argoproj.github.io/argo-workflows/) runs each workflow step as a Kubernetes pod, so every step benefits from Kubernetes features: resource limits, service accounts, RBAC, secrets, node selectors/affinity, and priority classes. Argo provides a focused set of capabilities for reliable platform automation: * DAGs and Steps: express dependencies and control parallelism or sequence. * Retries and timeouts: automatic retries, backoff, and per-step/workflow timeouts. * Artifacts: share files between steps using S3, GCS, MinIO, or other backends. * UI and CLI: visual workflow graphs, per-step logs and timings; submit and monitor from the CLI. Workflows are Kubernetes custom resources (YAML) that you can store in Git — declarative and portable. Two ways to express execution order 1. Steps (list-of-lists) * The outer list represents sequence. * Each inner list contains tasks that run in parallel. * Best for mostly-linear pipelines with occasional parallelism. Correct example steps layout (build → parallel tests → deploy): ```yaml theme={null} steps: - - name: build template: build - - name: test-a template: test-a - name: test-b template: test-b - - name: deploy template: deploy ``` 2. DAGs (Directed Acyclic Graphs) * Tasks declare explicit dependencies with `dependencies`. * Any task whose dependencies are satisfied will run immediately, providing maximal parallelism. * Use DAGs for complex dependency graphs. Example DAG: ```yaml theme={null} dag: tasks: - name: build template: build - name: test-a template: test-a dependencies: ["build"] - name: test-b template: test-b dependencies: ["build"] - name: deploy template: deploy dependencies: ["test-a", "test-b"] ``` Use Steps for simple, mostly-linear pipelines. Choose DAGs when you need complex dependencies and maximum parallelism. Templates — reusable workflow building blocks Templates are the primary building block in Argo Workflows. They let you package a task once and reuse it across multiple workflows. Typical template types: | Template type | Purpose | When to use | | - | - | - | | Containers | Run a container image with args, resources, env | Standard unit of work (build, test, deploy) | | Scripts | Run inline code inside a container image | Short, single-file tasks that don’t need a dedicated image | | Resource | Create/patch/delete Kubernetes resources and optionally wait | Declarative infra steps (apply manifests, wait for readiness) | | Suspend | Pause workflow for manual approval or external signal | Gate long-running or sensitive changes | ![A slide titled "Templates – Reusable Workflow Building Blocks" showing four numbered blocks: 01 Containers, 02 Script, 03 Resources, and 04 Suspend. Each block has a short caption underneath (e.g., "Run a container image", "Inline script in any language", "Create/Patch K8s resources", "Wait for manual approval").](https://mintcdn.com/kodekloud-c4ac6d9a/jFwNGqGMK4QeJs-g/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Platform-APIs-and-Self-Service-Capabilities/Workflow-Orchestration-DAGs-Steps-and-Event-Driven-Automation/workflow-templates-containers-script-resources-suspend.jpg?fit=max\&auto=format\&n=jFwNGqGMK4QeJs-g\&q=85\&s=29aea1d5e1d608c9ba47546b6eb17d5c) WorkflowTemplates are cluster-scoped resources that let platform teams register reusable templates once and reference them from many Workflows — ideal for building a platform task library. Parameters make templates flexible: pass values at runtime or wire outputs from previous steps into later inputs. Example WorkflowTemplate (kaniko build) that accepts a `repo` parameter: ```yaml theme={null} apiVersion: argoproj.io/v1alpha1 kind: WorkflowTemplate metadata: name: build-and-push spec: templates: - name: kaniko-build inputs: parameters: - name: repo container: image: gcr.io/kaniko-project/executor:latest args: - "--destination={{inputs.parameters.repo}}" ``` Artifacts — passing data between steps Because each step runs in its own pod and pods do not share a filesystem, Argo uses artifacts to move files between steps. A producing step declares an output artifact and writes a file to a known path; Argo uploads that file to the configured artifact repository. A consuming step declares the artifact as an input and Argo downloads it into the consumer pod. Producer and consumer example templates: ```yaml theme={null} templates: - name: produce container: image: alpine:3.18 command: [sh, -c] args: ['echo "hello from producer" > /tmp/msg.txt'] outputs: artifacts: - name: message path: /tmp/msg.txt - name: consume inputs: artifacts: - name: message path: /tmp/msg.txt container: image: alpine:3.18 command: [cat, /tmp/msg.txt] ``` ![A slide titled "Artifacts – Passing Data Between Steps" showing a simple flow where step A generates an artifact.tar file that is passed to and consumed by step B. The diagram uses an orange circle for A (Generate), a blue rectangle for artifact.tar, and a purple circle for B (Consume).](https://mintcdn.com/kodekloud-c4ac6d9a/jFwNGqGMK4QeJs-g/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Platform-APIs-and-Self-Service-Capabilities/Workflow-Orchestration-DAGs-Steps-and-Event-Driven-Automation/artifacts-pass-data-a-to-b.jpg?fit=max\&auto=format\&n=jFwNGqGMK4QeJs-g\&q=85\&s=f00ce743f0c3357757717898cb96539f) Storage backends are configured in the artifact repository ConfigMap in the Argo namespace. For local development or CI testing you can use MinIO; in production use S3, GCS, or another durable object store. Use artifacts for build outputs, test reports, generated configuration, and any files that must be transferred between steps. Argo Events — event-driven automation [Argo Events](https://argoproj.github.io/argo-events/) extends Argo Workflows so workflows can be triggered by external events. The main components are: * EventSource: connects to an external source (Git, webhooks, S3, Kubernetes, etc.) and listens for raw events. * Sensor: filters, correlates, and transforms events; extracts useful data (for example: commit SHA or filename). * Trigger: the action started by the sensor, commonly submitting an Argo Workflow with parameters populated from the event. ![A slide diagram titled "Event-Driven Automation With Argo Events" showing an architecture flow. It depicts an Event Source (Webhook, S3, K8s) → Sensor (Filter + Transform) → Trigger (Start Workflow).](https://mintcdn.com/kodekloud-c4ac6d9a/jFwNGqGMK4QeJs-g/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Platform-APIs-and-Self-Service-Capabilities/Workflow-Orchestration-DAGs-Steps-and-Event-Driven-Automation/argo-events-event-source-sensor-trigger.jpg?fit=max\&auto=format\&n=jFwNGqGMK4QeJs-g\&q=85\&s=e3df9c9628f241b59a1928e18831cd75) A sensor can extract values from an event and pass them as workflow parameters. Common real-world triggers include: ![Slide titled "Event-Driven Automation With Argo Events" showing a panel of real use cases. It lists examples: "Git push → Build pipeline", "File uploaded → Process data", and "CRD created → Provision infra."](https://mintcdn.com/kodekloud-c4ac6d9a/jFwNGqGMK4QeJs-g/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Platform-APIs-and-Self-Service-Capabilities/Workflow-Orchestration-DAGs-Steps-and-Event-Driven-Automation/argo-events-use-cases.jpg?fit=max\&auto=format\&n=jFwNGqGMK4QeJs-g\&q=85\&s=f88100bd5d478b6b8e48609f64e891c5) * Git push → build and test pipeline * File uploaded to storage → data processing workflow * CRD created in Kubernetes → provision infrastructure * Cron schedule → nightly cleanup or maintenance Argo Events + Argo Workflows give you a fully event-driven automation platform: declare what should happen when an event occurs, and the system will run, retry on failures, store artifacts, and provide visual and log-based visibility. Quick comparisons | Aspect | Scripts / CI | Argo Workflows | | - | - | - | | Execution model | Sequential or limited parallelism | Steps (sequenced/paralleled) or DAG (dependency-based) | | Retry/Backoff | Manual or CI-limited | Per-step retries and backoff policies | | Artifacts | Ad-hoc copying | Built-in artifact passing via object store | | Observability | Logs only | UI with graphs, per-step logs, timings | | Triggering | Cron or manual | Event-driven via Argo Events, cron, or API | Summary * Use a workflow engine like Argo Workflows for complex, multi-step platform automation instead of long Bash scripts. * Choose Steps for simple, mostly-linear flows; choose DAGs when you need explicit dependencies and high parallelism. * Build reusable WorkflowTemplates and use parameters to make platform tasks flexible and composable. * Use artifacts to pass files between isolated pods; configure a durable object store (S3/GCS/MinIO) for artifact storage. * Extend automation with Argo Events to trigger workflows from webhooks, storage events, Kubernetes resources, or schedules. Further reading and references * Argo Workflows — [https://argoproj.github.io/argo-workflows/](https://argoproj.github.io/argo-workflows/) * Argo Events — [https://argoproj.github.io/argo-events/](https://argoproj.github.io/argo-events/) * Kubernetes Concepts — [https://kubernetes.io/docs/concepts/](https://kubernetes.io/docs/concepts/) * Kaniko — [https://github.com/GoogleContainerTools/kaniko](https://github.com/GoogleContainerTools/kaniko) * MinIO — [https://min.io/](https://min.io/) > **[Watch Video](https://learn.kodekloud.com/user/courses/prep-course-certified-cloud-native-platform-engineer-cnpe/module/756ffaae-767b-4743-9724-c05d3fbf9a18/lesson/12ee1a17-ba58-41bc-a43d-b7091cd52324)** This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.