Skip to main content
In this lesson we focus on making software releases secure and resilient. Security is not only about what features you ship — it’s about ensuring the release process itself does not introduce risk. Developers write code, security teams define controls, and SREs operationalize and enforce those controls. When ownership gets assumed by others, SREs often become the last line of defense; many postmortems note “this should have been caught at deployment.”
A presentation slide titled "Security in Software Releases Is Essential" showing an "Expensive Hall of Fame" of three major incidents. It lists SolarWinds hack (2020), Log4Shell/Log4j (2021), and the Target breach (2013) with short notes on their impacts.
Real incidents demonstrate how the release path can amplify risk: the SolarWinds supply-chain compromise (2020) propagated a poisoned build broadly; Log4Shell (2021) showed how a single vulnerable library can cascade across ecosystems; and the Target breach (2013) involved stolen deployment credentials with costly business impact. SREs validate deployments, enforce security policies, and stop misconfigurations before they become incidents.
A presentation slide titled "Security in Software Releases Is Essential" stating "SREs are the last line of defense." It lists three points showing developers write code (SREs ensure secure deployment), security teams set policies (SREs enforce them), and everyone assumes someone else checks (SREs actually check).
Common release-path risks and mitigations
RiskWhy it mattersTypical mitigation
Vulnerable third-party dependenciesLarge dependency trees mean most code is third-party; one vulnerable package can affect many servicesAutomated dependency scanning (pip-audit, npm audit), SBOM generation
Misconfigured permissions (e.g., public S3 buckets)Overly-broad permissions expose data and infrastructurePrinciple of least privilege, scoped IAM policies, automated policy checks
Lack of automation or visibilityManual audits don’t scale; blind spots allow vulnerabilities to slip to productionCI-driven scanning, artifact signing, centralized logs/alerts
Weak authentication for registries/pipelinesHardcoded credentials or weak tokens lead to credential theftToken-based auth, short-lived credentials, secrets management
A small misstep can expose everything. For example: 
# What someone meant to do:
aws s3 cp file.txt s3://internal-bucket/

# What actually got deployed:
aws s3api put-bucket-acl --bucket internal-bucket --acl public-read

# ⚠️ Now every file in the bucket is publicly accessible
A slide titled "Common Risks in the Release Path" showing a "Vulnerability Domino Effect" graphic (a cracked shield on a monitor). It lists five numbered risks: widely used library, vulnerability found in production, no automated dependency scanning, manual audits of many services, and resulting panic/ executive escalations.
Build security into CI/CD (shift left)
  • Add automated checks in CI so vulnerabilities are detected before deployment.
  • SAST (Static Application Security Testing) inspects source code for issues (SQL injection, XSS, hard-coded credentials, buffer overflows, weak crypto) without running programs.
The image is a presentation slide titled "Security Verification in CI/CD" explaining Static Application Security Testing (SAST) which scans source code for security issues without running the program. A sidebar lists common findings: SQL injection, cross-site scripting (XSS), buffer overflow, and hardcoded credentials.
Example: GitHub CodeQL can be used as a SAST step in GitHub Actions. Keep CI configured to fail or require remediation for critical issues where appropriate. 
# GitHub Actions example for CodeQL SAST scan (snippet)
name: Run SAST Scan
on: [push, pull_request]
jobs:
  codeql-analysis:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Initialize CodeQL
        uses: github/codeql-action/init@v2
        with:
          languages: javascript, python

      - name: Run CodeQL analysis
        uses: github/codeql-action/analyze@v2

      - name: Fail on critical issues
        if: env.CRITICAL_ISSUES != ''
        run: |
          echo "Critical security issues found. Failing build."
          exit 1
Dependencies and SBOMs
  • Most modern apps include large third-party dependency trees. Automated dependency scanning prevents known vulnerable packages from reaching production.
  • Tools: pip-audit, npm audit, GitHub Dependabot, SCA (Software Composition Analysis) solutions.
# For Python
pip-audit --strict

# For Node.js
npm audit --audit-level=moderate
A Software Bill of Materials (SBOM) is an “ingredients list” for your software. With an SBOM you can rapidly identify which services are affected by a vulnerability (for example, during Log4Shell), rather than manually tracing dependency trees service by service.
A presentation slide titled "Security Verification in CI/CD" explaining Software Bill of Materials (SBOM) as a complete inventory of software components. It notes why SBOMs matter — they helped teams quickly identify at-risk apps during Log4Shell, whereas without them teams spent days or weeks manually checking services.
Core principles and common practices
  • Principle of Least Privilege: grant only the minimum permissions required.
// Bad: effectively full admin
{
  "Effect": "Allow",
  "Action": "*",
  "Resource": "*"
}

// Better: scoped to S3 object read/write in one bucket
{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:PutObject"
  ],
  "Resource": "arn:aws:s3:::app-uploads/*"
}
  • Automate security checks: dependency scanning, secret detection, SAST/DAST, container image scanning, and infrastructure-as-code (IaC) checks should run in CI.
  • Review and sign artifacts: build-time signing ensures the artifact you deploy is the artifact you built. cosign and sigstore are industry-standard tools for container/image signing and verification.
# Example artifact signing with cosign (bash)
cosign sign --key cosign.key ghcr.io/my-org/myapp:v1.2.3

# Verify during deployment
cosign verify --key cosign.pub ghcr.io/my-org/myapp:v1.2.3
  • Continuous monitoring: subscribe to advisories, configure automated CVE alerts, and periodically re-scan deployed images and running systems.
An infographic titled "Best Practices for Secure Releases" that lists four principles: Principle of Least Privilege, Automate Security Checks, Review and Sign Artifacts, and Monitor for New Vulnerabilities. A checklist below recommends subscribing to security advisories, setting up automated CVE alerts, and regularly reviewing infrastructure and apps.
Progressive security pipeline — step by step Evolve pipelines incrementally to reduce risk without blocking delivery:
  1. Start with a basic build/deploy pipeline.
  2. Add dependency vulnerability scanning so bad packages are blocked early.
  3. Generate SBOMs so you can quickly identify affected apps when vulnerabilities surface.
  4. Improve authentication and restrict permissions (least privilege).
  5. Add container image scanning (Trivy, Grype, etc.).
  6. Enforce environment separation and promotion gates (staging → production).
A presentation slide titled "Progressive Security Pipeline Evolution" that lists five steps: The Problem (Basic Insecure Pipeline), Dependency Vulnerability Scanning, Software Bill of Materials (SBOM), Secure Authentication and Permissions, and Container Security Scanning. A side note reads "Scan image for vulnerabilities" and the slide is copyrighted by KodeKloud.
Working with the provided repository
  • Fork and clone the repository locally to iterate on release workflows and test changes.
git clone https://github.com/YOUR_USERNAME/kodekloud-records-store-web-app.git
cd kodekloud-records-store-web-app

# Create a branch for testing
git checkout -b develop-test
Design a consolidated, secure GitHub Actions workflow with:
  • Explicit triggers (push branches and workflow_dispatch for manual runs).
  • Limited permissions using the top-level permissions block.
  • Jobs: build-and-test, build-container (secure registry login, image build, SBOM generation, container scan), deploy-staging, and deploy-production (with manual approval).
Example workflow header and permissions: 
# Workflow header + limited token permissions
name: Progressive Config Management & Secure Release Pipeline
on:
  push:
    branches:
      - main
      - develop-test
  pull_request:
    types: [closed]
    branches:
      - main
  workflow_dispatch:
    inputs:
      environment:
        description: 'Target environment for deployment'
        required: true
        default: 'staging'
        type: choice
        options: [development, staging, production]
permissions:
  contents: read
  packages: write
  id-token: write
Build-and-test (important parts only): 
jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install dependencies
        run: |
          python -m venv venv
          source venv/bin/activate
          pip install --upgrade pip setuptools
          pip install -r requirements.txt

      - name: Dependency vulnerability scan (pip-audit)
        run: |
          source venv/bin/activate
          pip install pip-audit
          pip-audit --format json > security-report.json || echo "vulnerabilities found"
Authentication to container registry — avoid hardcoded credentials
  • Never store plaintext credentials in workflows or source code.
Never store plaintext credentials in workflows or in source code. Use the provided runtime tokens and secrets.
Do not commit static credentials. Hardcoded passwords in CI can be exposed in logs, forks, or via leaked access. Rotate any credentials that were committed immediately.
Bad example (do not use): 
# ❌ INSECURE: hardcoded password (do not use)
echo "password123" | docker login ghcr.io -u admin --password-stdin
Good example (use GitHub token at runtime): 
- name: Log in to GitHub Container Registry
  run: |
    echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u $GITHUB_ACTOR --password-stdin
    echo "✅ Using secure GitHub token authentication"
Install Docker Compose and build/push images (use fixed URL for reproducibility): 
- name: Install Docker Compose
  run: |
    sudo curl -L "https://github.com/docker/compose/releases/download/v2.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
    sudo chmod +x /usr/local/bin/docker-compose
    docker-compose version

- name: Build and tag Docker image
  run: |
    docker build -t ghcr.io/${{ github.repository }}/kodekloud-records:${{ github.sha }} .
    docker push ghcr.io/${{ github.repository }}/kodekloud-records:${{ github.sha }}
Generate SBOM and upload as an artifact (example using syft): 
- name: Generate Software Bill of Materials (SBOM)
  run: |
    curl -sSFL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin
    syft -o cyclonedx-json . > sbom.json

- name: Upload SBOM as artifact
  uses: actions/upload-artifact@v4
  with:
    name: software-bill-of-materials
    path: sbom.json
Container scanning (example using Grype): 
- name: Install Grype
  run: |
    curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin

- name: Scan image for vulnerabilities
  run: |
    grype ghcr.io/${{ github.repository }}/kodekloud-records:${{ github.sha }} -o json > grype-report.json || true
Promotion gates and deploy jobs
  • Deploy to staging automatically if scans pass.
  • Require manual approval for production promotion (explicit human gate).
jobs:
  deploy-staging:
    environment: staging
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to staging
        run: |
          echo "Deploying to staging..."
          # deployment commands here

  deploy-production:
    environment: production
    runs-on: ubuntu-latest
    steps:
      - name: Production Promotion Gate
        run: |
          echo "🔒 MANUAL APPROVAL REQUIRED FOR PRODUCTION"
          echo "Approved by: ${{ github.actor }}"
Troubleshooting — failed authentication due to insecure login
  • Common error when using hardcoded login:
Error response from daemon: Get "https://ghcr.io/v2/": denied: denied
Error: Process completed with exit code 1.
Use runtime tokens and secrets as shown above; replace insecure credentials and re-run the workflow. When the pipeline runs successfully you will see GitHub Actions output indicating build, scans, SBOM upload, and staged deployment succeeded.
A screenshot of a GitHub repository's Actions page showing a workflow run titled "run secure pipeline #76." The right pane shows the "deploy-staging" job and its steps (Set up job, Checkout code, Install Docker Compose, Create Staging Environment, Deploy to Staging, etc.).
Summary — what changed and why it matters
  • Replaced insecure patterns with managed, auditable controls:
    • Token-based authentication instead of hard-coded secrets
    • Automated dependency and container scanning in CI
    • SBOM generation and artifact signing to ensure provenance
    • Principle of least privilege and environment-based promotion gates
Each layer reduces the chance that a single mistake or vulnerable dependency becomes a full-scale breach. Use the pipeline pattern shown here as a starting point: customize scanners, severity thresholds, and promotion gates to your organization’s risk profile.
A presentation slide titled "Progressive Security Pipeline Evolution" showing a Before vs After comparison: Before lists issues like hardcoded credentials, no scanning/SBOM, overly broad permissions, and no deployment controls; After lists fixes like token auth, automated/SBOM/container scanning, least privilege, and environment controls.
Further reading and references Next, we’ll wrap up this section with a concise checklist of best practices and additional resources for hardening CI/CD pipelines.

Watch Video