Skip to main content
This lesson walks through designing a robust background job processing system for a common user flow: a mobile app sign-up that must create an account, return success to the user quickly, and send a welcome email asynchronously. We build the design incrementally, highlight failure modes in each naive approach, and show production-grade mitigations. Why this matters: blocking user requests on slow external services leads to poor UX and brittle systems. Using queues, outboxes, visibility timeouts, idempotent workers, and dead-letter queues yields a resilient, observable pipeline for asynchronous work like emails, notifications, and other background tasks.

​
Version 1 — The naive synchronous approach

Flow:
  1. Mobile app submits sign-up details.
  2. Account service creates the user in the database.
  3. Account service calls the email service to send a welcome email.
  4. Only after the email is sent does the service return success to the mobile app.
Problems:
  • Perceived latency: the user must wait while the email provider responds.
  • Fragility: if the email service is slow or down, sign-up requests may time out even though the account was created.
  • Poor user experience and unexpected failure modes (account created, but user sees an error).
This approach works locally or in low-latency environments but does not scale or tolerate typical production failures.

​
Version 2 — Add a queue (asynchronous processing)

Improve perceived latency and decouple account creation from email delivery: Flow:
  1. Account service persists the user to the database.
  2. Account service enqueues a job to a queue (e.g., SQS) to send the welcome email.
  3. The request returns success to the mobile app immediately.
  4. Background workers consume jobs from the queue and send the emails.
Benefits:
  • The user experience is fast because signup confirms immediately.
  • Email sending can be retried and scaled independently from account creation.
A hand-drawn system architecture titled "V2: Add a Queue" showing a mobile app calling an account_setup service that writes to a database and pushes messages to an SQS queue. A Lambda worker consumes the queue (job) and triggers an email-send component.
Subtle consistency problem introduced:
  • The account service now performs two operations: save the account and enqueue the job. If it saves the account but crashes before enqueuing, the account exists but the welcome email is never scheduled.

​
Make the two actions atomic — Transactional Outbox

Use the transactional outbox pattern: write the email job to an outbox table inside the same database transaction that creates the account. A separate background publisher reads the outbox table and reliably publishes messages to the queue.

​
Version 3 — Visibility timeout and at-least-once delivery

Another failure mode: a worker dequeues a job and crashes while sending the email. If the message is removed/marked handled for that worker, the email gets lost and is never retried. Queue systems solve this with a visibility timeout: when a worker receives a message, the message is hidden for a configurable interval (e.g., 30s). The worker deletes the message only after successful completion. If the worker crashes, the message becomes visible again and another worker can retry. This provides reliability but introduces at-least-once delivery semantics: the same job may be redelivered and processed multiple times. Workers must tolerate duplicates.
A hand-drawn style architecture diagram titled "V3: VISIBILITY TIMEOUT." It shows a mobile app calling an account_setup service that writes to a database/outbox, enqueues a job to SQS, and a Lambda worker processes the queue to send an email.
Make workers idempotent: include a unique job ID with each job. When a worker completes the work, record the job ID (for example, in a processed_jobs table with a unique constraint). On receiving a job, check whether that job ID was already processed and skip duplicate processing if so. Idempotency is the standard pattern to handle at-least-once delivery.

​
Handling persistent failures — Dead Letter Queues (DLQ)

Some jobs will keep failing (invalid addresses, permanent external errors, malformed payloads). Retrying forever is not useful. Configure a maximum receive count; when exceeded, move the message to a dead letter queue (DLQ) for manual inspection and remediation. DLQs let engineers:
  • Inspect failing payloads and root causes.
  • Reprocess messages after fixing data or configuration.
  • Avoid clogging the main queue with repeatedly failing work.
A hand-drawn style system architecture diagram titled "V1: HANDLE FAILURE" showing a mobile app calling an account_setup service that writes to a database with an outbox table. A separate process enqueues messages to SQS, a Lambda worker consumes them to send email, and failures go to a dead-letter queue for engineer investigation.

​
Failure modes and mitigations (quick reference)

  • Use a transactional outbox to make DB writes + enqueue atomic.
  • Enqueue jobs to a durable queue (e.g., Amazon SQS, RabbitMQ, Google Pub/Sub).
  • Use visibility timeouts and require workers to explicitly delete messages after success.
  • Design workers to be idempotent (unique job IDs + processed-records or dedupe logic).
  • Configure a dead letter queue and business-appropriate retry/backoff policy.
  • Add monitoring, alerting, and observability for queue depth, DLQ growth, and processing latency.
Following these patterns produces a robust, observable, and production-ready background job system suitable for sending emails and other asynchronous workloads.

Watch Video