Services

Background jobs and queue reliability

Operate asynchronous work with idempotency, retries, dead letters, and clear SLOs.

Queues make systems resilient when work can happen asynchronously. They also hide failure until lag, duplicates, or poison messages break customer workflows. Treat queue design as production infrastructure, not a utility detail.

Queue use cases#

  • Email, notifications, and webhooks.
  • Billing, provisioning, and account lifecycle jobs.
  • Data imports, exports, and enrichment.
  • AI document processing and batch evaluation.
  • Integration syncs and reverse ETL.
  • Event fan-out to analytics and automation consumers.

Reliability baseline#

AreaPractice
IdempotencyJobs can be safely retried without duplicate side effects.
Retry policyBounded attempts with exponential backoff and jitter.
Dead lettersFailed messages are retained with reason, payload pointer, and replay path.
BackpressureProducers slow down or degrade gracefully when consumers lag.
OrderingOrdering requirements are explicit and partitioned by stable key.
ObservabilityLag, age, throughput, failures, retries, and handler latency are measured.
OwnershipEvery queue has an owner, SLO, and incident response path.

Implementation playbook#

1. Define job semantics#

For each queue, document:

  • Trigger source.
  • Payload schema and version.
  • Idempotency key.
  • Maximum processing time.
  • Retryable vs terminal errors.
  • Side effects.
  • Ownership and escalation path.

2. Make retries safe#

Retries are required because networks and dependencies fail. They are dangerous when handlers are not idempotent.

Common techniques:

  • Store idempotency keys with operation outcome.
  • Use database uniqueness constraints for once-only effects.
  • Make external API calls with provider-supported idempotency keys.
  • Separate validation errors from transient dependency errors.
  • Avoid infinite retries for poison messages.

3. Plan for dead-letter recovery#

A dead-letter queue is useful only if operators know what to do with it.

Include:

  • Failure classification.
  • Payload redaction or secure payload pointer.
  • Replay command or admin workflow.
  • Bulk replay safety checks.
  • Alert thresholds by message age and count.
  • Runbook for common terminal failures.

4. Control queue growth#

Add backpressure before lag becomes an outage:

  • Rate-limit producers.
  • Split high-priority and low-priority work.
  • Scale consumers on queue age, not just queue length.
  • Shed optional work when dependencies are degraded.
  • Alert on oldest-message age and processing latency.

Metrics and alerts#

Minimum metrics:

  • Enqueued, processed, failed, retried, and dead-lettered count.
  • Queue depth and oldest-message age.
  • Handler duration and dependency latency.
  • Consumer concurrency and saturation.
  • Retry attempts by error class.

Alert examples:

  • Oldest message age exceeds the business SLO.
  • Dead-letter count increases for a critical queue.
  • Handler error rate exceeds baseline.
  • Consumer count drops to zero.
  • Retry storm detected for one dependency.

Example queue contract#

yaml
1
queue: invoice-provisioning
2
owner: billing-platform
3
slo: 99% processed within 5 minutes
4
payload_schema: invoice_provisioning.v2
5
idempotency_key: invoice_id
6
ordering_key: tenant_id
7
retry:
8
  attempts: 6
9
  backoff: exponential_with_jitter
10
  max_delay: 30m
11
dead_letter:
12
  retention: 14d
13
  replay_requires: operator-approval

References#