Realtime operations playbook

Run WebSocket, streaming, and live-collaboration features with clear reliability controls.

Realtime features make product experiences feel immediate, but they introduce long-lived connections, fan-out pressure, ordering questions, and operational complexity. Use this playbook for WebSockets, server-sent events, live dashboards, event streams, collaboration, and presence.

Realtime patterns#

Pattern	Use when	Operational concern
WebSocket	Bidirectional low-latency interaction.	Connection state, authentication refresh, fan-out, backpressure.
Server-sent events	Server-to-client updates over HTTP.	Reconnect behavior, event IDs, proxy timeouts.
Pub/sub fan-out	Many clients receive topic updates.	Topic authorization, hot partitions, message loss.
Event stream	Durable ordered events for services.	Retention, consumer lag, schema evolution.
Polling with cache	Realtime not actually required.	Simpler operations, higher latency.

Design decisions#

Before implementation, decide:

What freshness does the user actually need?
Is delivery best-effort or guaranteed?
Does ordering matter globally, per tenant, per document, or not at all?
Can clients recover state after disconnect?
Which topics or rooms require authorization?
How will mobile networks, browser tab suspension, and proxy timeouts behave?

Implementation controls#

Connection lifecycle#

Authenticate on connect and re-check authorization on sensitive subscriptions.
Expire or refresh long-lived credentials.
Use heartbeats and idle timeouts.
Require clients to resume from a last-seen event ID when correctness matters.
Avoid storing critical state only in memory on a single realtime node.

Ordering and recovery#

Partition streams by a key that matches the ordering requirement.
Include monotonically increasing sequence numbers where clients need reconciliation.
Provide a snapshot endpoint so clients can recover after missed events.
Make events immutable and versioned.

Backpressure and fan-out#

Cap subscriptions, connection count, and messages per client.
Drop or coalesce non-critical updates such as typing indicators.
Separate high-volume presence signals from critical business events.
Monitor slow consumers and disconnect clients that cannot keep up.

Data governance notes#

Realtime channels often bypass normal page/API boundaries. Apply the same tenant, role, and data-classification checks to subscriptions as to REST or GraphQL endpoints.

Do not broadcast sensitive payloads to broad topics and rely on the client to filter.
Redact payloads for users with partial access.
Keep audit logs for privileged subscriptions.
Define retention for durable streams and replay buffers.

Observability#

Minimum dashboards:

Active connections by region, tenant, and client version.
Connect, disconnect, auth failure, and reconnect rates.
Message publish rate, delivery latency, and dropped/coalesced messages.
Subscription count by topic and hot-topic fan-out.
Consumer lag for durable streams.
Error rate by close code or protocol error.

Alert on connection storms, zero consumers for critical streams, rising reconnect loops, and oldest unprocessed event age.