How do I design eventual consistency with the Saga Pattern in a distributed system?

Short answer: avoiding 2PC is the right call — it blocks and doesn’t scale. A Saga turns one atomic transaction into a sequence of local transactions, each with a compensating action. For your flow, an orchestrator-based Saga is the right choice.

Short answer

The real issue is this: you can no longer roll everything back inside one transaction. Instead, you have to enforce the discipline of “if a step breaks, compensate the steps that came before it.” I covered the operational side of the queue that carries those steps in the Laravel queue and Supervisor post.

Why

1. Compensation is not a rollback. A refund is a real transaction — a forward-direction balancing step. Accepting that shapes the whole saga design.

2. Retries are guaranteed on queues. The same “decrement stock” command can arrive twice; without idempotency a saga produces double refunds / double stock on the second attempt. I covered the webhook version of that same discipline in the duplicate-notification and idempotency answer.

3. If “do the work” and “publish the event” come apart, the saga strands. The work commits, the event never publishes, and the flow simply stops.

What to do

1. Let a central coordinator drive the steps over a queue. The orchestrator (Order Service) pushes commands: reserve payment → decrement stock → create shipment. Each service reports back when it finishes. The flow lives in one place — explicit and readable.

2. On failure, run compensations in reverse. If the shipping step blows up, the orchestrator walks backwards: restock the items, refund the payment.

3. Make every step idempotent. Put a deterministic saga_id + step_id on each command and skip it if already processed.

4. Preserve atomicity with the outbox pattern. Write the work and the event to an outbox table in the same DB in one transaction, and let a separate relay move it to the queue.

5. Persist the saga state. Which step you’re on, which ones completed — keep it in a table so the coordinator resumes where it left off after a crash.

Bottom line: prefer orchestration over choreography; the flow is explicit and the compensation ordering is clear. Accept eventual consistency: stock may look inconsistent for a few seconds, then reconcile. I go deeper on the architecture on sade.dev.