How do I prevent PostgreSQL split-brain with quorum and consensus?

Short answer: the only cure for split-brain is quorum — a node may only become or stay primary if it holds a majority. That’s exactly what was missing in your scenario.

The root of the problem: a replica promoted itself based on local information alone (“I can’t reach the Master, so it must be dead”). But the Master wasn’t dead, it was just network-partitioned. Two primaries, diverged data.

1. Nobody becomes primary without a majority. Use a failover manager that decides by consensus: Patroni + etcd/Consul. Raft holds the leader lock; to become the new primary, a node has to take that lock from the quorum. A node that can’t reach the majority can’t get promoted.
2. Let the old primary demote itself (fencing). The old Master, isolated from the network, can’t renew its lock in etcd, so it expires and the node automatically demotes itself to a replica. This is fencing/STONITH; it makes “two primaries writing at once” physically impossible.
3. Use an odd number of voting members. Build the consensus layer (etcd) with an odd number of members spread across failure domains (availability zones) — 3 or 5. When the network partitions, there’s a clear majority side; the minority side stops writing.
4. Add synchronous replication if you can’t lose the last transactions. With synchronous_commit and quorum-based synchronous replication, a commit doesn’t return until at least one replica acknowledges. The cost is latency, the gain is near-zero data loss.
5. Never trust a hand-rolled failover script. Home-made scripts with “if the Master doesn’t ping, promote” logic produce exactly the split-brain you hit. This is a solved problem; don’t reinvent the solution.

Bottom line: I’d move to Patroni + etcd quorum + fencing, and not manage failover by hand. I go deeper on DB operations on sade.dev; but the one line is this: a promotion decision is made by the majority, not by local information.