Reproducing split brain on CloudNativePG

We run Postgres under an operator for automatic failover. That is a promise about what happens during a failure, so the only way to know you have it is to cause the failure and watch.

The docs tell you what should happen. A config review tells you which knobs are set. Neither tells you how long an isolated primary keeps accepting writes after its replacement has been promoted, and that number decides whether a failover is clean or leaves you with two versions of your data. So we keep chaos testing Postgres operators instead of trusting the datasheet.

This time we tested CloudNativePG, on defaults, with nothing tuned. It should handle this well. Since v1.27 the liveness probe of a primary also asks "can I still reach the API server, and at least one replica?" A primary that cannot say yes fails its own probe and gets killed, and the check is on by default. So we cut the primary off from the API server, the operator and its replicas, and expected a short, boring incident.

Two ways a failover can go wrong #

These two often get treated as the same thing. They are not the same failure, and they do not have the same fix.

Data loss at failover is prefix truncation. The standby was at LSN Y, the primary was at X, you promote, and everything between the two is gone. What survives is still a valid history, just a shorter one, and how much went missing is bounded by how far behind the standby was. That makes it something you can plan for: it is what RPO (recovery point objective) measures, the amount of recent data you have agreed you can afford to lose when something fails. Say "we may lose up to five seconds of writes" and you have set an RPO.

Split brain is forked history. Two nodes both accept writes, and each one looks fine on its own, but put together they could never have come from a single database: the same sequence values handed out twice, the same primary keys holding different rows, side effects firing on both branches. You cannot merge them back. pg_rewind does not even try. It picks a winner and deletes the other branch.

There is exactly one way to avoid the fork: the old primary has to stop accepting writes before anyone else starts. Which is why the experiment has to record two things rather than one, when each side was writing, and what each client was told while it was.

The setup #

A 5-node k3s cluster (v1.34.5+k3s1) running CloudNativePG 1.30.0, PostgreSQL 18.4 and Chaos Mesh 2.7.2.

The database is three instances with pod anti-affinity, one per node, and nothing tuned. No probes: section, so the isolation check is on. The three defaults that turn out to matter:

$ kubectl get cluster postgres-splitbrain -o json | jq '.spec | {failoverDelay, smartShutdownTimeout, probes}'
{
  "failoverDelay": 0,
  "smartShutdownTimeout": 180,
  "probes": {
    "liveness": {
      "isolationCheck": {"connectionTimeout": 1000, "enabled": true, "requestTimeout": 1000}
    }
  }
}

The clients are three small Go processes, written to behave like a normal application. Each connects through the postgres-splitbrain-rw Service and then keeps that connection:

// Established once, then held - deliberately - for the lifetime of the process.
conn := connect(ctx, cfg.clusterDSN, "cluster")

for {
    <-ticker.C
    post, err := writePost(ctx, conn, cfg.clientID)
    if err != nil {
        // Only a broken session gets us a new one. During a partition: as long
        // as the old primary keeps answering, there is no error, so there is no
        // reconnect, so we never learn that a new primary exists.
        conn = reconnect(ctx, cfg.clusterDSN)
        continue
    }
    // Postgres said this is durable. Tell the outside world.
    ledger.Exec(ctx, `INSERT INTO acked_posts (...) VALUES (...)`, ...)
}

No pool recycling, no max lifetime, nothing that would re-resolve the Service. We did not break it on purpose. This is how a connection pool behaves with default settings, because a Service endpoint change does not tear down a TCP session that is already open.

Each client inserts into a posts table with a bigserial primary key, and after every commit Postgres acknowledges, records what it was told into a separate single-instance Postgres called sb-truth. That second database is how we measure anything at all. It is not part of the cluster under test, so pg_rewind cannot reach it, and it stands in for everything an application touches that a database failover cannot undo: the notification it sent, the row it wrote to another service, the charge it made.

Cutting the primary off #

Picture a multi-AZ cluster, one instance per zone, and one zone loses its links to the other two. Inside that zone nothing looks broken: the node is up, and the application pods next to the primary can still reach it. What the primary cannot reach is the API server, the operator and its two replicas.

A single node losing its uplink does the same thing, as does a control plane outage or a NetworkPolicy applied to the wrong selector. All of these are rare. But rare is not never, and the cost here does not scale with the frequency, because nothing alerts and nobody finds out for weeks.

So, three Chaos Mesh NetworkChaos objects, all targeting the primary pod: one partitions it from the two replicas, one from the operator pods in cnpg-system, and one from the control plane. The third is the interesting one:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: sb-partition-external
spec:
  action: partition
  direction: both
  mode: all
  selector:
    namespaces: [default]
    labelSelectors:
      cnpg.io/instanceName: postgres-splitbrain-1   # the current primary
  externalTargets:
    # public addresses below are replaced with the RFC 5737 documentation range
    - 10.43.0.1     # kubernetes Service ClusterIP
    - 203.0.113.1   # API server endpoint
    - 10.42.0.1     # node CNI bridges: where kubelet probes come from
    - 10.42.1.1
    - 10.42.3.1
    - 10.42.4.1
    - 10.42.5.1
    - 203.0.113.2   # node IPs
    - 203.0.113.3
    - 203.0.113.4
    - 203.0.113.5

The CNI bridge addresses are in there for a reason worth stealing: CNPG refuses to fail over while Kubernetes still reports the pod as Ready, and says so itself.

Warning  PrimaryStatusCheckFailed  cluster/postgres-splitbrain 
Primary pod postgres-splitbrain-1 is Ready but the operator's /pg/status check failed; 
failover deferred until Kubernetes marks the primary as not Ready.

Kubelet's probes originate from the bridge, so cutting it is what eventually gets the pod marked NotReady. Note also what is not in any of this: the clients. They are untouched. They stay on the old primary because their TCP session survives, not because we pin them there.

What the events say, and what the ledger says #

Partition applied at 18:11:20. Inside the first minute, both mechanisms do exactly what they are supposed to:

18:11:54  Killing         Container postgres failed liveness probe, will be restarted
18:11:57  FailingOver     Current primary isn't healthy, initiating a failover from postgres-splitbrain-1
18:11:58  FailoverTarget  Failing over from postgres-splitbrain-1 to postgres-splitbrain-2

The isolation check fired 34 seconds in and Kubernetes decided to kill the pod. Four seconds later the operator promoted a replacement. Every component behaved as designed, and it was all over inside 40 seconds.

Except it was not, and the ledger shows it. It recorded which server acknowledged every write:

kubectl exec sb-truth-0 -- psql -U truth -d truth -c "
  select seen_server, seen_timeline, count(*) writes,
         to_char(min(acked_at),'HH24:MI:SS') first_ack,
         to_char(max(acked_at),'HH24:MI:SS') last_ack,
         min(post_id), max(post_id)
    from acked_posts group by 1,2 order by first_ack"

 seen_server | writes | first_ack | last_ack | min | max
-------------+--------+-----------+----------+-----+------
 10.42.4.248 |   1033 | 18:10:38  | 18:14:55 |   1 | 1033
 10.42.0.28  |    562 | 18:13:17  | 18:15:56 | 199 |  760

The isolated primary acknowledged its last write at 18:14:55, which is t+215s and 177 seconds after its replacement was promoted. And read the two rows against each other: the new primary started at 18:13:17 while the old one carried on until 18:14:55. Both were acknowledging commits at once, handing out the same ids. Grouping the same table by second, and keeping only the seconds where both servers appear:

 overlap_start | overlap_end | seconds
---------------+-------------+---------
 18:13:17      | 18:14:55    |      99

99 seconds in which both primaries were taking writes.

One caveat, because it is easy to overclaim here. A client only finds the new primary if it reconnects, and a client with a working connection never does. So how did one of ours end up there? We restarted a client pod shortly after the partition. That is the most ordinary event in Kubernetes: a deploy, a node drain, an OOM kill, an autoscaler. One restart was enough to put writes on both primaries at the same time.

If nobody had restarted anything, all three clients would have stayed on the old primary until it died, then moved across together. Writes would have gone to one primary at a time instead of both at once. As we are about to see, that barely changes the outcome.

Why the old primary outlived its replacement by 177 seconds #

Two numbers have now turned up, and they measure different things.

The 99 seconds is how long our clients actually wrote to both primaries. It depended on when one of them happened to restart, so it is partly luck. Restart a client earlier and it is longer. Restart nothing and it is zero.

The 177 seconds does not depend on luck at all. The new primary was promoted at t+38s and the old one went quiet at t+215s, and for every second in between there were two primaries able to accept writes, whether anybody was using them or not. Any client that reconnects in that window can fork the data. So it is the 177 seconds you should plan around, not the 99.

That gap comes from two separate timers, and nothing makes them agree.

One timer starts the new primary. The readiness probe fails three times, ten seconds apart, so Kubernetes marks the pod NotReady at around t+34s. Because failoverDelay is 0, the operator promotes a replacement straight away. Done by t+38s.

The other timer stops the old one. The liveness probe, which is where the isolation check lives, also fails three times ten seconds apart, and kubelet sends the kill at t+34s. So the fence starts at the same moment. So far this all sounds fine.

The catch is what CNPG does with that kill signal. It runs a smart shutdown, and a smart shutdown does not disconnect anyone. It refuses new connections, then waits for the existing sessions to finish on their own, for as long as smartShutdownTimeout allows. The default is 180 seconds.

Our clients never finish. They hold their connection on purpose, the way a connection pool does, so they get all 180 seconds. The kill landed at 18:11:54 and the last write was acknowledged at 18:14:55.

So failoverDelay and the shutdown budget are not consistent with each other. The operator promotes at t+38s, while the old primary is guaranteed to keep serving its existing clients until t+215s. There is no setting for "do not promote until the old primary is confirmed down", and there could not be, because the operator cannot reach the old primary to confirm anything about it.

The isolation check is not broken. It starts on time. It just takes six times longer to finish than the promotion it is racing.

What this looks like from the application #

Alice writes a post, gets id=123456, and shares /posts/123456. On the isolated primary that id is her post. On the new primary it is somebody else's. Same URL, two different objects, both returning 200.

The partition heals, pg_rewind runs, and Alice's branch is the one that loses. The next day she opens her own link.

It is not a 404. A 404 would be honest, and monitoring would catch it. She gets 200 OK and a stranger's post. The preview card in her friends' chat still shows her title, cached when she shared it, so the chat is lying too. And if the row saying "post 123456 is visible to Alice's friends" survived from her branch while the post came from the other one, this is no longer a data problem. It is a privacy incident.

And the collisions do not stop when the incident does. The isolated primary gave Alice id 123456, but the surviving branch never saw that post, so its counter is still below 123456. It will hand that id out again, to a brand new post, days later.

How many ids does this affect? Nobody can say. The branch that issued them has been deleted, so there is no record of how far it got. Ten? Ten thousand? Our schema is one table with no foreign keys, which makes the damage easy to describe. A real one has rows referencing rows, ids copied into other services, event streams and warehouses, and links people have already shared. Every one of those references now points at something that may not be what it was written for, and nothing in the database can tell you which ones.

Counting the damage #

Nothing in the database can tell you. But the ledger is not in the database.

It sat outside the cluster for the whole experiment, writing down what every client was told was durable, and pg_rewind never got near it. That is the only reason this post has numbers in it instead of adjectives.

The check is simple. Take every write the ledger says was acknowledged after the two branches split, look it up in the surviving cluster, and see which of three things happened to it. It is still there and unchanged. It is gone. Or the id is still there, but the row under it now belongs to a different client.

writes acknowledged to clients after the split:       1472
still there, unchanged:                                619
GONE - the post the client was promised is absent:     291
STOLEN - that id now belongs to another client:        562

client                          intact   gone  stolen
sb-client-7974695569-xqhz6         141    145     281
sb-client-7974695569-rqvld         140    145     281
sb-client-7974695569-7htdt          19      1       0
sb-client-7974695569-w2pls         319      0       0

562 writes that a client was told were durable now point at a row belonging to somebody else. And look at the last two lines. 7htdt is the pod we restarted, w2pls is its replacement, and it is the one that landed on the new primary. It lost nothing at all. The two clients that did nothing but hold their connection lost or had reassigned three quarters of everything they were promised.

Meanwhile:

$ kubectl get cluster postgres-splitbrain -o custom-columns='READY:.status.readyInstances,PHASE:.status.phase'
READY   PHASE
3       Cluster in healthy state

$ kubectl exec postgres-splitbrain-2 -c postgres -- psql -qAt -d sbdemo -c \
    "select count(*) from (select id from posts group by id having count(*)>1) x"
0

Three of three ready, no duplicate keys, no constraint violations, not one error in any Postgres log.

The cluster is not lying. pg_rewind did its job properly. It rolled the old primary all the way back to the point where the two branches split, leaving nothing half-written behind. What you are looking at is a completely consistent database. It is just not the one the clients were told about.

That is what makes this so hard to spot. The broken references are not in Postgres at all. They are in the other service that stored an id, the cache, the search index, the email that already went out. And you cannot go back and find them later, because the branch that would tell you which ids were affected has been deleted. We only have the numbers above because something outside the cluster was writing down what each client was told.

Making the two budgets agree #

The fix is to stop the two timers working against each other. Both probes default to failureThreshold: 3 and periodSeconds: 10, so each takes about 30 seconds to give up. After that, one starts smartShutdownTimeout and the other starts failoverDelay:

old primary stops  = 30s + smartShutdownTimeout  = 30 + 180 = 210s
new primary starts = 30s + failoverDelay         = 30 +   0 =  30s

Which is the gap we measured. You need failoverDelay larger than smartShutdownTimeout, and larger by a margin, because those two 30-second timers are separate and will not fire in the same instant. In our runs the kill landed 3 seconds before the operator's decision, which happened to help us, but nothing guarantees it. Twenty or thirty seconds of slack is plenty.

There are two ways to satisfy it. Leave the graceful shutdown alone and raise failoverDelay past 180, to something like 210. Or make the fence itself fast with smartShutdownTimeout: 0, and put a smaller failoverDelay on top. The first keeps graceful drains on healthy switchovers and costs a three-minute failover. The second gives up the drains and fails over in about a minute. Which one you want depends on your workload, and we have not tested them against each other. We tested the second:

spec:
  smartShutdownTimeout: 0   # stop waiting for clients to leave, drop them
  failoverDelay: 30         # and do not promote until that has certainly happened

Same cluster, same chaos, same clients, partition at 18:18:52:

 seen_server | writes | first_ack | last_ack | min | max
-------------+--------+-----------+----------+-----+-----
 10.42.0.28  |    307 | 18:18:13  | 18:19:26 |   1 | 307
 10.42.4.248 |    350 | 18:20:24  | 18:21:24 | 199 | 548

The old primary went quiet at 18:19:26, the same second kubelet killed it. The new primary took its first write 58 seconds later, and nothing accepted writes in between. That gap is the price of putting the two in the right order.

	old primary stops	overlap	GONE	STOLEN
defaults	t+215s	99s	291	562
smartShutdownTimeout: 0 + failoverDelay: 30	t+34s	0s	21	109

Five times less damage. Now look at the second row again: it is not zero.

The old primary spent 34 seconds accepting writes its replicas never received, because replication is asynchronous. Those writes were acknowledged and then thrown away, and the new primary handed their ids out again. None of that needs an overlap. It happens whenever the old primary gets ahead before it is stopped.

So timers only make the window smaller. Closing it altogether means the primary must not acknowledge a write its replicas have not seen, and no timer can promise that. Only synchronous replication can, which is a subject of its own.

What Patroni does differently, and what it does not #

It is worth asking how another HA stack handles the same partition, because the difference is instructive in both directions.

Patroni does not infer leadership from pod readiness. It holds a leader lock in a distributed configuration store (etcd, Consul, ZooKeeper, or the Kubernetes control plane), and that lock is a lease with a time to live. The ttl parameter defaults to 30 seconds, with a minimum of 20, and it sets the dead time before an automatic failover may begin.

The losing side is the interesting half. The primary has to renew its lease every loop_wait, 10 seconds by default. If it cannot reach the store, which is exactly our partition, it cannot renew, and as the lease runs out it demotes itself, without needing to hear from anybody. Only once the lease has actually expired can another node take the lock and promote.

So the order is guaranteed by the design. The old primary has stopped before the new one can start, because both wait on the same lease rather than on two separate timers that happen to be running near each other. That is the part CloudNativePG leaves you to configure, and by default configures against you.

To be clear about what that buys: it rules out split brain. Two primaries never take writes at the same time, because the lease makes that impossible rather than unlikely. We reached zero overlap on CloudNativePG too, but only by setting two timers correctly, and that is a weaker thing. It held in our runs, and it holds for as long as the arithmetic stays right.

The lease does not protect the writes, though. An isolated Patroni primary still spends up to ttl seconds accepting writes its replicas never receive, for the same reason ours did: replication is asynchronous, and nothing is checking. When it steps down those writes are gone and the new leader hands their ids out again. About 30 seconds of that, against the 34 we were left with after tuning the timers.

So stopping the old primary and keeping its writes are two different problems. A lease solves the first outright, which is more than a pair of timers can promise. Neither solves the second without synchronous replication.

Closing #

The failure is real, and it is not exotic. A partition like this is rare, but the probability is never zero, and on default settings the window is already there in your cluster.

So go and look at your own setup. Compare failoverDelay with smartShutdownTimeout, and if an acknowledged write matters to you, consider synchronous replication.

And this is what chaos testing is for. None of it would have turned up in the docs or in a config review. Breaking a cluster you do not mind breaking is the cheapest way to find out whether the guarantees you think you have are the ones you actually have.

Thanks to Nikolay Samokhvalov for the inspiration and the in-depth details.