drbd: create new uuid even we dont have quorum
PATCH 1/1 drbd: create new uuid even we dont have quorum
We cant skip creating new uuid, when we are making new uuid.
If there is an IO blocked by creating new uuid, but we found we dont have quorum, then NEW_CUR_UUID will be cleared but new uuid was not actually created.
After one peer online, we have enough quorum to continue IO, then this IO will reach primary and the online peer.
But if there is another peer join later, this peer will found same uuid, and will lose data.
--- drbd/drbd_sender.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drbd/drbd_sender.c +1 -1
@@ -2360,7 +2360,7 @@ void drbd_check_peers_new_current_uuid(struct drbd_device *device)drbd_check_peers (resource); -if (device->have_quorum[NOW ] &&drbd_data_accessible (device,NOW )) +if (drbd_data_accessible (device,NOW ))drbd_uuid_new_current (device, false); }
Signature · 1 line
1.8.3.1
Hi,
When quorum is configured, we expect not to have split-brain situations. Hence it is important that we do not generate new UUIDs until they are definitely necessary. When we do not have quorum, no writes should complete, so we do not need to generate a new UUID. We may need to generate one before we regain quorum instead.
That said, there have been various bugs in the implementation. Try to reproduce your issue with the latest drbd-9.1. If it can still be reproduced, then describe the exact reproduction steps and we can discuss the appropriate fix.
Best regards,
Joel
Hi Joel,
Thanks for the reply. I have a 3 node cluster and all 3 nodes running with DRBD9.0: Primary node is node-1, and Secondary nodes are node-2 and node-3, quorum is configured as 2.
After some more investigation, I found the root cause of this problem we met is that NEW_CUR_UUID will clear whether a new uuid was actually generated in make_new_current_uuid.
You can reproduce this problem on DRBD9.0 through the following steps:
(1) network failure on node-2
(2) network failure on node-3
(3) write on node-1(will suspend since quorum loss)
(4) node-2 network recovery
(5) node-3 network recovery
you will find that data on node-3 is inconsistent with two other nodes.
I also find this problem is not exist in DRBD9.1 since the code of inc_ap_bio_cond in DRBD9.1 is different from that in DRBD9.0, so i have sent a patch for this bug in DRBD9.0:
https://lists.linbit.com/pipermail/drbd-dev/2022-April/006576.html
So you can go to review my new patch directly.
By the way, have you focused on DRBD9.1 and is there a stable verison of it? Thanks,
Xu
From: Joel Colledge <[email protected]>
Date: 2022-04-07 18:48:12
To: Rui Xu <[email protected]>
Cc: Philipp Reisner <[email protected]>,[email protected],[email protected]
Subject: Re: [PATCH] drbd: create new uuid even we dont have quorum>Hi,
Quoted 14 lines
When quorum is configured, we expect not to have split-brain situations. Hence it is important that we do not generate new UUIDs until they are definitely necessary. When we do not have quorum, no writes should complete, so we do not need to generate a new UUID. We may need to generate one before we regain quorum instead.
That said, there have been various bugs in the implementation. Try to reproduce your issue with the latest drbd-9.1. If it can still be reproduced, then describe the exact reproduction steps and we can discuss the appropriate fix.
Best regards,
Joel
Hi Xu,
Quoted 1 line
By the way, have you focused on DRBD9.1 and is there a stable verison of it?
We have stopped working on 9.0. The final release was 9.0.32 (see the release announcement). For now, we are continuing to apply fixes to allow it to be built for newer kernels, but otherwise the 9.0 branch is end-of-life.
DRBD 9.1 is as stable as 9.0 was. The most recent release is 9.1.6.
Since 9.1.6, the 9.1 branch has received significant work in the area of handling suspended IO due to quorum loss. It looks like this might be relevant for you. There will be a 9.1.7 release soon containing these fixes.
Best regards,
Joel