Skip to contentSkip to conversation

drbd: create new uuid even we dont have quorum

PATCH In review Rui Xu 2 participants 4 messages Mar 3, 2022 – Apr 11, 2022 DRBD driver

lore
Share & tools
lore

Rui Xu

author
#0 lore raw

PATCH 1/1 drbd: create new uuid even we dont have quorum

We cant skip creating new uuid, when we are making new uuid.

If there is an IO blocked by creating new uuid, but we found we dont have quorum, then NEW_CUR_UUID will be cleared but new uuid was not actually created.

After one peer online, we have enough quorum to continue IO, then this IO will reach primary and the online peer.

But if there is another peer join later, this peer will found same uuid, and will lose data.

---
 drbd/drbd_sender.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drbd/drbd_sender.c +1 -1
index 7238730..88d49d9 100644
@@ -2360,7 +2360,7 @@ void drbd_check_peers_new_current_uuid(struct drbd_device *device)
 
 	drbd_check_peers(resource);
 
-	if (device->have_quorum[NOW] && drbd_data_accessible(device, NOW))
+	if (drbd_data_accessible(device, NOW))
 		drbd_uuid_new_current(device, false);
 }
 
Signature · 1 line
1.8.3.1
#1 lore raw

Hi,

When quorum is configured, we expect not to have split-brain situations. Hence it is important that we do not generate new UUIDs until they are definitely necessary. When we do not have quorum, no writes should complete, so we do not need to generate a new UUID. We may need to generate one before we regain quorum instead.

That said, there have been various bugs in the implementation. Try to reproduce your issue with the latest drbd-9.1. If it can still be reproduced, then describe the exact reproduction steps and we can discuss the appropriate fix.

Best regards,
Joel

author
#2 lore raw

Hi Joel,
Thanks for the reply. I have a 3 node cluster and all 3 nodes running with DRBD9.0: Primary node is node-1, and Secondary nodes are node-2 and node-3, quorum is configured as 2.

After some more investigation, I found the root cause of this problem we met is that NEW_CUR_UUID will clear whether a new uuid was actually generated in make_new_current_uuid.

You can reproduce this problem on DRBD9.0 through the following steps:
(1) network failure on node-2
(2) network failure on node-3
(3) write on node-1(will suspend since quorum loss)
(4) node-2 network recovery
(5) node-3 network recovery
you will find that data on node-3 is inconsistent with two other nodes.

I also find this problem is not exist in DRBD9.1 since the code of inc_ap_bio_cond in DRBD9.1 is different from that in DRBD9.0, so i have sent a patch for this bug in DRBD9.0:

https://lists.linbit.com/pipermail/drbd-dev/2022-April/006576.html

So you can go to review my new patch directly.
By the way, have you focused on DRBD9.1 and is there a stable verison of it? Thanks,
Xu

From: Joel Colledge <[email protected]>
Date: 2022-04-07 18:48:12
To: Rui Xu <[email protected]>
Cc: Philipp Reisner <[email protected]>,[email protected],[email protected]
Subject: Re: [PATCH] drbd: create new uuid even we dont have quorum>Hi,

Quoted 14 lines

When quorum is configured, we expect not to have split-brain situations. Hence it is important that we do not generate new UUIDs until they are definitely necessary. When we do not have quorum, no writes should complete, so we do not need to generate a new UUID. We may need to generate one before we regain quorum instead.

That said, there have been various bugs in the implementation. Try to reproduce your issue with the latest drbd-9.1. If it can still be reproduced, then describe the exact reproduction steps and we can discuss the appropriate fix.

Best regards,
Joel

#3 lore raw

Hi Xu,

Quoted 1 line

By the way, have you focused on DRBD9.1 and is there a stable verison of it?

We have stopped working on 9.0. The final release was 9.0.32 (see the release announcement). For now, we are continuing to apply fixes to allow it to be built for newer kernels, but otherwise the 9.0 branch is end-of-life.

DRBD 9.1 is as stable as 9.0 was. The most recent release is 9.1.6.

Since 9.1.6, the 9.1 branch has received significant work in the area of handling suspended IO due to quorum loss. It looks like this might be relevant for you. There will be a 9.1.7 release soon containing these fixes.

Best regards,
Joel