Skip to contentSkip to conversation

drbd: do not finish the stable resync when repl state is L_PAUSED_SYNC_T

PATCH In review Rui Xu 2 participants 4 messages Sep 1, 2021 – Sep 10, 2021 DRBD driver

lore
Share & tools
lore

Rui Xu

author
#0 lore raw

PATCH 1/1 drbd: do not finish the stable resync when repl state is L_PAUSED_SYNC_T

The assumption was, when a unstable resync start first, then a stable resync starts and be paused. If the stable resync finish ahead of the unstable one from paused directly, it will leaving the resync end into INCONSISTENT finally.

So don't finish the stable resync when repl state is L_PAUSED_SYNC_T, finish it when the repl state turns into L_SYNC_TARGET.

---
 drbd/drbd_actlog.c | 15 +++++++++++++++
 drbd/drbd_int.h    |  1 +
 drbd/drbd_sender.c |  5 +++++
 3 files changed, 21 insertions(+)
drbd/drbd_actlog.c +15 -0
index 1676c355..3900b426 100644
@@ -1117,7 +1117,22 @@ static int update_sync_bits(struct drbd_peer_device *peer_device,
 		if (mode == SET_IN_SYNC) {
 			unsigned long still_to_go = drbd_bm_total_weight(peer_device);
 			bool rs_is_done = (still_to_go <= peer_device->rs_failed);
+			enum drbd_repl_state repl_state = peer_device->repl_state[NOW];
 			drbd_advance_rs_marks(peer_device, still_to_go);
+                        if (rs_is_done && repl_state == L_PAUSED_SYNC_T && !test_bit(UNSTABLE_RESYNC, &peer_device->flags)) {
+                                struct drbd_device *device = peer_device->device;
+                                struct drbd_peer_device *p;
+                                for_each_peer_device(p, device) {
+                                    if (p == peer_device)
+                                        continue;
+
+                                    if (test_bit(UNSTABLE_RESYNC, &p->flags) && p->repl_state[NOW] == L_SYNC_TARGET) {
+                                        rs_is_done = false;
+                                        peer_device->finish_resync_delay = true;
+                                        break;
+                                    }
+                                }
+                        }
 			if (cleared || rs_is_done)
 				maybe_schedule_on_disk_bitmap_update(peer_device, rs_is_done);
 		} else if (mode == RECORD_RS_FAILED) {
drbd/drbd_int.h +1 -0
index 8db0007e..a3a7d241 100644
@@ -1095,6 +1095,7 @@ struct drbd_peer_device {
 	bool resync_susp_peer[2];
 	bool resync_susp_dependency[2];
 	bool resync_susp_other_c[2];
+	bool finish_resync_delay;
 	enum drbd_repl_state negotiation_result; /* To find disk state after attach */
 	unsigned int send_cnt;
 	unsigned int recv_cnt;
drbd/drbd_sender.c +5 -0
index f3c39b0f..01ce46b3 100644
@@ -735,6 +735,7 @@ next_sector:
 			goto request_done;
 		}
 
+		peer_device->finish_resync_delay = false;
 		sector = BM_BIT_TO_SECT(bit);
 
 		if (drbd_try_rs_begin_io(peer_device, sector, true)) {
@@ -833,6 +834,10 @@ request_done:
 		 * resync data block, and the last bit is cleared.
 		 * until then resync "work" is "inactive" ...
 		 */
+                if (peer_device->finish_resync_delay) {
+                	drbd_resync_finished(peer_device, D_MASK);
+                	peer_device->finish_resync_delay = false;
+                }
 		put_ldev(device);
 		return 0;
 	}
Signature · 1 line
2.25.1
#1 lore raw

Hi,

Thanks for the contribution. Please describe the steps necessary to reproduce the issue. I tried briefly to reproduce it without success; getting a paused sync to finish before a running one is not that easy. A test in https://github.com/LINBIT/drbd9-tests would be ideal, but a written description is OK too.

Please also verify that the issue occurs on the latest drbd-9.0 or master branch. I see from the patch that you are working on a version that is at least a couple of months old.

Best regards,
Joel

author
#2 lore raw

Hi Joel,<br/> Thanks for the reply and sorry for the delay. After some more <br/>investigation, I found the root cause of this problem we met is that <br/>there is a race condition in update_sync_bits() and receive_bitmap().<br/>This race condition will lead the problem I was trying to solve in this <br/>patch above.<br/><br/>We confirmed the race condition existing in upstream and I have sent a <br/>patch for it:<br/>https://lists.linbit.com/pipermail/drbd-dev/2021-September/005840.html<br/><br/>So you can go to review my new patch directly.<br/><br/>Thanks,<br/>Xu
From: Joel Colledge <[email protected]>
Date: 2021-09-01 16:40:32
To: Rui Xu <[email protected]>
Cc: Philipp Reisner <[email protected]>,[email protected]
Subject: Re: [Drbd-dev] [PATCH] drbd: do not finish the stable resync when repl state is L_PAUSED_SYNC_T

author
#3 lore raw
In reply to Joel Colledge

Hi Joel,
I'm sorry there were some format problems in my last email.

Thanks for the reply and sorry for the delay. After some more investigation, I found the root cause

of this problem we met is that there is a race condition in update_sync_bits() and receive_bitmap(). This race condition will lead the problem I was trying to solve in this patch above.

We confirmed the race condition existing in upstream and I have sent a patch for it:
https://lists.linbit.com/pipermail/drbd-dev/2021-September/005840.html

So you can go to review my new patch directly.
Thanks,
Xu

From: Joel Colledge <[email protected]>
Date: 2021-09-01 16:40:32
To: Rui Xu <[email protected]>
Cc: Philipp Reisner <[email protected]>,[email protected]
Subject: Re: [Drbd-dev] [PATCH] drbd: do not finish the stable resync when repl state is L_PAUSED_SYNC_T>Hi,

Quoted 13 lines

Thanks for the contribution. Please describe the steps necessary to reproduce the issue. I tried briefly to reproduce it without success; getting a paused sync to finish before a running one is not that easy. A test in https://github.com/LINBIT/drbd9-tests would be ideal, but a written description is OK too.

Please also verify that the issue occurs on the latest drbd-9.0 or master branch. I see from the patch that you are working on a version that is at least a couple of months old.

Best regards,
Joel