Skip to contentSkip to conversation

md/raid10: wake pending freeze in raid10d()

PATCH NAK Kevin Vigor 2 participants 3 messages Dec 31, 2020 – Jan 18, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 md/raid10: wake pending freeze in raid10d()

The freeze_array() function waits for changes to nr_queued, but the raid10d() function alters it without sending a wake.

This can lead to freeze_array() never being woken.

---
 drivers/md/raid10.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/md/raid10.c +7 -0
index c5d88ef6a45c..f98df9f084c2 100644
@@ -2698,6 +2698,7 @@ static void raid10d(struct md_thread *thread)
 	struct r10conf *conf = mddev->private;
 	struct list_head *head = &conf->retry_list;
 	struct blk_plug plug;
+	int dequeued = 0;
 
 	md_check_recovery(mddev);
 
@@ -2709,6 +2710,7 @@ static void raid10d(struct md_thread *thread)
 			while (!list_empty(&conf->bio_end_io_list)) {
 				list_move(conf->bio_end_io_list.prev, &tmp);
 				conf->nr_queued--;
+				dequeued++;
 			}
 		}
 		spin_unlock_irqrestore(&conf->device_lock, flags);
@@ -2739,6 +2741,7 @@ static void raid10d(struct md_thread *thread)
 		r10_bio = list_entry(head->prev, struct r10bio, retry_list);
 		list_del(head->prev);
 		conf->nr_queued--;
+		dequeued++;
 		spin_unlock_irqrestore(&conf->device_lock, flags);
 
 		mddev = r10_bio->mddev;
@@ -2762,6 +2765,10 @@ static void raid10d(struct md_thread *thread)
 			md_check_recovery(mddev);
 	}
 	blk_finish_plug(&plug);
+
+	/* in case freeze_array is waiting for changes to nr_queued. */
+	if (conf->array_freeze_pending && dequeued)
+		wake_up(&conf->wait_barrier);
 }
 
 static int init_resync(struct r10conf *conf)
Signature · 1 line
2.26.2
#1 lore raw

PATCH 2/2 md/raid10: introducing range-based barriers

Current resync code uses heavyweight barriers: sync requests call raise_barrier(), which waits for all non-sync traffic to drain; user requests call wait_barrier() which waits for all sync traffic to drain. This ensures that there is no consistency error caused by user writes sneaking in the middle of a resync request, but has severe performance impact (in test we see user write bandwidth reduced by 70% during check, even with sync_speed_max set to ridiculously small value such as 1MB/s).

This patch optionally enables range based locking, where only the portion of the device being synched is locked, significantly improving performance during a sync operation (in same test above we see user bandwidth reduction of 15% with resync achieving rates in the 100MB/s range).

The principle is that during a resync, all sync and user write requests obtain an exclusive lock on the virtual sector range they alter, blocking if necessary (it is assumed that user reads may safely proceed even when there is an outstanding sync request, since a resync by definition cannot change existing correct data). The lock is released when the owning rbio is completed, i.e. when the entire mirrored operation is complete.

The existing barrier code is left in place since it is used during drains and reshape operations. The only alteration to the pre-existing barrier code is that when range based barriers are enabled, sync requests issued in raid10_sync_request() do not call raise_barrier(), but instead call lock_range().

---
 Documentation/admin-guide/md.rst |  12 ++
 drivers/md/Kconfig               |  20 ++
 drivers/md/raid10.c              | 304 ++++++++++++++++++++++++++++++-
 drivers/md/raid10.h              |  54 ++++++
 4 files changed, 386 insertions(+), 4 deletions(-)
Documentation/admin-guide/md.rst +12 -0
index d8fc9a59c086..ebabd2dad00f 100644
@@ -763,3 +763,15 @@ These currently include:
 
   ppl_write_hint
       NVMe stream ID to be set for each PPL write request.
+
+  rangebased_barriers/enabled (currently raid10 only)
+      uses range-based locking instead of coarse barriers to prevent
+      resync IO and user writes from colliding. May improve performance
+      during a resync.
+
+  rangebased_barriers/max_locked_ranges (currently raid10 only)
+      maximum number of sector ranges that may be locked at one time in
+      range-based locking mode.
+
+  rangebased_barriers/stats (currently raid10 only)
+      counters pertinent to range-based locking.
drivers/md/Kconfig +20 -0
index b7e2d9666614..f6301448fd84 100644
@@ -120,6 +120,26 @@ config MD_RAID10
 
 	  If unsure, say Y.
 
+config MD_RAID10_RBB
+	bool "RAID-10 range-based barriers"
+	depends on MD_RAID10
+	default n
+	help
+	  Enable optional range-based barriers in RAID-10 resync code.
+	  This may improve performance during a resync operation.
+
+	  If unsure, say N.
+
+config MD_RAID10_RBB_CHECKS
+	bool "Sanity checks for RAID-10 range-based barriers"
+	depends on MD_RAID10_RBB
+	default n
+	help
+	  Enable this to turn on some internal sanity checks in RAID-10
+	  range-based barrier code.
+
+	  If unsure, say N.
+
 config MD_RAID456
 	tristate "RAID-4/RAID-5/RAID-6 mode"
 	depends on BLK_DEV_MD
drivers/md/raid10.c +300 -4

401 lines not on this page · raw message

drivers/md/raid10.h +54 -0

83 lines not on this page · raw message

Signature · 1 line
2.26.2
#2 lore raw
In reply to Kevin Vigor
Quoted 4 lines · Kevin Vigor

The freeze_array() function waits for changes to nr_queued, but the raid10d() function alters it without sending a wake.

This can lead to freeze_array() never being woken.

No it cannot.
freeze_array() is only called from handle_read_error() which is only called from raid10d(). So it cannot possibly be sleeping at the point where you added the wake_up().
handle_read_error() passes '1' as the 'extra' arg to freeze_array() which exactly compensates for the recent decrement of ->nr_queued.

NAK.

Thanks,
NeilBrown

Quoted 47 lines
---
 drivers/md/raid10.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/md/raid10.c +7 -0
index c5d88ef6a45c..f98df9f084c2 100644
@@ -2698,6 +2698,7 @@ static void raid10d(struct md_thread *thread)
 	struct r10conf *conf = mddev->private;
 	struct list_head *head = &conf->retry_list;
 	struct blk_plug plug;
+	int dequeued = 0;
 
 	md_check_recovery(mddev);
 
@@ -2709,6 +2710,7 @@ static void raid10d(struct md_thread *thread)
 			while (!list_empty(&conf->bio_end_io_list)) {
 				list_move(conf->bio_end_io_list.prev, &tmp);
 				conf->nr_queued--;
+				dequeued++;
 			}
 		}
 		spin_unlock_irqrestore(&conf->device_lock, flags);
@@ -2739,6 +2741,7 @@ static void raid10d(struct md_thread *thread)
 		r10_bio = list_entry(head->prev, struct r10bio, retry_list);
 		list_del(head->prev);
 		conf->nr_queued--;
+		dequeued++;
 		spin_unlock_irqrestore(&conf->device_lock, flags);
 
 		mddev = r10_bio->mddev;
@@ -2762,6 +2765,10 @@ static void raid10d(struct md_thread *thread)
 			md_check_recovery(mddev);
 	}
 	blk_finish_plug(&plug);
+
+	/* in case freeze_array is waiting for changes to nr_queued. */
+	if (conf->array_freeze_pending && dequeued)
+		wake_up(&conf->wait_barrier);
 }
 
 static int init_resync(struct r10conf *conf)
Signature · 1 line
2.26.2