Improve handling raid10 discard request
v2 cover Improve handling raid10 discard request
Hi all
Now mkfs on raid10 which is combined with ssd/nvme disks takes a long time. This patch set tries to resolve this problem.
Xiao Ni (3):
Move codes related with submitting discard bio into one function extend r10bio devs to raid disks
improve raid10 discard request
drivers/md/md.c | 22 ++++ drivers/md/md.h | 3 + drivers/md/raid0.c | 15 +-- drivers/md/raid10.c | 297 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 318 insertions(+), 19 deletions(-)
Signature · 1 line
2.7.5
PATCH v2 1/3 Move codes related with submitting discard bio into one function
These codes can be used in raid10. So we can move these codes into md.c. raid0 and raid10 can share these codes.
--- drivers/md/md.c | 22 ++++++++++++++++++++++ drivers/md/md.h | 3 +++ drivers/md/raid0.c | 15 ++------------- 3 files changed, 27 insertions(+), 13 deletions(-)
drivers/md/md.c +22 -0
@@ -8559,6 +8559,28 @@ void md_write_end(struct mddev *mddev)EXPORT_SYMBOL (md_write_end); +void md_submit_discard_bio (struct mddev *mddev,struct md_rdev *rdev, +struct bio *bio, +sector_t dev_start,sector_t dev_end) +{ +struct bio *discard_bio =NULL ; + +if (__blkdev_issue_discard (rdev->bdev, + dev_start + rdev->data_offset, + dev_end - dev_start,GFP_NOIO ,0 , &discard_bio) || + !discard_bio) +return ; + +bio_chain (discard_bio, bio); +bio_clone_blkg_association (discard_bio, bio); +if (mddev->gendisk) +trace_block_bio_remap (bdev_get_queue (rdev->bdev), + discard_bio,disk_devt (mddev->gendisk), + bio->bi_iter.bi_sector); +submit_bio_noacct (discard_bio); +} +EXPORT_SYMBOL (md_submit_discard_bio); +/* md_allow_write(mddev) * Calling this ensures that the array is marked 'active' so that writes * may proceed without blocking. It is important to call this before
drivers/md/md.h +3 -0
@@ -710,6 +710,9 @@ extern void md_write_end(struct mddev *mddev);extern void md_done_sync (struct mddev *mddev,int blocks,int ok);extern void md_error (struct mddev *mddev,struct md_rdev *rdev);extern void md_finish_reshape (struct mddev *mddev); +extern void md_submit_discard_bio (struct mddev *mddev,struct md_rdev *rdev, +struct bio *bio, +sector_t dev_start,sector_t dev_end);extern int mddev_congested (struct mddev *mddev,int bits);extern bool __must_check md_flush_request (struct mddev *mddev,struct bio *bio);
drivers/md/raid0.c +2 -13
@@ -525,7 +525,6 @@ static void raid0_handle_discard(struct mddev *mddev, struct bio *bio)for (disk =0 ; disk < zone->nb_dev; disk++) {sector_t dev_start, dev_end; -struct bio *discard_bio =NULL ;struct md_rdev *rdev;if (disk < start_disk_index)
@@ -548,18 +547,8 @@ static void raid0_handle_discard(struct mddev *mddev, struct bio *bio) rdev = conf->devlist[(zone - conf->strip_zone) * conf->strip_zone[0 ].nb_dev + disk]; -if (__blkdev_issue_discard (rdev->bdev, - dev_start + zone->dev_start + rdev->data_offset, - dev_end - dev_start,GFP_NOIO ,0 , &discard_bio) || - !discard_bio) -continue ; -bio_chain (discard_bio, bio); -bio_clone_blkg_association (discard_bio, bio); -if (mddev->gendisk) -trace_block_bio_remap (bdev_get_queue (rdev->bdev), - discard_bio,disk_devt (mddev->gendisk), - bio->bi_iter.bi_sector); -submit_bio_noacct (discard_bio); + dev_start += zone->dev_start; +md_submit_discard_bio (mddev, rdev, bio, dev_start, dev_end); }bio_endio (bio); }
Signature · 1 line
2.7.5
Re: [PATCH V2 1/3] Move codes related with submitting discard bio into one function
Hi Xiao,
Better to add a prefix before subject like "md" ...", the same to other patches.
Quoted 16 lines · Xiao Ni
These codes can be used in raid10. So we can move these codes into
md.c. raid0 and raid10 can share these codes.Reviewed-by Coly Li <[email protected]>Signed-off-by Xiao Ni <[email protected]>--- drivers/md/md.c | 22 ++++++++++++++++++++++ drivers/md/md.h | 3 +++ drivers/md/raid0.c | 15 ++------------- 3 files changed, 27 insertions(+), 13 deletions(-)drivers/md/md.c +0
-0
drivers/md/md.c void md_write_end(struct mddev *mddev) · line 8561
Quoted 3 lines
drivers/md/md.c +0
-0@@ -8559,6 +8559,28 @@ void md_write_end(struct mddev *mddev)EXPORT_SYMBOL (md_write_end);
Could you add comment before the function to inform it is used by raid0 and raid10?
drivers/md/md.c void md_write_end(struct mddev *mddev) · line 8564
Quoted 3 lines
drivers/md/md.c +3
-0+void md_submit_discard_bio (struct mddev *mddev,struct md_rdev *rdev, +struct bio *bio, +sector_t dev_start,sector_t dev_end)
Thanks,
Guoqing
Re: [PATCH V2 1/3] Move codes related with submitting discard bio into one function
Hi Guoqing
Thanks for reviewing this patch set.
Quoted 3 lines · Guoqing Jiang
Hi Xiao,
Better to add a prefix before subject like "md" ...", the same to other patches.
Sure.
Quoted 13 lines · Xiao Ni
Quoted 11 lines
These codes can be used in raid10. So we can move these codes into
md.c. raid0 and raid10 can share these codes.Reviewed-by Coly Li <[email protected]>Signed-off-by Xiao Ni <[email protected]>--- drivers/md/md.c | 22 ++++++++++++++++++++++ drivers/md/md.h | 3 +++ drivers/md/raid0.c | 15 ++------------- 3 files changed, 27 insertions(+), 13 deletions(-)
Quoted 8 lines
drivers/md/md.c void md_write_end(struct mddev *mddev) · line 8559
Quoted 6 lines
drivers/md/md.c +0
-0@@ -8559,6 +8559,28 @@ void md_write_end(struct mddev *mddev)EXPORT_SYMBOL (md_write_end);Could you add comment before the function to inform it is used by raid0 and raid10?
I'll add comments in next version.
Regards
Xiao
Quoted 8 lines
drivers/md/md.c void md_write_end(struct mddev *mddev) · line 8564
Quoted 4 lines
drivers/md/md.c +3
-0+void md_submit_discard_bio (struct mddev *mddev,struct md_rdev *rdev, +struct bio *bio, +sector_t dev_start,sector_t dev_end)Thanks,
Guoqing
PATCH v2 2/3 extend r10bio devs to raid disks
In reply to Xiao NiNow it allocs r10bio->devs[conf->copies]. Discard bio needs to submit to all member disks and it needs to use r10bio. So extend to r10bio->devs[geo.raid_disks].
--- drivers/md/raid10.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/md/raid10.c +5 -5
@@ -91,7 +91,7 @@ static inline struct r10bio *get_resync_r10bio(struct bio *bio)static void *r10bio_pool_alloc (gfp_t gfp_flags,void *data) {struct r10conf *conf = data; - int size = offsetof(struct r10bio, devs[conf->copies]); + int size = offsetof(struct r10bio, devs[conf->geo.raid_disks]);/* allocate a r10bio with room for raid_disks entries in the * bios array */
@@ -238,7 +238,7 @@ static void put_all_bios(struct r10conf *conf, struct r10bio *r10_bio) {int i; -for (i =0 ; i < conf->copies; i++) { +for (i =0 ; i < conf->geo.raid_disks; i++) {struct bio **bio = & r10_bio->devs[i].bio;if (!BIO_SPECIAL (*bio))bio_put (*bio);
@@ -327,7 +327,7 @@ static int find_bio_disk(struct r10conf *conf, struct r10bio *r10_bio,int slot;int repl =0 ; -for (slot =0 ; slot < conf->copies; slot++) { +for (slot =0 ; slot < conf->geo.raid_disks; slot++) {if (r10_bio->devs[slot].bio == bio)break ;if (r10_bio->devs[slot].repl_bio == bio) {
@@ -336,7 +336,7 @@ static int find_bio_disk(struct r10conf *conf, struct r10bio *r10_bio, } } -BUG_ON (slot == conf->copies); +BUG_ON (slot == conf->geo.raid_disks);update_head_pos (slot, r10_bio);if (slotp)
@@ -1510,7 +1510,7 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors) r10_bio->mddev = mddev; r10_bio->sector = bio->bi_iter.bi_sector; r10_bio->state =0 ; -memset (r10_bio->devs,0 ,sizeof (r10_bio->devs[0 ]) * conf->copies); +memset (r10_bio->devs,0 ,sizeof (r10_bio->devs[0 ]) * conf->geo.raid_disks);if (bio_data_dir (bio) ==READ )raid10_read_request (mddev, bio, r10_bio);
Signature · 1 line
2.7.5
PATCH v2 3/3 improve raid10 discard request
In reply to Xiao NiNow the discard request is split by chunk size. So it takes a long time to finish mkfs on disks which support discard function. This patch improve handling raid10 discard request. It uses the similar way with patch 29efc390b (md/md0: optimize raid0 discard handling).
But it's a little complex than raid0. Because raid10 has different layout. If raid10 is offset layout and the discard request is smaller than stripe size. There are some holes when we submit discard bio to underlayer disks.
For example: five disks (disk1 - disk5)
D01 D02 D03 D04 D05
D05 D01 D02 D03 D04
D06 D07 D08 D09 D10
D10 D06 D07 D08 D09
The discard bio just wants to discard from D03 to D10. For disk3, there is a hole between D03 and D08. For disk4, there is a hole between D04 and D09. D03 is a chunk, raid10_write_request can handle one chunk perfectly. So the part that is not aligned with stripe size is still handled by raid10_write_request.
If reshape is running when discard bio comes and the discard bio spans the reshape position, raid10_write_request is responsible to handle this discard bio.
I did a test with this patch set.
Without patch:
time mkfs.xfs /dev/md0
real4m39.775s
user0m0.000s
sys0m0.298s
With patch:
time mkfs.xfs /dev/md0
real0m0.105s
user0m0.000s
sys0m0.007s
nvme3n1 259:1 0 477G 0 disk └─nvme3n1p1 259:10 0 50G 0 part nvme4n1 259:2 0 477G 0 disk └─nvme4n1p1 259:11 0 50G 0 part nvme5n1 259:6 0 477G 0 disk └─nvme5n1p1 259:12 0 50G 0 part nvme2n1 259:9 0 477G 0 disk └─nvme2n1p1 259:15 0 50G 0 part nvme0n1 259:13 0 477G 0 disk └─nvme0n1p1 259:14 0 50G 0 part
v1:
Coly helps to review these patches and give some suggestions:
One bug is found. If discard bio is across one stripe but bio size is bigger than one stripe size. After spliting, the bio will be NULL. In this version, it checks whether discard bio size is bigger than 2*stripe_size. In raid10_end_discard_request, it's better to check R10BIO_Uptodate is set or not. It can avoid write memory to improve performance.
Add more comments for calculating addresses.
v2:
Fix error by checkpatch.pl
Fix one bug for offset layout. v1 calculates wrongly split size Add more comments to explain how the discard range of each component disk is decided.
--- drivers/md/raid10.c | 287 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 286 insertions(+), 1 deletion(-)
drivers/md/raid10.c +286 -1
@@ -1518,6 +1518,287 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)raid10_write_request (mddev, bio, r10_bio); } +static void wait_blocked_dev (struct mddev *mddev,int cnt) +{ +int i; +struct r10conf *conf = mddev->private; +struct md_rdev *blocked_rdev =NULL ; + +retry_wait: +rcu_read_lock (); +for (i =0 ; i < cnt; i++) { +struct md_rdev *rdev =rcu_dereference (conf->mirrors[i].rdev); +struct md_rdev *rrdev =rcu_dereference ( + conf->mirrors[i].replacement); +if (rdev == rrdev) + rrdev =NULL ; +if (rdev &&unlikely (test_bit (Blocked, &rdev->flags))) { +atomic_inc (&rdev->nr_pending); + blocked_rdev = rdev; +break ; + } +if (rrdev &&unlikely (test_bit (Blocked, &rrdev->flags))) { +atomic_inc (&rrdev->nr_pending); + blocked_rdev = rrdev; +break ; + } + } +rcu_read_unlock (); + +if (unlikely (blocked_rdev)) { +/* Have to wait for this device to get unblocked, then retry */ +allow_barrier (conf); +raid10_log (conf->mddev,"%s wait rdev %d blocked" , + __func__, blocked_rdev->raid_disk); +md_wait_for_blocked_rdev (blocked_rdev, mddev); +wait_barrier (conf); + goto retry_wait; + } +} + +static struct bio *raid10_split_bio (struct r10conf *conf, +struct bio *bio,sector_t sectors,bool want_first) +{ +struct bio *split; + + split =bio_split (bio, sectors,GFP_NOIO , &conf->bio_split); +bio_chain (split, bio); +allow_barrier (conf); +if (want_first) { +submit_bio_noacct (bio); + bio = split; + }else +submit_bio_noacct (split); +wait_barrier (conf); + +return bio; +} + +static void raid10_end_discard_request (struct bio *bio) +{ +struct r10bio *r10_bio = bio->bi_private; +struct r10conf *conf = r10_bio->mddev->private; +struct md_rdev *rdev =NULL ; +int dev; +int slot, repl; + +/* +* We don't care the return value of discard bio +*/ +if (!test_bit (R10BIO_Uptodate, &r10_bio->state)) +set_bit (R10BIO_Uptodate, &r10_bio->state); + + dev =find_bio_disk (conf, r10_bio, bio, &slot, &repl); +if (repl) + rdev = conf->mirrors[dev].replacement; +if (!rdev) { +smp_rmb (); + repl =0 ; + rdev = conf->mirrors[dev].rdev; + } + +if (atomic_dec_and_test (&r10_bio->remaining)) { +md_write_end (r10_bio->mddev); +raid_end_bio_io (r10_bio); + } + +rdev_dec_pending (rdev, conf->mddev); +} + +/* There are some limitations to handle discard bio +* 1st, the discard size is bigger than stripe_size*2. +* 2st, if the discard bio spans reshape progress, we use the old way to +* handle discard bio +*/ +static bool raid10_handle_discard (struct mddev *mddev,struct bio *bio) +{ +struct r10conf *conf = mddev->private; +struct geom geo = conf->geo; +struct r10bio *r10_bio; + +int disk; +sector_t chunk; +int stripe_size, stripe_mask; + +sector_t bio_start, bio_end; +sector_t first_stripe_index, last_stripe_index; +sector_t start_disk_offset; +unsigned int start_disk_index; +sector_t end_disk_offset; +unsigned int end_disk_index; + +wait_barrier (conf); + +if (conf->reshape_progress != MaxSector && + ((bio->bi_iter.bi_sector >= conf->reshape_progress) != + conf->mddev->reshape_backwards)) + geo = conf->prev; + + stripe_size = (1 <<geo.chunk_shift) * geo.raid_disks; + stripe_mask = stripe_size -1 ; + + bio_start = bio->bi_iter.bi_sector; + bio_end =bio_end_sector (bio); + +/* Maybe one discard bio is smaller than strip size or across one stripe +* and discard region is larger than one stripe size. For far offset layout, +* if the discard region is not aligned with stripe size, there is hole +* when we submit discard bio to member disk. For simplicity, we only +* handle discard bio which discard region is bigger than stripe_size*2 +*/ +if (bio_sectors (bio) < stripe_size*2 ) + goto out; + +if (test_bit (MD_RECOVERY_RESHAPE , &mddev->recovery) && + bio_start < conf->reshape_progress && + bio_end > conf->reshape_progress) + goto out; + +/* For far offset layout, if bio is not aligned with stripe size, it splits +* the part that is not aligned with strip size. +*/ +if (geo.far_offset && (bio_start & stripe_mask)) { +sector_t split_size; + split_size =round_up (bio_start, stripe_size) - bio_start; + bio =raid10_split_bio (conf, bio, split_size, false); + } +if (geo.far_offset && (bio_end & stripe_mask)) { +sector_t split_size; + split_size =bio_sectors (bio) - (bio_end & stripe_mask); + bio =raid10_split_bio (conf, bio, split_size, true); + } + + r10_bio =mempool_alloc (&conf->r10bio_pool,GFP_NOIO ); + r10_bio->mddev = mddev; + r10_bio->state =0 ; +memset (r10_bio->devs,0 ,sizeof (r10_bio->devs[0 ]) * conf->geo.raid_disks); + +wait_blocked_dev (mddev, geo.raid_disks); + + r10_bio->master_bio = bio; + + bio_start = bio->bi_iter.bi_sector; + bio_end =bio_end_sector (bio); + +/* raid10 uses chunk as the unit to store data. It's similar like raid0. +* One stripe contains the chunks from all member disk (one chunk from +* one disk at the same HAB address). For layout detail, see 'man md 4' +*/ + chunk = bio_start >> geo.chunk_shift; + chunk *= geo.near_copies; + first_stripe_index = chunk; + start_disk_index =sector_div (first_stripe_index, geo.raid_disks); +if (geo.far_offset) + first_stripe_index *= geo.far_copies; + start_disk_offset = (bio_start & geo.chunk_mask) + + (first_stripe_index << geo.chunk_shift); + + chunk = bio_end >> geo.chunk_shift; + chunk *= geo.near_copies; + last_stripe_index = chunk; + end_disk_index =sector_div (last_stripe_index, geo.raid_disks); +if (geo.far_offset) + last_stripe_index *= geo.far_copies; + end_disk_offset = (bio_end & geo.chunk_mask) + + (last_stripe_index << geo.chunk_shift); + +rcu_read_lock (); +for (disk =0 ; disk < geo.raid_disks; disk++) { +struct md_rdev *rdev =rcu_dereference (conf->mirrors[disk].rdev); +struct md_rdev *rrdev =rcu_dereference ( + conf->mirrors[disk].replacement); + + r10_bio->devs[disk].bio =NULL ; + r10_bio->devs[disk].repl_bio =NULL ; + +if (rdev && (test_bit (Faulty, &rdev->flags))) + rdev =NULL ; +if (rrdev && (test_bit (Faulty, &rrdev->flags))) + rrdev =NULL ; +if (!rdev && !rrdev) +continue ; + +if (rdev) { + r10_bio->devs[disk].bio = bio; +atomic_inc (&rdev->nr_pending); + } +if (rrdev) { + r10_bio->devs[disk].repl_bio = bio; +atomic_inc (&rrdev->nr_pending); + } + } +rcu_read_unlock (); + +atomic_set (&r10_bio->remaining,1 ); +for (disk =0 ; disk < geo.raid_disks; disk++) { +sector_t dev_start, dev_end; +struct bio *mbio, *rbio =NULL ; +struct md_rdev *rdev =rcu_dereference (conf->mirrors[disk].rdev); +struct md_rdev *rrdev =rcu_dereference ( + conf->mirrors[disk].replacement); + +/* +* Now start to calculate the start and end address for each disk. +* The space between dev_start and dev_end is the discard region. +* +* For dev_start, it needs to consider three conditions: +* 1st, the disk is before start_disk, you can imagine the disk in +* the next stripe. So the dev_start is the start address of next +* stripe. +* 2st, the disk is after start_disk, it means the disk is at the +* same stripe of first disk +* 3st, the first disk itself, we can use start_disk_offset directly +*/ +if (disk < start_disk_index) + dev_start = (first_stripe_index +1 ) * mddev->chunk_sectors; +else if (disk > start_disk_index) + dev_start = first_stripe_index * mddev->chunk_sectors; +else + dev_start = start_disk_offset; + +if (disk < end_disk_index) + dev_end = (last_stripe_index +1 ) * mddev->chunk_sectors; +else if (disk > end_disk_index) + dev_end = last_stripe_index * mddev->chunk_sectors; +else + dev_end = end_disk_offset; + +/* It only handles discard bio which size is >= stripe size, so +* dev_end > dev_start all the time +*/ +if (r10_bio->devs[disk].bio) { + mbio =bio_clone_fast (bio,GFP_NOIO , &mddev->bio_set); + mbio->bi_end_io = raid10_end_discard_request; + mbio->bi_private = r10_bio; + r10_bio->devs[disk].bio = mbio; + r10_bio->devs[disk].devnum = disk; +atomic_inc (&r10_bio->remaining); +md_submit_discard_bio (mddev, rdev, mbio, dev_start, dev_end); +bio_endio (mbio); + } +if (r10_bio->devs[disk].repl_bio) { + rbio =bio_clone_fast (bio,GFP_NOIO , &mddev->bio_set); + rbio->bi_end_io = raid10_end_discard_request; + rbio->bi_private = r10_bio; + r10_bio->devs[disk].repl_bio = rbio; + r10_bio->devs[disk].devnum = disk; +atomic_inc (&r10_bio->remaining); +md_submit_discard_bio (mddev, rrdev, rbio, dev_start, dev_end); +bio_endio (rbio); + } + } + +if (atomic_dec_and_test (&r10_bio->remaining)) { +md_write_end (r10_bio->mddev); +raid_end_bio_io (r10_bio); + } + +return 0 ; +out: +allow_barrier (conf); +return -EAGAIN ; +} +static bool raid10_make_request (struct mddev *mddev,struct bio *bio) {struct r10conf *conf = mddev->private;
@@ -1532,6 +1813,10 @@ static bool raid10_make_request(struct mddev *mddev, struct bio *bio)if (!md_write_start (mddev, bio))return false; +if (unlikely (bio_op (bio) ==REQ_OP_DISCARD )) +if (!raid10_handle_discard (mddev, bio)) +return true; +/* * If this request crosses a chunk boundary, we need to split * it.
@@ -3762,7 +4047,7 @@ static int raid10_run(struct mddev *mddev) chunk_size = mddev->chunk_sectors <<9 ;if (mddev->queue) {blk_queue_max_discard_sectors (mddev->queue, - mddev->chunk_sectors); +UINT_MAX );blk_queue_max_write_same_sectors (mddev->queue,0 );blk_queue_max_write_zeroes_sectors (mddev->queue,0 );blk_queue_io_min (mddev->queue, chunk_size);
Signature · 1 line
2.7.5
Re: [PATCH V2 3/3] improve raid10 discard request
Quoted 9 lines · Xiao Ni
Now the discard request is split by chunk size. So it takes a long time to finish mkfs on disks which support discard function. This patch improve handling raid10 discard request. It uses the similar way with patch
29efc390b(md/md0: optimize raid0 discard handling).But it's a little complex than raid0. Because raid10 has different layout. If raid10 is offset layout and the discard request is smaller than stripe size. There are some holes when we submit discard bio to underlayer disks.
Quoted 10 lines
For example: five disks (disk1 - disk5)
D01 D02 D03 D04 D05
D05 D01 D02 D03 D04
D06 D07 D08 D09 D10
D10 D06 D07 D08 D09
The discard bio just wants to discard from D03 to D10. For disk3, there is a hole between D03 and D08. For disk4, there is a hole between D04 and D09. D03 is a chunk,raid10_write_requestcan handle one chunk perfectly. So the part that is not aligned with stripe size is still handled byraid10_write_request.
One question, is far layout handled by raid10_write_request or the discard request?
Quoted 38 lines
If reshape is running when discard bio comes and the discard bio spans the reshape position,
raid10_write_requestis responsible to handle this discard bio.I did a test with this patch set.
Without patch:
time mkfs.xfs/dev/md0
real4m39.775s
user0m0.000s
sys0m0.298sWith patch:
time mkfs.xfs/dev/md0
real0m0.105s
user0m0.000s
sys0m0.007snvme3n1 259:1 0 477G 0 disk └─nvme3n1p1 259:10 0 50G 0 part nvme4n1 259:2 0 477G 0 disk └─nvme4n1p1 259:11 0 50G 0 part nvme5n1 259:6 0 477G 0 disk └─nvme5n1p1 259:12 0 50G 0 part nvme2n1 259:9 0 477G 0 disk └─nvme2n1p1 259:15 0 50G 0 part nvme0n1 259:13 0 477G 0 disk └─nvme0n1p1 259:14 0 50G 0 partv1:
Coly helps to review these patches and give some suggestions:
One bug is found. If discard bio is across one stripe but bio size is bigger than one stripe size. After spliting, the bio will be NULL. In this version, it checks whether discard bio size is bigger than 2*stripe_size. Inraid10_end_discard_request, it's better to checkR10BIO_Uptodateis set or not. It can avoid write memory to improve performance.
Add more comments for calculating addresses.
Quoted 5 lines
v2:
Fix error by checkpatch.pl
Fix one bug for offset layout. v1 calculates wrongly split size Add more comments to explain how the discard range of each component disk is decided.
The above change log are usually put under "---".
Quoted 3 lines
Reviewed-by Coly Li <[email protected]>Signed-off-by Xiao Ni <[email protected]>---
[...]
Quoted 8 lines
drivers/md/raid10.c +8
-0+ +/* There are some limitations to handle discard bio +* 1st, the discard size is bigger than stripe_size*2. +* 2st, if the discard bio spans reshape progress, we use the old way to +* handle discard bio +*/ +static bool raid10_handle_discard (struct mddev *mddev,struct bio *bio) +{
[...]
drivers/md/raid10.c static void __make_request(struct mddev *mddev, struct bio *bio, int sectors) · line 1669
Quoted 13 lines
drivers/md/raid10.c +13
-0+/* For far offset layout, if bio is not aligned with stripe size, it splits +* the part that is not aligned with strip size. +*/ +if (geo.far_offset && (bio_start & stripe_mask)) { +sector_t split_size; + split_size =round_up (bio_start, stripe_size) - bio_start; + bio =raid10_split_bio (conf, bio, split_size, false); + } +if (geo.far_offset && (bio_end & stripe_mask)) { +sector_t split_size; + split_size =bio_sectors (bio) - (bio_end & stripe_mask); + bio =raid10_split_bio (conf, bio, split_size, true); + }
So far layout is handled here. I think the hole issue is existed for far layout, Just FYI.
Quoted 7 lines
drivers/md/raid10.c +7
-0+ + r10_bio =mempool_alloc (&conf->r10bio_pool,GFP_NOIO ); + r10_bio->mddev = mddev; + r10_bio->state =0 ; +memset (r10_bio->devs,0 ,sizeof (r10_bio->devs[0 ]) * conf->geo.raid_disks); + +wait_blocked_dev (mddev, geo.raid_disks);
drivers/md/raid10.c static void __make_request(struct mddev *mddev, struct bio *bio, int sectors) · line 1686
Quoted 10 lines
drivers/md/raid10.c +10
-0+ + r10_bio->master_bio = bio; + + bio_start = bio->bi_iter.bi_sector; + bio_end =bio_end_sector (bio); + +/* raid10 uses chunk as the unit to store data. It's similar like raid0. +* One stripe contains the chunks from all member disk (one chunk from +* one disk at the same HAB address). For layout detail, see 'man md 4' +*/
s/HAB/HBA/
Thanks,
Guoqing
Re: [PATCH V2 3/3] improve raid10 discard request
Quoted 64 lines · Guoqing Jiang
Quoted 9 lines · Xiao Ni
Now the discard request is split by chunk size. So it takes a long time to finish mkfs on disks which support discard function. This patch improve handling raid10 discard request. It uses the similar way with patch
29efc390b(md/md0: optimize raid0 discard handling).But it's a little complex than raid0. Because raid10 has different layout. If raid10 is offset layout and the discard request is smaller than stripe size. There are some holes when we submit discard bio to underlayer disks.
Quoted 10 lines
For example: five disks (disk1 - disk5)
D01 D02 D03 D04 D05
D05 D01 D02 D03 D04
D06 D07 D08 D09 D10
D10 D06 D07 D08 D09
The discard bio just wants to discard from D03 to D10. For disk3, there is a hole between D03 and D08. For disk4, there is a hole between D04 and D09. D03 is a chunk,raid10_write_requestcan handle one chunk perfectly. So the part that is not aligned with stripe size is still handled byraid10_write_request.One question, is far layout handled by
raid10_write_requestor the discard request?Quoted 38 lines
If reshape is running when discard bio comes and the discard bio spans the reshape position,
raid10_write_requestis responsible to handle this discard bio.I did a test with this patch set.
Without patch:
time mkfs.xfs/dev/md0
real4m39.775s
user0m0.000s
sys0m0.298sWith patch:
time mkfs.xfs/dev/md0
real0m0.105s
user0m0.000s
sys0m0.007snvme3n1 259:1 0 477G 0 disk └─nvme3n1p1 259:10 0 50G 0 part nvme4n1 259:2 0 477G 0 disk └─nvme4n1p1 259:11 0 50G 0 part nvme5n1 259:6 0 477G 0 disk └─nvme5n1p1 259:12 0 50G 0 part nvme2n1 259:9 0 477G 0 disk └─nvme2n1p1 259:15 0 50G 0 part nvme0n1 259:13 0 477G 0 disk └─nvme0n1p1 259:14 0 50G 0 partv1:
Coly helps to review these patches and give some suggestions:
One bug is found. If discard bio is across one stripe but bio size is bigger than one stripe size. After spliting, the bio will be NULL. In this version, it checks whether discard bio size is bigger than 2*stripe_size. Inraid10_end_discard_request, it's better to checkR10BIO_Uptodateis set or not. It can avoid write memory to improve performance.
Add more comments for calculating addresses.
Quoted 7 lines
Quoted 5 lines
v2:
Fix error by checkpatch.pl
Fix one bug for offset layout. v1 calculates wrongly split size Add more comments to explain how the discard range of each component disk is decided.The above change log are usually put under "---".
Thanks for reminding me about this.
Quoted 34 lines
Quoted 4 lines
Reviewed-by Coly Li <[email protected]>Signed-off-by Xiao Ni <[email protected]>---
[...]
Quoted 9 lines
drivers/md/raid10.c +4
-0+ +/* There are some limitations to handle discard bio +* 1st, the discard size is bigger than stripe_size*2. +* 2st, if the discard bio spans reshape progress, we use the old way to
+ * handle discard bio
+ */drivers/md/raid10.c +2
-0+static bool raid10_handle_discard (struct mddev *mddev,struct bio *bio) +{[...]
Quoted 4 lines
+ /* For far offset layout, if bio is not aligned with stripe
size, it splitsdrivers/md/raid10.c +2
-0+* the part that is not aligned with strip size. +*/ drivers/md/raid10.c static void __make_request(struct mddev *mddev, struct bio *bio, int sectors) · line 1669
Quoted 10 lines
drivers/md/raid10.c +10
-0+if (geo.far_offset && (bio_start & stripe_mask)) { +sector_t split_size; + split_size =round_up (bio_start, stripe_size) - bio_start; + bio =raid10_split_bio (conf, bio, split_size, false); + } +if (geo.far_offset && (bio_end & stripe_mask)) { +sector_t split_size; + split_size =bio_sectors (bio) - (bio_end & stripe_mask); + bio =raid10_split_bio (conf, bio, split_size, true); + }
Quoted 2 lines
So far layout is handled here. I think the hole issue is existed for far layout, Just FYI.
It looks like I missed the far layout. Far layout is different with far offset layout. It doesn't need to consider the write hole problem. This patch only consider the write hole in one copy. For far layout, the discard region is sequential. So we just need to find the start/end address of next copy. I'll fix this in next version.
Regards
Xiao
Quoted 28 lines
Quoted 16 lines
drivers/md/raid10.c +5
-0+ + r10_bio =mempool_alloc (&conf->r10bio_pool,GFP_NOIO ); + r10_bio->mddev = mddev; + r10_bio->state =0 ; + memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) *
conf->geo.raid_disks);drivers/md/raid10.c +9
-0+ +wait_blocked_dev (mddev, geo.raid_disks); + + r10_bio->master_bio = bio; + + bio_start = bio->bi_iter.bi_sector; + bio_end =bio_end_sector (bio); + +/* raid10 uses chunk as the unit to store data. It's similar Quoted 6 lines
like raid0.
+ * One stripe contains the chunks from all member disk (one
chunk from
+ * one disk at the same HAB address). For layout detail, see
'man md 4'
+ */s/HAB/HBA/
Thanks,
Guoqing