Skip to contentSkip to conversation

md/raid5: Improve performance for sequential IO

PATCH In review Jan Kara 5 participants 12 messages Apr 17, 2023 – Apr 24, 2023 MD RAID

lore
Share & tools
lore

Jan Kara

author
#0 lore raw

PATCH 1/1 md/raid5: Improve performance for sequential IO

Commit 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()") changed the order in which requests for underlying disks are created. Since for large sequential IO adding of requests frequently races with md_raid5 thread submitting bios to underlying disks, this results in a change in IO pattern because intermediate states of new order of request creation result in more smaller discontiguous requests. For RAID5 on top of three rotational disks our performance testing revealed this results in regression in write throughput:

iozone -a -s 131072000 -y 4 -q 8 -i 0 -i 1 -R

before 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  493670  525964   524575   513384
       131072000       8  540467  532880   512028   513703

after 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  421785  456184   531278   509248
       131072000       8  459283  456354   528449   543834

To reduce the amount of discontiguous requests we can start generating requests with the stripe with the lowest chunk offset as that has the best chance of being adjacent to IO queued previously. This improves the performance to:

              KB  reclen   write rewrite    read    reread
       131072000       4  497682  506317   518043   514559
       131072000       8  514048  501886   506453   504319

restoring big part of the regression.

Fixes 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()")
---
 drivers/md/raid5.c | 45 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 44 insertions(+), 1 deletion(-)

I'm by no means raid5 expert but this is what I was able to come up with. Any opinion or ideas how to fix the problem in a better way?

drivers/md/raid5.c +44 -1
index 7b820b81d8c2..f787c9e5b10e 100644
@@ -6079,6 +6079,38 @@ static enum stripe_result make_stripe_request(struct mddev *mddev,
 	return ret;
 }
 
+/*
+ * If the bio covers multiple data disks, find sector within the bio that has
+ * the lowest chunk offset in the first chunk.
+ */
+static sector_t raid5_bio_lowest_chunk_sector(struct r5conf *conf,
+					      struct bio *bi)
+{
+	int sectors_per_chunk = conf->chunk_sectors;
+	int raid_disks = conf->raid_disks;
+	int dd_idx;
+	struct stripe_head sh;
+	unsigned int chunk_offset;
+	sector_t r_sector = bi->bi_iter.bi_sector & ~((sector_t)RAID5_STRIPE_SECTORS(conf)-1);
+	sector_t sector;
+
+	/* We pass in fake stripe_head to get back parity disk numbers */
+	sector = raid5_compute_sector(conf, r_sector, 0, &dd_idx, &sh);
+	chunk_offset = sector_div(sector, sectors_per_chunk);
+	if (sectors_per_chunk - chunk_offset >= bio_sectors(bi))
+		return r_sector;
+	/*
+	 * Bio crosses to the next data disk. Check whether it's in the same
+	 * chunk.
+	 */
+	dd_idx++;
+	while (dd_idx == sh.pd_idx || dd_idx == sh.qd_idx)
+		dd_idx++;
+	if (dd_idx >= raid_disks)
+		return r_sector;
+	return r_sector + sectors_per_chunk - chunk_offset;
+}
+
 static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
 {
 	DEFINE_WAIT_FUNC(wait, woken_wake_function);
@@ -6150,6 +6182,17 @@ static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
 	}
 	md_account_bio(mddev, &bi);
 
+	/*
+	 * Lets start with the stripe with the lowest chunk offset in the first
+	 * chunk. That has the best chances of creating IOs adjacent to
+	 * previous IOs in case of sequential IO and thus creates the most
+	 * sequential IO pattern. We don't bother with the optimization when
+	 * reshaping as the performance benefit is not worth the complexity.
+	 */
+	if (likely(conf->reshape_progress == MaxSector))
+		logical_sector = raid5_bio_lowest_chunk_sector(conf, bi);
+	s = (logical_sector - ctx.first_sector) >> RAID5_STRIPE_SHIFT(conf);
+
 	add_wait_queue(&conf->wait_for_overlap, &wait);
 	while (1) {
 		res = make_stripe_request(mddev, conf, &ctx, logical_sector,
@@ -6178,7 +6221,7 @@ static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
 			continue;
 		}
 
-		s = find_first_bit(ctx.sectors_to_do, stripe_cnt);
+		s = find_next_bit_wrap(ctx.sectors_to_do, stripe_cnt, s);
 		if (s == stripe_cnt)
 			break;
 
Signature · 1 line
2.35.3
Reviewed-by on v1 1/1
#1 lore raw
Quoted 25 lines · Jan Kara

Commit 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()") changed the order in which requests for underlying disks are created. Since for large sequential IO adding of requests frequently races with md_raid5 thread submitting bios to underlying disks, this results in a change in IO pattern because intermediate states of new order of request creation result in more smaller discontiguous requests. For RAID5 on top of three rotational disks our performance testing revealed this results in regression in write throughput:

iozone -a -s 131072000 -y 4 -q 8 -i 0 -i 1 -R

before 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  493670  525964   524575   513384
       131072000       8  540467  532880   512028   513703

after 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  421785  456184   531278   509248
       131072000       8  459283  456354   528449   543834

To reduce the amount of discontiguous requests we can start generating requests with the stripe with the lowest chunk offset as that has the best chance of being adjacent to IO queued previously. This improves the performance to:

Quoted 8 lines
              KB  reclen   write rewrite    read    reread
       131072000       4  497682  506317   518043   514559
       131072000       8  514048  501886   506453   504319

restoring big part of the regression.

Fixes 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()")

Looks good to me. I ran it through some of the functional tests I used to develop the patch in question and found no issues.

Quoted 6 lines
---
 drivers/md/raid5.c | 45 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 44 insertions(+), 1 deletion(-)

I'm by no means raid5 expert but this is what I was able to come up with. Any opinion or ideas how to fix the problem in a better way?

The other option would be to revert to the old method for spinning disks and use the pivot option only on SSDs. The pivot optimization really only applies at IO speeds that can be achieved by fast solid state disks anyway, and there isn't likely enough IOPS possible on spinning disks to get enough lock contention that the optimization would provide any benefit.

So it could make sense to just fall back to the old method of preparing the stripes in logical block order if we are running on spinning disks. Though, that might be a bit more involved than what this patch does. So I think this is probably a good approach, unless we want to recover more of the lost throughput.

Logan

maintainer
#2 lore raw
Quoted 35 lines · Logan Gunthorpe
Quoted 31 lines · Jan Kara

Commit 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()") changed the order in which requests for underlying disks are created. Since for large sequential IO adding of requests frequently races with md_raid5 thread submitting bios to underlying disks, this results in a change in IO pattern because intermediate states of new order of request creation result in more smaller discontiguous requests. For RAID5 on top of three rotational disks our performance testing revealed this results in regression in write throughput:

iozone -a -s 131072000 -y 4 -q 8 -i 0 -i 1 -R

before 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  493670  525964   524575   513384
       131072000       8  540467  532880   512028   513703

after 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  421785  456184   531278   509248
       131072000       8  459283  456354   528449   543834

To reduce the amount of discontiguous requests we can start generating requests with the stripe with the lowest chunk offset as that has the best chance of being adjacent to IO queued previously. This improves the performance to:

              KB  reclen   write rewrite    read    reread
       131072000       4  497682  506317   518043   514559
       131072000       8  514048  501886   506453   504319

restoring big part of the regression.

Quoted 7 lines
Quoted 2 lines
Fixes 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()")

Looks good to me. I ran it through some of the functional tests I used to develop the patch in question and found no issues.

Thanks Jan and Logan! I will apply this to md-next. But it may not make 6.4 release, as we are already at rc7.

Quoted 14 lines
Quoted 7 lines
---
 drivers/md/raid5.c | 45 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 44 insertions(+), 1 deletion(-)

I'm by no means raid5 expert but this is what I was able to come up with. Any opinion or ideas how to fix the problem in a better way?

The other option would be to revert to the old method for spinning disks and use the pivot option only on SSDs. The pivot optimization really only applies at IO speeds that can be achieved by fast solid state disks anyway, and there isn't likely enough IOPS possible on spinning disks to get enough lock contention that the optimization would provide any benefit.

Quoted 5 lines

So it could make sense to just fall back to the old method of preparing the stripes in logical block order if we are running on spinning disks. Though, that might be a bit more involved than what this patch does. So I think this is probably a good approach, unless we want to recover more of the lost throughput.

How about we only do the optimization in this patch for spinning disks? Something like:

        if (likely(conf->reshape_progress == MaxSector) &&
            !blk_queue_nonrot(mddev->queue))
                logical_sector = raid5_bio_lowest_chunk_sector(conf, bi);

Thanks,
Song

author
#3 lore raw
Quoted 36 lines · Song Liu
Quoted 35 lines · Logan Gunthorpe
Quoted 33 lines · Jan Kara

Commit 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()") changed the order in which requests for underlying disks are created. Since for large sequential IO adding of requests frequently races with md_raid5 thread submitting bios to underlying disks, this results in a change in IO pattern because intermediate states of new order of request creation result in more smaller discontiguous requests. For RAID5 on top of three rotational disks our performance testing revealed this results in regression in write throughput:

iozone -a -s 131072000 -y 4 -q 8 -i 0 -i 1 -R

before 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  493670  525964   524575   513384
       131072000       8  540467  532880   512028   513703

after 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  421785  456184   531278   509248
       131072000       8  459283  456354   528449   543834

To reduce the amount of discontiguous requests we can start generating requests with the stripe with the lowest chunk offset as that has the best chance of being adjacent to IO queued previously. This improves the performance to:

              KB  reclen   write rewrite    read    reread
       131072000       4  497682  506317   518043   514559
       131072000       8  514048  501886   506453   504319

restoring big part of the regression.

Fixes 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()")
Quoted 7 lines
Quoted 4 lines

Looks good to me. I ran it through some of the functional tests I used to develop the patch in question and found no issues.

Thanks Jan and Logan! I will apply this to md-next. But it may not make 6.4 release, as we are already at rc7.

OK, sure, this is not a critical issue.

Quoted 19 lines
Quoted 13 lines
Quoted 6 lines
---
 drivers/md/raid5.c | 45 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 44 insertions(+), 1 deletion(-)

I'm by no means raid5 expert but this is what I was able to come up with. Any opinion or ideas how to fix the problem in a better way?

The other option would be to revert to the old method for spinning disks and use the pivot option only on SSDs. The pivot optimization really only applies at IO speeds that can be achieved by fast solid state disks anyway, and there isn't likely enough IOPS possible on spinning disks to get enough lock contention that the optimization would provide any benefit.

Quoted 5 lines

So it could make sense to just fall back to the old method of preparing the stripes in logical block order if we are running on spinning disks. Though, that might be a bit more involved than what this patch does. So I think this is probably a good approach, unless we want to recover more of the lost throughput.

Quoted 6 lines

How about we only do the optimization in this patch for spinning disks? Something like:

        if (likely(conf->reshape_progress == MaxSector) &&
            !blk_queue_nonrot(mddev->queue))
                logical_sector = raid5_bio_lowest_chunk_sector(conf, bi);

Yeah, makes sense. On SSD disks I was not able to observe any adverse effects of the different stripe order. Will you update the patch or should I respin it?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#4 lore raw
Quoted 69 lines · Jan Kara
Quoted 36 lines · Song Liu
Quoted 35 lines · Logan Gunthorpe
Quoted 33 lines · Jan Kara

Commit 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()") changed the order in which requests for underlying disks are created. Since for large sequential IO adding of requests frequently races with md_raid5 thread submitting bios to underlying disks, this results in a change in IO pattern because intermediate states of new order of request creation result in more smaller discontiguous requests. For RAID5 on top of three rotational disks our performance testing revealed this results in regression in write throughput:

iozone -a -s 131072000 -y 4 -q 8 -i 0 -i 1 -R

before 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  493670  525964   524575   513384
       131072000       8  540467  532880   512028   513703

after 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  421785  456184   531278   509248
       131072000       8  459283  456354   528449   543834

To reduce the amount of discontiguous requests we can start generating requests with the stripe with the lowest chunk offset as that has the best chance of being adjacent to IO queued previously. This improves the performance to:

              KB  reclen   write rewrite    read    reread
       131072000       4  497682  506317   518043   514559
       131072000       8  514048  501886   506453   504319

restoring big part of the regression.

Fixes 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()")
Quoted 7 lines
Quoted 4 lines

Looks good to me. I ran it through some of the functional tests I used to develop the patch in question and found no issues.

Thanks Jan and Logan! I will apply this to md-next. But it may not make 6.4 release, as we are already at rc7.

OK, sure, this is not a critical issue.

Quoted 22 lines
Quoted 13 lines
Quoted 6 lines
---
 drivers/md/raid5.c | 45 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 44 insertions(+), 1 deletion(-)

I'm by no means raid5 expert but this is what I was able to come up with. Any opinion or ideas how to fix the problem in a better way?

The other option would be to revert to the old method for spinning disks and use the pivot option only on SSDs. The pivot optimization really only applies at IO speeds that can be achieved by fast solid state disks anyway, and there isn't likely enough IOPS possible on spinning disks to get enough lock contention that the optimization would provide any benefit.

Quoted 5 lines

So it could make sense to just fall back to the old method of preparing the stripes in logical block order if we are running on spinning disks. Though, that might be a bit more involved than what this patch does. So I think this is probably a good approach, unless we want to recover more of the lost throughput.

How about we only do the optimization in this patch for spinning disks? Something like:

Quoted 7 lines
Quoted 3 lines
        if (likely(conf->reshape_progress == MaxSector) &&
            !blk_queue_nonrot(mddev->queue))
                logical_sector = raid5_bio_lowest_chunk_sector(conf, bi);

Yeah, makes sense. On SSD disks I was not able to observe any adverse effects of the different stripe order. Will you update the patch or should I respin it?

Does it make sense? If SSDs work fine with the new stripe order, why do things different for them? So I don't see a benefit of making the fix only work with non-rotational devices. It's my original change which could be made for non-rotationatial only, but that's much more involved.

Logan

maintainer
#5 lore raw
Quoted 77 lines · Logan Gunthorpe
Quoted 73 lines · Jan Kara
Quoted 36 lines · Song Liu
Quoted 35 lines · Logan Gunthorpe
Quoted 33 lines · Jan Kara

Commit 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()") changed the order in which requests for underlying disks are created. Since for large sequential IO adding of requests frequently races with md_raid5 thread submitting bios to underlying disks, this results in a change in IO pattern because intermediate states of new order of request creation result in more smaller discontiguous requests. For RAID5 on top of three rotational disks our performance testing revealed this results in regression in write throughput:

iozone -a -s 131072000 -y 4 -q 8 -i 0 -i 1 -R

before 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  493670  525964   524575   513384
       131072000       8  540467  532880   512028   513703

after 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  421785  456184   531278   509248
       131072000       8  459283  456354   528449   543834

To reduce the amount of discontiguous requests we can start generating requests with the stripe with the lowest chunk offset as that has the best chance of being adjacent to IO queued previously. This improves the performance to:

              KB  reclen   write rewrite    read    reread
       131072000       4  497682  506317   518043   514559
       131072000       8  514048  501886   506453   504319

restoring big part of the regression.

Fixes 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()")
Quoted 7 lines
Quoted 4 lines

Looks good to me. I ran it through some of the functional tests I used to develop the patch in question and found no issues.

Thanks Jan and Logan! I will apply this to md-next. But it may not make 6.4 release, as we are already at rc7.

OK, sure, this is not a critical issue.

Quoted 25 lines
Quoted 13 lines
Quoted 6 lines
---
 drivers/md/raid5.c | 45 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 44 insertions(+), 1 deletion(-)

I'm by no means raid5 expert but this is what I was able to come up with. Any opinion or ideas how to fix the problem in a better way?

The other option would be to revert to the old method for spinning disks and use the pivot option only on SSDs. The pivot optimization really only applies at IO speeds that can be achieved by fast solid state disks anyway, and there isn't likely enough IOPS possible on spinning disks to get enough lock contention that the optimization would provide any benefit.

Quoted 5 lines

So it could make sense to just fall back to the old method of preparing the stripes in logical block order if we are running on spinning disks. Though, that might be a bit more involved than what this patch does. So I think this is probably a good approach, unless we want to recover more of the lost throughput.

How about we only do the optimization in this patch for spinning disks? Something like:

        if (likely(conf->reshape_progress == MaxSector) &&
            !blk_queue_nonrot(mddev->queue))
                logical_sector = raid5_bio_lowest_chunk_sector(conf, bi);
Quoted 8 lines
Quoted 3 lines

Yeah, makes sense. On SSD disks I was not able to observe any adverse effects of the different stripe order. Will you update the patch or should I respin it?

Does it make sense? If SSDs work fine with the new stripe order, why do things different for them? So I don't see a benefit of making the fix only work with non-rotational devices. It's my original change which could be made for non-rotationatial only, but that's much more involved.

I am hoping to make raid5_make_request() a little faster for non-rotational devices. We may not easily observe a difference in performance, but things add up. Does this make sense?

Thanks,
Song

#6 lore raw
Quoted 82 lines · Song Liu
Quoted 81 lines · Logan Gunthorpe
Quoted 76 lines · Jan Kara
Quoted 36 lines · Song Liu
Quoted 35 lines · Logan Gunthorpe
Quoted 33 lines · Jan Kara

Commit 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()") changed the order in which requests for underlying disks are created. Since for large sequential IO adding of requests frequently races with md_raid5 thread submitting bios to underlying disks, this results in a change in IO pattern because intermediate states of new order of request creation result in more smaller discontiguous requests. For RAID5 on top of three rotational disks our performance testing revealed this results in regression in write throughput:

iozone -a -s 131072000 -y 4 -q 8 -i 0 -i 1 -R

before 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  493670  525964   524575   513384
       131072000       8  540467  532880   512028   513703

after 7e55c60acfbb:

              KB  reclen   write rewrite    read    reread
       131072000       4  421785  456184   531278   509248
       131072000       8  459283  456354   528449   543834

To reduce the amount of discontiguous requests we can start generating requests with the stripe with the lowest chunk offset as that has the best chance of being adjacent to IO queued previously. This improves the performance to:

              KB  reclen   write rewrite    read    reread
       131072000       4  497682  506317   518043   514559
       131072000       8  514048  501886   506453   504319

restoring big part of the regression.

Fixes 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()")
Quoted 7 lines
Quoted 4 lines

Looks good to me. I ran it through some of the functional tests I used to develop the patch in question and found no issues.

Thanks Jan and Logan! I will apply this to md-next. But it may not make 6.4 release, as we are already at rc7.

OK, sure, this is not a critical issue.

Quoted 19 lines
Quoted 13 lines
Quoted 6 lines
---
 drivers/md/raid5.c | 45 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 44 insertions(+), 1 deletion(-)

I'm by no means raid5 expert but this is what I was able to come up with. Any opinion or ideas how to fix the problem in a better way?

The other option would be to revert to the old method for spinning disks and use the pivot option only on SSDs. The pivot optimization really only applies at IO speeds that can be achieved by fast solid state disks anyway, and there isn't likely enough IOPS possible on spinning disks to get enough lock contention that the optimization would provide any benefit.

Quoted 5 lines

So it could make sense to just fall back to the old method of preparing the stripes in logical block order if we are running on spinning disks. Though, that might be a bit more involved than what this patch does. So I think this is probably a good approach, unless we want to recover more of the lost throughput.

Quoted 6 lines

How about we only do the optimization in this patch for spinning disks? Something like:

        if (likely(conf->reshape_progress == MaxSector) &&
            !blk_queue_nonrot(mddev->queue))
                logical_sector = raid5_bio_lowest_chunk_sector(conf, bi);

Yeah, makes sense. On SSD disks I was not able to observe any adverse effects of the different stripe order. Will you update the patch or should I respin it?

Quoted 8 lines
Quoted 4 lines

Does it make sense? If SSDs work fine with the new stripe order, why do things different for them? So I don't see a benefit of making the fix only work with non-rotational devices. It's my original change which could be made for non-rotationatial only, but that's much more involved.

I am hoping to make raid5_make_request() a little faster for non-rotational devices. We may not easily observe a difference in performance, but things add up. Does this make sense?

I guess. But without a performance test that shows that it makes an improvement, I'm hesitant about that. It could also be that it helps a tiny bit for non-rotational disks, but we just don't know.

Unfortunately, I don't have the time right now to do these performance tests.

Logan

#7 lore raw
Quoted 10 lines · Logan Gunthorpe
Quoted 3 lines

I am hoping to make raid5_make_request() a little faster for non-rotational devices. We may not easily observe a difference in performance, but things add up. Does this make sense?

I guess. But without a performance test that shows that it makes an improvement, I'm hesitant about that. It could also be that it helps a tiny bit for non-rotational disks, but we just don't know.

Unfortunately, I don't have the time right now to do these performance tests.

FYI, SSDs in general do prefer sequential write streams. For most you won't see a different in write performance itself, but it will help with reducing GC overhead later on.

#8 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Logan Gunthorpe
Quoted 3 lines

I am hoping to make raid5_make_request() a little faster for non-rotational devices. We may not easily observe a difference in performance, but things add up. Does this make sense?

I guess. But without a performance test that shows that it makes an improvement, I'm hesitant about that. It could also be that it helps a tiny bit for non-rotational disks, but we just don't know.

Quoted 6 lines
Quoted 2 lines

Unfortunately, I don't have the time right now to do these performance tests.

FYI, SSDs in general do prefer sequential write streams. For most you won't see a different in write performance itself, but it will help with reducing GC overhead later on.

Thanks. Yes, my colleague was able to run performance testing on this patch and didn't find any degradation with Jan's optimization turned on. So I don't think it's worth doing this only for rotational disks and Jan's original patch makes sense.

Logan

maintainer
#9 lore raw
Quoted 13 lines · Logan Gunthorpe
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Logan Gunthorpe
Quoted 3 lines

I am hoping to make raid5_make_request() a little faster for non-rotational devices. We may not easily observe a difference in performance, but things add up. Does this make sense?

I guess. But without a performance test that shows that it makes an improvement, I'm hesitant about that. It could also be that it helps a tiny bit for non-rotational disks, but we just don't know.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately, I don't have the time right now to do these performance tests.

FYI, SSDs in general do prefer sequential write streams. For most you won't see a different in write performance itself, but it will help with reducing GC overhead later on.

Yeah, this makes sense.

Quoted 5 lines

Thanks. Yes, my colleague was able to run performance testing on this patch and didn't find any degradation with Jan's optimization turned on. So I don't think it's worth doing this only for rotational disks and Jan's original patch makes sense.

I will ship Jan's original patch.

Thanks,
Song

#10 lore raw
In reply to Jan Kara

Jan Kara ([email protected]) wrote on Mon, Apr 17, 2023 at 02:15:37PM -03:

Quoted 2 lines

Commit 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()") changed the order in which requests for underlying disks are created.

In which version was this first applied?

We've observed a large drop in disk performance since 4.* to the point that 6.1.* is almost unusable for some tasks. For example, we have 2 backup machines, one using raid6/ext4 and another using zfs. The backup of a third machine with a filesystem with ~25M inodes takes 6x-8x longer on the one using raid6/ext4 than on the one using zfs... This is during the rsync phase. Afterwards it removes old trees and the rm with raid6 takes eons even though the disks are not at all busy (as shown by sar). Running several rm in parallel speeds things up a lot, showing the problem is not in the disks.

maintainer
#11 lore raw
Quoted 6 lines · Carlos Carvalho

Jan Kara ([email protected]) wrote on Mon, Apr 17, 2023 at 02:15:37PM -03:

Quoted 2 lines

Commit 7e55c60acfbb ("md/raid5: Pivot raid5_make_request()") changed the order in which requests for underlying disks are created.

In which version was this first applied?

7e55c60acfbb ("md/raid5: Pivot raid5_make_request()") is in 6.0+ kernels.

Quoted 9 lines

We've observed a large drop in disk performance since 4.* to the point that 6.1.* is almost unusable for some tasks. For example, we have 2 backup machines, one using raid6/ext4 and another using zfs. The backup of a third machine with a filesystem with ~25M inodes takes 6x-8x longer on the one using raid6/ext4 than on the one using zfs... This is during the rsync phase. Afterwards it removes old trees and the rm with raid6 takes eons even though the disks are not at all busy (as shown by sar). Running several rm in parallel speeds things up a lot, showing the problem is not in the disks.

I am hoping to look into raid performance soon. But I cannot promise anything at the moment.

Thanks,
Song