Optimize block_device utilization
Jens Axboe
maintainercover Optimize block_device utilization
Hi,
First is just a prep patch that moves struct device, which is quite the pig, out of line in block_device, and then shuffles a few fields for better layout. It also saves 16 bytes of space in there while at it.
Second patch caches the bdev->disk->fops->submit_bio state in the bdev, so we can avoid this long dependent memory load chain in the fast path.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/2 block: re-arrange the struct block_device fields for better layout
This moves struct device out-of-line as it's just used at open/close time, so we can keep some of the commonly used fields closer together. On a standard setup, it also reduces the size from 864 bytes to 848 bytes. Yes, struct device is a pig...
--- include/linux/blk_types.h | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
include/linux/blk_types.h +8 -8
@@ -40,26 +40,25 @@ struct bio_crypt_ctx;struct block_device {sector_t bd_start_sect;sector_t bd_nr_sectors; +struct gendisk * bd_disk; +struct request_queue * bd_queue;struct disk_stats __percpu *bd_stats;unsigned long bd_stamp;bool bd_read_only;/* read-only policy */ +u8 bd_partno; +bool bd_write_holder;dev_t bd_dev;atomic_t bd_openers; +spinlock_t bd_size_lock;/* for bd_inode->i_size updates */ struct inode * bd_inode;/* will die */ struct super_block * bd_super;void * bd_claiming; -struct device bd_device;void * bd_holder; +/* The counter of freeze processes */ +int bd_fsfreeze_count;int bd_holders; -bool bd_write_holder;struct kobject *bd_holder_dir; -u8 bd_partno; -spinlock_t bd_size_lock;/* for bd_inode->i_size updates */ -struct gendisk * bd_disk; -struct request_queue * bd_queue; -/* The counter of freeze processes */ -int bd_fsfreeze_count;/* Mutex for freeze */ struct mutex bd_fsfreeze_mutex;struct super_block *bd_fsfreeze_sb;
@@ -68,6 +67,7 @@ struct block_device {#ifdef CONFIG_FAIL_MAKE_REQUEST bool bd_make_it_fail;#endif +struct device bd_device; } __randomize_layout;#define bdev_whole (_bdev) \
Signature · 1 line
2.39.2
Jens Axboe
maintainerPATCH 2/2 block: store bdev->bd_disk->fops->submit_bio state in bdev
In reply to Jens AxboeWe have a long chain of memory dereferencing just to whether or not this disk has a special submit_bio helper. As that's not necessarily the common case, add a bd_submit_bio state in the bdev to avoid traversing this memory dependency chain if we don't need to.
--- block/bdev.c | 1 + block/blk-core.c | 8 ++++---- block/genhd.c | 4 ++++ include/linux/blk_types.h | 1 + 4 files changed, 10 insertions(+), 4 deletions(-)
block/bdev.c +1 -0
@@ -419,6 +419,7 @@ struct block_device *bdev_alloc(struct gendisk *disk, u8 partno) bdev->bd_inode = inode; bdev->bd_queue = disk->queue; bdev->bd_stats = alloc_percpu(struct disk_stats); + bdev->bd_submit_bio =0 ;if (!bdev->bd_stats) {iput (inode);return NULL ;
block/blk-core.c +4 -4
@@ -587,14 +587,14 @@ static inline blk_status_t blk_check_zone_append(struct request_queue *q,static void __submit_bio (struct bio *bio) { -struct gendisk *disk = bio->bi_bdev->bd_disk; -if (unlikely (!blk_crypto_bio_prep (&bio)))return ; -if (!disk->fops->submit_bio) { +if (!bio->bi_bdev->bd_submit_bio) {blk_mq_submit_bio (bio); }else if (likely (bio_queue_enter (bio) ==0 )) { +struct gendisk *disk = bio->bi_bdev->bd_disk; + disk->fops->submit_bio (bio);blk_queue_exit (disk->queue); }
@@ -698,7 +698,7 @@ void submit_bio_noacct_nocheck(struct bio *bio)*/ if (current->bio_list)bio_list_add (¤t->bio_list[0 ], bio); -else if (!bio->bi_bdev->bd_disk->fops->submit_bio) +else if (!bio->bi_bdev->bd_submit_bio)__submit_bio_noacct_mq (bio);else __submit_bio_noacct (bio);
block/genhd.c +4 -0
@@ -420,6 +420,10 @@ int __must_check device_add_disk(struct device *parent, struct gendisk *disk,*/ elevator_init_mq (disk->queue); +/* Mark bdev as having a submit_bio, if needed */ +if (disk->fops->submit_bio) + disk->part0->bd_submit_bio =1 ; +/* * If the driver provides an explicit major number it also must provide * the number of minors numbers supported, and those will be used to
include/linux/blk_types.h +1 -0
@@ -47,6 +47,7 @@ struct block_device {bool bd_read_only;/* read-only policy */ u8 bd_partno;bool bd_write_holder; +bool bd_submit_bio;dev_t bd_dev;atomic_t bd_openers;spinlock_t bd_size_lock;/* for bd_inode->i_size updates */
Signature · 1 line
2.39.2
Re: [PATCH 2/2] block: store bdev->bd_disk->fops->submit_bio state in bdev
Quoted 18 lines · Jens Axboe
We have a long chain of memory dereferencing just to whether or not this disk has a special
submit_biohelper. As that's not necessarily the common case, add abd_submit_biostate in the bdev to avoid traversing this memory dependency chain if we don't need to.Signed-off-by Jens Axboe <[email protected]>--- block/bdev.c | 1 + block/blk-core.c | 8 ++++---- block/genhd.c | 4 ++++ include/linux/blk_types.h | 1 + 4 files changed, 10 insertions(+), 4 deletions(-)block/bdev.c +0
-0
block/bdev.c struct block_device *bdev_alloc(struct gendisk *disk, u8 partno) · line 422
Quoted 4 lines
block/bdev.c +1
-0@@ -419,6 +419,7 @@ struct block_device *bdev_alloc(struct gendisk *disk, u8 partno) bdev->bd_inode = inode; bdev->bd_queue = disk->queue; bdev->bd_stats = alloc_percpu(struct disk_stats); + bdev->bd_submit_bio =0 ;
"= false;" would be better to match bd_submit_bio type.
[...]
block/genhd.c int __must_check device_add_disk(struct device *parent, struct gendisk *disk, · line 425
Quoted 11 lines
block/genhd.c +3
-0@@ -420,6 +420,10 @@ int __must_check device_add_disk(struct device *parent, struct gendisk *disk,*/ elevator_init_mq (disk->queue); +/* Mark bdev as having a submit_bio, if needed */ +if (disk->fops->submit_bio) + disk->part0->bd_submit_bio =1 ;
"= true;" would be better to match the type.
Note that this could also be:
disk->part0->bd_submit_bio = disk->fops->submit_bio;
thus removing the if.
Jens Axboe
maintainerRe: [PATCH 2/2] block: store bdev->bd_disk->fops->submit_bio state in bdev
Quoted 14 lines · Damien Le Moal
Quoted 13 lines · Jens Axboe
We have a long chain of memory dereferencing just to whether or not this disk has a special
submit_biohelper. As that's not necessarily the common case, add abd_submit_biostate in the bdev to avoid traversing this memory dependency chain if we don't need to.Signed-off-by Jens Axboe <[email protected]>--- block/bdev.c | 1 + block/blk-core.c | 8 ++++---- block/genhd.c | 4 ++++ include/linux/blk_types.h | 1 + 4 files changed, 10 insertions(+), 4 deletions(-)
Quoted 11 lines
block/bdev.c struct block_device *bdev_alloc(struct gendisk *disk, u8 partno) · line 422
Quoted 9 lines
block/bdev.c +1
-0@@ -419,6 +419,7 @@ struct block_device *bdev_alloc(struct gendisk *disk, u8 partno) bdev->bd_inode = inode; bdev->bd_queue = disk->queue; bdev->bd_stats = alloc_percpu(struct disk_stats); + bdev->bd_submit_bio =0 ;"= false;" would be better to match
bd_submit_biotype.
Done
Quoted 14 lines
block/genhd.c int __must_check device_add_disk(struct device *parent, struct gendisk *disk, · line 425
Quoted 11 lines
block/genhd.c +3
-0@@ -420,6 +420,10 @@ int __must_check device_add_disk(struct device *parent, struct gendisk *disk,*/ elevator_init_mq (disk->queue); +/* Mark bdev as having a submit_bio, if needed */ +if (disk->fops->submit_bio) + disk->part0->bd_submit_bio =1 ;"= true;" would be better to match the type.
Quoted 5 lines
Note that this could also be:
disk->part0->bd_submit_bio=disk->fops->submit_bio;thus removing the if.
I made it:
disk->part0->bd_submit_bio = disk->fops->submit_bio != NULL;
instead to make it explicit, I don't think that assignment would be happy otherwise.
Signature · 1 line
Jens Axboe
Re: [PATCH 1/2] block: re-arrange the struct block_device fields for better layout
In reply to Jens AxboeQuoted 4 lines · Jens Axboe
This moves struct device out-of-line as it's just used at open/close time, so we can keep some of the commonly used fields closer together. On a standard setup, it also reduces the size from 864 bytes to 848 bytes. Yes, struct device is a pig...
Maybe add a comment about keeping struct device last and why?
Otherwise looks good:
Re: [PATCH 2/2] block: store bdev->bd_disk->fops->submit_bio state in bdev
In reply to Jens AxboeQuoted 4 lines · Jens Axboe
We have a long chain of memory dereferencing just to whether or not this disk has a special
submit_biohelper. As that's not necessarily the common case, add abd_submit_biostate in the bdev to avoid traversing this memory dependency chain if we don't need to.
Do you have any numbers on how this helps?
block/bdev.c struct block_device *bdev_alloc(struct gendisk *disk, u8 partno) · line 422
Quoted 1 line
block/bdev.c +1
-0+ bdev->bd_submit_bio =0 ;
bd_submit_bio sounds like a function call, so I'd name this bd_has_submit_io.
But maybe it might make more sense to just add a bit that this is a blk-mq backed device into bd_state as that might be handy in other places as well?
Jens Axboe
maintainerRe: [PATCH 2/2] block: store bdev->bd_disk->fops->submit_bio state in bdev
Quoted 7 lines · Christoph Hellwig
Quoted 4 lines · Jens Axboe
We have a long chain of memory dereferencing just to whether or not this disk has a special
submit_biohelper. As that's not necessarily the common case, add abd_submit_biostate in the bdev to avoid traversing this memory dependency chain if we don't need to.Do you have any numbers on how this helps?
I didn't run any numbers, but seems obvious to me that we don't want to pull in 3 layers deep of pointer indirections when we can avoid it.
Quoted 4 lines
block/bdev.c struct block_device *bdev_alloc(struct gendisk *disk, u8 partno) · line 422
Quoted 1 line
block/bdev.c +1
-0+ bdev->bd_submit_bio =0 ;
bd_submit_biosounds like a function call, so I'd name thisbd_has_submit_io.
Good point, I'll rename it.
Quoted 3 lines
But maybe it might make more sense to just add a bit that this is a blk-mq backed device into
bd_stateas that might be handy in other places as well?
I'd rather just do that if needed.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerRe: [PATCH 1/2] block: re-arrange the struct block_device fields for better layout
In reply to Christoph HellwigQuoted 7 lines · Christoph Hellwig
Quoted 4 lines · Jens Axboe
This moves struct device out-of-line as it's just used at open/close time, so we can keep some of the commonly used fields closer together. On a standard setup, it also reduces the size from 864 bytes to 848 bytes. Yes, struct device is a pig...
Maybe add a comment about keeping struct device last and why?
Sure, done.
Signature · 1 line
Jens Axboe