can we reduce bio_set_dev overhead due to bio_associate_blkg?
Subsystems (2)
Mike Snitzer
maintainerHey Tejun and Dennis,
I recently found that due to bio_set_dev()'s call to bio_associate_blkg(), bio_set_dev() needs much more cpu than ideal; especially when doing 4K IOs via io_uring's HIPRI bio-polling.
I'm very naive about blk-cgroups.. so I'm hopeful you or others can help me cut through this to understand what the ideal outcome should be for DM's bio clone + remap heavy use-case as it relates to bio_associate_blkg.
If I hack dm-linear with a local __bio_set_dev that simply removes the call to bio_associate_blkg() my IOPS go from ~980K to 995K.
Looking at what is happening a bit, relative to this DM bio cloning usecase, it seems __bio_clone() calls bio_clone_blkg_association() to clone the blkg from DM device, then dm-linear.c:linear_map's call to bio_set_dev() will cause bio_associate_blkg(bio) to reuse the css but then it triggers an update because the bdev is being remapped in the bio (due to linear_map sending the IO to the real underlying device). End result _seems_ like collective wasteful effort to get the blk-cgroup resources setup properly in the face of a simple remap.
Seems the current DM pattern is causing repeat blkg work for _every_ remapped bio? Do you see a way to speed up repeat calls to bio_associate_blkg()?
Test kernel is my latest dm-5.19 branch (though latest Linus 5.18-rc0 kernel should be fine too):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19
I'm using dm-linear ontop on a 16G blk-mq null_blk device:
modprobe null_blk queue_mode=2 poll_queues=2 bs=4096 gb=16 SIZE=blockdev --getsz /dev/nullb0
echo "0 $SIZE linear /dev/nullb0 0" | dmsetup create linear
And running the workload with fio using this wrapper script:io_uring.sh 20 1 /dev/mapper/linear 4096
#!/bin/bash
RTIME=$1
JOBS=$2
DEV=$3
BS=$4
QD=64
BATCH=16
HI=1
fio --bs=$BS --ioengine=io_uring --fixedbufs --registerfiles --hipri=$HI \
--iodepth=$QD \
--iodepth_batch_submit=$BATCH \
--iodepth_batch_complete_min=$BATCH \
--filename=$DEV \
--direct=1 --runtime=$RTIME --numjobs=$JOBS --rw=randread \
--name=test --group_reportingHi Mike,
Quoted 23 lines · Mike Snitzer
Hey Tejun and Dennis,
I recently found that due to bio_set_dev()'s call to
bio_associate_blkg(),bio_set_dev()needs much more cpu than ideal; especially when doing 4K IOs via io_uring's HIPRI bio-polling.I'm very naive about blk-cgroups.. so I'm hopeful you or others can help me cut through this to understand what the ideal outcome should be for DM's bio clone + remap heavy use-case as it relates to
bio_associate_blkg.If I hack dm-linear with a local
__bio_set_devthat simply removes the call tobio_associate_blkg()my IOPS go from ~980K to 995K.Looking at what is happening a bit, relative to this DM bio cloning usecase, it seems
__bio_clone()callsbio_clone_blkg_association()to clone the blkg from DM device, then dm-linear.c:linear_map's call tobio_set_dev()will causebio_associate_blkg(bio)to reuse the css but then it triggers an update because the bdev is being remapped in the bio (due tolinear_mapsending the IO to the real underlying device). End result _seems_ like collective wasteful effort to get the blk-cgroup resources setup properly in the face of a simple remap.
Quoted 4 lines
Seems the current DM pattern is causing repeat blkg work for _every_ remapped bio? Do you see a way to speed up repeat calls to
bio_associate_blkg()?
I must admit I wrote this with limited knowledge of bio cloning at the time. I can fill in the thought process here.
The idea was every bio should have a blkg associated with it for io accounting and things like blk-iolatency and blk-iocost. The device abstraction I believe means we can set limits here as well on submission rate to the md device.
I think cloning is a special case that I might have gotten wrong. If there is a bio_set_dev() call after each clone(), then the bio_clone_blkg_association() is excess work. We'd need to audit how bio_alloc_clone() is being used to be safe. Alternatively, we could opt for a bio_alloc_clone_noblkg(), but that's a little bit uglier.
1. bio_set_dev() above md <- needed so we can do throttling on the md.
2. bio_alloc_clone() <- doesn't need to clone the blkg() info.
3. bio_set_dev() in md <- sets the right underlying device association.
Thanks,
Dennis
Quoted 31 lines
Test kernel is my latest dm-5.19 branch (though latest Linus 5.18-rc0 kernel should be fine too):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19I'm using dm-linear ontop on a 16G blk-mq
null_blkdevice:modprobe
null_blkqueue_mode=2 poll_queues=2 bs=4096 gb=16 SIZE=blockdev --getsz /dev/nullb0
echo "0 $SIZE linear/dev/nullb00" | dmsetup create linearAnd running the workload with fio using this wrapper script:
io_uring.sh20 1/dev/mapper/linear4096#!/bin/bash
RTIME=$1
JOBS=$2
DEV=$3
BS=$4QD=64
BATCH=16
HI=1fio --bs=$BS --ioengine=io_uring --fixedbufs --registerfiles --hipri=$HI \ --iodepth=$QD \ --iodepth_batch_submit=$BATCH \ --iodepth_batch_complete_min=$BATCH \ --filename=$DEV \ --direct=1 --runtime=$RTIME --numjobs=$JOBS --rw=randread \ --name=test --group_reporting
Quoted 5 lines · Dennis Zhou
I think cloning is a special case that I might have gotten wrong. If there is a
bio_set_dev()call after eachclone(), then thebio_clone_blkg_association()is excess work. We'd need to audit howbio_alloc_clone()is being used to be safe. Alternatively, we could opt for abio_alloc_clone_noblkg(), but that's a little bit uglier.
As of Linux 5.18, the cloning interfaces have changed and take a block devie that the clone is intended to be used for, and bio_set_dev is mostly (there is a few more sports to be cleaned up in dm/md/bcache/btrfs) only used for remapping to a new device.
That being said I've eyed the code in bio_associate_blkg a bit and I've been wondering about some of how it is implemented as well.
Is recursive throttling really a thing? i.e. we can have cgroup policies on the upper (e.g. dm) device and then again on the lower (e.g. nvme device)? I think the code currently supports that, and if we want to keep that I don't really see much of a way to avoid the lookup, but maybe we cn make it faster.
Hello,
Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · Dennis Zhou
I think cloning is a special case that I might have gotten wrong. If there is a
bio_set_dev()call after eachclone(), then thebio_clone_blkg_association()is excess work. We'd need to audit howbio_alloc_clone()is being used to be safe. Alternatively, we could opt for abio_alloc_clone_noblkg(), but that's a little bit uglier.
Quoted 5 lines
As of Linux 5.18, the cloning interfaces have changed and take a block devie that the clone is intended to be used for, and
bio_set_devis mostly (there is a few more sports to be cleaned up in dm/md/bcache/btrfs) only used for remapping to a new device.
I took a quick look. It seems with the new interface, bio_clone_blkg_association() is unnecessary given the correct association should be derived from the bio_alloc*() calls with the passed in bdev. Also, blkcg_bio_issue_init() in clone seems wrong.
Maybe the right thing to do here for md-linear and btrfs (what I've looked at) is to delay cloning until the map occurs and the right device is already in hand?
Quoted 3 lines
That being said I've eyed the code in
bio_associate_blkga bit and I've been wondering about some of how it is implemented as well.
I'm sure stuff has evolved since I've last been involved, but here is a brief explanation of the initial story. I suspect most of it holds true. Apologies if this isn't helpful.
For others, a blkcg is a block cgroup. A blkcg_gq, blkg for short, is the marrying of a blkcg and a request_queue. It takes a reference on both so IO associated with the cgroup is tracked to the appropriate cgroup and prevents the request_queue from going away. Punted IOs go here and writeback is managed here as well. On the hot path, this is the tagging that blk-rq-qos stuff might depend on.
The lookup itself is handled by blkg_lookup() which is a radix tree lookup of the request_queue. There is also a last hint which helps. blkg's are percpu-refcounted.
In terms of lifetimes and pinning. child_blkcg pins parent_blkcgs in a tree hierarchy up to the root_blkcg. blkgs pin the blkcg it's associated to, the request_queue, and the blkg_parent (parent_blkcg and request_queue). They die in hierarchical order, alive until all children have passed.
If there's anything else I can try to help answer please let me know.
Quoted 5 lines
Is recursive throttling really a thing? i.e. we can have cgroup policies on the upper (e.g. dm) device and then again on the lower (e.g. nvme device)? I think the code currently supports that, and if we want to keep that I don't really see much of a way to avoid the lookup, but maybe we cn make it faster.
I'm not sure. I've primarily dealt with physical devices. However, I'm sure there are more complex setups that use it. Is it a good idea is probably debatable.
Backing up though, I feel like the abstraction naturally alludes to this multiple association because you don't necessarily know when you hit physical devices until you finally submit through.
Thanks,
Dennis
Quoted 4 lines · Dennis Zhou
I took a quick look. It seems with the new interface,
bio_clone_blkg_association()is unnecessary given the correct association should be derived from the bio_alloc*() calls with the passed in bdev. Also,blkcg_bio_issue_init()in clone seems wrong.
Yes, I think you are right.
Quoted 3 lines
Maybe the right thing to do here for md-linear and btrfs (what I've looked at) is to delay cloning until the map occurs and the right device is already in hand?
That would in general be the preferred option where possible.
Quoted 6 lines · Christoph Hellwig
Quoted 4 lines · Dennis Zhou
I took a quick look. It seems with the new interface,
bio_clone_blkg_association()is unnecessary given the correct association should be derived from the bio_alloc*() calls with the passed in bdev. Also,blkcg_bio_issue_init()in clone seems wrong.
Quoted 7 lines
Yes, I think you are right.
Quoted 3 lines
Maybe the right thing to do here for md-linear and btrfs (what I've looked at) is to delay cloning until the map occurs and the right device is already in hand?
That would in general be the preferred option where possible.
Delaying cloning until remap is a problem for DM given the target_type ->map interface for all DM targets assumes the passed bio is already a clone that needs to be remapped accordingly.
I think we can achieve the goal of efficient cloning/remapping for both usecases simply by splitting out the bio_set_dev() and leaving it to the caller to pick which interface to use (e.g. clone vs clone_and_remap).
Christoph is this something you're open to doing as continuation of your bio alloc/clone related audit/changes? Or would you prefer someone else deal with it? I could take a closer look next week if needed.
Mike
--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Quoted 4 lines · Mike Snitzer
I think we can achieve the goal of efficient cloning/remapping for both usecases simply by splitting out the
bio_set_dev()and leaving it to the caller to pick which interface to use (e.g. clone vsclone_and_remap).
You can just pass a NULL bdev to bio_alloc_clone/bio_init_clone. I've been hoping to get rid of that, but if we have a clear use case it will have to stay.
Mike Snitzer
maintainerQuoted 9 lines · Christoph Hellwig
Quoted 4 lines · Mike Snitzer
I think we can achieve the goal of efficient cloning/remapping for both usecases simply by splitting out the
bio_set_dev()and leaving it to the caller to pick which interface to use (e.g. clone vsclone_and_remap).You can just pass a NULL bdev to bio_alloc_clone/bio_init_clone. I've been hoping to get rid of that, but if we have a clear use case it will have to stay.
DM core is just using bio_alloc_clone. And bio_alloc_bioset() allows bdev to be NULL -- so you're likely referring to that (which will skip bio_init's bio_associate_blkg).
Circling back to earlier in this thread, Dennis and you agreed that it doesn't make sense to have __bio_clone() do blkcg work if the clone bio will be remapped (via bio_set_dev). Given that, and the fact that bio_clone_blkg_association() assumes both bios are from same bdev, this change makes sense:
block/bio.c +0 -0
@@ -772,14 +772,16 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
bio_set_flag(bio, BIO_CLONED); if (bio_flagged(bio_src, BIO_THROTTLED)) bio_set_flag(bio, BIO_THROTTLED); - if (bio->bi_bdev == bio_src->bi_bdev && - bio_flagged(bio_src, BIO_REMAPPED)) - bio_set_flag(bio, BIO_REMAPPED); bio->bi_ioprio = bio_src->bi_ioprio; bio->bi_iter = bio_src->bi_iter;
- bio_clone_blkg_association(bio, bio_src);
- blkcg_bio_issue_init(bio);
+ if (bio->bi_bdev == bio_src->bi_bdev) {
+ if (bio_flagged(bio_src, BIO_REMAPPED))
+ bio_set_flag(bio, BIO_REMAPPED);
+
+ bio_clone_blkg_association(bio, bio_src);
+ blkcg_bio_issue_init(bio);
+ }
if (bio_crypt_clone(bio, bio_src, gfp) < 0) return -ENOMEM;
Think this will fix some of the performance penalty of redundant blkcg initialization that I reported (though like was also discussed: more work likely needed to further optimize bio_associate_blkg).
I'll audit DM targets and test to verify my changes and will post proper patch(es) once done.
Mike
Quoted 16 lines · Mike Snitzer
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines · Mike Snitzer
I think we can achieve the goal of efficient cloning/remapping for both usecases simply by splitting out the
bio_set_dev()and leaving it to the caller to pick which interface to use (e.g. clone vsclone_and_remap).You can just pass a NULL bdev to bio_alloc_clone/bio_init_clone. I've been hoping to get rid of that, but if we have a clear use case it will have to stay.
DM core is just using
bio_alloc_clone. Andbio_alloc_bioset()allows bdev to be NULL -- so you're likely referring to that (which will skip bio_init'sbio_associate_blkg).
...
Quoted 27 lines
block/bio.c +7
-5@@ -772,14 +772,16 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)bio_set_flag (bio,BIO_CLONED );if (bio_flagged (bio_src,BIO_THROTTLED ))bio_set_flag (bio,BIO_THROTTLED ); -if (bio->bi_bdev == bio_src->bi_bdev && -bio_flagged (bio_src,BIO_REMAPPED )) -bio_set_flag (bio,BIO_REMAPPED ); bio->bi_ioprio = bio_src->bi_ioprio; bio->bi_iter = bio_src->bi_iter; -bio_clone_blkg_association (bio, bio_src); -blkcg_bio_issue_init (bio); +if (bio->bi_bdev == bio_src->bi_bdev) { +if (bio_flagged (bio_src,BIO_REMAPPED )) +bio_set_flag (bio,BIO_REMAPPED ); + +bio_clone_blkg_association (bio, bio_src); +blkcg_bio_issue_init (bio); + }if (bio_crypt_clone (bio, bio_src, gfp) <0 )return -ENOMEM ;
Quoted 3 lines
Think this will fix some of the performance penalty of redundant blkcg initialization that I reported (though like was also discussed: more work likely needed to further optimize
bio_associate_blkg).
Looking closer at the case where bio_{alloc,init}_clone are passed a bdev, bio_init() will call bio_associate_blkg() so the __bio_clone() work to do anything with blkbg isn't needed at all. So this patch is best:
block/bio.c +0 -0
@@ -778,9 +778,6 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
bio->bi_ioprio = bio_src->bi_ioprio; bio->bi_iter = bio_src->bi_iter;
- bio_clone_blkg_association(bio, bio_src);
- blkcg_bio_issue_init(bio);
-
if (bio_crypt_clone(bio, bio_src, gfp) < 0) return -ENOMEM; if (bio_integrity(bio_src) &&
Mike Snitzer
maintainerPATCH 1/1 block: remove redundant blk-cgroup init from __bio_clone
When bio_{alloc,init}_clone are passed a bdev, bio_init() will call bio_associate_blkg() so the __bio_clone() work to initialize blkcg isn't needed.
--- block/bio.c | 3 --- 1 file changed, 3 deletions(-)
block/bio.c +0 -3
@@ -778,9 +778,6 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp) bio->bi_ioprio = bio_src->bi_ioprio; bio->bi_iter = bio_src->bi_iter; -bio_clone_blkg_association (bio, bio_src); -blkcg_bio_issue_init (bio); -if (bio_crypt_clone (bio, bio_src, gfp) <0 )return -ENOMEM ; if (bio_integrity(bio_src) &&
Signature · 1 line
2.30.0
Re: [PATCH] block: remove redundant blk-cgroup init from __bio_clone
Quoted 3 lines · Mike Snitzer
When bio_{alloc,init}_clone are passed a bdev,
bio_init()will callbio_associate_blkg()so the__bio_clone()work to initialize blkcg isn't needed.
No, unfortunately it isn't as simple as that. There are bios that do not use the default cgroup and thus blkg, e.g. those that come from cgroup writeback.
Re: [PATCH] block: remove redundant blk-cgroup init from __bio_clone
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Mike Snitzer
When bio_{alloc,init}_clone are passed a bdev,
bio_init()will callbio_associate_blkg()so the__bio_clone()work to initialize blkcg isn't needed.No, unfortunately it isn't as simple as that. There are bios that do not use the default cgroup and thus blkg, e.g. those that come from cgroup writeback.
Yeah I wasn't quite right earlier. But, the new api isn't in line with the original semantics. Cloning the blkg preserves the original bios request_queue which likely differs from the bdev passed into clone. This means an IO might be charged to the wrong device.
So, the blkg combines the who, blkcg, and the where, the corresponding request_queue. Before bios were inited in 2 phases:
bio_alloc();
bio_set_dev();This meant at clone time, we didn't have the where, but the who was encased in the blkg. So, after bio_clone_blkg_association() expected a bio_set_dev() call which called bio_associate_blkg(). When the bio already has a blkg, it attempts to reuse the blkcg while using the new bdev to find the correct blkg.
The tricky part seems to be how to seamlessly expose the appropriate blkcg without being intrusive to bio_alloc*() apis.
Regarding the NULL bdev, I think that works as long as we keep the bio_clone_blkg_association() call to carry the correct blkcg to the bio_set_dev() call.
Thanks,
Dennis
--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Re: [PATCH] block: remove redundant blk-cgroup init from __bio_clone
In reply to Mike SnitzerSo while I'm still diving into the somewhat invasive changes to optimize some of the cloning all we might relaly need for your use case should be this:
http://git.infradead.org/users/hch/block.git/shortlog/refs/heads/blk-clone-no-bdev
Can you check if this helps you?
Re: block: remove redundant blk-cgroup init from __bio_clone
Quoted 7 lines · Christoph Hellwig
So while I'm still diving into the somewhat invasive changes to optimize some of the cloning all we might relaly need for your use case should be this:
http://git.infradead.org/users/hch/block.git/shortlog/refs/heads/blk-clone-no-bdev
Can you check if this helps you?
Passing NULL will be useful for targets that do call bio_set_dev() but there are some that inherit the original bdev. I'll audit all targets and sort out how best to handle the conditional initialization to avoid needless work.
This change is useful and a requirement for relevant follow-on DM changes:
Thanks.