Skip to contentSkip to conversation

can we reduce bio_set_dev overhead due to bio_associate_blkg?

PATCH In review Mike Snitzer 4 participants 14 messages Mar 30, 2022 – Apr 26, 2022

Subsystems (2)
lore
Share & tools
lore

Mike Snitzer

author maintainer
#0 lore raw

Hey Tejun and Dennis,

I recently found that due to bio_set_dev()'s call to bio_associate_blkg(), bio_set_dev() needs much more cpu than ideal; especially when doing 4K IOs via io_uring's HIPRI bio-polling.

I'm very naive about blk-cgroups.. so I'm hopeful you or others can help me cut through this to understand what the ideal outcome should be for DM's bio clone + remap heavy use-case as it relates to bio_associate_blkg.

If I hack dm-linear with a local __bio_set_dev that simply removes the call to bio_associate_blkg() my IOPS go from ~980K to 995K.

Looking at what is happening a bit, relative to this DM bio cloning usecase, it seems __bio_clone() calls bio_clone_blkg_association() to clone the blkg from DM device, then dm-linear.c:linear_map's call to bio_set_dev() will cause bio_associate_blkg(bio) to reuse the css but then it triggers an update because the bdev is being remapped in the bio (due to linear_map sending the IO to the real underlying device). End result _seems_ like collective wasteful effort to get the blk-cgroup resources setup properly in the face of a simple remap.

Seems the current DM pattern is causing repeat blkg work for _every_ remapped bio? Do you see a way to speed up repeat calls to bio_associate_blkg()?

Test kernel is my latest dm-5.19 branch (though latest Linus 5.18-rc0 kernel should be fine too):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19

I'm using dm-linear ontop on a 16G blk-mq null_blk device:

modprobe null_blk queue_mode=2 poll_queues=2 bs=4096 gb=16 SIZE=blockdev --getsz /dev/nullb0
echo "0 $SIZE linear /dev/nullb0 0" | dmsetup create linear

And running the workload with fio using this wrapper script:
io_uring.sh 20 1 /dev/mapper/linear 4096

#!/bin/bash

RTIME=$1
JOBS=$2
DEV=$3
BS=$4

QD=64
BATCH=16
HI=1

fio --bs=$BS --ioengine=io_uring --fixedbufs --registerfiles --hipri=$HI \
        --iodepth=$QD \
        --iodepth_batch_submit=$BATCH \
        --iodepth_batch_complete_min=$BATCH \
        --filename=$DEV \
        --direct=1 --runtime=$RTIME --numjobs=$JOBS --rw=randread \
        --name=test --group_reporting
#1 lore raw

Hi Mike,

Quoted 23 lines · Mike Snitzer

Hey Tejun and Dennis,

I recently found that due to bio_set_dev()'s call to bio_associate_blkg(), bio_set_dev() needs much more cpu than ideal; especially when doing 4K IOs via io_uring's HIPRI bio-polling.

I'm very naive about blk-cgroups.. so I'm hopeful you or others can help me cut through this to understand what the ideal outcome should be for DM's bio clone + remap heavy use-case as it relates to bio_associate_blkg.

If I hack dm-linear with a local __bio_set_dev that simply removes the call to bio_associate_blkg() my IOPS go from ~980K to 995K.

Looking at what is happening a bit, relative to this DM bio cloning usecase, it seems __bio_clone() calls bio_clone_blkg_association() to clone the blkg from DM device, then dm-linear.c:linear_map's call to bio_set_dev() will cause bio_associate_blkg(bio) to reuse the css but then it triggers an update because the bdev is being remapped in the bio (due to linear_map sending the IO to the real underlying device). End result _seems_ like collective wasteful effort to get the blk-cgroup resources setup properly in the face of a simple remap.

Quoted 4 lines

Seems the current DM pattern is causing repeat blkg work for _every_ remapped bio? Do you see a way to speed up repeat calls to bio_associate_blkg()?

I must admit I wrote this with limited knowledge of bio cloning at the time. I can fill in the thought process here.

The idea was every bio should have a blkg associated with it for io accounting and things like blk-iolatency and blk-iocost. The device abstraction I believe means we can set limits here as well on submission rate to the md device.

I think cloning is a special case that I might have gotten wrong. If there is a bio_set_dev() call after each clone(), then the bio_clone_blkg_association() is excess work. We'd need to audit how bio_alloc_clone() is being used to be safe. Alternatively, we could opt for a bio_alloc_clone_noblkg(), but that's a little bit uglier.

1. bio_set_dev() above md <- needed so we can do throttling on the md.
2. bio_alloc_clone() <- doesn't need to clone the blkg() info.
3. bio_set_dev() in md <- sets the right underlying device association.

Thanks,
Dennis

Quoted 31 lines

Test kernel is my latest dm-5.19 branch (though latest Linus 5.18-rc0 kernel should be fine too):
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-5.19

I'm using dm-linear ontop on a 16G blk-mq null_blk device:

modprobe null_blk queue_mode=2 poll_queues=2 bs=4096 gb=16 SIZE=blockdev --getsz /dev/nullb0
echo "0 $SIZE linear /dev/nullb0 0" | dmsetup create linear

And running the workload with fio using this wrapper script:
io_uring.sh 20 1 /dev/mapper/linear 4096

#!/bin/bash

RTIME=$1
JOBS=$2
DEV=$3
BS=$4

QD=64
BATCH=16
HI=1

fio --bs=$BS --ioengine=io_uring --fixedbufs --registerfiles --hipri=$HI \
        --iodepth=$QD \
        --iodepth_batch_submit=$BATCH \
        --iodepth_batch_complete_min=$BATCH \
        --filename=$DEV \
        --direct=1 --runtime=$RTIME --numjobs=$JOBS --rw=randread \
        --name=test --group_reporting
#2 lore raw
Quoted 5 lines · Dennis Zhou

I think cloning is a special case that I might have gotten wrong. If there is a bio_set_dev() call after each clone(), then the bio_clone_blkg_association() is excess work. We'd need to audit how bio_alloc_clone() is being used to be safe. Alternatively, we could opt for a bio_alloc_clone_noblkg(), but that's a little bit uglier.

As of Linux 5.18, the cloning interfaces have changed and take a block devie that the clone is intended to be used for, and bio_set_dev is mostly (there is a few more sports to be cleaned up in dm/md/bcache/btrfs) only used for remapping to a new device.

That being said I've eyed the code in bio_associate_blkg a bit and I've been wondering about some of how it is implemented as well.

Is recursive throttling really a thing? i.e. we can have cgroup policies on the upper (e.g. dm) device and then again on the lower (e.g. nvme device)? I think the code currently supports that, and if we want to keep that I don't really see much of a way to avoid the lookup, but maybe we cn make it faster.

#3 lore raw

Hello,

Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · Dennis Zhou

I think cloning is a special case that I might have gotten wrong. If there is a bio_set_dev() call after each clone(), then the bio_clone_blkg_association() is excess work. We'd need to audit how bio_alloc_clone() is being used to be safe. Alternatively, we could opt for a bio_alloc_clone_noblkg(), but that's a little bit uglier.

Quoted 5 lines

As of Linux 5.18, the cloning interfaces have changed and take a block devie that the clone is intended to be used for, and bio_set_dev is mostly (there is a few more sports to be cleaned up in dm/md/bcache/btrfs) only used for remapping to a new device.

I took a quick look. It seems with the new interface, bio_clone_blkg_association() is unnecessary given the correct association should be derived from the bio_alloc*() calls with the passed in bdev. Also, blkcg_bio_issue_init() in clone seems wrong.

Maybe the right thing to do here for md-linear and btrfs (what I've looked at) is to delay cloning until the map occurs and the right device is already in hand?

Quoted 3 lines

That being said I've eyed the code in bio_associate_blkg a bit and I've been wondering about some of how it is implemented as well.

I'm sure stuff has evolved since I've last been involved, but here is a brief explanation of the initial story. I suspect most of it holds true. Apologies if this isn't helpful.

For others, a blkcg is a block cgroup. A blkcg_gq, blkg for short, is the marrying of a blkcg and a request_queue. It takes a reference on both so IO associated with the cgroup is tracked to the appropriate cgroup and prevents the request_queue from going away. Punted IOs go here and writeback is managed here as well. On the hot path, this is the tagging that blk-rq-qos stuff might depend on.

The lookup itself is handled by blkg_lookup() which is a radix tree lookup of the request_queue. There is also a last hint which helps. blkg's are percpu-refcounted.

In terms of lifetimes and pinning. child_blkcg pins parent_blkcgs in a tree hierarchy up to the root_blkcg. blkgs pin the blkcg it's associated to, the request_queue, and the blkg_parent (parent_blkcg and request_queue). They die in hierarchical order, alive until all children have passed.

If there's anything else I can try to help answer please let me know.

Quoted 5 lines

Is recursive throttling really a thing? i.e. we can have cgroup policies on the upper (e.g. dm) device and then again on the lower (e.g. nvme device)? I think the code currently supports that, and if we want to keep that I don't really see much of a way to avoid the lookup, but maybe we cn make it faster.

I'm not sure. I've primarily dealt with physical devices. However, I'm sure there are more complex setups that use it. Is it a good idea is probably debatable.

Backing up though, I feel like the abstraction naturally alludes to this multiple association because you don't necessarily know when you hit physical devices until you finally submit through.

Thanks,
Dennis

#4 lore raw
Quoted 4 lines · Dennis Zhou

I took a quick look. It seems with the new interface, bio_clone_blkg_association() is unnecessary given the correct association should be derived from the bio_alloc*() calls with the passed in bdev. Also, blkcg_bio_issue_init() in clone seems wrong.

Yes, I think you are right.

Quoted 3 lines

Maybe the right thing to do here for md-linear and btrfs (what I've looked at) is to delay cloning until the map occurs and the right device is already in hand?

That would in general be the preferred option where possible.

#5 lore raw
Quoted 6 lines · Christoph Hellwig
Quoted 4 lines · Dennis Zhou

I took a quick look. It seems with the new interface, bio_clone_blkg_association() is unnecessary given the correct association should be derived from the bio_alloc*() calls with the passed in bdev. Also, blkcg_bio_issue_init() in clone seems wrong.

Quoted 7 lines

Yes, I think you are right.

Quoted 3 lines

Maybe the right thing to do here for md-linear and btrfs (what I've looked at) is to delay cloning until the map occurs and the right device is already in hand?

That would in general be the preferred option where possible.

Delaying cloning until remap is a problem for DM given the target_type ->map interface for all DM targets assumes the passed bio is already a clone that needs to be remapped accordingly.

I think we can achieve the goal of efficient cloning/remapping for both usecases simply by splitting out the bio_set_dev() and leaving it to the caller to pick which interface to use (e.g. clone vs clone_and_remap).

Christoph is this something you're open to doing as continuation of your bio alloc/clone related audit/changes? Or would you prefer someone else deal with it? I could take a closer look next week if needed.

Mike

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

#6 lore raw
Quoted 4 lines · Mike Snitzer

I think we can achieve the goal of efficient cloning/remapping for both usecases simply by splitting out the bio_set_dev() and leaving it to the caller to pick which interface to use (e.g. clone vs clone_and_remap).

You can just pass a NULL bdev to bio_alloc_clone/bio_init_clone. I've been hoping to get rid of that, but if we have a clear use case it will have to stay.

author maintainer
#7 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines · Mike Snitzer

I think we can achieve the goal of efficient cloning/remapping for both usecases simply by splitting out the bio_set_dev() and leaving it to the caller to pick which interface to use (e.g. clone vs clone_and_remap).

You can just pass a NULL bdev to bio_alloc_clone/bio_init_clone. I've been hoping to get rid of that, but if we have a clear use case it will have to stay.

DM core is just using bio_alloc_clone. And bio_alloc_bioset() allows bdev to be NULL -- so you're likely referring to that (which will skip bio_init's bio_associate_blkg).

Circling back to earlier in this thread, Dennis and you agreed that it doesn't make sense to have __bio_clone() do blkcg work if the clone bio will be remapped (via bio_set_dev). Given that, and the fact that bio_clone_blkg_association() assumes both bios are from same bdev, this change makes sense:

block/bio.c +0 -0
index 7892f1108ca6..0340acc283a0 100644
@@ -772,14 +772,16 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
	bio_set_flag(bio, BIO_CLONED);
	if (bio_flagged(bio_src, BIO_THROTTLED))
		bio_set_flag(bio, BIO_THROTTLED);
-	if (bio->bi_bdev == bio_src->bi_bdev &&
-	    bio_flagged(bio_src, BIO_REMAPPED))
-		bio_set_flag(bio, BIO_REMAPPED);
	bio->bi_ioprio = bio_src->bi_ioprio;
	bio->bi_iter = bio_src->bi_iter;

- bio_clone_blkg_association(bio, bio_src);
- blkcg_bio_issue_init(bio);
+ if (bio->bi_bdev == bio_src->bi_bdev) {
+ if (bio_flagged(bio_src, BIO_REMAPPED))
+ bio_set_flag(bio, BIO_REMAPPED);
+
+ bio_clone_blkg_association(bio, bio_src);
+ blkcg_bio_issue_init(bio);
+ }

	if (bio_crypt_clone(bio, bio_src, gfp) < 0)
		return -ENOMEM;

Think this will fix some of the performance penalty of redundant blkcg initialization that I reported (though like was also discussed: more work likely needed to further optimize bio_associate_blkg).

I'll audit DM targets and test to verify my changes and will post proper patch(es) once done.

Mike

#8 lore raw
Quoted 16 lines · Mike Snitzer
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines · Mike Snitzer

I think we can achieve the goal of efficient cloning/remapping for both usecases simply by splitting out the bio_set_dev() and leaving it to the caller to pick which interface to use (e.g. clone vs clone_and_remap).

You can just pass a NULL bdev to bio_alloc_clone/bio_init_clone. I've been hoping to get rid of that, but if we have a clear use case it will have to stay.

DM core is just using bio_alloc_clone. And bio_alloc_bioset() allows bdev to be NULL -- so you're likely referring to that (which will skip bio_init's bio_associate_blkg).

...

Quoted 27 lines
block/bio.c +7 -5
index 7892f1108ca6..0340acc283a0 100644
@@ -772,14 +772,16 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
 	bio_set_flag(bio, BIO_CLONED);
 	if (bio_flagged(bio_src, BIO_THROTTLED))
 		bio_set_flag(bio, BIO_THROTTLED);
-	if (bio->bi_bdev == bio_src->bi_bdev &&
-	    bio_flagged(bio_src, BIO_REMAPPED))
-		bio_set_flag(bio, BIO_REMAPPED);
 	bio->bi_ioprio = bio_src->bi_ioprio;
 	bio->bi_iter = bio_src->bi_iter;
 
-	bio_clone_blkg_association(bio, bio_src);
-	blkcg_bio_issue_init(bio);
+	if (bio->bi_bdev == bio_src->bi_bdev) {
+		if (bio_flagged(bio_src, BIO_REMAPPED))
+			bio_set_flag(bio, BIO_REMAPPED);
+
+		bio_clone_blkg_association(bio, bio_src);
+		blkcg_bio_issue_init(bio);
+	}
 
 	if (bio_crypt_clone(bio, bio_src, gfp) < 0)
 		return -ENOMEM;
Quoted 3 lines

Think this will fix some of the performance penalty of redundant blkcg initialization that I reported (though like was also discussed: more work likely needed to further optimize bio_associate_blkg).

Looking closer at the case where bio_{alloc,init}_clone are passed a bdev, bio_init() will call bio_associate_blkg() so the __bio_clone() work to do anything with blkbg isn't needed at all. So this patch is best:

block/bio.c +0 -0
index 7892f1108ca6..6980f1b4b0f4 100644
@@ -778,9 +778,6 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
	bio->bi_ioprio = bio_src->bi_ioprio;
	bio->bi_iter = bio_src->bi_iter;

- bio_clone_blkg_association(bio, bio_src);
- blkcg_bio_issue_init(bio);
-

	if (bio_crypt_clone(bio, bio_src, gfp) < 0)
		return -ENOMEM;
	if (bio_integrity(bio_src) &&
author maintainer
#9 lore raw

PATCH 1/1 block: remove redundant blk-cgroup init from __bio_clone

When bio_{alloc,init}_clone are passed a bdev, bio_init() will call bio_associate_blkg() so the __bio_clone() work to initialize blkcg isn't needed.

---
 block/bio.c | 3 ---
 1 file changed, 3 deletions(-)
block/bio.c +0 -3
index 7892f1108ca6..6980f1b4b0f4 100644
@@ -778,9 +778,6 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
 	bio->bi_ioprio = bio_src->bi_ioprio;
 	bio->bi_iter = bio_src->bi_iter;
 
-	bio_clone_blkg_association(bio, bio_src);
-	blkcg_bio_issue_init(bio);
-
 	if (bio_crypt_clone(bio, bio_src, gfp) < 0)
 		return -ENOMEM;
 	if (bio_integrity(bio_src) &&
Signature · 1 line
2.30.0
#10 lore raw

Re: [PATCH] block: remove redundant blk-cgroup init from __bio_clone

Quoted 3 lines · Mike Snitzer

When bio_{alloc,init}_clone are passed a bdev, bio_init() will call bio_associate_blkg() so the __bio_clone() work to initialize blkcg isn't needed.

No, unfortunately it isn't as simple as that. There are bios that do not use the default cgroup and thus blkg, e.g. those that come from cgroup writeback.

#11 lore raw

Re: [PATCH] block: remove redundant blk-cgroup init from __bio_clone

Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Mike Snitzer

When bio_{alloc,init}_clone are passed a bdev, bio_init() will call bio_associate_blkg() so the __bio_clone() work to initialize blkcg isn't needed.

No, unfortunately it isn't as simple as that. There are bios that do not use the default cgroup and thus blkg, e.g. those that come from cgroup writeback.

Yeah I wasn't quite right earlier. But, the new api isn't in line with the original semantics. Cloning the blkg preserves the original bios request_queue which likely differs from the bdev passed into clone. This means an IO might be charged to the wrong device.

So, the blkg combines the who, blkcg, and the where, the corresponding request_queue. Before bios were inited in 2 phases:

    bio_alloc();
    bio_set_dev();

This meant at clone time, we didn't have the where, but the who was encased in the blkg. So, after bio_clone_blkg_association() expected a bio_set_dev() call which called bio_associate_blkg(). When the bio already has a blkg, it attempts to reuse the blkcg while using the new bdev to find the correct blkg.

The tricky part seems to be how to seamlessly expose the appropriate blkcg without being intrusive to bio_alloc*() apis.

Regarding the NULL bdev, I think that works as long as we keep the bio_clone_blkg_association() call to carry the correct blkcg to the bio_set_dev() call.

Thanks,
Dennis

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

author maintainer Reviewed-by
#13 lore raw

Re: block: remove redundant blk-cgroup init from __bio_clone

Quoted 7 lines · Christoph Hellwig

So while I'm still diving into the somewhat invasive changes to optimize some of the cloning all we might relaly need for your use case should be this:

http://git.infradead.org/users/hch/block.git/shortlog/refs/heads/blk-clone-no-bdev

Can you check if this helps you?

Passing NULL will be useful for targets that do call bio_set_dev() but there are some that inherit the original bdev. I'll audit all targets and sort out how best to handle the conditional initialization to avoid needless work.

This change is useful and a requirement for relevant follow-on DM changes:

Thanks.