blkio cgroups controller doesn't work with LVM?

Hi,

Are there known limitations with the blkio cgroup controller when used with LVM?

I'm using Ubuntu 15.10 with the 4.2 kernel. I got the same results with CentOS 7.

I set up two groups, /sys/fs/cgroup/blkio/test1 and /sys/fs/cgroup/blkio/test2. I set the weight for test1 to 500, and the weight for test2 to 1000.

If I then read/write to a file on /dev/sda1 from a process in each group, then everything works as expected and test1 gets half the bandwidth of test2.

If I do the same but accessing an LVM logical volume, where the underlying physical volume is /dev/sda4, then both groups get the same bandwidth.

It appears that the group weight doesn't get propagated down through all the layers. Am I missing something? If not, is this documented anywhere?

Thanks,
Chris

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe cgroups" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 6 lines · Chris Friesen

Hi,

Are there known limitations with the blkio cgroup controller when used with LVM?

Quoted 6 lines

I'm using Ubuntu 15.10 with the 4.2 kernel. I got the same results with CentOS 7.

I set up two groups, /sys/fs/cgroup/blkio/test1 and /sys/fs/cgroup/blkio/test2. I set the weight for test1 to 500, and the weight for test2 to 1000.

The weighed mode of blkio works only with CFQ scheduler. And as far as I have seen you cannot set CFQ to be the scheduler of DM devices. In this case you can use the BLK io throttling mechanism. That's what I've encountered in my practice. Though I'd be happy to be proven wrong by someone. I believe the following sentence in the blkio controller states that:
"
First one is proportional weight time based division of disk policy. It is implemented in CFQ. Hence this policy takes effect only on leaf nodes when CFQ is being used.
"

Quoted 12 lines · Nikolay Borisov

Quoted 10 lines · Chris Friesen

Are there known limitations with the blkio cgroup controller when used with LVM?

I'm using Ubuntu 15.10 with the 4.2 kernel. I got the same results with CentOS 7.

I set up two groups, /sys/fs/cgroup/blkio/test1 and /sys/fs/cgroup/blkio/test2. I set the weight for test1 to 500, and the weight for test2 to 1000.

Quoted 11 lines

The weighed mode of blkio works only with CFQ scheduler. And as far as I have seen you cannot set CFQ to be the scheduler of DM devices. In this case you can use the BLK io throttling mechanism. That's what I've encountered in my practice. Though I'd be happy to be proven wrong by someone. I believe the following sentence in the blkio controller states that:
"
First one is proportional weight time based division of disk policy. It is implemented in CFQ. Hence this policy takes effect only on leaf nodes when CFQ is being used.
"

Right, LVM created devices are bio-based DM devices in the kernel. bio-based block devices do _not_ have an IO scheduler. Their underlying request-based device does.

I'm not well-versed on the top-level cgroup interface and how it maps to associated resources that are established in the kernel. But it could be that the configuration of blkio cgroup against a bio-based LVM device needs to be passed through to the underlying request-based device (e.g. /dev/sda4 in Chris's case)?

Quoted 16 lines · Nikolay Borisov

Quoted 6 lines · Chris Friesen

Hi,

Are there known limitations with the blkio cgroup controller when used with LVM?

Quoted 6 lines

I'm using Ubuntu 15.10 with the 4.2 kernel. I got the same results with CentOS 7.

I set up two groups, /sys/fs/cgroup/blkio/test1 and /sys/fs/cgroup/blkio/test2. I set the weight for test1 to 500, and the weight for test2 to 1000.

Quoted 11 lines

The weighed mode of blkio works only with CFQ scheduler. And as far as I have seen you cannot set CFQ to be the scheduler of DM devices. In this case you can use the BLK io throttling mechanism. That's what I've encountered in my practice. Though I'd be happy to be proven wrong by someone. I believe the following sentence in the blkio controller states that:
"
First one is proportional weight time based division of disk policy. It is implemented in CFQ. Hence this policy takes effect only on leaf nodes when CFQ is being used.
"

Right, LVM created devices are bio-based DM devices in the kernel. bio-based block devices do _not_ have an IO scheduler. Their underlying request-based device does.

I'm not well-versed on the top-level cgroup interface and how it maps to associated resources that are established in the kernel. But it could be that the configuration of blkio cgroup against a bio-based LVM device needs to be passed through to the underlying request-based device (e.g. /dev/sda4 in Chris's case)?

I'm also wondering whether the latest cgroup work that Tejun has just finished (afaik to support buffered IO in the IO controller) will afford us a more meaningful reason to work to make cgroups' blkio controller actually work with bio-based devices like LVM's DM devices?

I'm very much open to advice on how to proceed with investigating this integration work. Tejun, Vivek, anyone else: if you have advice on next steps for DM on this front _please_ yell, thanks!

Quoted 41 lines · Mike Snitzer

Quoted 16 lines · Nikolay Borisov

Quoted 6 lines · Chris Friesen

Hi,

Are there known limitations with the blkio cgroup controller when used with LVM?

Quoted 6 lines

I'm using Ubuntu 15.10 with the 4.2 kernel. I got the same results with CentOS 7.

I set up two groups, /sys/fs/cgroup/blkio/test1 and /sys/fs/cgroup/blkio/test2. I set the weight for test1 to 500, and the weight for test2 to 1000.

Quoted 11 lines

The weighed mode of blkio works only with CFQ scheduler. And as far as I have seen you cannot set CFQ to be the scheduler of DM devices. In this case you can use the BLK io throttling mechanism. That's what I've encountered in my practice. Though I'd be happy to be proven wrong by someone. I believe the following sentence in the blkio controller states that:
"
First one is proportional weight time based division of disk policy. It is implemented in CFQ. Hence this policy takes effect only on leaf nodes when CFQ is being used.
"

Right, LVM created devices are bio-based DM devices in the kernel. bio-based block devices do _not_ have an IO scheduler. Their underlying request-based device does.

I'm not well-versed on the top-level cgroup interface and how it maps to associated resources that are established in the kernel. But it could be that the configuration of blkio cgroup against a bio-based LVM device needs to be passed through to the underlying request-based device (e.g. /dev/sda4 in Chris's case)?

Quoted 8 lines

I'm also wondering whether the latest cgroup work that Tejun has just finished (afaik to support buffered IO in the IO controller) will afford us a more meaningful reason to work to make cgroups' blkio controller actually work with bio-based devices like LVM's DM devices?

I'm very much open to advice on how to proceed with investigating this integration work. Tejun, Vivek, anyone else: if you have advice on next steps for DM on this front _please_ yell, thanks!

Ok, here is my understanding. Tejun, please correct me if that's not the case anymore. I have not been able to keep pace with all the recent work.

IO throttling policies should be applied on top level dm devices and these should work for reads and direct writes.

For IO throttling buffered writes, I think it might not work on dm devices as it because we might not be copying cgroup information when cloning happens in dm layer.

IIRC, one concern with cloning cgroup info from parent bio was that how would one take care of any priority inversion issues. For example, we are waiting for a clone to finish IO which is in severely throttled IO cgroup and rest of the IO can't proceed till that IO finishes).

IIUC, there might not be a straight forward answer to that question. We probably will have to look at all the dm code closely and if that serialization is possible in any of the paths, then reset the cgroup info.

For CFQ's proportional policy, it might not work well when a dm device is sitting on top. And reason being that for all reads and direct writes we inherit cgroup from submitter and dm might be submitting IO from an internal thread, hence losing the cgroup of submitter hence IO gets misclassified at dm level.

To solve this, we will have to carry submitter's cgroup info in bio and clones and again think of priority inversion issues.

Thanks
Vivek

Right, LVM created devices are bio-based DM devices in the kernel. bio-based block devices do _not_ have an IO scheduler. Their underlying request-based device does.

I'm not well-versed on the top-level cgroup interface and how it maps to associated resources that are established in the kernel. But it could be that the configuration of blkio cgroup against a bio-based LVM device needs to be passed through to the underlying request-based device (e.g. /dev/sda4 in Chris's case)?

I'm also wondering whether the latest cgroup work that Tejun has just finished (afaik to support buffered IO in the IO controller) will afford us a more meaningful reason to work to make cgroups' blkio controller actually work with bio-based devices like LVM's DM devices?

I'm very much open to advice on how to proceed with investigating this integration work. Tejun, Vivek, anyone else: if you have advice on next steps for DM on this front _please_ yell, thanks!

Hello,

Quoted 3 lines · Mike Snitzer

Right, LVM created devices are bio-based DM devices in the kernel. bio-based block devices do _not_ have an IO scheduler. Their underlying request-based device does.

dm devices are not the actual resource source, so I don't think it'd work too well to put io controllers on them (can't really do things like proportional control without owning the queue).

Quoted 6 lines

I'm not well-versed on the top-level cgroup interface and how it maps to associated resources that are established in the kernel. But it could be that the configuration of blkio cgroup against a bio-based LVM device needs to be passed through to the underlying request-based device (e.g. /dev/sda4 in Chris's case)?

Quoted 8 lines

I'm also wondering whether the latest cgroup work that Tejun has just finished (afaik to support buffered IO in the IO controller) will afford us a more meaningful reason to work to make cgroups' blkio controller actually work with bio-based devices like LVM's DM devices?

I'm very much open to advice on how to proceed with investigating this integration work. Tejun, Vivek, anyone else: if you have advice on next steps for DM on this front _please_ yell, thanks!

I think the only thing necessary is dm transferring bio cgroup tags to the bio's that it ends up passing down the stack. Please take a look at fs/btrfs/extent_io.c::btrfs_bio_clone() for an example. We probably should introduce a wrapper for this so that each site doesn't need to ifdef it.

Thanks.

Thanks for the patch I will likely have time to test this sometime next week. But just to be sure - the expected behavior would be that processes writing to dm-based devices would experience the fair-shair scheduling of CFQ (provided that the physical devices that back those DM devices use CFQ), correct?

Thanks for the patch I will likely have time to test this sometime next week. But just to be sure - the expected behavior would be that processes writing to dm-based devices would experience the fair-shair scheduling of CFQ (provided that the physical devices that back those DM devices use CFQ), correct?

Quoted 1 line · Nikolay Borisov

Thanks for the patch I will likely have time to test this sometime next

week.

Quoted 4 lines

But just to be sure - the expected behavior would be that processes writing to dm-based devices would experience the fair-shair scheduling of CFQ (provided that the physical devices that back those DM devices use CFQ), correct?

Nikolay,

I am not sure how well it will work with CFQ of underlying device. It will get cgroup information right for buffered writes. But cgroup information

layer gets in between, then many a times submitter might be a worker thread and IO will be attributed to that worker's cgroup (root cgroup).

Give it a try anyway.

Thanks
Vivek

Quoted 8 lines · Vivek Goyal

Quoted 1 line · Nikolay Borisov

Thanks for the patch I will likely have time to test this sometime next

week.

Quoted 4 lines

But just to be sure - the expected behavior would be that processes writing to dm-based devices would experience the fair-shair scheduling of CFQ (provided that the physical devices that back those DM devices use CFQ), correct?

Quoted 4 lines

Nikolay,

I am not sure how well it will work with CFQ of underlying device. It will get cgroup information right for buffered writes. But cgroup information

Right, what's your definition of buffered writes?

My mental model is that
when a process submits a write request to a dm device , the bio is going to be put on a devi e workqueue which would then be serviced by a background worker thread and later the submitter notified. Do you refer to this whole gamut of operations as buffered writes?

for reads and direct writes will come from submitter's context and if dm

Quoted 2 lines

layer gets in between, then many a times submitter might be a worker thread and IO will be attributed to that worker's cgroup (root cgroup).

Be that as it may, proivded that the worker thread is in the 'correct' cgroup, then the appropriate babdwidth policies should apply, no?

Quoted 1 line · Nikolay Borisov

On Wednesday, March 2, 2016, Vivek Goyal <[email protected]

Quoted 3 lines · <javascript:;>> wrote:

Quoted 3 lines · Nikolay Borisov

Quoted 1 line

Thanks for the patch I will likely have time to test this sometime

next

Quoted 9 lines

Quoted 9 lines

week.

Quoted 4 lines

But just to be sure - the expected behavior would be that processes writing to dm-based devices would experience the fair-shair scheduling of CFQ (provided that the physical devices that back those DM devices use CFQ), correct?

Nikolay,

I am not sure how well it will work with CFQ of underlying device. It

will

Quoted 1 line

Quoted 1 line

get cgroup information right for buffered writes. But cgroup

information

Quoted 3 lines

Right, what's your definition of buffered writes?

Writes which go through page cache.

Quoted 2 lines

My mental model is that
when a process submits a write request to a dm device , the bio is going

to

Quoted 1 line

be put on a devi e workqueue which would then be serviced by a

background

Quoted 1 line

worker thread and later the submitter notified. Do you refer to this

whole

Quoted 1 line

gamut of operations as buffered writes?

No, once the bio is submitted to dm device it could be a buffered write or a direct write.

Quoted 8 lines

for reads and direct writes will come from submitter's context and if dm

Quoted 2 lines

layer gets in between, then many a times submitter might be a worker thread and IO will be attributed to that worker's cgroup (root cgroup).

Be that as it may, proivded that the worker thread is in the 'correct' cgroup, then the appropriate babdwidth policies should apply, no?

Worker thread will most likely be in root cgroup. So if a worker thread is submitting bio, it will be attributed to root cgroup.

We had similar issue with IO priority and it did not work reliably with CFQ on underlying device when dm devices were sitting on top.

If we really want to give it a try, I guess we will have to put cgroup info of submitter early in bio at the time of bio creation even for all kind of IO. Not sure if it is worth the effort.

For the case of IO throttling, I think you should put throttling rules on the dm device itself. That means as long as filesystem supports the cgroups, you should be getting right cgroup information for all kind of IO and throttling should work just fine.

Thanks
Vivek

Quoted 52 lines · Vivek Goyal

Quoted 1 line · Nikolay Borisov

On Wednesday, March 2, 2016, Vivek Goyal <[email protected]

Quoted 3 lines · <javascript:;>> wrote:

Quoted 3 lines · Nikolay Borisov

Quoted 1 line

Thanks for the patch I will likely have time to test this sometime

next

Quoted 9 lines

Quoted 9 lines

week.

Quoted 4 lines

But just to be sure - the expected behavior would be that processes writing to dm-based devices would experience the fair-shair scheduling of CFQ (provided that the physical devices that back those DM devices use CFQ), correct?

Nikolay,

I am not sure how well it will work with CFQ of underlying device. It

will

Quoted 1 line

Quoted 1 line

get cgroup information right for buffered writes. But cgroup

information

Quoted 3 lines

Right, what's your definition of buffered writes?

Writes which go through page cache.

Quoted 2 lines

My mental model is that
when a process submits a write request to a dm device , the bio is going

to

Quoted 1 line

be put on a devi e workqueue which would then be serviced by a

background

Quoted 1 line

worker thread and later the submitter notified. Do you refer to this

whole

Quoted 1 line

gamut of operations as buffered writes?

No, once the bio is submitted to dm device it could be a buffered write or a direct write.

Quoted 8 lines

for reads and direct writes will come from submitter's context and if dm

Quoted 2 lines

layer gets in between, then many a times submitter might be a worker thread and IO will be attributed to that worker's cgroup (root cgroup).

Be that as it may, proivded that the worker thread is in the 'correct' cgroup, then the appropriate babdwidth policies should apply, no?

Worker thread will most likely be in root cgroup. So if a worker thread is submitting bio, it will be attributed to root cgroup.

We had similar issue with IO priority and it did not work reliably with CFQ on underlying device when dm devices were sitting on top.

Quoted 8 lines

If we really want to give it a try, I guess we will have to put cgroup info of submitter early in bio at the time of bio creation even for all kind of IO. Not sure if it is worth the effort.

For the case of IO throttling, I think you should put throttling rules on the dm device itself. That means as long as filesystem supports the cgroups, you should be getting right cgroup information for all kind of IO and throttling should work just fine.

Throttling does work even now, but the use case I had in mind was proportional
distribution of IO. Imagine 50 or so dm devices, hosting IO intensive workloads. In
this situation, I'd be interested each of them getting proportional IO based on the weights
set in the blkcg controller for each respective cgroup for every workload.

On Wed, Mar 02, 2016 at 09:59:13PM +0200, Nikolay Borisov wrote:

We had similar issue with IO priority and it did not work reliably with CFQ on underlying device when dm devices were sitting on top.

If we really want to give it a try, I guess we will have to put cgroup info of submitter early in bio at the time of bio creation even for all kind of IO. Not sure if it is worth the effort.

For the case of IO throttling, I think you should put throttling rules on the dm device itself. That means as long as filesystem supports the cgroups, you should be getting right cgroup information for all kind of IO and throttling should work just fine.

Quoted 8 lines · Vivek Goyal

On Wed, Mar 02, 2016 at 09:59:13PM +0200, Nikolay Borisov wrote:

We had similar issue with IO priority and it did not work reliably with CFQ on underlying device when dm devices were sitting on top.

If we really want to give it a try, I guess we will have to put cgroup info of submitter early in bio at the time of bio creation even for all kind of IO. Not sure if it is worth the effort.

As it stands, imagine that you have a hypervisor node running many VMs (or containers), each of which is assigned a separate logical volume (possibly thin-provisioned) as its rootfs.

Ideally we want the disk accesses by those VMs to be "fair" relative to each other, and we want to guarantee a certain amount of bandwidth for the host as well.

Without this sort of feature, how can we accomplish that?

Quoted 5 lines

For the case of IO throttling, I think you should put throttling rules on the dm device itself. That means as long as filesystem supports the cgroups, you should be getting right cgroup information for all kind of IO and throttling should work just fine.

IO throttling isn't all that useful, since it requires you to know in advance what your IO rate is. And it doesn't adjust nicely as the number of competing entities changes the way that weight-based schemes do.

Quoted 14 lines · Chris Friesen

Quoted 8 lines · Vivek Goyal

On Wed, Mar 02, 2016 at 09:59:13PM +0200, Nikolay Borisov wrote:

We had similar issue with IO priority and it did not work reliably with CFQ on underlying device when dm devices were sitting on top.

If we really want to give it a try, I guess we will have to put cgroup info of submitter early in bio at the time of bio creation even for all kind of IO. Not sure if it is worth the effort.

As it stands, imagine that you have a hypervisor node running many VMs (or containers), each of which is assigned a separate logical volume (possibly thin-provisioned) as its rootfs.

Quoted 5 lines

Ideally we want the disk accesses by those VMs to be "fair" relative to each other, and we want to guarantee a certain amount of bandwidth for the host as well.

Without this sort of feature, how can we accomplish that?

As of now, you can't. I will try adding bio_associate_current() and see if that along with Mike's patches gets you what you are looking for.

Quoted 26 lines · Tejun Heo

Hello,

Quoted 3 lines · Mike Snitzer

Right, LVM created devices are bio-based DM devices in the kernel. bio-based block devices do _not_ have an IO scheduler. Their underlying request-based device does.

dm devices are not the actual resource source, so I don't think it'd work too well to put io controllers on them (can't really do things like proportional control without owning the queue).

Quoted 6 lines

I'm not well-versed on the top-level cgroup interface and how it maps to associated resources that are established in the kernel. But it could be that the configuration of blkio cgroup against a bio-based LVM device needs to be passed through to the underlying request-based device (e.g. /dev/sda4 in Chris's case)?

Quoted 8 lines

I'm also wondering whether the latest cgroup work that Tejun has just finished (afaik to support buffered IO in the IO controller) will afford us a more meaningful reason to work to make cgroups' blkio controller actually work with bio-based devices like LVM's DM devices?

I'm very much open to advice on how to proceed with investigating this integration work. Tejun, Vivek, anyone else: if you have advice on next steps for DM on this front _please_ yell, thanks!

Quoted 7 lines

I think the only thing necessary is dm transferring bio cgroup tags to the bio's that it ends up passing down the stack. Please take a look at fs/btrfs/extent_io.c::btrfs_bio_clone() for an example. We probably should introduce a wrapper for this so that each site doesn't need to ifdef it.

Thanks.

OK, I think this should do it. Nikolay and/or others can you test this patch using blkio cgroups controller with LVM devices and report back?

From: Mike Snitzer <[email protected]>
Date: Wed, 2 Mar 2016 12:37:39 -0500
Subject: [PATCH] block: transfer source bio's cgroup tags to clone via bio_associate_blkcg()

Move btrfs_bio_clone()'s support for transferring a source bio's cgroup tags to a clone into both bio_clone_bioset() and __bio_clone_fast(). The former is used by btrfs (MD and blk-core also use it via bio_split). The latter is used by both DM and bcache.

This should enable the blkio cgroups controller to work with all stacking bio-based block devices.

Reported-by Nikolay Borisov <[email protected]>

Suggested-by Tejun Heo <[email protected]>

Signed-off-by Mike Snitzer <[email protected]>

---
 block/bio.c          | 10 ++++++++++
 fs/btrfs/extent_io.c |  6 ------
 2 files changed, 10 insertions(+), 6 deletions(-)

Quoted 52 lines · Mike Snitzer

Quoted 26 lines · Tejun Heo

Hello,

Quoted 3 lines · Mike Snitzer

Right, LVM created devices are bio-based DM devices in the kernel. bio-based block devices do _not_ have an IO scheduler. Their underlying request-based device does.

dm devices are not the actual resource source, so I don't think it'd work too well to put io controllers on them (can't really do things like proportional control without owning the queue).

Quoted 6 lines

I'm not well-versed on the top-level cgroup interface and how it maps to associated resources that are established in the kernel. But it could be that the configuration of blkio cgroup against a bio-based LVM device needs to be passed through to the underlying request-based device (e.g. /dev/sda4 in Chris's case)?

Quoted 8 lines

I'm also wondering whether the latest cgroup work that Tejun has just finished (afaik to support buffered IO in the IO controller) will afford us a more meaningful reason to work to make cgroups' blkio controller actually work with bio-based devices like LVM's DM devices?

I'm very much open to advice on how to proceed with investigating this integration work. Tejun, Vivek, anyone else: if you have advice on next steps for DM on this front _please_ yell, thanks!

Quoted 7 lines

I think the only thing necessary is dm transferring bio cgroup tags to the bio's that it ends up passing down the stack. Please take a look at fs/btrfs/extent_io.c::btrfs_bio_clone() for an example. We probably should introduce a wrapper for this so that each site doesn't need to ifdef it.

Thanks.

OK, I think this should do it. Nikolay and/or others can you test this patch using blkio cgroups controller with LVM devices and report back?

From: Mike Snitzer <[email protected]>
Date: Wed, 2 Mar 2016 12:37:39 -0500
Subject: [PATCH] block: transfer source bio's cgroup tags to clone via bio_associate_blkcg()

Move btrfs_bio_clone()'s support for transferring a source bio's cgroup tags to a clone into both bio_clone_bioset() and __bio_clone_fast(). The former is used by btrfs (MD and blk-core also use it via bio_split). The latter is used by both DM and bcache.

This should enable the blkio cgroups controller to work with all stacking bio-based block devices.

Quoted 7 lines

Reported-by Nikolay Borisov <[email protected]>

Suggested-by Tejun Heo <[email protected]>

Signed-off-by Mike Snitzer <[email protected]>

---
 block/bio.c          | 10 ++++++++++
 fs/btrfs/extent_io.c |  6 ------
 2 files changed, 10 insertions(+), 6 deletions(-)

So I had a chance to test the settings here is what I got when running 2 container, using LVM-thin for their root device and having applied your patch:

When the 2 containers are using the same blkio.weight values (500) I get the following from running DD simultaneously on the 2 containers:

[root@c1501 ~]# dd if=test.img of=test2.img bs=1M count=3000 oflag=direct 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 165.171 s, 19.0 MB/s

[root@c1500 ~]# dd if=test.img of=test2.img bs=1M count=3000 oflag=direct 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 166.165 s, 18.9 MB/s

Also iostat showed the 2 volumes using almost the same amount of IO (around 20mb r/w). I then increase the weight for c1501 to 1000 i.e. twice the bandwidth that c1500 has, so I would expect its dd to complete twice as fast:

[root@c1501 ~]# dd if=test.img of=test2.img bs=1M count=3000 oflag=direct 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 150.892 s, 20.8 MB/s

[root@c1500 ~]# dd if=test.img of=test2.img bs=1M count=3000 oflag=direct 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 157.167 s, 20.0 MB/s

Now repeating the same tests but this time using the page-cache (echo 3 > /proc/sys/vm/drop_caches) was executed before each test run:

With equal weights (500):
[root@c1501 ~]# dd if=test.img of=test2.img bs=1M count=3000 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 114.923 s, 27.4 MB/s

[root@c1500 ~]# dd if=test.img of=test2.img bs=1M count=3000 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 120.245 s, 26.2 MB/s

With (c1501's weight equal to twice that of c1500 (1000)):

[root@c1501 ~]# dd if=test.img of=test2.img bs=1M count=3000 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 99.0181 s, 31.8 MB/s

[root@c1500 ~]# dd if=test.img of=test2.img bs=1M count=3000 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 122.872 s, 25.6 MB/s

I'd say that for buffered IO your patch does indeed make a difference, and this sort of aligns with what Vivek said about the patch working for buffered writes but not for direct.

I will proceed now and test his patch applied for the case of direct writes.

Hope this helps.

Quoted 118 lines · Nikolay Borisov

Quoted 52 lines · Mike Snitzer

Quoted 26 lines · Tejun Heo

Hello,

Quoted 3 lines · Mike Snitzer

Right, LVM created devices are bio-based DM devices in the kernel. bio-based block devices do _not_ have an IO scheduler. Their underlying request-based device does.

dm devices are not the actual resource source, so I don't think it'd work too well to put io controllers on them (can't really do things like proportional control without owning the queue).

Quoted 6 lines

I'm not well-versed on the top-level cgroup interface and how it maps to associated resources that are established in the kernel. But it could be that the configuration of blkio cgroup against a bio-based LVM device needs to be passed through to the underlying request-based device (e.g. /dev/sda4 in Chris's case)?

Quoted 8 lines

I'm also wondering whether the latest cgroup work that Tejun has just finished (afaik to support buffered IO in the IO controller) will afford us a more meaningful reason to work to make cgroups' blkio controller actually work with bio-based devices like LVM's DM devices?

I'm very much open to advice on how to proceed with investigating this integration work. Tejun, Vivek, anyone else: if you have advice on next steps for DM on this front _please_ yell, thanks!

Quoted 7 lines

I think the only thing necessary is dm transferring bio cgroup tags to the bio's that it ends up passing down the stack. Please take a look at fs/btrfs/extent_io.c::btrfs_bio_clone() for an example. We probably should introduce a wrapper for this so that each site doesn't need to ifdef it.

Thanks.

OK, I think this should do it. Nikolay and/or others can you test this patch using blkio cgroups controller with LVM devices and report back?

From: Mike Snitzer <[email protected]>
Date: Wed, 2 Mar 2016 12:37:39 -0500
Subject: [PATCH] block: transfer source bio's cgroup tags to clone via bio_associate_blkcg()

Move btrfs_bio_clone()'s support for transferring a source bio's cgroup tags to a clone into both bio_clone_bioset() and __bio_clone_fast(). The former is used by btrfs (MD and blk-core also use it via bio_split). The latter is used by both DM and bcache.

This should enable the blkio cgroups controller to work with all stacking bio-based block devices.

Quoted 7 lines

Reported-by Nikolay Borisov <[email protected]>

Suggested-by Tejun Heo <[email protected]>

Signed-off-by Mike Snitzer <[email protected]>

---
 block/bio.c          | 10 ++++++++++
 fs/btrfs/extent_io.c |  6 ------
 2 files changed, 10 insertions(+), 6 deletions(-)

So I had a chance to test the settings here is what I got when running 2 container, using LVM-thin for their root device and having applied your patch:

When the 2 containers are using the same blkio.weight values (500) I get the following from running DD simultaneously on the 2 containers:

[root@c1501 ~]# dd if=test.img of=test2.img bs=1M count=3000 oflag=direct 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 165.171 s, 19.0 MB/s

[root@c1500 ~]# dd if=test.img of=test2.img bs=1M count=3000 oflag=direct 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 166.165 s, 18.9 MB/s

Also iostat showed the 2 volumes using almost the same amount of IO (around 20mb r/w). I then increase the weight for c1501 to 1000 i.e. twice the bandwidth that c1500 has, so I would expect its dd to complete twice as fast:

[root@c1501 ~]# dd if=test.img of=test2.img bs=1M count=3000 oflag=direct 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 150.892 s, 20.8 MB/s

[root@c1500 ~]# dd if=test.img of=test2.img bs=1M count=3000 oflag=direct 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 157.167 s, 20.0 MB/s

Now repeating the same tests but this time using the page-cache (echo 3 > /proc/sys/vm/drop_caches) was executed before each test run:

With equal weights (500):
[root@c1501 ~]# dd if=test.img of=test2.img bs=1M count=3000 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 114.923 s, 27.4 MB/s

[root@c1500 ~]# dd if=test.img of=test2.img bs=1M count=3000 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 120.245 s, 26.2 MB/s

With (c1501's weight equal to twice that of c1500 (1000)):

[root@c1501 ~]# dd if=test.img of=test2.img bs=1M count=3000 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 99.0181 s, 31.8 MB/s

Quoted 4 lines

[root@c1500 ~]# dd if=test.img of=test2.img bs=1M count=3000 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 122.872 s, 25.6 MB/s

And another test which makes it obvious that your patch works:

[root@c1501 ~]# dd if=test.img of=test2.img bs=1M count=6000 6000+0 records in
6000+0 records out
6291456000 bytes (6.3 GB) copied, 210.466 s, 29.9 MB/s

[root@c1500 ~]# dd if=test.img of=test2.img bs=1M count=3000 3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 201.118 s, 15.6 MB/s

So a file that is twice the size of another one (6vs3 g) is copied for almost the same amount of time with 2x the bandwidth.

Great.

Jens, can you pick up the patch in question ("[PATCH] block: transfer source bio's cgroup tags to clone via bio_associate_blkcg()") that I posted in this thread?