Skip to contentSkip to conversation

blkcg throttle settings page cache writes

DISCUSSION Nikolay Borisov 2 participants 4 messages Sep 14, 2016 – Sep 27, 2016 3 messages not in the archive cgroup

lore
Share & tools
lore
#0 lore raw

Hello,

I'm observing blkcg throttle limits to not apply on writes which go through the page cache. E.g. I have a container whose write iops limits are set to 300 I then run a fio job with direct=0 (the default) and can see that writes/s (via iostat) are around 6-7k. Tracing confirmed thae AIO that fio is submitting is being handled by a kworker thread, which is in the root cgroup and as such has no lio limits. Direct writes/reads work as expected and are throttled.

After Paolo Valente's patch
(https://patchwork.kernel.org/patch/9249247/) I was left with the impression that bios submitted would just carry the information of the submitted throughout the storage stack. Apparently this is not the case. I also went through an old thread I started on dm-devel dealing with this exact problem (even before Paolo's patch), where Vivel Goyal hinted at this particular problem :
https://www.redhat.com/archives/dm-devel/2016-March/msg00012.html

So Tejun, my question is what is the problem for the blkcg information to be transferred alongside the bio (which I believed to be happening with the blkcg_association calls) so that even buffered writes are throttled via the cgroup policies? Or are the numbers I'm seeing in iostat actually the writes to the page-cache and not the underlying block device?

Regards,
Nikolay

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

Hello,

Quoted 7 lines · Nikolay Borisov

Hello,
So Tejun, my question is what is the problem for the blkcg information to be transferred alongside the bio (which I believed to be happening with the blkcg_association calls) so that even buffered writes are throttled via the cgroup policies? Or are the numbers I'm seeing in iostat actually the writes to the page-cache and not the underlying block device?

You gotta use cgroup v2 for that. Controlling buffered writes requires coordination between memory and io controllers which isn't possible with cgroup v1.

Thanks.

Signature · 1 line
tejun

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 11 lines · Tejun Heo

Hello,

Quoted 7 lines · Nikolay Borisov

Hello,
So Tejun, my question is what is the problem for the blkcg information to be transferred alongside the bio (which I believed to be happening with the blkcg_association calls) so that even buffered writes are throttled via the cgroup policies? Or are the numbers I'm seeing in iostat actually the writes to the page-cache and not the underlying block device?

Quoted 3 lines

You gotta use cgroup v2 for that. Controlling buffered writes requires coordination between memory and io controllers which isn't possible with cgroup v1.

Thanks for the information Tejun, that makes sense. However, in my testing I observed that when a throttle limit is set on IOPS it's not being enforced for buffered writes, but when it's set on bps it is actually working as expected. Is this expected or are there conflating factors which I'm not taking into account and as such what I'm seeing - iostat reports perfect utilisation of a particular device according to the bps throttle limit.

Quoted 3 lines

Thanks.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

Hello,

Quoted 7 lines · Nikolay Borisov

Thanks for the information Tejun, that makes sense. However, in my testing I observed that when a throttle limit is set on IOPS it's not being enforced for buffered writes, but when it's set on bps it is actually working as expected. Is this expected or are there conflating factors which I'm not taking into account and as such what I'm seeing - iostat reports perfect utilisation of a particular device according to the bps throttle limit.

If you're saying that you're seeing buffered writes controlled on cgroup v1 in non-root cgroups, there's something very wrong with the test setup. The code simply isn't there.

Thanks.

Signature · 1 line
tejun