blkcg throttle settings page cache writes
Hello,
I'm observing blkcg throttle limits to not apply on writes which go through the page cache. E.g. I have a container whose write iops limits are set to 300 I then run a fio job with direct=0 (the default) and can see that writes/s (via iostat) are around 6-7k. Tracing confirmed thae AIO that fio is submitting is being handled by a kworker thread, which is in the root cgroup and as such has no lio limits. Direct writes/reads work as expected and are throttled.
After Paolo Valente's patch
(https://patchwork.kernel.org/patch/9249247/) I was left with the impression that bios submitted would just carry the information of the submitted throughout the storage stack. Apparently this is not the case. I also went through an old thread I started on dm-devel dealing with this exact problem (even before Paolo's patch), where Vivel Goyal hinted at this particular problem :
https://www.redhat.com/archives/dm-devel/2016-March/msg00012.html
So Tejun, my question is what is the problem for the blkcg information to be transferred alongside the bio (which I believed to be happening with the blkcg_association calls) so that even buffered writes are throttled via the cgroup policies? Or are the numbers I'm seeing in iostat actually the writes to the page-cache and not the underlying block device?
Regards,
Nikolay
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hello,
Quoted 7 lines · Nikolay Borisov
Hello,
So Tejun, my question is what is the problem for the blkcg information to be transferred alongside the bio (which I believed to be happening with theblkcg_associationcalls) so that even buffered writes are throttled via the cgroup policies? Or are the numbers I'm seeing in iostat actually the writes to the page-cache and not the underlying block device?
You gotta use cgroup v2 for that. Controlling buffered writes requires coordination between memory and io controllers which isn't possible with cgroup v1.
Thanks.
Signature · 1 line
tejun
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 11 lines · Tejun Heo
Hello,
Quoted 7 lines · Nikolay Borisov
Hello,
So Tejun, my question is what is the problem for the blkcg information to be transferred alongside the bio (which I believed to be happening with theblkcg_associationcalls) so that even buffered writes are throttled via the cgroup policies? Or are the numbers I'm seeing in iostat actually the writes to the page-cache and not the underlying block device?
Quoted 3 lines
You gotta use cgroup v2 for that. Controlling buffered writes requires coordination between memory and io controllers which isn't possible with cgroup v1.
Thanks for the information Tejun, that makes sense. However, in my testing I observed that when a throttle limit is set on IOPS it's not being enforced for buffered writes, but when it's set on bps it is actually working as expected. Is this expected or are there conflating factors which I'm not taking into account and as such what I'm seeing - iostat reports perfect utilisation of a particular device according to the bps throttle limit.
Quoted 3 lines
Thanks.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hello,
Quoted 7 lines · Nikolay Borisov
Thanks for the information Tejun, that makes sense. However, in my testing I observed that when a throttle limit is set on IOPS it's not being enforced for buffered writes, but when it's set on bps it is actually working as expected. Is this expected or are there conflating factors which I'm not taking into account and as such what I'm seeing - iostat reports perfect utilisation of a particular device according to the bps throttle limit.
If you're saying that you're seeing buffered writes controlled on cgroup v1 in non-root cgroups, there's something very wrong with the test setup. The code simply isn't there.
Thanks.
Signature · 1 line
tejun