memcg: per cgroup dirty_ratio

The goal of the patch is to control how much dirty file pages a cgroup can have at any given time (see also [1]).

Dirty file and writeback pages are accounted for each cgroup using the memory controller statistics. Moreover, the dirty_ratio parameter is added to the memory controller. It contains, as a percentage of the cgroup memory, the number of dirty pages at which the processes belonging to the cgroup which are generating disk writes will start writing out dirty data.

So, the behaviour is actually the same as the global dirty_ratio, except that it works per cgroup.

Interface:
- two new entries "writeback" and "filedirty" are added to the file memory.stat, to export to userspace respectively the number of pages under writeback and the number of dirty file pages in the cgroup

- the new file memory.dirty_ratio is added in the cgroup filesystem to show/set the memcg dirty_ratio

[ This patch is still experimental and I only did few quick tests. I'd like to do run more detailed benchmarks and compare the results, I guess the overhead introduced by this patch shouldn't be so small... and BTW I would prefer a dirty limit in bytes, intead of using a percentage of memory. Bytes are hugely more flexible IMHO, they allow to define more fine-grained limits and so this would work better on large memory machines. ]

[1] http://lkml.org/lkml/2008/9/9/245

Quoted 12 lines · On Fri, 12 Sep 2008 17:09:50 +0200 Andrea Righi…

The goal of the patch is to control how much dirty file pages a cgroup can have at any given time (see also [1]).

Dirty file and writeback pages are accounted for each cgroup using the memory controller statistics. Moreover, the dirty_ratio parameter is added to the memory controller. It contains, as a percentage of the cgroup memory, the number of dirty pages at which the processes belonging to the cgroup which are generating disk writes will start writing out dirty data.

So, the behaviour is actually the same as the global dirty_ratio, except that it works per cgroup.

Quoted 7 lines

Interface:
- two new entries "writeback" and "filedirty" are added to the file memory.stat, to export to userspace respectively the number of pages under writeback and the number of dirty file pages in the cgroup

- the new file memory.dirty_ratio is added in the cgroup filesystem to show/set the memcg dirty_ratio

Seems like a desirable objective.

Quoted 8 lines

[ This patch is still experimental and I only did few quick tests. I'd like to do run more detailed benchmarks and compare the results, I guess the overhead introduced by this patch shouldn't be so small... and BTW I would prefer a dirty limit in bytes, intead of using a percentage of memory. Bytes are hugely more flexible IMHO, they allow to define more fine-grained limits and so this would work better on large memory machines. ]

[1] http://lkml.org/lkml/2008/9/9/245

I tend to duck experimental and rfc patches ;)

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

So after thinking about it a bit I encouraged him to propose a patch which adds a new /proc/sys/vm/hires-dirty-ratio (for some value of "hires" ;)) which simply offers a higher-resolution interface to the same internal kernel machinery.

How does this affect you? I don't think we should be adding new interfaces which have the old 1%-resolution problem. Once we get this higher-resolution interface sorted out, your new interface should do it the same way.

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

Quoted 12 lines · On Fri, 12 Sep 2008 17:09:50 +0200 Andrea Righi <[email protected]> wrote:

The goal of the patch is to control how much dirty file pages a cgroup can have at any given time (see also [1]).

Dirty file and writeback pages are accounted for each cgroup using the memory controller statistics. Moreover, the dirty_ratio parameter is added to the memory controller. It contains, as a percentage of the cgroup memory, the number of dirty pages at which the processes belonging to the cgroup which are generating disk writes will start writing out dirty data.

So, the behaviour is actually the same as the global dirty_ratio, except that it works per cgroup.

Quoted 7 lines

Interface:
- two new entries "writeback" and "filedirty" are added to the file memory.stat, to export to userspace respectively the number of pages under writeback and the number of dirty file pages in the cgroup

- the new file memory.dirty_ratio is added in the cgroup filesystem to show/set the memcg dirty_ratio

Seems like a desirable objective.

Quoted 8 lines

[ This patch is still experimental and I only did few quick tests. I'd like to do run more detailed benchmarks and compare the results, I guess the overhead introduced by this patch shouldn't be so small... and BTW I would prefer a dirty limit in bytes, intead of using a percentage of memory. Bytes are hugely more flexible IMHO, they allow to define more fine-grained limits and so this would work better on large memory machines. ]

[1] http://lkml.org/lkml/2008/9/9/245

I tend to duck experimental and rfc patches ;)

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

So after thinking about it a bit I encouraged him to propose a patch which adds a new /proc/sys/vm/hires-dirty-ratio (for some value of "hires" ;)) which simply offers a higher-resolution interface to the same internal kernel machinery.

How does this affect you? I don't think we should be adding new interfaces which have the old 1%-resolution problem. Once we get this higher-resolution interface sorted out, your new interface should do it the same way.

BTW why not use a simple dirty-ratio-in-bytes?

Quoted 1 line · On Sat, 13 Sep 2008 01:04:35 +0200 Andrea Righi…

BTW why not use a simple dirty-ratio-in-bytes?

s/ratio/amount/ ;)

No particular reason - I haven't really thought about it frankly.

A "ratio" might make more sense in a containerised setup, particularly if the container can be resized on the fly.

Quoted 1 line · On Sat, 13 Sep 2008 01:04:35 +0200 Andrea Righi <[email protected]> wrote:

BTW why not use a simple dirty-ratio-in-bytes?

s/ratio/amount/ ;)

No particular reason - I haven't really thought about it frankly.

A "ratio" might make more sense in a containerised setup, particularly if the container can be resized on the fly.

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

Quoted 5 lines · Andrew Morton

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

Re-sending since I top-posted before. Never again. Also adding more thoughts on a byte based interface.

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

The same for the background_ratio.

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Any feedback?

mrubin

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

The same for the background_ratio.

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Any feedback?

Quoted 5 lines · Andrew Morton

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

Re-sending since I top-posted before. Never again. Also adding more thoughts on a byte based interface.

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

The same for the background_ratio.

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Quoted 5 lines · Andrew Morton

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

Re-sending since I top-posted before. Never again. Also adding more thoughts on a byte based interface.

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

The same for the background_ratio.

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Any feedback?

mrubin

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

The same for the background_ratio.

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Any feedback?

Why vm_dirty_ratio = 0.125 is wrong?
it is hardly for parser maker, but it have nicer user experience.

We don't have any motivation of its interface change.

Quoted 12 lines

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

Quoted 5 lines

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

The same for the background_ratio.

Why vm_dirty_ratio = 0.125 is wrong?
it is hardly for parser maker, but it have nicer user experience.

Quoted 5 lines

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Quoted 1 line

Any feedback?

Sure.
We don't have any motivation of its interface change.

Why vm_dirty_ratio = 0.125 is wrong?
it is hardly for parser maker, but it have nicer user experience.

We don't have any motivation of its interface change.

Quoted 1 line

We don't have any motivation of its interface change.

We are seeing problems where we are generating a lot of dirty memory from asynchronous background writes while more important traffic is operating with DIRECT_IO. The DIRECT_IO traffic will incur high latency spikes as the pdflush hits the background threshold and tries to write a lot of dirty buffers at once.

What we want to do is lower the background threshold low enough so that we don't end up writing a lot of data at one time. As systems get more and more memory this is and will become difficult. 1% of system RAM could tie up a disk.

Quoted 1 line

We don't have any motivation of its interface change.

We are seeing problems where we are generating a lot of dirty memory from asynchronous background writes while more important traffic is operating with DIRECT_IO. The DIRECT_IO traffic will incur high latency spikes as the pdflush hits the background threshold and tries to write a lot of dirty buffers at once.

What we want to do is lower the background threshold low enough so that we don't end up writing a lot of data at one time. As systems get more and more memory this is and will become difficult. 1% of system RAM could tie up a disk.

Quoted 12 lines

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

Quoted 5 lines

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

The same for the background_ratio.

Why vm_dirty_ratio = 0.125 is wrong?
it is hardly for parser maker, but it have nicer user experience.

Quoted 5 lines

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Quoted 1 line

Any feedback?

Sure.
We don't have any motivation of its interface change.

Quoted 5 lines · Andrew Morton

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

Re-sending since I top-posted before. Never again. Also adding more thoughts on a byte based interface.

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

The same for the background_ratio.

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Quoted 26 lines · Michael Rubin

Quoted 5 lines · Andrew Morton

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

Re-sending since I top-posted before. Never again. Also adding more thoughts on a byte based interface.

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

Quoted 7 lines

The same for the background_ratio.

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Just provide a vm_dirty_ration_in_bytes interface and keep it in sync with vm_dirty_ratio (they are just two representations of the same internal value) and for higher resolution propose that users use the bytes interface.

Quoted 26 lines · Michael Rubin

Quoted 5 lines · Andrew Morton

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

Re-sending since I top-posted before. Never again. Also adding more thoughts on a byte based interface.

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

Quoted 7 lines

The same for the background_ratio.

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Just provide a vm_dirty_ration_in_bytes interface and keep it in sync with vm_dirty_ratio (they are just two representations of the same internal value) and for higher resolution propose that users use the bytes interface.

Quoted 35 lines · Balbir Singh

Quoted 26 lines · Michael Rubin

Quoted 5 lines · Andrew Morton

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

Re-sending since I top-posted before. Never again. Also adding more thoughts on a byte based interface.

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

Quoted 7 lines

The same for the background_ratio.

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Quoted 3 lines

Just provide a vm_dirty_ration_in_bytes interface and keep it in sync with vm_dirty_ratio (they are just two representations of the same internal value) and for higher resolution propose that users use the bytes interface.

Hi Balbir,

now that I read carefully the documentation, the description in Documentation/filesystems/proc.txt seems to be a bit misleading. In proc.txt we say that dirty_ratio and dirty_background_ratio are "a percentage of total system memory", but in mm/page-writeback.c we apply the percentages to the dirtyable memory: free pages + reclaimable pages. So, first of all I think we should clarify this in the documentation...

Saying that, keeping in sync the vm_dirty_amount_in_bytes according to dirty_ratio_in_percentage is not a trivial task. One is a static value, the other depends on the dirtyable memory in the system. If we want to preserve the same behaviour we should do the following:

dirty_ratio = x => dirty_amount_in_bytes = x * dirtyable_memory / 100

dirty_amount_in_bytes = y => dirty_ratio = y / dirtyable_memory * 100

But anytime the dirtyable memory (or the total memory in the system) changes we should update both values accordingly to preserve the coherency between them (ouch!).

Possible solutions:

1) introduce fine-grained dirty_ratio handling decimals by an opportune parser (disadvantage: this would break the compatibility with all the userspace apps that expect to read an int from vm_dirty_ratio)

2) introduce dirty_ratio + dirty_ratio_millis (disadvantage: can generate unexpected behaviours when something is written to dirty_ratio ignoring the existence of dirty_ratio_millis)

3) introduce dirty_ratio + dirty_amount_in_bytes mutually exclusive, writing to one automatically "disable" the other (disadvantage:
writing to dirty_ratio ignoring dirty_amount_in_bytes can cause unexpected behaviours)

4) introduce dirty_ratio + dirty_amount_in_bytes and change the old behaviour: when something is written to dirty_ratio, dirty_amount_in_bytes is evaluated in function of totalram_pages (or the memcg limit) and then we always use this static value, instead of something that depends on the dirtyable memory - we can easily update dirty_amount_in_bytes also when totalram_pages or the memcg limit changes (disadvantage: change an old - working - behaviour).

5) handle fine-grained dirty_ratio decimals by an opportune parser when writing something to dirty_ratio; export the percentage units via dirty_ratio, and the decimals via dirty_ratio_decimals; writing to dirty_ratio_decimals is not allowed.

I tend to choose 5. The same for dirty_background_ratio.

Quoted 35 lines · Balbir Singh

Quoted 26 lines · Michael Rubin

Quoted 5 lines · Andrew Morton

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

Re-sending since I top-posted before. Never again. Also adding more thoughts on a byte based interface.

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

Quoted 7 lines

The same for the background_ratio.

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Quoted 3 lines

Just provide a vm_dirty_ration_in_bytes interface and keep it in sync with vm_dirty_ratio (they are just two representations of the same internal value) and for higher resolution propose that users use the bytes interface.

Hi Balbir,

now that I read carefully the documentation, the description in Documentation/filesystems/proc.txt seems to be a bit misleading. In proc.txt we say that dirty_ratio and dirty_background_ratio are "a percentage of total system memory", but in mm/page-writeback.c we apply the percentages to the dirtyable memory: free pages + reclaimable pages. So, first of all I think we should clarify this in the documentation...

Saying that, keeping in sync the vm_dirty_amount_in_bytes according to dirty_ratio_in_percentage is not a trivial task. One is a static value, the other depends on the dirtyable memory in the system. If we want to preserve the same behaviour we should do the following:

dirty_ratio = x => dirty_amount_in_bytes = x * dirtyable_memory / 100

dirty_amount_in_bytes = y => dirty_ratio = y / dirtyable_memory * 100

But anytime the dirtyable memory (or the total memory in the system) changes we should update both values accordingly to preserve the coherency between them (ouch!).

Possible solutions:

1) introduce fine-grained dirty_ratio handling decimals by an opportune parser (disadvantage: this would break the compatibility with all the userspace apps that expect to read an int from vm_dirty_ratio)

2) introduce dirty_ratio + dirty_ratio_millis (disadvantage: can generate unexpected behaviours when something is written to dirty_ratio ignoring the existence of dirty_ratio_millis)

3) introduce dirty_ratio + dirty_amount_in_bytes mutually exclusive, writing to one automatically "disable" the other (disadvantage:
writing to dirty_ratio ignoring dirty_amount_in_bytes can cause unexpected behaviours)

4) introduce dirty_ratio + dirty_amount_in_bytes and change the old behaviour: when something is written to dirty_ratio, dirty_amount_in_bytes is evaluated in function of totalram_pages (or the memcg limit) and then we always use this static value, instead of something that depends on the dirtyable memory - we can easily update dirty_amount_in_bytes also when totalram_pages or the memcg limit changes (disadvantage: change an old - working - behaviour).

5) handle fine-grained dirty_ratio decimals by an opportune parser when writing something to dirty_ratio; export the percentage units via dirty_ratio, and the decimals via dirty_ratio_decimals; writing to dirty_ratio_decimals is not allowed.

I tend to choose 5. The same for dirty_background_ratio.

Quoted 52 lines · On Tue, 07 Oct 2008 17:49:49 +0200 Andrea Righi…

Quoted 29 lines · Balbir Singh

Quoted 28 lines · Michael Rubin

Quoted 5 lines · Andrew Morton

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

Re-sending since I top-posted before. Never again. Also adding more thoughts on a byte based interface.

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

The same for the background_ratio.

Quoted 8 lines

Quoted 5 lines

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Just provide a vm_dirty_ration_in_bytes interface and keep it in sync with vm_dirty_ratio (they are just two representations of the same internal value) and for higher resolution propose that users use the bytes interface.

Hi Balbir,

now that I read carefully the documentation, the description in Documentation/filesystems/proc.txt seems to be a bit misleading. In proc.txt we say that dirty_ratio and dirty_background_ratio are "a percentage of total system memory", but in mm/page-writeback.c we apply the percentages to the dirtyable memory: free pages + reclaimable pages. So, first of all I think we should clarify this in the documentation...

Saying that, keeping in sync the vm_dirty_amount_in_bytes according to dirty_ratio_in_percentage is not a trivial task. One is a static value, the other depends on the dirtyable memory in the system. If we want to preserve the same behaviour we should do the following:

Quoted 8 lines

dirty_ratio = x => dirty_amount_in_bytes = x * dirtyable_memory / 100

dirty_amount_in_bytes = y => dirty_ratio = y / dirtyable_memory * 100

But anytime the dirtyable memory (or the total memory in the system) changes we should update both values accordingly to preserve the coherency between them (ouch!).

Quoted 23 lines

Possible solutions:

1) introduce fine-grained dirty_ratio handling decimals by an opportune parser (disadvantage: this would break the compatibility with all the userspace apps that expect to read an int from vm_dirty_ratio)

2) introduce dirty_ratio + dirty_ratio_millis (disadvantage: can generate unexpected behaviours when something is written to dirty_ratio ignoring the existence of dirty_ratio_millis)

3) introduce dirty_ratio + dirty_amount_in_bytes mutually exclusive, writing to one automatically "disable" the other (disadvantage:
writing to dirty_ratio ignoring dirty_amount_in_bytes can cause unexpected behaviours)

4) introduce dirty_ratio + dirty_amount_in_bytes and change the old behaviour: when something is written to dirty_ratio, dirty_amount_in_bytes is evaluated in function of totalram_pages (or the memcg limit) and then we always use this static value, instead of something that depends on the dirtyable memory - we can easily update dirty_amount_in_bytes also when totalram_pages or the memcg limit changes (disadvantage: change an old - working - behaviour).

Quoted 7 lines

5) handle fine-grained dirty_ratio decimals by an opportune parser when writing something to dirty_ratio; export the percentage units via dirty_ratio, and the decimals via dirty_ratio_decimals; writing to dirty_ratio_decimals is not allowed.

I tend to choose 5. The same for dirty_background_ratio.

Hmm... I agree to "5"... like this ?
==
prvoides
- vm.dirty_ratio (1/100)
- vm.dirty_ratio_percentmille(1/100,000, pcm)

and allow
#echo 0.05 > vm/dirty_ratio
#cat vm/dirty_ratio
0
#cat vm/dirty_ratio_percentmille
500
==

Quoted 52 lines · On Tue, 07 Oct 2008 17:49:49 +0200 Andrea Righi <[email protected]> wrote:

Quoted 29 lines · Balbir Singh

Quoted 28 lines · Michael Rubin

Quoted 5 lines · Andrew Morton

One thing to think about please: Michael Rubin is hitting problems with the existing /proc/sys/vm/dirty-ratio. Its present granularity of 1% is just too coarse for really large machines, and as memory-size/disk-speed ratios continue to increase, this will just get worse.

Re-sending since I top-posted before. Never again. Also adding more thoughts on a byte based interface.

Currently the problem we are hitting is that we cannot specify pdflush to have background limits less than 1% of memory. I am currently finishing up a patch right now that adds a dirty_ratio_millis interface. I hope to submit the patch to LKML by the end of the week.

The idea is that we don't want to break backwards compatibility and we also don't want to have two conflicting knobs in the sysctl or /proc/sys/vm/ space. I thought adding a new knob for those who want to specify finer grained functionality was a compromise. So the patch has a vm_dirty_ratio and a vm_dirty_ratio_millis interface. The first to specify 0-100% and the second to specify .0 to .999%.

So to represent 0.125% of RAM we set
vm_dirty_ratio = 0
vm_dirty_ratio_millis = 125

The same for the background_ratio.

Quoted 8 lines

Quoted 5 lines

I would also prefer using a bytes interface but I am not sure how to offer that without either removing the legacy interface of the ratios or by offering a concurrent interface that might be confusing such as when users are looking at the old one and not aware of a new one.

Just provide a vm_dirty_ration_in_bytes interface and keep it in sync with vm_dirty_ratio (they are just two representations of the same internal value) and for higher resolution propose that users use the bytes interface.

Hi Balbir,

now that I read carefully the documentation, the description in Documentation/filesystems/proc.txt seems to be a bit misleading. In proc.txt we say that dirty_ratio and dirty_background_ratio are "a percentage of total system memory", but in mm/page-writeback.c we apply the percentages to the dirtyable memory: free pages + reclaimable pages. So, first of all I think we should clarify this in the documentation...

Saying that, keeping in sync the vm_dirty_amount_in_bytes according to dirty_ratio_in_percentage is not a trivial task. One is a static value, the other depends on the dirtyable memory in the system. If we want to preserve the same behaviour we should do the following:

Quoted 8 lines

dirty_ratio = x => dirty_amount_in_bytes = x * dirtyable_memory / 100

dirty_amount_in_bytes = y => dirty_ratio = y / dirtyable_memory * 100

But anytime the dirtyable memory (or the total memory in the system) changes we should update both values accordingly to preserve the coherency between them (ouch!).

Quoted 23 lines

Possible solutions:

1) introduce fine-grained dirty_ratio handling decimals by an opportune parser (disadvantage: this would break the compatibility with all the userspace apps that expect to read an int from vm_dirty_ratio)

2) introduce dirty_ratio + dirty_ratio_millis (disadvantage: can generate unexpected behaviours when something is written to dirty_ratio ignoring the existence of dirty_ratio_millis)

3) introduce dirty_ratio + dirty_amount_in_bytes mutually exclusive, writing to one automatically "disable" the other (disadvantage:
writing to dirty_ratio ignoring dirty_amount_in_bytes can cause unexpected behaviours)

4) introduce dirty_ratio + dirty_amount_in_bytes and change the old behaviour: when something is written to dirty_ratio, dirty_amount_in_bytes is evaluated in function of totalram_pages (or the memcg limit) and then we always use this static value, instead of something that depends on the dirtyable memory - we can easily update dirty_amount_in_bytes also when totalram_pages or the memcg limit changes (disadvantage: change an old - working - behaviour).

Quoted 7 lines

5) handle fine-grained dirty_ratio decimals by an opportune parser when writing something to dirty_ratio; export the percentage units via dirty_ratio, and the decimals via dirty_ratio_decimals; writing to dirty_ratio_decimals is not allowed.

I tend to choose 5. The same for dirty_background_ratio.

Hmm... I agree to "5"... like this ?
==
prvoides
- vm.dirty_ratio (1/100)
- vm.dirty_ratio_percentmille(1/100,000, pcm)

and allow
#echo 0.05 > vm/dirty_ratio
#cat vm/dirty_ratio
0
#cat vm/dirty_ratio_percentmille
500
==

The current granularity of 5% of dirtyable memory for dirty pages writeback is too coarse for large memory machines and this will get worse as memory-size/disk-speed ratio continues to increase.

These large writebacks can be unpleasant for desktop or latency-sensitive environments, where the time to complete a writeback can be perceived as a lack of responsiveness by the whole system.

So, something to define fine grained settings is needed.

Following there's a similar solution as discussed in [1], but I tried to simplify the things a little bit, in order to provide the same functionality (in particular try to avoid backward compatibility problems) and reduce the amount of code needed to implement an in-kernel parser to handle percentages with decimals digits.

The kernel provides the following parameters:
- dirty_ratio, dirty_background_ratio in percentage
(1 ... 100)
- dirty_ratio_pcm, dirty_background_ratio_pcm in units of percent mille (1 ... 100,000)

Both dirty_ratio and dirty_ratio_pcm refer to the same vm_dirty_ratio variable, only the interface to read/write this value is different. The same is valid for dirty_background_ratio and dirty_background_ratio_pcm.

In this way it's possible to provide a fine grained interface to configure the writeback policy and at the same time preserve the compatibility with the old coarse grained dirty_ratio / dirty_background_ratio users.

Examples:

 # echo 5 > /proc/sys/vm/dirty_ratio
 # cat /proc/sys/vm/dirty_ratio
 5
 # cat /proc/sys/vm/dirty_ratio_pcm
 5000

 # echo 500 > /proc/sys/vm/dirty_ratio_pcm
 # cat /proc/sys/vm/dirty_ratio
 0
 # cat /proc/sys/vm/dirty_ratio_pcm
 500

 # echo 5500 > /proc/sys/vm/dirty_ratio_pcm
 # cat /proc/sys/vm/dirty_ratio
 5
 # cat /proc/sys/vm/dirty_ratio_pcm
 5500

mm/page-writeback.c +2 -2

-int dirty_background_ratio = 5;
+int dirty_background_ratio = 5 * PERCENT_PCM;
 
 /*
  * free highmem will not be subtracted from the total free memory

@@ -77,7 +77,7 @@ int vm_highmem_is_dirtyable;
 /*
  * The generator of dirty data starts writeback at this percentage
  */
-int vm_dirty_ratio = 10;
+int vm_dirty_ratio = 10 * PERCENT_PCM;
 
 /*
  * The interval between `kupdate'-style writebacks, in jiffies

<snip>

Quoted 18 lines

(quoted diff) +2 -2

-int dirty_background_ratio = 5;
+int dirty_background_ratio = 5 * PERCENT_PCM;
 
 /*
  * free highmem will not be subtracted from the total free memory

@@ -77,7 +77,7 @@ int vm_highmem_is_dirtyable;
 /*
  * The generator of dirty data starts writeback at this percentage
  */
-int vm_dirty_ratio = 10;
+int vm_dirty_ratio = 10 * PERCENT_PCM;
 
 /*
  * The interval between `kupdate'-style writebacks, in jiffies

@@ -135,7 +135,8 @@ static int calc_period_shift(void)
 {
 	unsigned long dirty_total;
 

Review static int calc_period_shift(void) · line 142

Quoted 6 lines

(quoted diff) +2 -1

-	dirty_total = (vm_dirty_ratio * determine_dirtyable_memory()) / 100;
+	dirty_total = (vm_dirty_ratio * determine_dirtyable_memory())
+			/ ONE_HUNDRED_PCM;
 	return 2 + ilog2(dirty_total - 1);
 }
 

I wonder...isn't this overflow in 32bit system ?

Quoted 20 lines · KAMEZAWA Hiroyuki

<snip>

Quoted 18 lines

(quoted diff) +2 -2

-int dirty_background_ratio = 5;
+int dirty_background_ratio = 5 * PERCENT_PCM;
 
 /*
  * free highmem will not be subtracted from the total free memory

@@ -77,7 +77,7 @@ int vm_highmem_is_dirtyable;
 /*
  * The generator of dirty data starts writeback at this percentage
  */
-int vm_dirty_ratio = 10;
+int vm_dirty_ratio = 10 * PERCENT_PCM;
 
 /*
  * The interval between `kupdate'-style writebacks, in jiffies

@@ -135,7 +135,8 @@ static int calc_period_shift(void)
 {
 	unsigned long dirty_total;
 

Quoted 7 lines

Review static int calc_period_shift(void) · line 142

Quoted 6 lines

(quoted diff) +2 -1

-	dirty_total = (vm_dirty_ratio * determine_dirtyable_memory()) / 100;
+	dirty_total = (vm_dirty_ratio * determine_dirtyable_memory())
+			/ ONE_HUNDRED_PCM;
 	return 2 + ilog2(dirty_total - 1);
 }
 

I wonder...isn't this overflow in 32bit system ?

Correct! the worst case is (in pages):

4GB = 100,000 * determine_dirtyable_memory()

that means 42950 pages (~168MB) of dirtyable memory is enough to overflow :(. Using an u64 for dirty_total should resolve.

Delta patch is below.

Unfortunately I have all 64-bit machines right now. Maybe tomorrow I'll be able to get a 32-bit box, if someone doesn't test this before.

Thanks!
-Andrea

---
Subject: fix overflow in 32-bit systems using fine-grained dirty_ratio

Signed-off-by Andrea Righi <[email protected]>

Signed-off-by KAMEZAWA Hiroyuki <[email protected]>

---
 mm/page-writeback.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

mm/page-writeback.c +0 -0

index 6bc8c9b..29913e5 100644

The current granularity of 5% of dirtyable memory for dirty pages writeback is too coarse for large memory machines and this will get worse as memory-size/disk-speed ratio continues to increase.

These large writebacks can be unpleasant for desktop or latency-sensitive environments, where the time to complete each writeback can be perceived as a lack of responsiveness by the whole system.

Following there's a similar solution as discussed in [1], but a little bit simplified in order to provide the same functionality (in particular to avoid backward compatibility problems) and reduce the amount of code needed to implement an in-kernel parser to handle percentages with decimals digits.

The kernel provides the following parameters:
- dirty_ratio, dirty_background_ratio in percentage (1 ... 100)
- dirty_ratio_pcm, dirty_background_ratio_pcm in units of percent mille (1 ... 100,000)

Quoted 14 lines · On Mon, 10 Nov 2008 21:58:28 +0100 Andrea Righi <[email protected]> wrote:

The current granularity of 5% of dirtyable memory for dirty pages writeback is too coarse for large memory machines and this will get worse as memory-size/disk-speed ratio continues to increase.

These large writebacks can be unpleasant for desktop or latency-sensitive environments, where the time to complete each writeback can be perceived as a lack of responsiveness by the whole system.

Following there's a similar solution as discussed in [1], but a little bit simplified in order to provide the same functionality (in particular to avoid backward compatibility problems) and reduce the amount of code needed to implement an in-kernel parser to handle percentages with decimals digits.

Quoted 3 lines

The kernel provides the following parameters:
- dirty_ratio, dirty_background_ratio in percentage (1 ... 100)
- dirty_ratio_pcm, dirty_background_ratio_pcm in units of percent mille (1 ... 100,000)

hm, so how long until dirty_ratio_pcm becomes too coarse...

What happened to the idea of specifying these in units of kilobytes?

Quoted 17 lines · Andrew Morton

Quoted 14 lines · On Mon, 10 Nov 2008 21:58:28 +0100 Andrea Righi…

The current granularity of 5% of dirtyable memory for dirty pages writeback is too coarse for large memory machines and this will get worse as memory-size/disk-speed ratio continues to increase.

These large writebacks can be unpleasant for desktop or latency-sensitive environments, where the time to complete each writeback can be perceived as a lack of responsiveness by the whole system.

Following there's a similar solution as discussed in [1], but a little bit simplified in order to provide the same functionality (in particular to avoid backward compatibility problems) and reduce the amount of code needed to implement an in-kernel parser to handle percentages with decimals digits.

Quoted 7 lines

Quoted 3 lines

The kernel provides the following parameters:
- dirty_ratio, dirty_background_ratio in percentage (1 ... 100)
- dirty_ratio_pcm, dirty_background_ratio_pcm in units of percent mille (1 ... 100,000)

hm, so how long until dirty_ratio_pcm becomes too coarse...

What happened to the idea of specifying these in units of kilobytes?

The conclusion was that with units in KB requires much more complexity to keep in sync the old dirty_ratio (and dirty_background_ratio) interface with the new one.

The KB limit is a static value, the other depends on the dirtyable memory. If we want to preserve the same behaviour we should do the following:

- when dirty_ratio changes to x:
dirty_amount_in_bytes = x * dirtyable_memory / 100.

- when dirty_amount_in_bytes changes to x:
dirty_ratio = x / dirtyable_memory * 100

But anytime the dirtyable memory changes (as well as the total memory in the system) we should update both values accordingly to preserve the coherency between them.

I wonder if setting also PERCENT_PCM (that is 1% expressed in fine-grained units) as a parameter could be a better long-term solution. And also use another name for it, because in this case this would be not a milli-percent value anymore.

The KB limit is a static value, the other depends on the dirtyable memory. If we want to preserve the same behaviour we should do the following:

- when dirty_ratio changes to x:
dirty_amount_in_bytes = x * dirtyable_memory / 100.

- when dirty_amount_in_bytes changes to x:
dirty_ratio = x / dirtyable_memory * 100

But anytime the dirtyable memory changes (as well as the total memory in the system) we should update both values accordingly to preserve the coherency between them.