VM throttling: avoid blocking occasional writers

Which problem are we trying to solve here? afaik our two uppermost problems are:

a) Heavy write to queue A causes light writer to queue B to blok for a long time in balance_dirty_pages(). Even if the devices have the same speed.

b) heavy write to device A causes light write to device A to block for a long time in balance_dirty_pages(), occasionally. Harder to fix.

...

-Solution:

I consider that all of the dirty pages for the disk have been written back and that the disk is clean if a process cannot write 'write_chunk' pages in balance_dirty_pages().

To avoid using up the free memory with dirty pages by passing blocking, this patchset adds a new threshold named vm.dirty_limit_ratio to sysctl.

It modifies balance_dirty_pages() not to block when the amount of Dirty+Writeback is less than vm.dirty_limit_ratio percent of the memory. In the other cases, writers are throttled as current Linux does.

In this patchset, vm.dirty_limit_ratio, instead of vm.dirty_ratio, is used as the clamping level of Dirty+Writeback. And, vm.dirty_ratio is used as the level at which a writers will itself start writeback of the dirty pages.

Quoted 19 lines · On Wed, 14 Mar 2007 21:42:46 +0900 Tomoki Sekiyama…

...

-Solution:

I consider that all of the dirty pages for the disk have been written back and that the disk is clean if a process cannot write 'write_chunk' pages in balance_dirty_pages().

To avoid using up the free memory with dirty pages by passing blocking, this patchset adds a new threshold named vm.dirty_limit_ratio to sysctl.

It modifies balance_dirty_pages() not to block when the amount of Dirty+Writeback is less than vm.dirty_limit_ratio percent of the memory. In the other cases, writers are throttled as current Linux does.

Quoted 4 lines

In this patchset, vm.dirty_limit_ratio, instead of vm.dirty_ratio, is used as the clamping level of Dirty+Writeback. And, vm.dirty_ratio is used as the level at which a writers will itself start writeback of the dirty pages.

Might be a reasonable solution - let's see what Peter comes up with too.

Comments on the patch:

- Please don't VM_DIRTY_LIMIT_RATIO: just use CTL_UNNUMBERED and leave sysctl.h alone.

- The 40% default is already too high. Let's set this new upper limit to 40% and decrease he non-blocking ratio.

- Please update the procfs documentation in ./Docmentation/

- I wonder if dirty_limit_ratio is the best name we could choose. vm_dirty_blocking_ratio, perhaps? Dunno.

Quoted 29 lines · Andrew Morton

Quoted 20 lines

On Wed, 14 Mar 2007 21:42:46 +0900 Tomoki Sekiyama <[email protected]> wrote:

...

-Solution:

I consider that all of the dirty pages for the disk have been written back and that the disk is clean if a process cannot write 'write_chunk' pages in balance_dirty_pages().

To avoid using up the free memory with dirty pages by passing blocking, this patchset adds a new threshold named vm.dirty_limit_ratio to sysctl.

It modifies balance_dirty_pages() not to block when the amount of Dirty+Writeback is less than vm.dirty_limit_ratio percent of the memory. In the other cases, writers are throttled as current Linux does.

Quoted 4 lines

In this patchset, vm.dirty_limit_ratio, instead of vm.dirty_ratio, is used as the clamping level of Dirty+Writeback. And, vm.dirty_ratio is used as the level at which a writers will itself start writeback of the dirty pages.

Might be a reasonable solution - let's see what Peter comes up with too.

Comments on the patch:

Quoted 7 lines

- Please don't VM_DIRTY_LIMIT_RATIO: just use CTL_UNNUMBERED and leave sysctl.h alone.

- The 40% default is already too high. Let's set this new upper limit to 40% and decrease he non-blocking ratio.

- Please update the procfs documentation in ./Docmentation/

Quoted 3 lines

- I wonder if dirty_limit_ratio is the best name we could choose. vm_dirty_blocking_ratio, perhaps? Dunno.

I don't like it, but I dislike it less than "dirty_limit_ratio" I guess. It would probably break things to change it now, including my sysctl.conf on a number of systems :-(

Hi,

Thanks for your comments.
I'm sorry for my late reply.

Quoted 6 lines · Bill Davidsen

Quoted 3 lines · Andrew Morton

- I wonder if dirty_limit_ratio is the best name we could choose. vm_dirty_blocking_ratio, perhaps? Dunno.

I don't like it, but I dislike it less than "dirty_limit_ratio" I

guess.

Quoted 2 lines

It would probably break things to change it now, including my sysctl.conf on a number of systems :-(

I'm wondering which interface is preferred...

1) Just rename "dirty_limit_ratio" to "dirty_blocking_ratio." Those who had been changing dirty_ratio should additionally modify dirty_blocking_ratio in order to determine the upper limit of dirty
pages.

2) Change "dirty_ratio" to a vector, consists of 2 values; {blocking ratio, writeback starting ratio}. For example, to change the both values:

     # echo 40 35 > /proc/sys/vm/dirty_ratio
   And to change only the first one:
     # echo 20 > /proc/sys/vm/dirty_ratio
   In the latter way the writeback starting ratio is regarded as the 
same as the
   blocking ratio if the writeback starting ratio is smaller. And 
then, the kernel behaves
   similarly as the current kernel.

3) Use "dirty_ratio" as the blocking ratio. And add "start_writeback_ratio", and start writeback at start_writeback_ratio(default:90) * dirty_ratio / 100 [%]. In this way, specifying blocking ratio can be done in the same way as current kernel, but high/low watermark algorithm is enabled.

Quoted 5 lines

3) Use "dirty_ratio" as the blocking ratio. And add "start_writeback_ratio", and start writeback at start_writeback_ratio(default:90) * dirty_ratio / 100 [%]. In this way, specifying blocking ratio can be done in the same way as current kernel, but high/low watermark algorithm is enabled.

I like 3 better, it should make tuning behavior more precise.

You can make an argument for absolute values for writeback, if my disk will only write 70MB/s I may only want 203 sec of pending writes, regardless of available memory.

Hi,
Thanks for your reply.

Quoted 8 lines

Quoted 6 lines

3) Use "dirty_ratio" as the blocking ratio. And add "start_writeback_ratio", and start writeback at start_writeback_ratio(default:90) * dirty_ratio / 100 [%]. In this way, specifying blocking ratio can be done in the same way as current kernel, but high/low watermark algorithm is enabled.

I like 3 better, it should make tuning behavior more precise.

Then, what do you think of the following idea?

(4) add `dirty_start_writeback_ratio' as percentage of memory, at which a generator of dirty pages itself starts writeback (that is, non-blocking ratio).

In this way, dirty_ratio' is used as the blocking ratio, so we don't need to modify the sysctl.conf etc. I think it's easier to understand for administrators of systems, because the interface is similar as dirty_background_ratio' and`dirty_ratio.'

If this is OK, I'll repost the patch.

Quoted 5 lines

You can make an argument for absolute values for writeback, if my disk will only write 70MB/s I may only want 203 sec of pending writes, regardless of available memory.

To realize tuning with absolute values, I consider that we need to modify handling of dirty_background_ratio,' dirty_ratio' and so on as well as `dirty_start_writeback_ratio.' I think this should be done in another patch if this feature is required.

Regards,
--
Tomoki Sekiyama
Hitachi, Ltd., Systems Development Laboratory

Quoted 6 lines

Then, what do you think of the following idea?

(4) add `dirty_start_writeback_ratio' as percentage of memory, at which a generator of dirty pages itself starts writeback (that is, non-blocking ratio).

Quoted 5 lines

In this way, dirty_ratio' is used as the blocking ratio, so we don't need to modify the sysctl.conf etc. I think it's easier to understand for administrators of systems, because the interface is similar as dirty_background_ratio' and`dirty_ratio.'

It sounds good to me, just be sure behavior is sane for for both blocking less than start_writeback and vice versa.

This patchset is to avoid the problem that write(2) can be blocked for a long time if a system has several disks with different speed and is under heavy I/O pressure.

-Description of the problem:
While Dirty+Writeback pages get more than 40%(dirty_ratio') of memory, generators of dirty pages are blocked in balance_dirty_pages() until they start writeback of a specific number (write_chunk', typically=1536) of dirty pages on the disks they write to.

Under this rule, if a process writes to the disk which has only a few (less than 1536) dirty pages, that process will be blocked until writeback of the other disks is completed and % of Dirty+Writeback goes below 40%.

Thus, if a slow device (such as a USB disk) has many dirty pages, the processes which write small data to the other disks can be blocked for quite a long time.

-Solution:
This patch introduces high/low-watermark algorithm in balance_dirty_pages() in order to throttle only the processes which write to disks with heavy load.

This patch adds dirty_start_writeback_ratio' for the low-watermark, and modifies get_dirty_limits() to calculate and return the writeback starting level of dirty pages based on dirty_start_writeback_ratio'.

If % of Dirty+Writeback > `dirty_writeback_start_ratio', generators of dirty pages start writeback of dirty pages by themselves. At that time, these processes are not blocked in balance_dirty_pages(), but they may be blocked if the write-requests-queue of the written disk is full (that is, the length of the queue > `nr_requests'). By this behavior, we can throttle only processes which write to the disks with heavy load, and can allow processes to write to the other disks without blocking.

If % of Dirty+Writeback > `dirty_ratio', generators of dirty pages are throttled as current Linux does, not to fill up memory with dirty pages.

Quoted 11 lines · On Tue, 03 Apr 2007 19:46:04 +0900 Tomoki Sekiyama…

If % of Dirty+Writeback > `dirty_writeback_start_ratio', generators of dirty pages start writeback of dirty pages by themselves. At that time, these processes are not blocked in balance_dirty_pages(), but they may be blocked if the write-requests-queue of the written disk is full (that is, the length of the queue > `nr_requests'). By this behavior, we can throttle only processes which write to the disks with heavy load, and can allow processes to write to the other disks without blocking.

If % of Dirty+Writeback > `dirty_ratio', generators of dirty pages are throttled as current Linux does, not to fill up memory with dirty pages.

Does this actually solve the problem? If the request queue is sufficiently large (relative to the various dirty-memory thresholds) then I'd expect that a heavy-writer will be able to very quickly take the total dirty+writeback memory up to the dirty_ratio (should be renamed throttle_threshold, but it's too late for that).

I suspect the reason why this patch was successful in your testing was because dirty_start_writeback_ratio happens to exceed the size of the disk request queues, so the heavy writer is getting stuck on disk request queue exhaustion.

But that won't work if we have a lot of processes writing to a lot of disks, and it won't work if the request queue size is large, or if the dirty-memory thresholds are small (relative to the request queue size).

Do the patches still work after
`echo 10000 > /sys/block/sda/queue/nr_requests'?

Hello Andrew,
Thank you for your comments.

Quoted 25 lines · Andrew Morton

Quoted 11 lines · On Tue, 03 Apr 2007 19:46:04 +0900 Tomoki Sekiyama…

If % of Dirty+Writeback > `dirty_writeback_start_ratio', generators of dirty pages start writeback of dirty pages by themselves. At that time, these processes are not blocked in balance_dirty_pages(), but they may be blocked if the write-requests-queue of the written disk is full (that is, the length of the queue > `nr_requests'). By this behavior, we can throttle only processes which write to the disks with heavy load, and can allow processes to write to the other disks without blocking.

If % of Dirty+Writeback > `dirty_ratio', generators of dirty pages are throttled as current Linux does, not to fill up memory with dirty pages.

Does this actually solve the problem? If the request queue is sufficiently large (relative to the various dirty-memory thresholds) then I'd expect that a heavy-writer will be able to very quickly take the total dirty+writeback memory up to the dirty_ratio (should be renamed throttle_threshold, but it's too late for that).

I suspect the reason why this patch was successful in your testing was because dirty_start_writeback_ratio happens to exceed the size of the disk request queues, so the heavy writer is getting stuck on disk request queue exhaustion.

Quoted 6 lines

But that won't work if we have a lot of processes writing to a lot of disks, and it won't work if the request queue size is large, or if the dirty-memory thresholds are small (relative to the request queue size).

Do the patches still work after
`echo 10000 > /sys/block/sda/queue/nr_requests'?

As you pointed out, this patch has no effect if nr_requests is too large, because it distinguishes heavy disks depending on the length of the write- requests queue of each disk.

This patch is for providing the system administrators with room to avoid the problem by adjusting parameters appropriately, rather than an automatic solution for any possible situations.

Could you please tell me some situations in which we should set nr_request that large?