Page Cache writeback too slow, SSD/noop scheduler/ext2
A message that is not in the archive (sent off-list or not yet received): C5E989F1.C0C7%[email protected]
Hi,
We have hit a problem where the page-cache writeback algorithm is not keeping up.
When memory gets low this will result in very irregular performance drops.
Our setup is as follows:
30 x Quad core machine with 64GB ram.
These are single purpose machines running MySQL. Kernel version: 2.6.28.7
A dedicated SSD drive for the ext2 database partition Noop scheduler for the ssd drive.
The current hypothesis is as follows:
The wk_update function does not write enough dirty pages, which allows the number of dirty pages to grow to the dirty_background limit. When memory is low, Œbackground_writeout() comes around and Œforcefully¹ writes dirty pages to disk.
This forced write fills the disk queue and starves read calls that MySQL is trying to do: basically killing performance for a few seconds. This pattern repeats as soon as the cleared memory is filled again.
Decreasing the dirty_writeback_centisecs to 100 doesn¹t help
I don¹t know why this is, but I did some preliminary tracing using systemtap and it seems that the majority of times wk_update calls decides to do nothing.
Doubling /sys/block/sdb/queue/nr_requests to 256, seems to help abit: the nr_dirty pages is increasing more slowly.
But I am unsure of side-effects and am afraid of increasing the starvation problem for mysql.
I¹am very much willing to work on this issue and see it fixed, but would like to tap into the knowledge of people here.
So:
* Have more people seen this or simular issues?
* Is the hypothesis above a viable one?
* Suggestions/pointers for further research and statistics I should measure to improve the understanding of this problem.
With regards,
Jos
Quoted 35 lines · On Fri, 20 Mar 2009 19:26:06 +0100 Jos Houtman <[email protected]> wrote:
Hi,
We have hit a problem where the page-cache writeback algorithm is not keeping up.
When memory gets low this will result in very irregular performance drops.Our setup is as follows:
30 x Quad core machine with 64GB ram.
These are single purpose machines running MySQL. Kernel version: 2.6.28.7
A dedicated SSD drive for the ext2 database partition Noop scheduler for the ssd drive.The current hypothesis is as follows:
Thewk_updatefunction does not write enough dirty pages, which allows the number of dirty pages to grow to thedirty_backgroundlimit. When memory is low,__background_writeout()comes around and__forcefully__writes dirty pages to disk.
This forced write fills the disk queue and starves read calls that MySQL is trying to do: basically killing performance for a few seconds. This pattern repeats as soon as the cleared memory is filled again.Decreasing the
dirty_writeback_centisecsto 100doesn__thelpI
don__tknow why this is, but I did some preliminary tracing using systemtap and it seems that the majority of timeswk_updatecalls decides to do nothing.Doubling
/sys/block/sdb/queue/nr_requeststo 256, seems to help abit: thenr_dirtypages is increasing more slowly.
But I am unsure of side-effects and am afraid of increasing the starvation problem for mysql.
Quoted 8 lines
I__amvery much willing to work on this issue and see it fixed, but would like to tap into the knowledge of people here.
So:
* Have more people seen this or simular issues?
* Is the hypothesis above a viable one?
* Suggestions/pointers for further research and statistics I should measure to improve the understanding of this problem.
I don't think that noop-iosched tries to do anything to prevent writes-starve-reads. Do you get better behaviour from any of the other IO schedulers?