RFC: dirty_ratio back to 40%

PATCH Larry Woodman 9 participants 13 messages May 20, 2010 – Jun 8, 2010

Subsystems (2)

Memory management Page cache

lore

Share & tools

lore

Larry Woodman

author

2010-05-20 11:25 UTC #0 lore raw

We've seen multiple performance regressions linked to the lower(20%)
dirty_ratio. When performing enough IO to overwhelm the background flush daemons the percent of dirty pagecache memory quickly climbs to the new/lower dirty_ratio value of 20%. At that point all writing processes are forced to stop and write dirty pagecache pages back to disk. This causes performance regressions in several benchmarks as well as causing a noticeable overall sluggishness. We all know that the dirty_ratio is
an integrity vs performance trade-off but the file system journaling
will cover any devastating effects in the event of a system crash.

Increasing the dirty_ratio to 40% will regain the performance loss seen in several benchmarks. Whats everyone think about this???

------------------------------------------------------------------------

mm/page-writeback.c +0 -0

index ef27e73..645a462 100644
@@ -78,7 +78,7 @@ int vm_highmem_is_dirtyable;

/*
* The generator of dirty data starts writeback at this percentage */
-int vm_dirty_ratio = 20;
+int vm_dirty_ratio = 40;

/*
* vm_dirty_bytes starts at 0 (disabled) so that it is a function of

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Heinz Diehl

2010-05-20 12:29 UTC #1 lore raw

Quoted 2 lines · Larry Woodman

Increasing the dirty_ratio to 40% will regain the performance loss seen in several benchmarks. Whats everyone think about this???

These are tuneable via sysctl. What I have in my /etc/sysctl.conf is

vm.dirty_ratio = 4
vm.dirty_background_ratio = 2

This writes back the data more often and frequently, thus preventing the system from long stalls.

Works at least for me. AMD Quadcore, 8 GB RAM.

Richard Kennedy

2010-05-20 13:47 UTC #2 lore raw

Quoted 7 lines · Heinz Diehl

On 20.05.2010, Larry Woodman wrote:
[email protected]
Quoted 2 lines
Increasing the dirty_ratio to 40% will regain the performance loss seen in several benchmarks. Whats everyone think about this???
These are tuneable via sysctl. What I have in my /etc/sysctl.conf is

Quoted 8 lines

vm.dirty_ratio = 4
vm.dirty_background_ratio = 2
This writes back the data more often and frequently, thus preventing the system from long stalls.
Works at least for me. AMD Quadcore, 8 GB RAM.

get_dirty_limits uses a minimum vm_dirty_ratio of 5, so you can't set it lower than that (unless you use vm_dirty_bytes).
But it's interesting that you find lowering the dirty_ratio helpful. Do you have any benchmark results you can share?
regards
Richard

Heinz Diehl

2010-05-20 17:12 UTC #3 lore raw

Quoted 2 lines · Richard Kennedy

But it's interesting that you find lowering the dirty_ratio helpful. Do you have any benchmark results you can share?

No, unfortunately not. My machine is mainly designed to run squid, and it suffered from very long stalls. After posting the problem, I was advised to read about vm.dirty_ratio and vm.dirty_background_ratio, and lower these. I tried several combinations, and with 4/2, the stalls vanished completely.

KOSAKI Motohiro

Acked-by

2010-05-20 23:49 UTC #4 lore raw

In reply to Larry Woodman

CC to Nick and Jan

Quoted 10 lines

We've seen multiple performance regressions linked to the lower(20%) dirty_ratio. When performing enough IO to overwhelm the background flush daemons the percent of dirty pagecache memory quickly climbs to the new/lower dirty_ratio value of 20%. At that point all writing processes are forced to stop and write dirty pagecache pages back to disk. This causes performance regressions in several benchmarks as well as causing a noticeable overall sluggishness. We all know that the dirty_ratio is an integrity vs performance trade-off but the file system journaling will cover any devastating effects in the event of a system crash.

Quoted 2 lines

Increasing the dirty_ratio to 40% will regain the performance loss seen in several benchmarks. Whats everyone think about this???

In past, Jan Kara also claim the exactly same thing.

Subject: [LSF/VM TOPIC] Dynamic sizing of dirty_limit
Date: Wed, 24 Feb 2010 15:34:42 +0100

> (*) We ended up increasing dirty_limit in SLES 11 to 40% as it used to be > with old kernels because customers running e.g. LDAP (using BerkelyDB > heavily) were complaining about performance problems.

So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly same distro specific patch. because we can easily imazine other users will face the same issue in the future.

Acked-by KOSAKI Motohiro <[email protected]>

Nick, Jan, if the above is too old and your distro have been dropped the patch, please correct me.

My motivation is, distro specific patches should keep minimum as far as possible. It exactly help I and other MM developers handle MM bug report.

Thanks.

Quoted 17 lines

------------------------------------------------------------------------

mm/page-writeback.c +1 -2

index ef27e73..645a462 100644
@@ -78,7 +78,7 @@ int vm_highmem_is_dirtyable;
 /*
  * The generator of dirty data starts writeback at this percentage
  */
-int vm_dirty_ratio = 20;
+int vm_dirty_ratio = 40;
 
 /*
  * vm_dirty_bytes starts at 0 (disabled) so that it is a function of
 
--

Quoted 4 lines

To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Zan Lynx

2010-05-21 00:49 UTC #5 lore raw

Quoted 22 lines · KOSAKI Motohiro

Hi
CC to Nick and Jan
Quoted 10 lines
We've seen multiple performance regressions linked to the lower(20%) dirty_ratio. When performing enough IO to overwhelm the background flush daemons the percent of dirty pagecache memory quickly climbs to the new/lower dirty_ratio value of 20%. At that point all writing processes are forced to stop and write dirty pagecache pages back to disk. This causes performance regressions in several benchmarks as well as causing a noticeable overall sluggishness. We all know that the dirty_ratio is an integrity vs performance trade-off but the file system journaling will cover any devastating effects in the event of a system crash.
Quoted 2 lines
Increasing the dirty_ratio to 40% will regain the performance loss seen in several benchmarks. Whats everyone think about this???
In past, Jan Kara also claim the exactly same thing.
Subject: [LSF/VM TOPIC] Dynamic sizing of dirty_limit
Date: Wed, 24 Feb 2010 15:34:42 +0100

Quoted 7 lines

> (*) We ended up increasing dirty_limit in SLES 11 to 40% as it used to be > with old kernels because customers running e.g. LDAP (using BerkelyDB > heavily) were complaining about performance problems.
So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly same distro specific patch. because we can easily imazine other users will face the same issue in the future.

On desktop systems the low dirty limits help maintain interactive feel. Users expect applications that are saving data to be slow. They do not like it when every application in the system randomly comes to a halt because of one program stuffing data up to the dirty limit.

The cause and effect for the system slowdown is clear when the dirty limit is low. "I saved data and now the system is slow until it is done." When the dirty page ratio is very high, the cause and effect is disconnected. "I was just web surfing and the system came to a halt."

I think we should expect server admins to do more tuning than desktop users, so the default limits should stay low in my opinion.

Signature · 4 lines

Zan Lynx
[email protected]

"Knowledge is Power.  Power Corrupts.  Study Hard.  Be Evil."

KOSAKI Motohiro

2010-05-21 01:12 UTC #6 lore raw

Quoted 8 lines

Quoted 3 lines
So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly same distro specific patch. because we can easily imazine other users will face the same issue in the future.
On desktop systems the low dirty limits help maintain interactive feel. Users expect applications that are saving data to be slow. They do not like it when every application in the system randomly comes to a halt because of one program stuffing data up to the dirty limit.

really?
Do you mean our per-task dirty limit wouldn't works?

If so, I think we need fix it. IOW sane per-task dirty limitation seems independent issue from per-system dirty limit.

Quoted 7 lines

The cause and effect for the system slowdown is clear when the dirty limit is low. "I saved data and now the system is slow until it is done." When the dirty page ratio is very high, the cause and effect is disconnected. "I was just web surfing and the system came to a halt."
I think we should expect server admins to do more tuning than desktop users, so the default limits should stay low in my opinion.

Jan Kara

maintainer

2010-05-21 16:00 UTC #7 lore raw

Quoted 9 lines · KOSAKI Motohiro

Quoted 8 lines
Quoted 3 lines
So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly same distro specific patch. because we can easily imazine other users will face the same issue in the future.
On desktop systems the low dirty limits help maintain interactive feel. Users expect applications that are saving data to be slow. They do not like it when every application in the system randomly comes to a halt because of one program stuffing data up to the dirty limit.

Quoted 5 lines

really?
Do you mean our per-task dirty limit wouldn't works?
If so, I think we need fix it. IOW sane per-task dirty limitation seems independent issue from per-system dirty limit.

Well, I don't know about any per-task dirty limits. What function
implements them? Any application that dirties a single page can be caught and forced to call balance_dirty_pages() and do writeback.
But generally what we observe on a desktop with lots of dirty memory is
that application needs to allocate memory (either private one or for page cache) and that triggers direct reclaim so the allocation takes a long time to finish and thus the application is sluggish...

Honza

Signature · 8 lines

Jan Kara <[email protected]>
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Ric Wheeler

2010-05-24 19:51 UTC #8 lore raw

In reply to Zan Lynx

Quoted 32 lines · Zan Lynx

Quoted 22 lines · KOSAKI Motohiro
Hi
CC to Nick and Jan
Quoted 10 lines
We've seen multiple performance regressions linked to the lower(20%) dirty_ratio. When performing enough IO to overwhelm the background flush daemons the percent of dirty pagecache memory quickly climbs to the new/lower dirty_ratio value of 20%. At that point all writing processes are forced to stop and write dirty pagecache pages back to disk. This causes performance regressions in several benchmarks as well as causing a noticeable overall sluggishness. We all know that the dirty_ratio is an integrity vs performance trade-off but the file system journaling will cover any devastating effects in the event of a system crash.
Quoted 2 lines
Increasing the dirty_ratio to 40% will regain the performance loss seen in several benchmarks. Whats everyone think about this???
In past, Jan Kara also claim the exactly same thing.
Subject: [LSF/VM TOPIC] Dynamic sizing of dirty_limit
Date: Wed, 24 Feb 2010 15:34:42 +0100
Quoted 8 lines
Quoted 1 line
(*) We ended up increasing dirty_limit in SLES 11 to 40% as it
used to be
Quoted 2 lines
with old kernels because customers running e.g. LDAP (using BerkelyDB heavily) were complaining about performance problems.
So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly same distro specific patch. because we can easily imazine other users will face the same issue in the future.

Quoted 6 lines

On desktop systems the low dirty limits help maintain interactive feel. Users expect applications that are saving data to be slow. They do not like it when every application in the system randomly comes to a halt because of one program stuffing data up to the dirty limit.
The cause and effect for the system slowdown is clear when the dirty limit is low. "I saved data and now the system is slow until it is done." When the dirty page ratio is very high, the cause and effect is disconnected. "I was just web surfing and the system came to a halt."
I think we should expect server admins to do more tuning than desktop users, so the default limits should stay low in my opinion.

Have you done any performance testing that shows this?

A laptop the smaller default would spin up drives more often and greatly decrease your battery life.

Note that both SLES and RHEL default away from the upstream default.

Ric

Jan Kara

maintainer

2010-05-21 15:51 UTC #9 lore raw

In reply to KOSAKI Motohiro

Hi,

Quoted 1 line · KOSAKI Motohiro

CC to Nick and Jan

Thanks.

Quoted 22 lines

Quoted 10 lines
We've seen multiple performance regressions linked to the lower(20%) dirty_ratio. When performing enough IO to overwhelm the background flush daemons the percent of dirty pagecache memory quickly climbs to the new/lower dirty_ratio value of 20%. At that point all writing processes are forced to stop and write dirty pagecache pages back to disk. This causes performance regressions in several benchmarks as well as causing a noticeable overall sluggishness. We all know that the dirty_ratio is an integrity vs performance trade-off but the file system journaling will cover any devastating effects in the event of a system crash.
Quoted 2 lines
Increasing the dirty_ratio to 40% will regain the performance loss seen in several benchmarks. Whats everyone think about this???
In past, Jan Kara also claim the exactly same thing.
Subject: [LSF/VM TOPIC] Dynamic sizing of dirty_limit
Date: Wed, 24 Feb 2010 15:34:42 +0100
> (*) We ended up increasing dirty_limit in SLES 11 to 40% as it used to be > with old kernels because customers running e.g. LDAP (using BerkelyDB > heavily) were complaining about performance problems.

Quoted 8 lines

So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly same distro specific patch. because we can easily imazine other users will face the same issue in the future.
Acked-by KOSAKI Motohiro <[email protected]>
Nick, Jan, if the above is too old and your distro have been dropped the patch, please correct me.

No, SLE11 SP1 still has a patch that increases dirty_ratio to 40. But on
the other hand I agree with Zan that for desktop, 40% of memory for dirty data is a lot these days and takes a long time to write out (it could easily be 30s - 1m). On a desktop the memory is much better used as a read-only pagecache or for memory hungry apps like Firefox or Acrobat Reader. So I believe for a desktop the current setting (20) is a better choice. So until we find a way how to dynamically size the dirty limit, we have to decide whether we want to have a default setting for a server or for a desktop... Personally, I don't care very much and I feel my time would be better spent thinking about dynamic limit sizing rather than arguing what is better default ;).

Honza

Signature · 8 lines

Jan Kara <[email protected]>
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

David Miller

Acked-by

2010-05-21 06:17 UTC #10 lore raw

In reply to Larry Woodman

From: Larry Woodman <[email protected]>
Date: Thu, 20 May 2010 07:20:42 -0400

Quoted 2 lines

Increasing the dirty_ratio to 40% will regain the performance loss seen in several benchmarks. Whats everyone think about this???

I've been making this change via sysctl on every single system I have, and have been doing so for quite some time.

When doing a lot of GIT operations to a non-SSD disk the kernel simply can't submit the writes early enough to prevent everything getting backlogged, and then processes pile up being forced to sleep on I/O for several seconds at a time.

I therefore totally support making this the default, but I know some people will be against it :-)

Acked-by David S. Miller <[email protected]>

Christoph Hellwig

2010-06-08 18:49 UTC #11 lore raw

In reply to Larry Woodman

Did this patch get merged somewhere?

Quoted 35 lines · Larry Woodman

We've seen multiple performance regressions linked to the lower(20%) dirty_ratio. When performing enough IO to overwhelm the background flush daemons the percent of dirty pagecache memory quickly climbs to the new/lower dirty_ratio value of 20%. At that point all writing processes are forced to stop and write dirty pagecache pages back to disk. This causes performance regressions in several benchmarks as well as causing
a noticeable overall sluggishness. We all know that the dirty_ratio is an integrity vs performance trade-off but the file system journaling will cover any devastating effects in the event of a system crash.
Increasing the dirty_ratio to 40% will regain the performance loss seen in several benchmarks. Whats everyone think about this???
------------------------------------------------------------------------
mm/page-writeback.c +1 -2
index ef27e73..645a462 100644
@@ -78,7 +78,7 @@ int vm_highmem_is_dirtyable;
 /*
 * The generator of dirty data starts writeback at this percentage
 */
-int vm_dirty_ratio = 20;
+int vm_dirty_ratio = 40;
 
 /*
 * vm_dirty_bytes starts at 0 (disabled) so that it is a function of
 
--

Quoted 4 lines

To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

---end quoted text---

Larry Woodman

author

2010-06-08 18:56 UTC #12 lore raw

Quoted 1 line · Christoph Hellwig

Did this patch get merged somewhere?

I dont think it ever did, about 1/2 of responses were for it and the other 1/2 against it.

Larry

Quoted 43 lines · Larry Woodman

Quoted 35 lines
We've seen multiple performance regressions linked to the lower(20%) dirty_ratio. When performing enough IO to overwhelm the background flush daemons the percent of dirty pagecache memory quickly climbs to the new/lower dirty_ratio value of 20%. At that point all writing processes are forced to stop and write dirty pagecache pages back to disk. This causes performance regressions in several benchmarks as well as causing
a noticeable overall sluggishness. We all know that the dirty_ratio is an integrity vs performance trade-off but the file system journaling will cover any devastating effects in the event of a system crash.
Increasing the dirty_ratio to 40% will regain the performance loss seen in several benchmarks. Whats everyone think about this???
------------------------------------------------------------------------
mm/page-writeback.c +1 -2
index ef27e73..645a462 100644
@@ -78,7 +78,7 @@ int vm_highmem_is_dirtyable;
 /*
 * The generator of dirty data starts writeback at this percentage
 */
-int vm_dirty_ratio = 20;
+int vm_dirty_ratio = 40;
 
 /*
 * vm_dirty_bytes starts at 0 (disabled) so that it is a function of
 
--
Quoted 4 lines
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
---end quoted text---

Quoted 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>