VM balancing problems under 2.4.2-ac1
Hiya.
The VM balancing updates in the recent ac kernels seem to have caused some interesting performance problems on my desktop machine. I've got 160Mb of RAM, and 2.4.2-ac1 appears to be using excessively large amounts of it for buffers and cache while pushing stuff out to swap. This means that Mozilla, for instance, runs significantly worse than under 2.4.0, since bits of it are being swapped in and out.
After the machine had been sitting for a while not doing very much:
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us sy id 1 0 0 97184 2116 12844 111768 5 6 15 11 154 791
29 4 67
After some heavy reiserfs disk IO (deleting lots of small files):
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us sy id 1 0 0 102620 1796 85836 43880 100 0 25 0 190 587
12 3 85
Signature · 2 lines
Adam Sampson [email protected]
Quoted 6 lines · Adam Sampson
The VM balancing updates in the recent ac kernels seem to have caused some interesting performance problems on my desktop machine. I've got 160Mb of RAM, and 2.4.2-ac1 appears to be using excessively large amounts of it for buffers and cache while pushing stuff out to swap. This means that Mozilla, for instance, runs significantly worse than under 2.4.0, since bits of it are being swapped in and out.
This is a known problem which I'll fix as soon as I have a solution.
The problem is that we still have no good way to balance how much memory we take from the cache and how much memory we take from processes.
This means that for some workloads we'll be evicting too much cache while for other workloads we'll be evicting too much process pages...
If anybody as a good idea to make this code auto-balancing, please let me know.
regards,
Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
http://www.surriel.com/
Quoted 31 lines · Rik van Riel
Quoted 6 lines · Adam Sampson
The VM balancing updates in the recent ac kernels seem to have caused some interesting performance problems on my desktop machine. I've got 160Mb of RAM, and 2.4.2-ac1 appears to be using excessively large amounts of it for buffers and cache while pushing stuff out to swap. This means that Mozilla, for instance, runs significantly worse than under 2.4.0, since bits of it are being swapped in and out.
This is a known problem which I'll fix as soon as I have a solution.
The problem is that we still have no good way to balance how much memory we take from the cache and how much memory we take from processes.
This means that for some workloads we'll be evicting too much cache while for other workloads we'll be evicting too much process pages...
If anybody as a good idea to make this code auto-balancing, please let me know.
regards,
Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
Quoted 8 lines
http://www.surriel.com/http://www.conectiva.com/ http://distro.conectiva.com.br/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Hi Rik,
I understand that auto-balancing code that deals with all situations is very hard to design; so let me share my experience on other Unix systems (from a user/administrator point of view):
I have used Unix systems (mainly HPUX) for several years as personal workstations or servers and buffer cache usage were very differents:
On workstations, you are mainly looking for fast interactive response time and you want to dedicate as much memory as possible to running processes so limiting buffer cache to 10% of physical memory (these workstations had typically 32 - 64 Mb of RAM) was good.
On file servers, interactive response time is much less important than file/network througput. In this case, having 80% of RAM used for buffer cache is good and you may even want to not let it go below 50% even if it slows down some batch processes.
Both cases were easily handled by 2 HPUX kernel tunable parameters that defined minimum and maximum number of pages that could be used by the buffer cache.
This could be implemented on Linux via /proc. I know it is already done for minimum limit (in 2.2, I have no experience with 2.4 yet). I have found some situations where not being able to force a maximum limit was a problem.
You could argue that with a good load balancing algorithm user defined limits are useless. Believe me, my experience on HPUX workstations showed that lowering its max. limit from 50% (default value) to 10% turned some sluggish machines into speed daemons !
Just my 0.02$
Hope this helps
--
Joseph Bueno
Quoted 21 lines · Joseph Bueno
Quoted 9 lines · Rik van Riel
Quoted 6 lines · Adam Sampson
The VM balancing updates in the recent ac kernels seem to have caused some interesting performance problems on my desktop machine. I've got 160Mb of RAM, and 2.4.2-ac1 appears to be using excessively large amounts of it for buffers and cache while pushing stuff out to swap. This means that Mozilla, for instance, runs significantly worse than under 2.4.0, since bits of it are being swapped in and out.
Quoted 6 lines
This is a known problem which I'll fix as soon as I have a solution.
The problem is that we still have no good way to balance how much memory we take from the cache and how much memory we take from processes.
I understand that auto-balancing code that deals with all situations is very hard to design; so let me share my experience on other Unix systems (from a user/administrator point of view):
Quoted 7 lines
I have used Unix systems (mainly HPUX) for several years as personal workstations or servers and buffer cache usage were very differents:
On workstations, you are mainly looking for fast interactive response time and you want to dedicate as much memory as possible to running processes so limiting buffer cache to 10% of physical memory (these workstations had typically 32 - 64 Mb of RAM) was good.
"Unfortunately" the cache also contains _process memory_ in Linux. Limiting the cache to 10% also means limiting the code size of all your processes to something smaller than that.
Also, read-in swap pages are in the so-called swap cache, which is also part of the page cache.
This means that simple limits on cache size probably won't do much good on Linux.
regards,
Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
http://www.surriel.com/
Quoted 2 lines · Rik van Riel
If anybody as a good idea to make this code auto-balancing, please let me know.
(I haven't seen even one suggestion posted.. must be a real bugger)
I haven't found any silver bullets ;) but the one line bend-adjust below does help the cache problem a little on my wimpy 128mb box.
Worth trying out on other boxen with other loads? It doesn't get swap in/out to a 1:1 ratio, but does improve it considerably by scanning a larger portion of active pages prior to swapout.
(the other two lines are there only because it seemed reasonable;)
-Mike
against 2.4.2-ac4
vmscan.c renamed from vmscan.c.org +3 -1
@@ -278,6 +278,8 @@/* Always start by trying to penalize the process that is allocating memory */ if (mm) retval =swap_out_mm (mm,swap_amount (mm)); +if (retval) +return retval;/* Then, look at the other mm's */ counter = (mmlist_nr <<SWAP_SHIFT ) >> priority;
@@ -846,7 +848,7 @@* continue with its real work sooner. It also helps balancing when we * have multiple processes in try_to_free_pages simultaneously. */ -#define DEF_PRIORITY (6) +#define DEF_PRIORITY (2)static int refill_inactive (unsigned int gfp_mask,int user) {int count, start_count, maxtry;
Quoted 9 lines · Rik van Riel
Quoted 6 lines · Adam Sampson
The VM balancing updates in the recent ac kernels seem to have caused some interesting performance problems on my desktop machine. I've got 160Mb of RAM, and 2.4.2-ac1 appears to be using excessively large amounts of it for buffers and cache while pushing stuff out to swap. This means that Mozilla, for instance, runs significantly worse than under 2.4.0, since bits of it are being swapped in and out.
Quoted 6 lines
This is a known problem which I'll fix as soon as I have a solution.
The problem is that we still have no good way to balance how much memory we take from the cache and how much memory we take from processes.
I have the same problem Adam has: I'm running 3-5 applications on my computer. I have 64 MB of RAM and I use usually less than 50 MB. I have swap for the rare cases where I need more RAM than I have. But with 2.4.x-acyz kernels I do often have to wait several seconds after I switched to another running application before it's swapped in again because it seems this application was swapped out to cache some MP3 I surely won't listen to before the next reboot...
Quoted 6 lines
This means that for some workloads we'll be evicting too much cache while for other workloads we'll be evicting too much process pages...
If anybody as a good idea to make this code auto-balancing, please let me know.
I have no idea for auto-balancing but another idea: It's one possibility to let the user choose when doing "make *config" what he wants:
- A VM optimized for servers that swaps out applications in favor of caching.
or
- A VM optimized for workstations that won't swap out applications in favor of caching.
I know that's not a perfect solution but it would make the situation much better.
Quoted 3 lines
regards,
Rik
cu
Adrian
Signature · 2 lines
Nicht weil die Dinge schwierig sind wagen wir sie nicht, sondern weil wir sie nicht wagen sind sie schwierig.
Quoted 11 lines · Adrian Bunk
Quoted 2 lines
If anybody as a good idea to make this code auto-balancing, please let me know.
I have no idea for auto-balancing but another idea: It's one possibility to let the user choose when doing "make *config" what he wants:
- A VM optimized for servers that swaps out applications in favor of caching.
or
- A VM optimized for workstations that won't swap out applications in favor of caching.
I thought about the same thing sometimes (but for other troughput vs. latency decisions, too).
But I realized, that my very own workstation is also a server, since it runs an httpd, mysqld, smbd, ftpd etc.
And somtimes the servers become very busy in our LAN[1].
IF we want that tuning, we should have it as a sysctl. Most of it is already possible with /proc/sys/vm/*, but balancing decisions are still missing.
And even for servers we need to reduce caching sometimes. Think of an httpd serving _very_ dynamic content. Or any other application (e.g. DMBS), that doesn't rely on file system caching.
A anonymous/file-backed[2] ratio would be VERY handy ;-)
But maybe this will be implemented one day along the lines of QoS in the VM...
Regards
Ingo Oeser
[1] >1500 possible clients for these servers.
[2] Not counting swaps as file backed. We have a special inode for the swapper anyway, right?
Signature · 2 lines
10.+11.03.2001 - 3. Chemnitzer LinuxTag <http://www.tu-chemnitz.de/linux/tag>
<<<<<<<<<<<< come and join the fun >>>>>>>>>>>>Quoted 16 lines · Ingo Oeser
Quoted 11 lines · Adrian Bunk
Quoted 2 lines
If anybody as a good idea to make this code auto-balancing, please let me know.
I have no idea for auto-balancing but another idea: It's one possibility to let the user choose when doing "make *config" what he wants:
- A VM optimized for servers that swaps out applications in favor of caching.
or
- A VM optimized for workstations that won't swap out applications in favor of caching.I thought about the same thing sometimes (but for other troughput vs. latency decisions, too).
Quoted 8 lines
But I realized, that my very own workstation is also a server, since it runs an httpd, mysqld, smbd, ftpd etc.
And somtimes the servers become very busy in our LAN[1].
IF we want that tuning, we should have it as a sysctl. Most of it is already possible with
/proc/sys/vm/*, but balancing decisions are still missing.
I think sysctls for balancing knobs is a great idea. The VM has no clue concerning the cost of rebuilding cache eg but a human may.
Automatic tuning would be wonderful, but it requires information which the VM flat doesn't have.. so it should ask the boss for help.
Three handy knobs I can think of off the top of my head are swap_size, flush_size [for page_launder().. bdflush has that] and cache_stickiness.
-Mike