"Cached" grows and grows and grows...
This is probably closely related to the vm work going on...and you're probably aware of it, but...
The "Cached" field in /proc/meminfo grows and grows and grows. (kernel 2.4.7, 2.4.9, 2.4.10pre4aa1) The kernel seems to be favoring buffer cache for the filesystem over programs. I recently purchased 256MB more memory for my machine, only to find that Linux is using 200-300MB to cache the filesystem. Over time it swaps out everything to disk, and "Cached" grows as large as 415MB on a 512MB machine. Every time I come back to my machine after not using the console for a while, it has to swap everything back into memory in order to be usable. (Note this machine is basically unloaded except for setiathome while I'm away)
Let me place my vote that the vm subsystem should place pages with program code, and their data at a higher priority to be in-memory than filesystem cache buffers. (Is the problem that the filesystem cache buffers never expire, and stay in memory forever?)
Cheers,
-- Bob
Bob McElrath ([email protected]) Univ. of Wisconsin at Madison, Department of Physics
Quoted 12 lines · On Fri, 7 Sep 2001 11:08:36 -0500 Bob McElrath…
This is probably closely related to the vm work going on...and you're probably aware of it, but...
The "Cached" field in
/proc/meminfogrows and grows and grows. (kernel 2.4.7, 2.4.9, 2.4.10pre4aa1) The kernel seems to be favoring buffer cache for the filesystem over programs. I recently purchased 256MB more memory for my machine, only to find that Linux is using 200-300MB to cache the filesystem. Over time it swaps out everything to disk, and "Cached" grows as large as 415MB on a 512MB machine. Every time I come back to my machine after not using the console for a while, it has to swap everything back into memory in order to be usable. (Note this machine is basically unloaded except for setiathome while I'm away)
To tell you the honest truth: you are not alone in cosmos (with this problem) ;-)
To give you that explicit hint for saving money: do not buy mem, it will be eaten up by recent kernels without any performance gain or other positive impact whatsoever.
Try using 2.4.4, if it doesn't succeed, forget 2.4 and use 2.2.19. That works. Unfortunately you may have to completely reinstall your system when going back to 2.2.
Regards,
Stephan
Quoted 5 lines
To tell you the honest truth: you are not alone in cosmos (with this problem) ;-)
To give you that explicit hint for saving money: do not buy mem, it will be eaten up by recent kernels without any performance gain or other positive impact whatsoever.
Pick up a 2.4.9-ac kernel, and you shouldnt be seeing the problem (I say shouldnt, I'm not 100% convinced its all under control)
Quoted 3 lines
Try using 2.4.4, if it doesn't succeed, forget 2.4 and use 2.2.19. That works. Unfortunately you may have to completely reinstall your system when going back to 2.2.
That should not be needed at all.
Quoted 1 line · On Fri, 7 Sep 2001 23:15:36 +0100 (BST) Alan Cox <[email protected]>…
Quoted 1 line
To tell you the honest truth: you are not alone in cosmos (with this
problem)
Quoted 7 lines
Quoted 4 lines
;-)
To give you that explicit hint for saving money: do not buy mem, it will be eaten up by recent kernels without any performance gain or other positive impact whatsoever.Pick up a 2.4.9-ac kernel, and you shouldnt be seeing the problem (I say shouldnt, I'm not 100% convinced its all under control)
VERY FUNNY, Alan!
2.4.9-ac9: __alloc_pages:
/* No luck.. */
// printk(KERN_ERR "__alloc_pages: %lu-order allocation failed.\n", order)
return NULL;If there is no printk, you will obviously not notice the problem. You can bet your car on not "seeing the problem".
Quoted 1 line
Quoted 1 line
Try using 2.4.4, if it doesn't succeed, forget 2.4 and use 2.2.19. That
works.
Quoted 1 line
Quoted 1 line
Unfortunately you may have to completely reinstall your system when going
back
Quoted 3 lines
Quoted 1 line
to 2.2.
That should not be needed at all.
Well, as long as you do not use any features that made you install 2.4 before, e.g. files > 2GB and some others. Of course, if you do not use these, you might be better of with 2.2 anyway.
That was not a very convincing comment, Alan.
But I must admit one thing: 2.4.9-ac9 runs smoother in my test. There are no delays experienced during which the system desperately seeks mem. In fact I can see a lot of inact_clean nearly all the time (a lot means 200-600 MB). Nevertheless there _is_ a problem, because nfs still fails on low mem situation when option "no_subtree_check" is _off_/not used.
I will have some closer looks on ac tree.
Regards,
Stephan
Quoted 5 lines
//
printk(KERN_ERR "__alloc_pages: %lu-order allocation failed.\n", order)return NULL;If there is no printk, you will obviously not notice the problem. You can bet your car on not "seeing the problem".
That printk is commented out because its pointless and bogus noise. It just causes confused bug reporting. The stuff that matters is cache sizes shrinking back when we need memory not slowly eating the computer alive.
Quoted 2 lines
Nevertheless there _is_ a problem, because nfs still fails on low mem situation when option "
no_subtree_check" is _off_/not used.
In certain cases NFS will struggle. It isnt as bad in -ac as we use GFP_KERNEL for nfs level reassembly. It happens in 2.2 as well tho.
Alan
Quoted 1 line · On Sat, 8 Sep 2001 18:14:46 +0100 (BST) Alan Cox <[email protected]>…
Quoted 1 line
// printk(KERN_ERR "
__alloc_pages: %lu-order allocation failed.\n",
order)
Quoted 3 lines
Quoted 3 lines
return NULL;If there is no printk, you will obviously not notice the problem. You can
bet
Quoted 5 lines
Quoted 1 line
your car on not "seeing the problem".
That printk is commented out because its pointless and bogus noise. It just causes confused bug reporting. The stuff that matters is cache sizes shrinking back when we need memory not slowly eating the computer alive.
Hm, I guess I had to find out, that you've the same problem than linus' tree, only on "the other side of the street". This is a meminfo from a 2.4.9-ac9 kernel working one day, and then going crazy on too low mem. "Going crazy" here means that kswapd took virtually over the cpu(s) and swapped the hell out the machine:
(BTW you can see that swap did not really grow, but seems to get in and out permanently)
total: used: free: shared: buffers: cached:
Mem: 921706496 894324736 27381760 4096 783917056 29102080 Swap: 271392768 13185024 258207744 MemTotal: 900104 kB MemFree: 26740 kB MemShared: 4 kB Buffers: 765544 kB Cached: 15744 kB SwapCached: 12676 kB Active: 502132 kB Inact_dirty: 291836 kB Inact_clean: 0 kB Inact_target: 80 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 900104 kB LowFree: 26740 kB SwapTotal: 265032 kB SwapFree: 252156 kB
You are right: page cache shrunk.
You are wrong: it does _not_ work, because now buffers increased and obviously cannot be shrunk to allow "normal" applications to run. I could not even shutdown the machine correctly. It looks like a deadlock in vm to me.
I switched back to Linus' tree, because it does have problems, but is not dead within one day.
Regards,
Stephan
Quoted 6 lines
only on "the other side of the street". This is a meminfo from a 2.4.9-ac9 kernel working one day, and then going crazy on too low mem. "Going crazy" here means that kswapd took virtually over the cpu(s) and swapped the hell out the machine:
(BTW you can see that swap did not really grow, but seems to get in and out permanently)
That buffers count looks like you have something leaking. Probably a file system. When kswapd is busy like that it is trying to free buffer cache. The lack of freeing implies something is still holding those buffer cache entries in memory.
Quoted 4 lines
You are right: page cache shrunk.
You are wrong: it does _not_ work, because now buffers increased and obviously cannot be shrunk to allow "normal" applications to run. I could not even shutdown the machine correctly. It looks like a deadlock in vm to me.
Nope. The VM
Quoted 2 lines
I switched back to Linus' tree, because it does have problems, but is not dead within one day.
Thats good, Linus tree currently lasts about 30 minutes for me. I'd be interested to know what your workload is and what drivers/filesystem you are using so I can look for leaks.
Alan
Quoted 8 lines · Alan Cox [[email protected]] wrote:
Quoted 5 lines
To tell you the honest truth: you are not alone in cosmos (with this problem) ;-)
To give you that explicit hint for saving money: do not buy mem, it will be eaten up by recent kernels without any performance gain or other positive impact whatsoever.Pick up a 2.4.9-ac kernel, and you shouldnt be seeing the problem (I say shouldnt, I'm not 100% convinced its all under control)
Hmmm...that will make my kernel 2.4.10pre4aa1acX-xfs. Ack! ;)
Maybe I can wait 'til 2.4.10. Then it will only be 2.4.10aa1acX-xfs. ;)
Cheers,
-- Bob
Bob McElrath ([email protected]) Univ. of Wisconsin at Madison, Department of Physics
VM improvement in -ac [was: "Cached" grows and grows and grows...]
In reply to Stephan von KrawczynskiQuoted 9 lines · Alan Cox
Quoted 5 lines
To tell you the honest truth: you are not alone in cosmos (with this problem) ;-)
To give you that explicit hint for saving money: do not buy mem, it will be eaten up by recent kernels without any performance gain or other positive impact whatsoever.Pick up a 2.4.9-ac kernel, and you shouldnt be seeing the problem (I say shouldnt, I'm not 100% convinced its all under control)
I have to agree for the most part. My system used to use a lot of cache, and swapped all of the time with 300+MBs of ram on a Linus 2.4 kernel, now with -ac my swap is under 1kB with the same apps and load. I can even run ext3 now without another patch :).
One thing I have noticed is that I still see my system accessing the drive softly for about 30 second time periods a few times a day. I didn't see this on 2.2, and vmstat doesn't show any paging traffic, but it does show about 48 avg pages/sec going out to the disk. I haven't really looked very hard to find the culprit. Has anyone else noticed this?
[OT] Hmm, why isn't ext3 in the Linus kernel yet? It seems more mature than reiserfs was when it was included back in 2.4.1...