Skip to contentSkip to conversation

should I trust 'free' or 'top'?

DISCUSSION Adam McKenna 8 participants 11 messages Feb 1, 2002 – Feb 2, 2002

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

There have been a lot of issues lately on our oracle servers with runaway processes and swapping, enough that I feel compelled to report them here. We have tried all different kernels from 2.4.6 to 2.4.16, and the problem seems to happen with all of them (but is more pronounced on certain kernels)

Basically, what happens is that after an unspecified amount of time, the boxes will become unresponsve and start swapping wildly. At that time, I will login to the box to see what is going on, and I generally see something like this:

adam@xpdb:~$ uptime
 11:21am  up 42 days, 18:53,  3 users,  load average: 54.72, 21.21, 17.60
adam@xpdb:~$ free
             total       used       free     shared    buffers     cached
Mem:       5528464    5522744       5720          0        476    5349784
-/+ buffers/cache:     172484    5355980
Swap:      2939804    1302368    1637436

As you can see, there are supposedly 5.3 gigs of memory free (not counting memory used for cache). However, the box is swapping like mad (about 10 megs every 2 seconds according to vmstat) and the load is skyrocketing.

Now top, on the other hand, has a very different idea about the amount of free memory:

CPU states:  0.0% user,  0.1% system,  0.1% nice,  0.0% idle
Mem:  5528464K av, 5523484K used,   4980K free,      0K shrd,    340K buff
Swap: 2939804K av, 1082008K used, 1857796K free                5351892K

cached

So, what am I supposed to believe? Is 'free' a useful tool? Is it providing accurate results? I'm constantly fielding questions from people who want to know why a box is swapping, even though 'free' reports a whole bunch of memory free, and I'm tired of not having an answer for them.

Thanks,

--Adam

Signature · 2 lines
Adam McKenna <[email protected]>   | GPG: 17A4 11F7 5E7E C2E7 08AA
http://flounder.net/publickey.html |      38B0 05D0 8BF7 2C6D 110A
#1 lore raw
Quoted 3 lines

processes and swapping, enough that I feel compelled to report them here. We have tried all different kernels from 2.4.6 to 2.4.16, and the problem seems to happen with all of them (but is more pronounced on certain kernels)

The only kernels you are likely to see that not happen on are

- The 2.4.9 kernel with Rik's patches that Linus didnt take

	(Red Hat 2.4.9-*)

- 2.4.17/18pre with the rmap11/rmap12 patches
- 2.4.17/18pre with the -aa patched VM
(which I believe is also in the SuSE kernel packages)
- 2.2

The base VM in Linus tree has been broken since before 2.4.0 and while somewhat better is still that - broken. The major vendors don't ship it for a reason.

Alan

#2 lore raw

Review

Quoted 4 lines

The only kernels you are likely to see that not happen on are

(quoted diff) +0 -1
-	The 2.4.9 kernel with Rik's patches that Linus didnt take
	(Red Hat 2.4.9-*)

Review

Quoted 2 lines
(quoted diff) +0 -2
-	2.4.17/18pre with the rmap11/rmap12 patches
-	2.4.17/18pre with the -aa patched VM

(which I believe is also in the SuSE kernel packages)

Quoted 5 lines
(quoted diff) +0 -1
-	2.2

The base VM in Linus tree has been broken since before 2.4.0 and while somewhat better is still that - broken. The major vendors don't ship it for a reason.

Why is this?

Is linus working toward what he believes will be a better impementation? Is he just being stubborn?
I guess I just can't imagine any reason why he would want
large enterprise applications running poorly when there are obvious fixes.

Believe it or not, im not trying to start a flame war, just trying to understand the logic.

--Buddy

#3 lore raw
Quoted 19 lines · Buddy Lumpkin

Review

Quoted 3 lines

The only kernels you are likely to see that not happen on are

(quoted diff) +0 -1
-	The 2.4.9 kernel with Rik's patches that Linus didnt take
	(Red Hat 2.4.9-*)

Review

Quoted 2 lines
(quoted diff) +0 -2
-	2.4.17/18pre with the rmap11/rmap12 patches
-	2.4.17/18pre with the -aa patched VM

(which I believe is also in the SuSE kernel packages)

Quoted 5 lines
(quoted diff) +0 -1
-	2.2

The base VM in Linus tree has been broken since before 2.4.0 and while somewhat better is still that - broken. The major vendors don't ship it for a reason.

Why is this?

Quoted 7 lines

Is linus working toward what he believes will be a better impementation? Is he just being stubborn?
I guess I just can't imagine any reason why he would want
large enterprise applications running poorly when there are obvious fixes.

Believe it or not, im not trying to start a flame war, just trying to understand the logic.

Let's not get into politics. I suggest trying -aa kernels. If that solves your problems, report to lkml, Linus and Marcelo - this will help Andrea patches to get in mainline 2.5/2.4 faster.

As a minimum, you may try attached patch.
--
vda

vmscan.patch.2.4.17.d (author: "M.H.VanLeeuwen" <[email protected]>)
====================================================================

mm/vmscan.c Fri Jan 11 18:03:05 2002 renamed from mm/vmscan.c Mon Dec 31 12:46:25 2001 +12 -6
@@ -394,9 +394,9 @@
                if (PageDirty(page) && is_page_cache_freeable(page) && page->mapping) {
                        /*
                         * It is not critical here to write it only if
-                        * the page is unmapped beause any direct writer
+                        * the page is unmapped because any direct writer
                         * like O_DIRECT would set the PG_dirty bitflag
-                        * on the phisical page after having successfully
+                        * on the physical page after having successfully
                         * pinned it and after the I/O to the page is finished,
                         * so the direct writes to the page cannot get lost.
                         */
@@ -480,11 +480,14 @@
 
                        /*
                         * Alert! We've found too many mapped pages on the
-                        * inactive list, so we start swapping out now!
+                        * inactive list.
+                        * Move referenced pages to the active list.
                         */
-                       spin_unlock(&pagemap_lru_lock);
-                       swap_out(priority, gfp_mask, classzone);
-                       return nr_pages;
+                       if (PageReferenced(page) && !PageLocked(page)) {
+                               del_page_from_inactive_list(page);
+                               add_page_to_active_list(page);
+                       }
+                       continue;
                }
 
                /*
@@ -521,6 +524,9 @@
        }
        spin_unlock(&pagemap_lru_lock);
 
+       if (max_mapped <= 0 && (nr_pages > 0 || priority < DEF_PRIORITY))
+               swap_out(priority, gfp_mask, classzone);
+
        return nr_pages;
 }
 
#4 lore raw
In reply to Buddy Lumpkin
Quoted 5 lines
Quoted 3 lines

The base VM in Linus tree has been broken since before 2.4.0 and while somewhat better is still that - broken. The major vendors don't ship it for a reason.

Why is this?

Linus kept ignoring Rik's patches and making other changes, then at 2.4.10 switched to Andrea's VM and ignored most of the follow up changes that made that one work

Quoted 2 lines

Believe it or not, im not trying to start a flame war, just trying to understand the logic.

You've got me there. I don't understand either.

Alan

#5 lore raw

as an outside observer during that time it looked like Linus was rejecting patches from Rik (at least in part) becouse the reasoning behind them wasn't fully explained and it appeared to be deteriorating into 'tweak these magic numbers to fix problem A, discover that that caused problem B, tweak them again to fix B anc cause C .... tweak them again to fix J and cause A' type circles with nobody (other then possibly rik) understanding what was really causing the problems (at least if they did understand them they weren't posted here)

the fundamental problem was that while the VM would work well most of the time, once in a while it would hit a pathalogical condition that would lockup the machine completely, the new VM was seen as not nessasarily being quite as good in the best cases, but avoiding the worst lockups (of course it had a few problems of it's own, but these seemed to be easier to fix without causing additional problems)

David Lang

Quoted 30 lines · Alan Cox

Date: Sat, 2 Feb 2002 17:11:43 +0000 (GMT)
From: Alan Cox <[email protected]>
To: [email protected]
Cc: [email protected], [email protected],
[email protected]
Subject: Re: should I trust 'free' or 'top'?

Quoted 5 lines
Quoted 3 lines

The base VM in Linus tree has been broken since before 2.4.0 and while somewhat better is still that - broken. The major vendors don't ship it for a reason.

Why is this?

Linus kept ignoring Rik's patches and making other changes, then at 2.4.10 switched to Andrea's VM and ignored most of the follow up changes that made that one work

Quoted 2 lines

Believe it or not, im not trying to start a flame war, just trying to understand the logic.

You've got me there. I don't understand either.

Alan

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#6 lore raw
Quoted 4 lines

tweak them again to fix B anc cause C .... tweak them again to fix J and cause A' type circles with nobody (other then possibly rik) understanding what was really causing the problems (at least if they did understand them they weren't posted here)

Plenty of people understood them. The continual changing was also not helped by the fact that at least three totally contradictory sets of patches were getting applied (Riks, the use once stuff and Linus own changes)

Quoted 6 lines

the fundamental problem was that while the VM would work well most of the time, once in a while it would hit a pathalogical condition that would lockup the machine completely, the new VM was seen as not nessasarily being quite as good in the best cases, but avoiding the worst lockups (of course it had a few problems of it's own, but these seemed to be easier to fix without causing additional problems)

The original Andrea vm was faster on light loads, and even less stable on anything sane. In the -aa patches it does quite well, but those aren't merged either.

Alan

#7 lore raw
In reply to Adam McKenna
Quoted 2 lines · [email protected] ("Adam McKenna") writes:

Now top, on the other hand, has a very different idea about the amount of free memory:

A very different idea? The difference is about 1M if I read it correctly.

Signature · 3 lines
Når folk spørger mig, om jeg er nørd, bliver jeg altid ilde til mode
og svarer lidt undskyldende: "Nej, jeg bruger RedHat".
                                -- Allan Olesen på dk.edb.system.unix
#8 lore raw
In reply to Adam McKenna
Quoted 10 lines · Adam McKenna
adam@xpdb:~$ uptime
 11:21am  up 42 days, 18:53,  3 users,  load average: 54.72, 21.21, 17.60
adam@xpdb:~$ free
             total       used       free     shared    buffers     cached
Mem:       5528464    5522744       5720          0        476    5349784
-/+ buffers/cache:     172484    5355980
Swap:      2939804    1302368    1637436
As you can see, there are supposedly 5.3 gigs of memory free (not counting
memory used for cache).  However, the box is swapping like mad (about 10 megs
every 2 seconds according to vmstat) and the load is skyrocketing.

That 5.3GB is without kernel caches. I see 5.7MB...

Quoted 6 lines · Adam McKenna

Now top, on the other hand, has a very different idea about the amount of free memory:

CPU states:  0.0% user,  0.1% system,  0.1% nice,  0.0% idle
Mem:  5528464K av, 5523484K used,   4980K free,      0K shrd,    340K buff
Swap: 2939804K av, 1082008K used, 1857796K free                5351892K

cached

They actually agree. The line you're reading with 5.3GB in it subtracts kernel caches from the memory in use.

The fun bit about swapping like mad is because kernel caches are not being flushed and shrunk properly in response to growth of the working set. In more concrete terms, the kernel is making decisions which prefer to keep things like the page cache, the dentry cache, the inode cache, and the buffer cache in memory over the working sets of your programs. There is some tradeoff: it is probably also not desirable to allow the working set to erode kernel caches to the absolute minimum (or at least not very easily), but obviously what tradeoffs are happening here are suboptimal for your workload (and generally insufficiently adaptive). It appears that when the kernel caches are done with you you've got 172MB out of 5.5GB of physical memory left for your programs' anonymous memory.

What kernel/VM are you using?
Could you follow up with /proc/slabinfo and /proc/meminfo?

Cheers,
Bill

#9 lore raw
In reply to Adam McKenna
Quoted 37 lines · William Lee Irwin III
Quoted 10 lines · Adam McKenna
adam@xpdb:~$ uptime
 11:21am  up 42 days, 18:53,  3 users,  load average: 54.72, 21.21, 17.60
adam@xpdb:~$ free
             total       used       free     shared    buffers     cached
Mem:       5528464    5522744       5720          0        476    5349784
-/+ buffers/cache:     172484    5355980
Swap:      2939804    1302368    1637436
As you can see, there are supposedly 5.3 gigs of memory free (not counting
memory used for cache).  However, the box is swapping like mad (about 10 megs
every 2 seconds according to vmstat) and the load is skyrocketing.

That 5.3GB is without kernel caches. I see 5.7MB...

Quoted 6 lines · Adam McKenna

Now top, on the other hand, has a very different idea about the amount of free memory:

CPU states:  0.0% user,  0.1% system,  0.1% nice,  0.0% idle
Mem:  5528464K av, 5523484K used,   4980K free,      0K shrd,    340K buff
Swap: 2939804K av, 1082008K used, 1857796K free                5351892K

cached

They actually agree. The line you're reading with 5.3GB in it subtracts kernel caches from the memory in use.

The fun bit about swapping like mad is because kernel caches are not being flushed and shrunk properly in response to growth of the working set. In more concrete terms, the kernel is making decisions which prefer to keep things like the page cache, the dentry cache, the inode cache, and the buffer cache in memory over the working sets of your programs. There is some tradeoff: it is probably also not desirable to allow the working set to erode kernel caches to the absolute minimum (or at least not very easily), but obviously what tradeoffs are happening here are suboptimal for your workload (and generally insufficiently adaptive). It appears that when the kernel caches are done with you you've got 172MB out of 5.5GB of physical memory left for your programs' anonymous memory.

Quoted 1 line

What kernel/VM are you using?

2.4.6-xfs but we've also seen this with 2.4.14-xfs (xfs 1.0.2 release)

Quoted 1 line

Could you follow up with /proc/slabinfo and /proc/meminfo?

We've already rebooted the box, next time we are experiencing the problem I'll send this info.

Meanwhile, is there any way to tune the kernel cache?

--Adam

Signature · 2 lines
Adam McKenna <[email protected]>   | GPG: 17A4 11F7 5E7E C2E7 08AA
http://flounder.net/publickey.html |      38B0 05D0 8BF7 2C6D 110A
#10 lore raw
In reply to Adam McKenna
Quoted 1 line · William Lee Irwin III
Quoted 1 line

What kernel/VM are you using?

Quoted 1 line · Adam McKenna

2.4.6-xfs but we've also seen this with 2.4.14-xfs (xfs 1.0.2 release)

You appear to be in more trouble than I can get you out of. Could you try again with -aa or -rmap against a recent kernel? (mainline VM appears not to behave as well as either of these).

Quoted 1 line · William Lee Irwin III
Quoted 1 line

Could you follow up with /proc/slabinfo and /proc/meminfo?

Quoted 3 lines · Adam McKenna

We've already rebooted the box, next time we are experiencing the problem I'll send this info.
Meanwhile, is there any way to tune the kernel cache?

Kernel hacking. Until you get yourself a stabler VM it probably won't be meaningful to try that directly, though making this tunable would be nice, too. I've heard it's difficult to merge xfs with VM changes due to its invasiveness in the VM, but I've heard of it being done several times.

Cheers,
Bill