Gigantic memory leak in linux-2.6.[789]!
Hi all!
After some more testing after the previous post of the OOPS in generic_delete_inode, we have now found a gigantic memory leak in Linux 2.6. [789]. The scenario is the same:
File system: EXT3
Unpack and delete linux-2.6.8.1.tar.bz2 with this Bash while loop:
let "i = 0"
while [ "$i" -lt 10 ]; do
tar jxf linux-2.6.8.1.tar.bz2;
rm -fr linux-2.6.8.1;
let "i = i + 1"
done
When the loop has completed, the system use 124 MB memory more _each_ time.... so it is pretty easy to make a denial-of-service attack :-(
We have tried the same test on a RHEL WS 3 host (running a RedHat 2.4 kernel)
- and there is no problem.
Any deas?
Signature · 5 lines
Kristian Sørensen - The Umbrella Project http://umbrella.sourceforge.net E-mail: [email protected], Phone: +45 29723816
Quoted 11 lines · Kristian Sørensen
Hi all!
After some more testing after the previous post of the OOPS in
generic_delete_inode, we have now found a gigantic memory leak in Linux 2.6. [789]. The scenario is the same:File system: EXT3
Unpack and delete linux-2.6.8.1.tar.bz2 with this Bash while loop:let "i = 0"
while [ "$i" -lt 10 ]; do
Quoted 7 lines
tar jxf linux-2.6.8.1.tar.bz2; rm -fr linux-2.6.8.1; let "i = i + 1"done
When the loop has completed, the system use 124 MB memory more _each_ time.... so it is pretty easy to make a denial-of-service attack :-(
well.. i could understand if it used the total size of a unpacked linux kernel, even after the loop stopped, since it would just keep it cached, however, it might not be that case when it adds 124mb each time...
Quoted 7 lines
We have tried the same test on a RHEL WS 3 host (running a RedHat 2.4 kernel)
- and there is no problem.Any deas?
Quoted 17 lines · Kasper Sandberg
Quoted 16 lines · Kristian Sørensen
Hi all!
After some more testing after the previous post of the OOPS in
generic_delete_inode, we have now found a gigantic memory leak in Linux 2.6. [789]. The scenario is the same:File system: EXT3
Unpack and delete linux-2.6.8.1.tar.bz2 with this Bash while loop:let "i = 0"
while [ "$i" -lt 10 ]; dotar jxf linux-2.6.8.1.tar.bz2; rm -fr linux-2.6.8.1; let "i = i + 1"done
Quoted 2 lines
Quoted 2 lines
When the loop has completed, the system use 124 MB memory more _each_ time.... so it is pretty easy to make a denial-of-service attack :-(
Do something like this with your favorite kernel version.....
while true ; do tar -xzf linux-2.6.9.tar.gz ; rm -rf linux-2.6.9 ; vmstat ; done
You can watch this for as long as you want. If there is no other activity, the values reported by vmstat remain, on the average, stable. If you throw in a sync command, the values rapidly converge to little memory usage as the disk-data gets flused to disk.
Quoted 8 lines
well.. i could understand if it used the total size of a unpacked linux kernel, even after the loop stopped, since it would just keep it cached, however, it might not be that case when it adds 124mb each time...
Quoted 5 lines
We have tried the same test on a RHEL WS 3 host (running a RedHat 2.4 kernel)
- and there is no problem.
Quoted 9 lines
Quoted 2 lines
Any deas?
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 GrumpyMips).
98.36% of all statistics are fiction.
Quoted 25 lines · Richard B. Johnson
Quoted 18 lines · Kasper Sandberg
Quoted 16 lines · Kristian Sørensen
Hi all!
After some more testing after the previous post of the OOPS in
generic_delete_inode, we have now found a gigantic memory leak in Linux 2.6. [789]. The scenario is the same:File system: EXT3
Unpack and delete linux-2.6.8.1.tar.bz2 with this Bash while loop:let "i = 0"
while [ "$i" -lt 10 ]; dotar jxf linux-2.6.8.1.tar.bz2; rm -fr linux-2.6.8.1; let "i = i + 1"done
Quoted 2 lines
Quoted 2 lines
When the loop has completed, the system use 124 MB memory more _each_ time.... so it is pretty easy to make a denial-of-service attack :-(
Quoted 8 lines
Do something like this with your favorite kernel version.....
while true ; do tar -xzf linux-2.6.9.tar.gz ; rm -rf linux-2.6.9 ; vmstat ; done
You can watch this for as long as you want. If there is no other activity, the values reported by vmstat remain, on the average, stable. If you throw in a
synccommand, the values rapidly converge to little memory usage as the disk-data gets flused to disk.
The problem is, that the free memory reported by vmstat is decresing by 124mb for each 10-iterations....
The allocated memory does not get freed even if the system has been left alone for three hours!
Cheers, Kristian.
Signature · 5 lines
Kristian Sørensen - The Umbrella Project http://umbrella.sourceforge.net E-mail: [email protected], Phone: +45 29723816
Quoted 30 lines · On Fri, 22 Oct 2004, [ISO-8859-15] Kristian Sørensen wrote:
Quoted 29 lines · Richard B. Johnson
Quoted 18 lines · Kasper Sandberg
Quoted 16 lines · Kristian Sørensen
Hi all!
After some more testing after the previous post of the OOPS in
generic_delete_inode, we have now found a gigantic memory leak in Linux 2.6. [789]. The scenario is the same:File system: EXT3
Unpack and delete linux-2.6.8.1.tar.bz2 with this Bash while loop:let "i = 0"
while [ "$i" -lt 10 ]; dotar jxf linux-2.6.8.1.tar.bz2; rm -fr linux-2.6.8.1; let "i = i + 1"done
Quoted 2 lines
Quoted 2 lines
When the loop has completed, the system use 124 MB memory more _each_ time.... so it is pretty easy to make a denial-of-service attack :-(
Do something like this with your favorite kernel version.....
while true ; do tar -xzf linux-2.6.9.tar.gz ; rm -rf linux-2.6.9 ; vmstat ; done
Quoted 8 lines
Quoted 4 lines
You can watch this for as long as you want. If there is no other activity, the values reported by vmstat remain, on the average, stable. If you throw in a
synccommand, the values rapidly converge to little memory usage as the disk-data gets flused to disk.The problem is, that the free memory reported by vmstat is decresing by 124mb for each 10-iterations....
The allocated memory does not get freed even if the system has been left alone for three hours!
Yes. So? Why would it be freed? It's left how it was until it is needed. Freeing it would waste CPU cycles.
This cannot be a problem unless you are inventing some sort of hot-swap memory thing. If so, you need to make a module that tells the kernel memory manager to free everything so you can remove and replace the RAM.
Quoted 12 lines
Cheers, Kristian.
Signature · 5 lines
Kristian Sørensen - The Umbrella Project http://umbrella.sourceforge.net E-mail: [email protected], Phone: +45 29723816
Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 GrumpyMips).
98.36% of all statistics are fiction.
Quoted 39 lines · Richard B. Johnson
Quoted 37 lines · On Fri, 22 Oct 2004, [ISO-8859-15] Kristian Sørensen wrote:
Quoted 34 lines · Richard B. Johnson
Quoted 18 lines · Kasper Sandberg
Quoted 16 lines · Kristian Sørensen
Hi all!
After some more testing after the previous post of the OOPS in
generic_delete_inode, we have now found a gigantic memory leak in Linux 2.6. [789]. The scenario is the same:File system: EXT3
Unpack and delete linux-2.6.8.1.tar.bz2 with this Bash while loop:let "i = 0"
while [ "$i" -lt 10 ]; dotar jxf linux-2.6.8.1.tar.bz2; rm -fr linux-2.6.8.1; let "i = i + 1"done
Quoted 3 lines
Quoted 2 lines
When the loop has completed, the system use 124 MB memory more _each_ time.... so it is pretty easy to make a denial-of-service attack :-(
Do something like this with your favorite kernel version.....
while true ; do tar -xzf linux-2.6.9.tar.gz ; rm -rf linux-2.6.9 ; vmstat ; done
You can watch this for as long as you want. If there is no other activity, the values reported by vmstat remain, on the average, stable. If you throw in a
synccommand, the values rapidly converge to little memory usage as the disk-data gets flused to disk.
Quoted 7 lines
Quoted 4 lines
The problem is, that the free memory reported by vmstat is decresing by 124mb for each 10-iterations....
The allocated memory does not get freed even if the system has been left alone for three hours!
Yes. So? Why would it be freed? It's left how it was until it is needed. Freeing it would waste CPU cycles.
Okay :-) So it looks like two of you says we have been mistaken :-D (and the behaviour has been changed since linux-2.4)
Anyway - How does this work in practice? Does the file system implementation use a wrapper for kfree or?
Is there any way to force instant free of kernel memory - when freed? Else it is quite hard testing for possible memory leaks in our Umbrella kernel module ... :-/
Best regards,
Signature · 5 lines
Kristian Sørensen - The Umbrella Project http://umbrella.sourceforge.net E-mail: [email protected], Phone: +45 29723816
Quoted 49 lines · On Fri, 22 Oct 2004, [ISO-8859-15] Kristian Sørensen wrote:
Quoted 39 lines · Richard B. Johnson
Quoted 37 lines · On Fri, 22 Oct 2004, [ISO-8859-15] Kristian Sørensen wrote:
Quoted 34 lines · Richard B. Johnson
Quoted 18 lines · Kasper Sandberg
Quoted 16 lines · Kristian Sørensen
Hi all!
After some more testing after the previous post of the OOPS in
generic_delete_inode, we have now found a gigantic memory leak in Linux 2.6. [789]. The scenario is the same:File system: EXT3
Unpack and delete linux-2.6.8.1.tar.bz2 with this Bash while loop:let "i = 0"
while [ "$i" -lt 10 ]; dotar jxf linux-2.6.8.1.tar.bz2; rm -fr linux-2.6.8.1; let "i = i + 1"done
Quoted 3 lines
Quoted 2 lines
When the loop has completed, the system use 124 MB memory more _each_ time.... so it is pretty easy to make a denial-of-service attack :-(
Do something like this with your favorite kernel version.....
while true ; do tar -xzf linux-2.6.9.tar.gz ; rm -rf linux-2.6.9 ; vmstat ; done
You can watch this for as long as you want. If there is no other activity, the values reported by vmstat remain, on the average, stable. If you throw in a
synccommand, the values rapidly converge to little memory usage as the disk-data gets flused to disk.Quoted 7 lines
Quoted 4 lines
The problem is, that the free memory reported by vmstat is decresing by 124mb for each 10-iterations....
The allocated memory does not get freed even if the system has been left alone for three hours!
Yes. So? Why would it be freed? It's left how it was until it is needed. Freeing it would waste CPU cycles.
Quoted 8 lines
Okay :-) So it looks like two of you says we have been mistaken :-D (and the behaviour has been changed since linux-2.4)
Anyway - How does this work in practice? Does the file system implementation use a wrapper for kfree or?
Is there any way to force instant free of kernel memory - when freed? Else it is quite hard testing for possible memory leaks in our Umbrella kernel module ... :-/Best regards,
First, you can always execute sync() and flush most of the file-buffers to disk. This frees up a lot.
In the kernel....
If you are doing a lot of kmalloc() allocation and free(), you can write out the pointer values using printk(). I use '0' before such ... printk("0 %p\n", ptr); do dmesg | sort >xxx.xxx. Now you can look at the file and see the sorted pointer values. If they repeat, chances are pretty good that you are not leaking memory.
In user space...
Periodically look at the break address.
If it keeps going up, you may have a leak. If it's stable you probably are okay.
The only sure way of detecting a memory leak is to use some substitute code (maybe a macro) that substitutes
for (intercepts) the allocator and deallocator. It eventually executes the allocator and deallocator after saving information somewhere you define (array, file, etc). You can sort that information and determine if there are as many (k)mallocs() as there are for (z)frees() (for instance) of the same pointer values.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 GrumpyMips).
98.36% of all statistics are fiction.
Quoted 1 line · Kristian Sørensen
Anyway - How does this work in practice? Does the file system implementation use a wrapper for kfree or?
When an app faults in new memory and there is no unused memory, the system will page out apps and/or filesystem data from the page cache so the memory can be given to the app requesting it.
Quoted 1 line
Is there any way to force instant free of kernel memory - when freed?
It's not free, it's in use by the page cache. This is a performance feature--we try and keep around as much stuff as possible that might be needed by running apps.
Quoted 1 line
Else it is quite hard testing for possible memory leaks in our Umbrella kernel module ... :-/
Such is life. As a crude workaround, on a swapless system you can start one or two memory hogs and they will force the system to free up as much memory as possible.
Chris
Quoted 2 lines · Richard B. Johnson
while true ; do tar -xzf linux-2.6.9.tar.gz ; rm -rf linux-2.6.9 ; vmstat ; done
Stable, yes. But only after about 3 or 4 iterations. The first 3 rather handily used 500+ megs of memory that I did not get back when I stopped it and cleaned up the mess.
Signature · 8 lines
Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) 99.28% setiathome rank, not too shabby for a WV hillbilly Yahoo.com attorneys please note, additions to this message by Gene Heskett are: Copyright 2004 by Maurice Eugene Heskett, all rights reserved.
Quoted 7 lines · Gene Heskett
Quoted 2 lines · Richard B. Johnson
while true ; do tar -xzf linux-2.6.9.tar.gz ; rm -rf linux-2.6.9 ; vmstat ; done
Stable, yes. But only after about 3 or 4 iterations. The first 3 rather handily used 500+ megs of memory that I did not get back when I stopped it and cleaned up the mess.
It should get freed when something else needs it. Usually not before.
Quoted 1 line · Gene Heskett
Stable, yes. But only after about 3 or 4 iterations. The first 3 rather handily used 500+ megs of memory that I did not get back when I stopped it and cleaned up the mess.
Did you run a memory hog to put memory pressure on the system?
The following is with 2.6.9-rc4
-bash-2.05b$ while true ; do tar -xjf linux-2.6.7.tar.bz2 ; rm -rf linux-2.6.7 ; vmstat ; done
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy wa id
1 0 0 1675768 104004 112576 0 0 0 1 11 2 0 0 0 10
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy wa id
1 1 0 1649032 110792 112724 0 0 0 1 11 3 0 0 0 10
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy wa id
1 0 0 1630472 118580 112620 0 0 0 2 11 3 0 0 0 10
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy wa id
1 0 0 1607560 125500 112636 0 0 0 2 11 3 0 0 0 10
After running a memory hog,
-bash-2.05b$ vmstat
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy wa id
0 0 0 1890248 672 4836 0 0 0 3 11 3 0 0 0 10
Looks like the cached memory all got freed, which is exactly as expected.
Chris
Quoted 5 lines · Kristian Sørensen
Hi all!
After some more testing after the previous post of the OOPS in
generic_delete_inode, we have now found a gigantic memory leak in Linux 2.6. [789]. The scenario is the same:
<SNIP>
Quoted 7 lines
When the loop has completed, the system use 124 MB memory more _each_ time.... so it is pretty easy to make a denial-of-service attack :-(
We have tried the same test on a RHEL WS 3 host (running a RedHat 2.4 kernel)
- and there is no problem.
Quoted 6 lines
Any deas?
--
Kristian Sørensen
- The Umbrella Project
http://umbrella.sourceforge.net
This is a common mistake that many people make when first looking at the Linux stats.
Linux starts off with most of the memory free, but rapidly uses it up. it keeps a small amount (a few megs) free at all times, but for the rest counts of freeing memory (possibly by swapping) when a new program asks for memory and there is less then the minimum amount left free.
It does this becouse there is a chance that the memory will be re-used (in your example where you were untaring the kernel source there is a chance that someone else would be reading that source and if they did it would already be in memory and not have to be re-read from disk) and becouse there is a chance that nothing will ever need to use that memory before the computer is shut off so it would be a waste of time to do the free (which includes zeroing out the memory, not just marking it as available).
This puts the cost of zeroing out and freeing memory on new programs that are allocating memory, which tends to scatter the work over time rather then having a large burst of work kick in when a program exits (it seems odd to think that if a large computer exits the machine would be pegged for a little while while it frees up and zeros the memory, not exactly what you would expect when you killed a program :-)
David Lang
Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare
Quoted 1 line · David Lang
This puts the cost of zeroing out and freeing memory on new programs that are allocating memory, which tends to scatter the work over time rather then having a large burst of work kick in when a program exits (it seems odd to think that if a large computer exits the machine would be pegged for a little while while it frees up and zeros the memory, not exactly what you would expect when you killed a program :-)
Any this partially explains why response is bad every morning when starting daily operation. Instead of using the totally unproductive time in the idle loop to zero and free those pages when it would not hurt response, the kernel saves that work for the next time the memory is needed lest it do work which might not be needed before the system is shutdown.
With all the work Nick, Ingo,Con and others are putting into latency and responsiveness, I don't understand why anyone thinks this is desirable behavior. The idle loop is the perfect place to perform things like this, to convert non-productive cycles into performing tasks which will directly improve response and performance when the task MUST be done. Things like zeroing these pages, perhaps defragmenting memory, anything which can be done in small parts.
It would seem that doing things like this in small inefficient steps in idle moments is still better than doing them efficiently while a process is waiting for the resources being freed.
Signature · 3 lines
bill davidsen <[email protected]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979
Uttered Bill Davidsen <[email protected]>, spake thus:
Quoted 5 lines
With all the work Nick, Ingo,Con and others are putting into latency and responsiveness, I don't understand why anyone thinks this is desirable behavior. The idle loop is the perfect place to perform things like this, to convert non-productive cycles into performing tasks which will directly improve response and performance when the task MUST be done.
Bill, with respect,
The idle loop is, by definition, the place to go when there is nothing else to do. Scrubbing memory is, by definition, not "nothing", so leave the idle loop alone.
That's why God, or maybe it was Linus, invented kernel threads.
Cheers!
Quoted 6 lines · Tommy Reynolds
Uttered Bill Davidsen <[email protected]>, spake thus:
Quoted 1 line
With all the work Nick, Ingo,Con and others are putting into latency and responsiveness, I don't understand why anyone thinks this is desirable behavior. The idle loop is the perfect place to perform things like this, to convert non-productive cycles into performing tasks which will directly improve response and performance when the task MUST be done.
Quoted 7 lines
Bill, with respect,
The idle loop is, by definition, the place to go when there is nothing else to do. Scrubbing memory is, by definition, not "nothing", so leave the idle loop alone.
That's why God, or maybe it was Linus, invented kernel threads.
Did you really not know what I meant here, or are you being pedantic about the nomenclature? Yes, obviously implement by thread(s) with priority lower than whale shit, the object of which is to do the work when no process is waiting for the CPU, and in very small steps so the CPU isn't tied up.
Signature · 3 lines
-bill davidsen ([email protected]) "The secret to procrastination is to put things off until the last possible moment - but no longer" -me
Quoted 5 lines · Bill Davidsen
Quoted 1 line · David Lang
This puts the cost of zeroing out and freeing memory on new programs that are allocating memory, which tends to scatter the work over time rather then having a large burst of work kick in when a program exits (it seems odd to think that if a large computer exits the machine would be pegged for a little while while it frees up and zeros the memory, not exactly what you would expect when you killed a program :-)
Any this partially explains why response is bad every morning when starting daily operation. Instead of using the totally unproductive time in the idle loop to zero and free those pages when it would not hurt response, the kernel saves that work for the next time the memory is needed lest it do work which might not be needed before the system is shutdown.
actually, what useually has happened is that updatedb ran overnight and used all your memory for it's work so all your application stuff got thrown away or swapped out as it appeared to be less useful then the then-active process. so first thing in the morning you need to do a lot of disk reads to get your desktop working set into memory. the cost of zeroing the pages is minor compared to the disk IO
Quoted 3 lines
With all the work Nick, Ingo,Con and others are putting into latency and responsiveness, I don't understand why anyone thinks this is desirable behavior. The idle loop is the perfect place to perform things like this, to convert non-productive cycles into performing tasks which will directly improve response and performance when the task MUST be done. Things like zeroing these pages, perhaps defragmenting memory, anything which can be done in small parts.
It would seem that doing things like this in small inefficient steps in idle moments is still better than doing them efficiently while a process is waiting for the resources being freed.
the problem is that you don't know that you need to throw away the data. the next thing that you try to do could re-use the data that's in the ram, how can the system know?
David Lang
Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare
Quoted 2 lines · In article <[email protected]> you wrote:
When the loop has completed, the system use 124 MB memory more _each_ time.... so it is pretty easy to make a denial-of-service attack :-(
for starters i recommend to look at "free" and only at the marked number:
total used free shared buffers cached
Mem: 126368 108432 17936 0 6532 42104
-/+ buffers/cache: 59796* 66572*
Swap: 262128 43400 218728
or at the swap numbers if you have low memory (like i do).
Gruss
Bernd
Signature · 2 lines
eckes privat - http://www.eckes.org/ Project Freefire - http://www.freefire.org/