2.6.19-rc1-mm1+ memory problem
Short description: when X is loaded (maybe any heavy application is sufficient, but I don't use anything heavy in console), 'free' says used memory is growing.
Keywords: memory.
Kernel: built locally, gcc 4.0.3
I have a strange problem with 2.6.19-rc-mm kernels. After I load X, I notice that memory is marked used at rate of tens of KB/s. Then it starts to swap very heavily, when physical memory is all used. I tried to verify it - it is so with all -mm kernels after 2.6.19-rc1-mm1, including 2.6.19-rc5-mm2. At the meantime everything works OK with kernels 2.6.18-mm3 and 2.6.19-rc1 through 2.6.19-rc6. I do not see any options that should be memory eating in my .config . Module list is short enough to include inline.
When I just run some things like periodical suck, oops proxy server etc with X shut down, I do not notice "leak" from console because of small fluctuations of memory use. When I run X and shut it down, used memory count goes up a few megs (consistent with speed of eating it by X).
I didn't find exactly this problem in lkml or www, though the problem with OOM on 2.6.19-rc-mm seems similar.
What should I check to fix problem or produce a useful bug report?
/etc/sysconfig/modules:
ehci-hcd, usb-storage, usbhid, ipaq, i915
Now loaded in 2.6.19-rc6:
i915, drm, ipaq, usbserial, usbhid, usb_storage, libusual, ehci_hcd, usbcore
Main configuration options:
http://bigtip.narod.ru/temp/xorg.conf.txt
http://bigtip.narod.ru/temp/config-2.6.19-rc2-mm5-swsusp-my-1.txt
http://bigtip.narod.ru/temp/lspci.txt
Drivers:
http://bigtip.narod.ru/temp/ioports.txt
http://bigtip.narod.ru/temp/iomem.txt
Quoted 2 lines · Michael Raskin
Short description: when X is loaded (maybe any heavy application is sufficient, but I don't use anything heavy in console), 'free' says used memory is growing.
Tried driver vesa. Leak still exists.
About leak size: with dri, xscreensaver, and nothing loaded while true; do free >>free.log; sleep 1; done shows ~100KB/s.
Quoted 22 lines · On Mon, 20 Nov 2006 09:26:29 +0300 Michael Raskin <[email protected]> wrote:
Short description: when X is loaded (maybe any heavy application is sufficient, but I don't use anything heavy in console), 'free' says used memory is growing.
Keywords: memory.
Kernel: built locally, gcc 4.0.3
I have a strange problem with 2.6.19-rc-mm kernels. After I load X, I notice that memory is marked used at rate of tens of KB/s. Then it starts to swap very heavily, when physical memory is all used. I tried to verify it - it is so with all -mm kernels after 2.6.19-rc1-mm1, including 2.6.19-rc5-mm2. At the meantime everything works OK with kernels 2.6.18-mm3 and 2.6.19-rc1 through 2.6.19-rc6. I do not see any options that should be memory eating in my .config . Module list is short enough to include inline.
When I just run some things like periodical suck, oops proxy server etc with X shut down, I do not notice "leak" from console because of small fluctuations of memory use. When I run X and shut it down, used memory count goes up a few megs (consistent with speed of eating it by X).
Quoted 4 lines
I didn't find exactly this problem in lkml or www, though the problem with OOM on 2.6.19-rc-mm seems similar.
What should I check to fix problem or produce a useful bug report?
Monitor /proc/meminfo
If the leak is slab, monitor /proc/slabinfo and /proc/slab_allocators. /proc/slab_allocators needs CONFIG_DEBUG_SLAB_LEAK.
Thanks.
Quoted 44 lines · On Tue, 21 Nov 2006 21:41:31 +0300 Michael Raskin <[email protected]> wrote:
Quoted 7 lines · Andrew Morton
Quoted 4 lines · On Mon, 20 Nov 2006 09:26:29 +0300 Michael Raskin <[email protected]> wrote:
Short description: when X is loaded (maybe any heavy application is sufficient, but I don't use anything heavy in console), 'free' says used memory is growing.
Thank you for reply.
Quoted 1 line
Monitor
/proc/meminfoThanks for advice. I didn't think of it. I should be ashamed.
Result: mysterious. All fields that grow can not account for even a third part.
In top I found a situation when 2MB (~half a minute) go to nowhere and no of first 50 processes changes resident memory usage at all. The rest have less than a MB each.
Quoted 1 line
If the leak is slab, monitor
/proc/slabinfoand/proc/slab_allocators.I hope no.
Quoted 4 lines
/proc/slab_allocatorsneedsCONFIG_DEBUG_SLAB_LEAK.Thanks.
I did a few cat
/proc/meminfo. Two of them are here:MemTotal: 763532 kB 763532 kB MemFree: 445956 kB 430932 kB Buffers: 20908 kB 21048 kB Cached: 77008 kB 77212 kB SwapCached: 0 kB 0 kB Active: 65916 kB 66120 kB Inactive: 54748 kB 54884 kB SwapTotal: 1052216 kB 1052216 kB SwapFree: 1052216 kB 1052216 kB Dirty: 264 kB 324 kB Writeback: 0 kB 0 kB AnonPages: 22752 kB 22748 kB Mapped: 14616 kB 14628 kB Slab: 23108 kB 23088 kB SReclaimable: 15360 kB 15364 kB SUnreclaim: 7748 kB 7724 kB
Quoted 8 lines
PageTables: 1216 kB 1216 kB NFS_Unstable: 0 kB 0 kB Bounce: 0 kB 0 kB CommitLimit: 1433980 kB 1433980 kB Committed_AS: 281456 kB 281448 kB VmallocTotal: 262104 kB 262104 kB VmallocUsed: 2876 kB 2876 kB VmallocChunk: 259028 kB 259028 kB
You lost 15MB and they didn't even turn up on the page LRU.
Can you try to determine exactly which activity causes this to happen? In particular, is it due to the X server? If so, does any particular client cause it to happen? Things which use 3d?
Quoted 2 lines · Andrew Morton
On Tue, 21 Nov 2006 21:41:31 +0300
Michael Raskin <[email protected]> wrote:
Sorry for leaving lkml out of "To: " in previous post.
Quoted 3 lines
Can you try to determine exactly which activity causes this to happen? In particular, is it due to the X server? If so, does any particular client cause it to happen? Things which use 3d?
You were right, it's not because of personally X, but because of environment I use.
Simplest example of reproducing code:
while true; do free | cat &>/dev/null; done
Looks like minimum (except of &>/dev/null not to involve console/xterm output - leaks well without it too).
Michael Raskin wrote:
Strange thing: when run from xterm,
while true; do free | cat &>/dev/null; done
causes leak. While X is not loaded - no.
Also I have uploaded contents of /proc/page_owner after loosing more than 100M. (220M used, 29M - on page_owner, lessthan 50M - for processes). I will study it also.
http://bigtip.narod.ru/temp/page_owner.bz2
http://bigtip.narod.ru/temp/page_owner.gz
Quoted 1 line · Michael Raskin
Also I have uploaded contents of
/proc/page_ownerafter loosing more than 100M. (220M used, 29M - onpage_owner, lessthan 50M - for processes).
Top 3 entries:
89361 times:
Page allocated via order 0, mask 0x280d2
[0xc0159f31] __handle_mm_fault+1809
[0xc011318a] do_page_fault+314
[0xc04111c4] error_code+116
Can be anything. But if I understand anything, this memory is used because someone has requested a page that is swapped out. So the memory must be used, but not reflected in meminfo, and not by a process?
35560 times:
Page allocated via order 0, mask 0x201d2 [0xc0152ec2] __do_page_cache_readahead+450 [0xc015309a] do_page_cache_readahead+74 [0xc014d7b5] filemap_nopage+325 [0xc0159919] __handle_mm_fault+249 [0xc011318a] do_page_fault+314
[0xc04111c4] error_code+116
- is reflected in cache usage statistics, I guess..
6185 times:
Page allocated via order 0, mask 0x200d2
[0xc014e069] generic_file_buffered_write+329
[0xc014e814] __generic_file_aio_write_nolock+612
[0xc014eb85] generic_file_aio_write+85
[0xc01b26ff] ext3_file_write+63
[0xc016b23c] do_sync_write+204
[0xc016b9a7] vfs_write+167
[0xc016c2a7] sys_write+71
[0xc010303a] sysenter_past_esp+95
- negligible, really..
Quoted 27 lines · On Sat, 25 Nov 2006 02:07:43 +0300 Michael Raskin <[email protected]> wrote:
Quoted 3 lines · Michael Raskin
Also I have uploaded contents of
/proc/page_ownerafter loosing more than 100M. (220M used, 29M - onpage_owner, lessthan 50M - for processes).Top 3 entries:
89361 times:
Page allocated via order 0, mask0x280d2
[0xc0159f31] __handle_mm_fault+1809
[0xc011318a] do_page_fault+314
[0xc04111c4] error_code+116
Can be anything. But if I understand anything, this memory is used because someone has requested a page that is swapped out. So the memory must be used, but not reflected in meminfo, and not by a process?35560 times:
Page allocated via order 0, mask0x201d2[0xc0152ec2] __do_page_cache_readahead+450 [0xc015309a] do_page_cache_readahead+74 [0xc014d7b5] filemap_nopage+325 [0xc0159919] __handle_mm_fault+249 [0xc011318a] do_page_fault+314
[0xc04111c4] error_code+116
- is reflected in cache usage statistics, I guess..
Quoted 11 lines
6185 times:
Page allocated via order 0, mask0x200d2
[0xc014e069] generic_file_buffered_write+329
[0xc014e814] __generic_file_aio_write_nolock+612
[0xc014eb85] generic_file_aio_write+85
[0xc01b26ff] ext3_file_write+63
[0xc016b23c] do_sync_write+204
[0xc016b9a7] vfs_write+167
[0xc016c2a7] sys_write+71
[0xc010303a] sysenter_past_esp+95
- negligible, really..
What you should do is to cause the system to free as many pages as possible before looking ad /proc/page_owner. For example, build `usemem' from
http://www.zip.com.au/~akpm/linux/patches/stuff/ext3-tools.tar.gz, run
usemem -m N (where N is the number of megabytes which the machine has)
a couple of times. Then check /proc/meminfo, and look to see which pages are left over in /proc/page_owner.
Thanks.
Quoted 7 lines · Andrew Morton
Quoted 6 lines
89361 times:
Page allocated via order 0, mask0x280d2
[0xc0159f31] __handle_mm_fault+1809
[0xc011318a] do_page_fault+314
[0xc04111c4] error_code+116
Can be anything. But if I understand anything, this memory is used because someone has requested a page that is swapped out. So the memory must be used, but not reflected in meminfo, and not by a process?
Quoted 8 lines
What you should do is to cause the system to free as many pages as possible before looking ad
/proc/page_owner. For example, build `usemem' from
http://www.zip.com.au/~akpm/linux/patches/stuff/ext3-tools.tar.gz, runusemem -m N (where N is the number of megabytes which the machine has)
a couple of times. Then check
/proc/meminfo, and look to see which pages are left over in/proc/page_owner.
Well, I was too lazy to get this utility, used my own to allocate and fill enough memory as to go some 50MB to deep swap (Did I understand correctly what usemem does?). Top 3 did not change, except for exact numbers.
Re: 2.6.19-rc6-mm2 is ok (2.6.19-rc1-mm1+ memory problem)
In reply to Michael RaskinQuoted 1 line · Michael Raskin
I have a strange problem with 2.6.19-rc-mm kernels. After I load X, I notice that memory is marked used at rate of tens of KB/s. Then it
Tried 2.6.19-rc6-mm2. Now the problem is gone. Sometimes memory is getting maked used as before, but when the loss reaches a few MB's it is all freed. After 3 hours of X+all those scripts that cause leak + ThunderBird I can still shut down everything except a few processes and have only 50MB used. Script that demonstrated leak is now working without problems and without eating memory.
Thanks.