Memory management issues with 2.4.4
Short version:
Under very heavy thrashing (about four hours) the system either lockups or OOM handler kills a task even when there is swap space left.
Long version:
My system is a dual 2x200MMX in a Gigabyte 586DX with 96Mb and 226Mb of swap this way:
[root@quartz ~]# swapon -s
Filename Type Size Used Priority /dev/hda7 partition 96348 6456 1 /dev/hdc6 partition 130276 6460 1
Well, I have tried to compile Mozilla 0.8.1 since the day it came out, but I always lockup in the same place, wich it's begining to be a bit frustrating ;-)
The problem is that compiling the file content/base/src/nsStyleContext.o makes cc1plus grow up to a size of 141M, at this point the system is in heavy thrashing, kswapd is using about 7-12% of CPU time and cc1plus is using 8-14%.
If I use a SMP kernel the system always ends up frozen, some times after about almost one day of uptime and compiling since booting, and another times it gets frozen in much less time, three or four hours.
If I use a non SMP kernel the system doesn't lokup but after some hours, it varies between 6-8h, the cc1plus procces get a kill signal by OOM killer, althought there is plenty of swap space left ((96Mb RAM + 226Mb swap) - (140Mb - tiny amount used by the system) = plenty ;-).
For this tries I have left the system in single user, so no cron jobs, no network trafic, no etc...
I suspect there is a race in the swap handling in SMP, and that the OOM doesn't take into account the swap space left sometimes.
I don't know what to try next, suggestions?
Signature · 2 lines
Jorge Nerin <[email protected]>
Quoted 3 lines · Jorge Nerin
Short version:
Under very heavy thrashing (about four hours) the system either lockups or OOM handler kills a task even when there is swap space left.
First of all, please try to reproduce the problem with 2.4.5-pre1.
If it still happens with pre1, please show us the output of "cat /proc/slabinfo" when the kernel is about to trigger the OOM killer.
Thanks.
Quoted 7 lines · Marcelo Tosatti
Quoted 2 lines · Jorge Nerin
Short version:
Under very heavy thrashing (about four hours) the system either lockups or OOM handler kills a task even when there is swap space left.
Quoted 7 lines
First of all, please try to reproduce the problem with 2.4.5-pre1.
If it still happens with pre1, please show us the output of "cat
/proc/slabinfo" when the kernel is about to trigger the OOM killer.Thanks.
Well, I have tried with 2.4.5-pre1 compiled form SMP, and the result has been that this morning when I wake up the system has the console black (is there any way to prevent cons.saver from blanking the screen) and the disks where quiet, so I SysRQ-Sync, Umount, powerOff and then, at the last command the console wake up and I have been saluted with:
Kernel BUG in sched.c:709!
Invalid operand: 0000
Dump copied by hand, but not yet filtered by ksymoops (I'm at work now). kernel panic Aiee, killing interrupt handler!
In interrupt not syncing.
This afternoon when I return home I will feed the stack dump to ksymoops and post the results, I mail this now just to see if someone sees the ligth.
Signature · 2 lines
Jorge Nerin <[email protected]>
Quoted 8 lines · Marcelo Tosatti
Quoted 3 lines · Jorge Nerin
Short version:
Under very heavy thrashing (about four hours) the system either lockups or OOM handler kills a task even when there is swap space left.
Quoted 7 lines
First of all, please try to reproduce the problem with 2.4.5-pre1.
If it still happens with pre1, please show us the output of "cat
/proc/slabinfo" when the kernel is about to trigger the OOM killer.Thanks.
Well, as I had said this morning I have feed the Oops to ksymoops, note that I may have mirtyped something, but anyway here is the output of ksymoops:
ksymoops 2.3.4 on i586 2.4.5-pre1. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.5-pre1/ (default)
-m /usr/src/linux/System.map (default)
Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options.
Warning (compare_maps): ksyms_base symbol __VERSIONED_SYMBOL(shmem_file_setup) not found in System.map. Ignoring ksyms_base entry invalid operand: 0000
CPU: 1
EIP: 0010:[<c0123c4>]
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010096
eax: 0000001b ebx: ffffffff ecx: 00000001 edx: 00000001
esi: c021b960 edi: c5fe2000 ebp: c5fe3ce0 esp: c5fe3c88
Stack: c01e89a5 c01c8af6 000002c5 00000000 c021b960 c021b954 00000000 00000001
00000020 000000cc c02961c0 00000120 c012ca08 c5fe2000 00000000
c216960
c21b954 c5fe2000 00000080 00000001 c5fe2000 00000001 00000001
c12da64
Call Trace: [<c012ca08>] [<c012da64>] [<c012dbcc>] [<c0113bbb>] [<c01058c9>] [<c0106d07>] [<c010e568>]
[<c01054ea>] [<c010e591>] [<c010e568>] [<c01746de>] [<c0172dfd>] [<c0173fcc>] [<c017405c>] [<c0108516>]
[<c0108709>] [<c0106de0>] [<c010fc60>] [<c0129e78>] [<c0129c78>]
[<c0129e52>] [<c0129e78>] [<c0129f56>]
[<c0129e78>] [<c0129f83>] [<c0129a29>] [<c01471a4>] [<c012cc9b>]
[<c012cd2b>] [<c0105000>] [<c01054f3>]
Code: 0f 0b 8d 65 b4 5b 5f 89 ec 5d c3 55 89 e5 83 ec 18 57 56
Quoted 1 line
Quoted 1 line
EIP;
0c0123c4Before first symbol <=====
Trace; c012ca08 <free_shortage+1c/100> Trace; c012da64 <__alloc_pages+16c/2c0> Trace; c012dbcc <__get_free_pages+14/20> Trace; c0113bbb <do_fork+93/770>
Trace; c01058c9 <sys_clone+1d/24>
Trace; c0106d07 <system_call+37/40>
Trace; c010e568 <apm_magic+0/8>
Trace; c01054ea <kernel_thread+1a/30> Trace; c010e591 <apm_power_off+21/3c> Trace; c010e568 <apm_magic+0/8>
Trace; c01746de <handle_sysrq+de/230> Trace; c0172dfd <handle_scancode+1bd/318> Trace; c0173fcc <handle_kbd_event+130/1a4> Trace; c017405c <keyboard_interrupt+1c/28> Trace; c0108516 <handle_IRQ_event+52/7c> Trace; c0108709 <do_IRQ+99/ec>
Trace; c0106de0 <ret_from_intr+0/20> Trace; c010fc60 <smp_call_function+8c/c0> Trace; c0129e78 <do_ccupdate_local+0/40> Trace; c0129c78 <kmem_cache_create+220/3e0> Trace; c0129e52 <smp_call_function_all_cpus+1a/40> Trace; c0129e78 <do_ccupdate_local+0/40> Trace; c0129f56 <drain_cpu_caches+9e/c0> Trace; c0129e78 <do_ccupdate_local+0/40> Trace; c0129f83 <__kmem_cache_shrink+b/6c> Trace; c0129a29 <kmem_slab_destroy+79/a8> Trace; c01471a4 <shrink_dcache_memory+2c/30> Trace; c012cc9b <do_try_to_free_pages+5f/7c> Trace; c012cd2b <kswapd+73/110>
Trace; c0105000 <init+0/1b0>
Trace; c01054f3 <kernel_thread+23/30> Code; 0c0123c4 Before first symbol 00000000 <_EIP>:
Code; 0c0123c4 Before first symbol <=====
0: 0f 0b ud2a <=====
Code; 0c0123c6 Before first symbol
2: 8d 65 b4 lea 0xffffffb4(%ebp),%esp
Code; 0c0123c9 Before first symbol
5: 5b pop %ebx
Code; 0c0123ca Before first symbol
6: 5f pop %edi
Code; 0c0123cb Before first symbol
7: 89 ec mov %ebp,%esp
Code; 0c0123cd Before first symbol
9: 5d pop %ebp
Code; 0c0123ce Before first symbol
a: c3 ret
Code; 0c0123cf Before first symbol
b: 55 push %ebp
Code; 0c0123d0 Before first symbol
c: 89 e5 mov %esp,%ebp
Code; 0c0123d2 Before first symbol
e: 83 ec 18 sub $0x18,%esp
Code; 0c0123d5 Before first symbol
11: 57 push %edi
Code; 0c0123d6 Before first symbol
12: 56 push %esi
Kernel panic Aiee killing interrupt handler
2 warnings issued. Results may not be reliable.
As I said I have tried to no make errors, but I copied it at 7 in the morning, so who knows ... ;-)
Signature · 2 lines
Jorge Nerin <[email protected]>