2.6.9, 64bit, 4GB memory => panics ...
Dear Kernel Gurus ! :*)
I just installed Fedora Core 3, x86_64. It boots fine, but crashes. The "crash" part occurs as soon as I start any app that apparently uses some "larger" amount of memory. At first I thought there was a problem with X, and my video card. So I tried to login on one of the virtual screens (ALT-F1). Here I was able to see a kernel panic screen when I started several random apps. I am working in duplicating that and capturing the full panic message.
My hardware is:
Tyan Tiger K8W (S2875) with two Opterons, 4 SATA drives (with softraid) and a low end Nvidia MX-400 video card. My BIOS version is 2.03.
I can use the system 100% with only 2GB of memory installed. I have run the memtest86+ for over a day and did not get any errors (with the full 4gb installed). Previously I was running Fedora Core 2 but 32 bit, also with the 2.6.9 kernel and did not have any problems with 4 GB of memory. My guess is the combination of 64 bit and 4GB is causing the problem.
I have tried to compile my own kernel, and also got the 667 and 681 releases (for 2.6.9) from Fedora.org. Problem seems to be persistent.
Here is the ver_linux output
-----------------------------------------------------------
Linux cpu10 2.6.9-1.681_FC3smp #1 SMP Thu Nov 18 15:30:11 EST 2004 x86_64 x86_64 x86_64 GNU
/Linux
Gnu C 3.4.2 Gnu make 3.80 binutils 2.15.92.0.2 util-linux 2.12a mount 2.12a module-init-tools 3.1-pre5 e2fsprogs 1.35 reiserfsprogs line reiser4progs line pcmcia-cs 3.2.7 quota-tools 3.12. PPP 2.4.2 isdn4k-utils 3.3 nfs-utils 1.0.6 Linux C Library 2.3.3 Dynamic linker (ldd) 2.3.3 Procps 3.2.3 Net-tools 1.60 Kbd 1.12 Sh-utils 5.2.1 Modules Loaded autofs4 i2c_dev i2c_core nfs lockd sunrpc ds
yenta_socket pcmcia_cor
e dm_mod button battery ac nvidia md5 ipv6 ohci1394 ieee1394 ohci_hcd uhci_hcd ehci_hcd hw_
random snd_intel8x0 snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_all
oc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore e1000 floppy ext3 jbd
raid5 xor sata_sil libata sd_mod scsi_mod
-----------------------------------------------------------
Any suggestions what may be causing the crashes ?
I do not subscribe to this list, so would sincerely appreciate a cc of any replies. I'm sure I forgot to mention something, please let me know what other information can be help full to nail this down.
Thanks a lot in advance !
Kind Regards,
rudi
Quoted 8 lines · Rudolf Usselmann
I just installed Fedora Core 3,
x86_64. It boots fine, but crashes. The "crash" part occurs as soon as I start any app that apparently uses some "larger" amount of memory. At first I thought there was a problem with X, and my video card. So I tried to login on one of the virtual screens (ALT-F1). Here I was able to see a kernel panic screen when I started several random apps. I am working in duplicating that and capturing the full panic message.
The full panic message would help a lot. We can't do much without it.
-- wli
Quoted 3 lines · William Lee Irwin III
The full panic message would help a lot. We can't do much without it.
-- wli
I was hoping to be able to get the entire panic screen, but am only getting the bottom of it ...
Does this help ?
FS: 0000002a95cea6e0(0000) GS: ffffffff80516b00(0000) knlGS: 0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000002 CR3: 0000000000011000 CR4: 00000000000006ed Process as (pid: 4312, threadinfo 00000101735e0000, task 00000101746c9070)
Stack: ffffffff80141160 0000000000000012 0000010176b87fa0 0000000000000000
000001000709a200 0000010175b37730 000001017aed32c0 000000000000005d
ffffffff801c6f1d 000001016bfd1310
Call Trace: <ffffffff80141160>{__mod_timer+160} <ffffffff801c6f1d>{start_this_handle+605}
<ffffffff80159bcd>{filemap_nopage+413} <ffffffff8019335c>{__d_lookup+300}
<ffffffff80187c47>{do_lookup+55} <ffffffff801c724f>{journal_start+239}
<ffffffff801c20bd>{ext3_create+45} <ffffffff8018957d>{vfs_create+237}
<ffffffff80189a00>{open_namei+432} <ffffffff801796a7>{filp_open+39}
<ffffffff801797be>{get_unused_fd+238} <ffffffff8017990c>{sys_open+76}
<ffffffff801105f6>{system_call+126}
Code: f0 fe 0f 0f 88 8e 02 00 00 c3 66 66 90 66 66 90 c6 07 01 c3RIP <ffffffff8038443e0>{spin_lock+0} RSP <00000101735e1cc0>
CR2: 0000000000000002
Kind Regards,
rudi
FYI, I tried 2.9.10-rc3 kernel, same problem.
Can anyone guide me or give me some pointers how I can enable login on serial ports - that would hopefully allow me to capture more/longer kernal panic screens. Doing an rlogin doesn't help, the kernel panic is still going to the root console (24 lines visible).
Any other tips/pointers as to how to capture kernel panics or what else I can do to help you guys to help me would be highly appreciated. :*)
Kind Regards,
rudi
Hello Rudolf,
Quoted 11 lines · Rudolf Usselmann
FYI, I tried 2.9.10-rc3 kernel, same problem.
Can anyone guide me or give me some pointers how I can enable login on serial ports - that would hopefully allow me to capture more/longer kernal panic screens. Doing an rlogin doesn't help, the kernel panic is still going to the root console (24 lines visible).
Any other tips/pointers as to how to capture kernel panics or what else I can do to help you guys to help me would be highly appreciated. :*)
Make sure that you have the following option in your kernel;
CONFIG_SERIAL_8250_CONSOLE=y
Then boot the kernel with this kernel parameter;
console=ttyS0,38400
Make necessary adjustments for baud rate and serial port. Then you may additionally add a getty on the console with something akin to the following /etc/inittab entry;
7:35:respawn:/sbin/agetty 38400 ttyS0
Finally all you need is a program on the monitoring system, minicom should suffice if you have nothing else.
Zwane
Quoted 4 lines · Zwane Mwaikambo
Make sure that you have the following option in your kernel;
CONFIG_SERIAL_8250_CONSOLE=y
...
Quoted 7 lines
7:35:respawn:/sbin/agetty 38400 ttyS0
Finally all you need is a program on the monitoring system, minicom should suffice if you have nothing else.
Zwane
Thanks a lot Zwane, appreciate these tips !
I am now trying to be better prepared for my next 4gb test, and have been working on a small program that should "eat memory". I'm not sure I am doing the right thing though ...
The idea was to create an app that would consume a lot of memory without touching any other system resources.
At first I used ksysguard to watch the memory consumption. Interestingly enough, it was "wired": The VmSize column seems to jump randomly, once I get 1,051 than I will see 3,148,615 than I will get 2,789 than I might see 1,073,976 than it will give me 3,148 .... all during the same run of my mem eater app.
... just mentioning it here in case there is a connection between my problem and this wired display ...
Now this testing was done with 2GB at first ... no kernel panics with 2GB.
Than I decided to use /proc/meminfo, but that suggests that I am not using all memory at all ! With 2GB malloc fails after 510 iterations (510 MB ???) ... (I have a 10GB swap partition) And /proc/meminfo confirms that:
MemTotal: 2056528 kB MemFree: 1526424 kB
About 1/2 gig is used, I checked limits, and that said unlimited ... What else can block memory usage ? May be I just need to start my app several times ?!
Hmm, after forking mem eater 6 times I get:
MemTotal: 2056528 kB MemFree: 610000 kB
Which is almost the same as after 4 and 5 spins ...
Any ideas/suggestion ?
Thanks a lot guys & gals !
Kind Regards,
rudi
// -------------------------- eat mem ---------------------------------
#include "stdio.h"
#include "stdlib.h"
int main() {
int mem[10000];
int i, n;
for(i=0;i<2000;i++) {
printf("Doing alloc %0d ...\n",i);
mem[i] = (int)malloc(1024*1024*1024);
if(mem[i] == NULL)
printf("Malloc failed ...\n");
else
for(n=0;n<(1024*1024*1024);n=n+640) mem[i] = n;}
while(1);
return(0);
}
// --------------------------------------------------------------------
Quoted 6 lines · Rudolf Usselmann
int mem[10000];
int i, n;for(i=0;i<2000;i++) { printf("Doing alloc %0d ...\n",i); mem[i] = (int)malloc(1024*1024*1024);
allocate a 1 gig block, but store it in an int array? That's wrong.
Quoted 4 lines
if(mem[i] == NULL)
printf("Malloc failed ...\n");
else
for(n=0;n<(1024*1024*1024);n=n+640) mem[i] = n;You alter n, but then always only set mem[i], without varying i.
Your program is buggy, and memleaks.
Plus, the kernel uses overcommit by default, and since you aren't actually touching the memory you are allocating, you are only limited by the address space size allowed per process(depends on how you configured the kernel).
Hmm, ok my eat mem was useless. Could not get the system to crash ...
Thank I was thinking and it occured to me that each time I was experiencing the crashes thgere was heavy file system activity.
So I tried two things:
tar cf /dev/null <large directory 500+MB>
That went ok, than I did:
tar cf archive.tar <large directory 500+MB>
And that gave me:
Unable to handle kernel paging request at 0000000000001170 RIP:
<ffffffff8015df54>{kmem_getpages+132}
PML4 17102c067 PGD 171058067 PMD 0
Oops: 0000 [1] SMP
CPU 1
Modules linked in: nvidia ipv6 autofs4 sunrpc binfmt_misc dm_mod button
battery ac ohci_hcy
Pid: 3454, comm: tar Tainted: P 2.6.9RU1.1
RIP: 0010:[<ffffffff8015df54>] <ffffffff8015df54>{kmem_getpages+132}
RSP: 0018:000001017100da38 EFLAGS: 00010213
RAX: ffffffff7fffffff RBX: 000001017ffc9680 RCX: 0000000000000000
RDX: 000001000000f500 RSI: 000001000000f580 RDI: 000001000000f840
RBP: 0000010037ffe000 R08: 000001016ac59000 R09: 000001017ffc96c8
R10: 0000000000001000 R11: 0000000000000000 R12: 000001017ffc9680
R13: 000001017ffc96a8 R14: 0000000000000000 R15: 0000000000000000
FS: 0000002a95572b00(0000) GS:ffffffff80512980(0000)knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000001170 CR3: 0000000037f08000 CR4: 00000000000006e0 Process tar (pid: 3454, threadinfo 000001017100c000, task 000001017fc5a7d0)
Stack: 000001016b3ca088 0000000000001f7d 0000000000000001
ffffffff8015e720
000001017ffc9708 0000000000000001 0000005000000050
0000000000000000
0000000000000000 0000000000001000
Call Trace:<ffffffff8015e720>{cache_alloc_refill+656}
<ffffffff8015e416>{kmem_cache_alloc+
<ffffffff8017ac91>{alloc_buffer_head+17}
<ffffffff8017b307>{create_buffers+39}
<ffffffff8017b3a4>{create_empty_buffers+20}
<ffffffff8017baa5>{__block_prepare_writ
<ffffffff801bb8f0>{ext3_get_block+0}
<ffffffff8017be0a>{block_prepare_write+26}
<ffffffff801b9a25>{ext3_prepare_write+101}
<ffffffff80157e12>{generic_file_buffered
<ffffffff80191f4e>{inode_update_time+158}
<ffffffff80158686>{generic_file_aio_write
<ffffffff8015875f>{generic_file_aio_write+127}
<ffffffff801b7f33>{ext3_file_write+3
<ffffffff80177a7d>{do_sync_write+173}
<ffffffff80186dc9>{may_open+105}
<ffffffff80136360>{autoremove_wake_function+0}
<ffffffff803803e9>{thread_return+41}
<ffffffff8019418e>{dnotify_parent+46}
<ffffffff80177b94>{vfs_write+228}
<ffffffff80177cc3>{sys_write+83}<ffffffff801105d6>{system_call+126}
Code: 48 8b 91 70 11 00 00 76 07 b8 00 00 00 80 eb 0a 48 b8 00 00
RIP <ffffffff8015df54>{kmem_getpages+132} RSP <000001017100da38>
CR2: 0000000000001170
Killed
[rudi@cpu10 /home]$ <1>Unable to handle kernel paging request at 0000000000001170 RIP:
<ffffffff8015ab8a>{free_pages+154}
PML4 179d73067 PGD 0
Oops: 0000 [2] SMP
CPU 1
Modules linked in: nvidia ipv6 autofs4 sunrpc binfmt_misc dm_mod button
battery ac ohci_hcy
Pid: 2078, comm: syslogd Tainted: P 2.6.9RU1.1
RIP: 0010:[<ffffffff8015ab8a>] <ffffffff8015ab8a>{free_pages+154}
RSP: 0018:000001017cdf9dd0 EFLAGS: 00010213
RAX: ffffffff7fffffff RBX: 000001016ac5b010 RCX: 0000000000000000
RDX: 000001007fc3b838 RSI: 0000000000000000 RDI: 000001016ac5b000
RBP: 000001016ac5b010 R08: 000001000000f000 R09: 000001017fd01f48
R10: 000001017fd01f40 R11: ffffffff8031b400 R12: 0000000000000000
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
FS: 0000002a957a3b00(0000) GS:ffffffff80512980(0000)knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000001170 CR3: 0000000037f08000 CR4: 00000000000006e0 Process syslogd (pid: 2078, threadinfo 000001017cdf8000, task 000001017e8757d0)
Stack: ffffffff8018b543 000001017e8cac80 0000000000000002
0000000000000001
ffffffff8018ba43 0000000000000000 0000000000000000
0000000000000001
0000000000000000 0000000000000000
Call Trace:<ffffffff8018b543>{poll_freewait+67}
<ffffffff8018ba43>{do_select+1027}
<ffffffff8018b550>{__pollwait+0}
<ffffffff80199868>{__writeback_single_inode+728}
<ffffffff8018bde5>{sys_select+885}<ffffffff801105d6>{system_call+126}
Code: 48 8b 91 70 11 00 00 76 0d b8 00 00 00 80 eb 10 66 66 90 66
RIP <ffffffff8015ab8a>{free_pages+154} RSP <000001017cdf9dd0>
CR2: 0000000000001170
Signature · 4 lines
rudi ============================================================= Rudolf Usselmann, ASICS World Services, http://www.asics.ws Your Partner for IP Cores, Design, Verification and Synthesis
Quoted 8 lines · Rudolf Usselmann
// -------------------------- eat mem ---------------------------------
#include "
stdio.h"
#include "stdlib.h"int
main(){int mem[10000];
Array of integers, you probably want int *mem[10000] if anything.
Quoted 5 lines
int i, n;
for(i=0;i<2000;i++) { printf("Doing alloc %0d ...\n",i); mem[i] = (int)malloc(1024*1024*1024);
Wrongly assigning pointer to signed integer, casting can sometimes be evil.
Quoted 4 lines
if(mem[i] == NULL)
printf("Malloc failed ...\n");
else
for(n=0;n<(1024*1024*1024);n=n+640) mem[i] = n;You lose the pointer value here and do not touch the allocated memory, which means that the VM isn't forced to commit the memory you allocated. You also want to be looking at the Committed_AS and Mapped fields of /proc/meminfo
Quoted 16 lines · Adam Heath
Quoted 6 lines · Rudolf Usselmann
int mem[10000];
int i, n;for(i=0;i<2000;i++) { printf("Doing alloc %0d ...\n",i); mem[i] = (int)malloc(1024*1024*1024);allocate a 1 gig block, but store it in an int array? That's wrong.
Quoted 4 lines
if(mem[i] == NULL) printf("Malloc failed ...\n"); else for(n=0;n<(1024*1024*1024);n=n+640) mem[i] = n;
Quoted 7 lines
You alter n, but then always only set mem[i], without varying i.
Your program is buggy, and memleaks.
Plus, the kernel uses overcommit by default, and since you aren't actually touching the memory you are allocating, you are only limited by the address space size allowed per process(depends on how you configured the kernel).
Thanks for pointing that out Adam ! Will this work better:
#include "stdio.h"
#include "stdlib.h"
int main() {
char *mem[10000];
int i, n;
for(i=0;i<2000;i++) {
printf("Doing alloc %0d ...\n",i);
mem[i] = (char *)malloc(1024*1024*1024);
if(mem[i] == NULL)
printf("Malloc failed ...\n");
else
for(n=0;n<(1024*1024*1024);n=n+640) mem[i][n] = n;}
while(1);
return(0);
}
Thanks !
rudi
I think I got it !!!
After fixing my mem eater program (per Adam and Zwane's suggestions), I get the attached "end-less" panic ... This happens immediately when I start the a.out ...
So does this make any sense to you guys ? What is wrong with my kernel/system ?
Thanks a lot guys !!!
Kind Regards,
rudi
Quoted 7 lines · Rudolf Usselmann
I think I got it !!!
After fixing my mem eater program (per Adam and Zwane's suggestions), I get the attached "end-less" panic ... This happens immediately when I start the a.out ... So does this make any sense to you guys ? What is wrong with my kernel/system ?
Thanks a lot guys !!!
Much better. Looks like someone broke the VM from that.
-- wli
Quoted 5 lines · William Lee Irwin III
Much better. Looks like someone broke the VM from that.
-- wli
:*)
Do I need to notify the maintainer of that specific code ? Or whats the standard procedure for bug reports ?
I could not identify a maintainer in the MAINTAINERS file, and memory.c/mm has a last modified in 1999 entry !
What should I do next ?
Thanks !
Kind Regards,
rudi