Oops linux 2.4.23-pre6 on amd64

It jumped to nirvana, probably because someone passed crap to pci_announce_device.

My first guess would be a non matching module. Do a make distclean and recompile/reinstall everything.

But the decode is useless because the module in question is not loaded.

Can you load the module whatever it is manually and then decode the oops while it's still loaded? Or better compile in all USB statically and see if it oopses too.

Your legacy USB problems are very likely BIOS bugs.

-Andi

Here's the decode with it loaded. The oops happens during the initialisation of ehci-hcd (from an init=/bin/bash clean system, I did modprobe usb-uhci then modprobe ehci-hcd and it oopsed).

Well this BIOS does insist I have 'sourth' bridge, so I don't have high hopes on that score :)

Tony

ksymoops 2.4.9 on x86_64 2.4.23-pre6-amd64. Options used

     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.23-pre6-amd64/ (default)
     -m /boot/System.map-2.4.23-pre6-amd64 (default)
     -t elf64-x86-64 -a x86-64

Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options.

Oct  4 19:23:06 (none) kernel: Unable to handle kernel paging request<1> at 00000000a000a700 RIP: [<00000000a000a700>]PML4 3ef32067 PGD 0
Oct  4 19:23:06 (none) kernel: Oops: 0010
Oct  4 19:23:06 (none) kernel: CPU 0
Oct  4 19:23:06 (none) kernel: Pid: 104, comm: modprobe Not tainted
Oct  4 19:23:06 (none) kernel: RIP: 0010:[<00000000a000a700>]
Oct  4 19:23:06 (none) kernel: RSP: 0000:000001003efddde0  EFLAGS: 00010246
Oct  4 19:23:06 (none) kernel: RAX: ffffffffa00140e0 RBX: ffffffffa0014560 RCX: 0000000000000007
Oct  4 19:23:06 (none) kernel: RDX: ffffffffa00140e0 RSI: ffffffffa00140e0 RDI: 000001000262f800
Oct  4 19:23:06 (none) kernel: RBP: 000001000262f800 R08: 0000000000001000 R09: 00000100025a2b08
Oct  4 19:23:06 (none) kernel: R10: 000000000003ef30 R11: 0000010001000048 R12: ffffffffa00140e0
Oct  4 19:23:06 (none) kernel: R13: 0000000000000000 R14: ffffff0000a11000 R15: 000001003eed9140
Oct  4 19:23:06 (none) kernel: FS:  0000000000000000(0000) GS:ffffffff8034b540(0000) knlGS:0000000000000000
Oct  4 19:23:06 (none) kernel: CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
Oct  4 19:23:06 (none) kernel: CR2: 00000000a000a700 CR3: 0000000000101000 CR4: 00000000000006e0
Oct  4 19:23:06 (none) kernel: Process modprobe (pid: 104, stackpage=1003efdd000)
Oct  4 19:23:06 (none) kernel: Stack: 000001003efddde0 0000000000000000 ffffffff80202fce 0000000000000007
Oct  4 19:23:06 (none) kernel:        000001000262f800 ffffffffa0014560 0000000000000000 00000000000000b8
Oct  4 19:23:06 (none) kernel:        ffffffff80203048 ffffffffffffffea ffffffffa0010000 0000000008086dc8
Oct  4 19:23:06 (none) kernel: Call Trace: [<ffffffff80202fce>] [<ffffffffa0014560>]
Oct  4 19:23:06 (none) kernel:        [<ffffffff80203048>] [<ffffffffa0013f4d>] [<ffffffff8012008e>]
Oct  4 19:23:06 (none) kernel:        [<ffffffff8012e21c>] [<ffffffffa00100b8>] [<ffffffff80189963>]
Oct  4 19:23:06 (none) kernel: Code:  Bad RIP value.
Oct  4 19:23:06 (none) kernel: CR2: 00000000a000a700

Warning (Oops_read): Code line not seen, dumping what data is available

Trace; ffffffff80202fce <pci_announce_device+3e/60> Trace; ffffffffa0014560 <[usb-uhci]uhci_pci_probe+d0/110> Trace; ffffffff80203048 <pci_register_driver+58/80> Trace; ffffffffa0013f4d <[usb-uhci]uhci_interrupt+15d/180> Trace; ffffffff8012008e <sys_init_module+62e/6f0> Trace; ffffffff8012e21c <handle_mm_fault+bc/180> Trace; ffffffffa00100b8 <[usbcore]hcd_data_lock+7a4/7ac> Trace; ffffffff80189963 <ia32_syscall+67/71>

2 warnings issued. Results may not be reliable.

Oops, scratch that last one... It's invalid too as I used the original oops rather than the new one. This one's the right one (honest!).

Tony

ksymoops 2.4.9 on x86_64 2.4.23-pre6-amd64. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.23-pre6-amd64/ (default) -m /boot/System.map-2.4.23-pre6-amd64 (default)

Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options.

Unable to handle kernel paging request<1> at 00000000a000a700 RIP: [<00000000a000a700>]PML4 3f145067 PGD 0
Oops: 0010
CPU 0
Pid: 146, comm: modprobe.moduti Not tainted
RIP: 0010:[<00000000a000a700>]
Using defaults from ksymoops -t elf32-i386 -a i386
RSP: 0000:000001003f165de0  EFLAGS: 00010246
RAX: ffffffffa00220e0 RBX: ffffffffa0022560 RCX: 0000000000000007
RDX: ffffffffa00220e0 RSI: ffffffffa00220e0 RDI: 000001000262f800
RBP: 000001000262f800 R08: 0000000000000001 R09: ffffffffa00224e0
R10: ffffffffa0021fa0 R11: 0000000000000000 R12: ffffffffa00220e0
R13: 0000000000000000 R14: ffffff0000a11000 R15: 0000010002767c00
FS:  0000000000000000(0000) GS:ffffffff8034b540(0000) knlGS:0000000000000000
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000a000a700 CR3: 0000000000101000 CR4: 00000000000006e0
Process modprobe.moduti (pid: 146, stackpage=1003f165000)
Stack: 000001003f165de0 0000000000000000 ffffffff80202fce 0000000000000007
       000001000262f800 ffffffffa0022560 0000000000000000 00000000000000b8
       ffffffff80203048 ffffffffffffffea ffffffffa001e000 0000000008088758
Call Trace: [<ffffffff80202fce>] [<ffffffffa0022560>]
       [<ffffffff80203048>] [<ffffffffa0021f4d>] [<ffffffff8012008e>]
       [<ffffffff8012e21c>] [<ffffffffa001e0b8>] [<ffffffff80189963>]
Code:  Bad RIP value.

CR2: 00000000a000a700
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; ffffffff80202fce <pci_announce_device+3e/60> Trace; ffffffffa0022560 <[ehci-hcd]ehci_pci_driver+0/4f> Trace; ffffffff80203048 <pci_register_driver+58/80> Trace; ffffffffa0021f4d <[ehci-hcd]init+d/40> Trace; ffffffff8012008e <sys_init_module+62e/6f0> Trace; ffffffff8012e21c <handle_mm_fault+bc/180> Trace; ffffffffa001e0b8 <[keybdev].data.end+919/921> Trace; ffffffff80189963 <ia32_syscall+67/71>

2 warnings issued. Results may not be reliable.

I cannot see anything obviously wrong. At which kernel version did the problem start? And is your compiler version known to be not buggy ?

-Andi

AFAIK there is only one version that supports compiling to amd64 (only one for debian anyway) - I'm on 3.3.2-0pre4.biarch1. It is a bit flaky (in 32bit it'll happily do a make -j255 without worrying... going to 64 bit needs about 10 attempts to do a single compile because it keeps falling over with internal compiler errors/segfaults/etc.).

The problem with a brand new architecture is until the toolset stabliizes I have to cross my fingers and hope. Certainly if it's compiling dodgy code it'll explain why Alsa is about as stable as a drunk in a high wind...

OK I'll forget it for now & go back to 32bit for a couple of months then try again.

Tony

That doesn't sound good. Why did you not mention this first, it's unlikely that such a compiler produces a working kernel. When the segfaults are not deterministic (go away when you try again) then you likely have some hardware problem, like bad DIMMs (run memtest86 for 12+hours to make sure)

To rule out the compiler you can use the compiler/binutils from

ftp.suse.com:/pub/suse/x86-64/supplementary/CrossTools/8.1-i386/

That's rpms for SuSE 8.1/i386, but I suspect you install it on Debian with rpm2cpio or somesuch. That's an older gcc 3.2 that is known to work.

Then just put /opt/cross/bin in your $PATH and compile with CROSS_COMPILE=x86_64-linux- ARCH=x86_64

-Andi

It's had 24 hours under memtest86 (I had to RMA one memory stick when I first got the machine) and as I mentioned handles a make -j255 in 32bit mode without a hitch. The kernel does work apart from that module (and floppy.o which I discovered later does much the same thing as the ehci-hcd.o).

OK I'll try that.

Tony

OK I built with that and here are the results:

1. The ehci-hcd driver fails in exactly the same place.
2. It was still v. unstable, which led me to investigate why (since I'm pretty sure the hardware is good & the suse compiler is supposed to be a good one). I started stripping out options until eventually I found that it's devfs that's the culprit - with that enabled I get random compile errors every few seconds. With it disabled the compile works perfectly, even with the debian compiler (tried -j20 and -j255 and both passed).

My first guess was you can't use a 32bit devfsd with a 64bit kernel, but stopping devfsd didn't seem to make a whole lot of difference to the stability... only compiling out the entire devfs system solved it.

I suppose it could be insmod breaking the ehci-hcd... I'll see if I can find a pure 64bit one (presumably suse have one) rather than the biarch one that debian uses.

Tony

Thanks for tracking this down. I would have never noticed because I don't use devfs.

Marcelo, any ideas? Do you get broken devfs reports for other 64bit architectures too?

AFAIK devfs is unmaintained and I don't really plan to maintain it myself. My proposal is to just disable it in the configuration for x86-64 for now.

Very likely the devfs code in the kernel is buggy. It is known to be race hell, I wouldn't be surprised if it has 64bit bugs too.

-Andi

OK well I can do without devfs anyway.

The ehci-hcd problem I've eventually tracked down to a bug in the biarch modutils - compiling a pure 64bit version of modutils solves all the module loading problems, and even alsa started working :)

Tony

Nope, never got such reports.

What problem are you seeing Tony? Oopsing right? Where is the oops output?

Nod

Yes.

Random segfaults, internal compiler errors, general instability. It was impossible to compile a kernel without something breaking... when oopses happend they usually happened 3 or 4 in a row.

It's been steady as a rock since I removed devfs...

Tony

Hello all,

I'm experiencing a kernel panic in RedHat Linux 8.0 with 2.4.20 kernel when calling the function devinet_ioctl() with the SIOCSIFADDR command from within a kernel module. The function is called within the scope of userspace addressing in the following manner.

int res;
mm_segment_t oldfs = get_fs();
set_fs(get_ds());
res = devinet_ioctl(cmd,arg);
set_fs(oldfs);

This is how ipconfig.c illustrates how to use this function from within a kernel module, see ic_dev_ioctl() function in the source file.

When running in the scope of a kgdb patched kernel, I see the kernel panic when this function is called from within my kernel module. The panic happens because alloc_skb is being called non-atomically from an interrupt.

Is there anything grossly wrong that I am doing? Am I supposed to do something else before calling this function?

Thanks,
Vishwas.