kernel panic and then oops
I am running Centos 4 with the latest kernel (updated yesterday) 2.6.9-5.0.5.EL on a P4 2.4 machine w/1 Gb ram and for the last couple weeks the machine has just been randomly hanging. I did upgrade the memory from a 512m stick to 2 1 gig sticks because the machine was regularly using 100% of the swap and would simply crawl. I know the thing to look at here obviously is the memory but i've already run memtest86+ and it reported that there was nothing wrong with the memory. I have tried running with a single stick of the 1Gb for the last couple days and tomorrow will be putting the 512 back in just for testing.
I searched the archives for the error, but as far as kernel debugging goes, i'm very new to it. I saw a lot of errors that looked similar but as i'm not exactly sure how to read all of the data from a crash I hoped i could get some help from the experts.
Generally when the machine hangs i get __nothing__ in the logs as far as a crash trace goes...the machine just seems to hang (sometimes i can ping it..other times not) and a hard reset is forced. When it comes up..the log is void of any info. Today the machine reset, I wasn't onsite, but as it was hanging i got the onsite person to give me the error:
<0> kernel panic not syncing: fatal exception in interrupt <0> kernel panic not syncing: arch/i386/kernel/irq.c:590 spin_is_locked on
initialized spinlock C03a5098
I'm sure there was other messages but this is all i got from him before he needed to get the machine running again. He reset the machine and about 2 minutes later the following messages showed up in the log:
Unable to handle kernel paging request at virtual address b9e91c8a Apr 21 16:12:44 wolverine kernel: printing eip: Apr 21 16:12:44 wolverine kernel: f88aed9a Apr 21 16:12:44 wolverine kernel: *pde = 00000000 Apr 21 16:12:44 wolverine kernel: Oops: 0002 [#1] Apr 21 16:12:44 wolverine kernel: Modules linked in: md5 ipv6 autofs4 iptable_mangle iptable_nat ipt_LOG ipt_state ip_conntrack iptable_filter ip_tables uhci_hcd ehci_hcd 8139too mii floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod Apr 21 16:12:44 wolverine kernel: CPU: 0 Apr 21 16:12:44 wolverine kernel: EIP: 0060:[<f88aed9a>] Not tainted VLI Apr 21 16:12:44 wolverine kernel: EFLAGS: 00010246 (2.6.9-5.0.5.EL) Apr 21 16:12:44 wolverine kernel: EIP is at ext3_try_to_allocate_with_rsv+0xd1/0x358 [ext3] Apr 21 16:12:44 wolverine kernel: eax: 00000000 ebx: f7e01aa8 ecx: 00158000 edx: 00000000 Apr 21 16:12:44 wolverine kernel: esi: e85bb669 edi: 00007000 ebp: 00000000 esp: e64abbd5 Apr 21 16:12:44 wolverine kernel: ds: 007b es: 007b ss: 0068 Apr 21 16:12:44 wolverine kernel: Process imapd (pid: 3502, threadinfo=e64ab000 task=e3ae38f0) Apr 21 16:12:44 wolverine kernel: Stack: 00000000 2b001580 94000000 00f69a32 00f6e4d8 00f6e4d8 01000000 00000000 Apr 21 16:12:44 wolverine kernel: 00f6e104 00f7e01a 00000070 5bf6e4d8 ecf88af2 00f5d512 68000070 4ce85bb6 Apr 21 16:12:44 wolverine kernel: e4e64abc 80c02a10 2bf500fc 68000000 00e85bb6 60f6e104 00f6e785 00000000 Apr 21 16:12:44 wolverine kernel: Call Trace: Apr 21 16:12:44 wolverine kernel: Code: 8d 98 a8 00 00 00 8b 42 14 01 c1
89 4c 24 04 8b 56 00 80 14 24 8b 46 34 89 44 24 14 8b 46 38 89 44 24 00 8b 04 24 8b
14 24 22 06 <18> 83 e2 01 09 c2 64 84 83 7c 24 18 00 74 20 84 ed 78 1c ff 74
Apr 21 16:12:44 wolverine kernel: <1>Unable to handle kernel paging request at virtual address b9e9168a Apr 21 16:12:44 wolverine kernel: printing eip: Apr 21 16:12:44 wolverine kernel: f88aed9a Apr 21 16:12:44 wolverine kernel: *pde = 00000000 Apr 21 16:12:44 wolverine kernel: Oops: 0002 [#2] Apr 21 16:12:44 wolverine kernel: Modules linked in: md5 ipv6 autofs4 iptable_mangle iptable_nat ipt_LOG ipt_state ip_conntrack iptable_filter ip_tables uhci_hcd ehci_hcd 8139too mii floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod Apr 21 16:12:44 wolverine kernel: CPU: 0 Apr 21 16:12:44 wolverine kernel: EIP: 0060:[<f88aed9a>] Not tainted VLI Apr 21 16:12:44 wolverine kernel: EFLAGS: 00010206 (2.6.9-5.0.5.EL) Apr 21 16:12:44 wolverine kernel: EIP is at ext3_try_to_allocate_with_rsv+0xd1/0x358 [ext3] Apr 21 16:12:44 wolverine kernel: eax: 00000430 ebx: f7e014a8 ecx: 00048000 edx: 000004b0 Apr 21 16:12:44 wolverine kernel: esi: f3f0ccd1 edi: 00002f54 ebp: 00000000 esp: f5f4fbd5 Apr 21 16:12:44 wolverine kernel: ds: 007b es: 007b ss: 0068 Apr 21 16:12:44 wolverine kernel: Process cleanup (pid: 2005, threadinfo=f5f4f000 task=f594e1b0) Apr 21 16:12:44 wolverine kernel: Stack: 000004b0 09000480 64000000 00f69a37 00f6ebac 00f6ebac 00000000 00000000 Apr 21 16:12:44 wolverine kernel: 00f6f3c4 54f7e014 0000002f 5bf6ebac a0f88af2 54f64e89 d000002f 4cf3f0cc Apr 21 16:12:44 wolverine kernel: 08f5f4fc 7e000000 09c01655 d0000000 00f3f0cc 20f6f3c4 00f69e31 d4000000 Apr 21 16:12:44 wolverine kernel: Call Trace: Apr 21 16:12:44 wolverine kernel: Code: 8d 98 a8 00 00 00 8b 42 14 01 c1
89 4c 24 04 8b 56 00 80 14 24 8b 46 34 89 44 24 14 8b 46 38 89 44 24 00 8b 04 24 8b
14 24 22 06 <18> 83 e2 01 09 c2 64 84 83 7c 24 18 00 74 20 84 ed 78 1c ff 74
at this point i was still logged in but after issuing a shutdown command..it just kept running. Same thing when the onsite person tried to shut down as well.
After the reboot everything seems fine of course but i get these errors every time that also concern me:
Apr 21 15:31:01 wolverine kernel: PCI: Probing PCI hardware Apr 21 15:31:01 wolverine kernel: PCI: Probing PCI hardware (bus 00) Apr 21 15:31:01 wolverine kernel: PCI: Using IRQ router default [1106/3128] at 0000:00:00.0 Apr 21 15:31:01 wolverine kernel: PCI: IRQ 0 for device 0000:00:0f.0 doesn't match PIRQ mask - try pci=usepirqmask Apr 21 15:31:01 wolverine kernel: PCI: IRQ 0 for device 0000:00:0f.1 doesn't match PIRQ mask - try pci=usepirqmask Apr 21 15:31:01 wolverine kernel: PCI: IRQ 0 for device 0000:00:0f.2
doesn't match PIRQ mask - try pci=usepirqmask
and further down:
Apr 21 15:31:03 wolverine kernel: 8139too Fast Ethernet driver 0.9.27 Apr 21 15:31:03 wolverine kernel: eth0: RealTek RTL8139 at 0xb400, 00:40:63:c0:2c:fb, IRQ 10 Apr 21 15:31:03 wolverine kernel: PCI: IRQ 0 for device 0000:00:0f.2 doesn't match PIRQ mask - try pci=usepirqmask Apr 21 15:31:03 wolverine kernel: PCI: No IRQ known for interrupt pin C of device 0000:00:0f.2. Please try using pci=biosirq. Apr 21 15:31:03 wolverine kernel: ehci_hcd 0000:00:0f.2: Found HC with no IRQ. Check BIOS/PCI 0000:00:0f.2 setup! Apr 21 15:31:04 wolverine kernel: USB Universal Host Controller Interface driver v2.2 Apr 21 15:31:04 wolverine kernel: PCI: IRQ 0 for device 0000:00:0f.0 doesn't match PIRQ mask - try pci=usepirqmask Apr 21 15:31:04 wolverine kernel: PCI: No IRQ known for interrupt pin A of device 0000:00:0f.0. Please try using pci=biosirq. Apr 21 15:31:04 wolverine kernel: uhci_hcd 0000:00:0f.0: Found HC with no IRQ. Check BIOS/PCI 0000:00:0f.0 setup! Apr 21 15:31:04 wolverine kernel: PCI: IRQ 0 for device 0000:00:0f.1 doesn't match PIRQ mask - try pci=usepirqmask Apr 21 15:31:04 wolverine kernel: PCI: No IRQ known for interrupt pin B of device 0000:00:0f.1. Please try using pci=biosirq. Apr 21 15:31:04 wolverine kernel: uhci_hcd 0000:00:0f.1: Found HC with
no IRQ. Check BIOS/PCI 0000:00:0f.1 setup!
I will repost w/the full dmesg if it's needed.
TIA,
Aaron Martinez
P.S.
I'm not a developer, but i would like to learn more about how to debug these errors, at least so that i may post better questions in the future, any pointers to info in this regard would be appreciated.
Hi Aaron,
Quoted 2 lines
Unable to handle kernel pagingrequest at virtual address
b9e9168a
This message is comming because a page fault has occured at virtual address b9e9168a . This means the requested page was not available in the virtual memory. When i got these kind of errors in my WLAN driver kernel was crashing. After decoding the oops i found out that i was tryng to free some invalid memory !! Once i fixed this bug my module was working fine. Regards,
Lavin
Quoted 167 lines · Aaron P. Martinez
I am running Centos 4 with the latest kernel (updated yesterday) 2.6.9-5.0.5.EL on a P4 2.4 machine w/1 Gb ram and for the last couple weeks the machine has just been randomly hanging. I did upgrade the memory from a 512m stick to 2 1 gig sticks because the machine was regularly using 100% of the swap and would simply crawl. I know the thing to look at here obviously is the memory but i've already run memtest86+ and it reported that there was nothing wrong with the memory. I have tried running with a single stick of the 1Gb for the last couple days and tomorrow will be putting the 512 back in just for testing.
I searched the archives for the error, but as far as kernel debugging goes, i'm very new to it. I saw a lot of errors that looked similar but as i'm not exactly sure how to read all of the data from a crash I hoped i could get some help from the experts.
Generally when the machine hangs i get
__nothing__in the logs as far as a crash trace goes...the machine just seems to hang (sometimes i can ping it..other times not) and a hard reset is forced. When it comes up..the log is void of any info. Today the machine reset, I wasn't onsite, but as it was hanging i got the onsite person to give me the error:<0> kernel panic not syncing: fatal exception in interrupt <0> kernel panic not syncing:
arch/i386/kernel/irq.c:590spin_is_lockedon
initialized spinlock C03a5098I'm sure there was other messages but this is all i got from him before he needed to get the machine running again. He reset the machine and about 2 minutes later the following messages showed up in the log:
Unable to handle kernel paging request at virtual address b9e91c8a Apr 21 16:12:44 wolverine kernel: printing eip: Apr 21 16:12:44 wolverine kernel: f88aed9a Apr 21 16:12:44 wolverine kernel: *pde = 00000000 Apr 21 16:12:44 wolverine kernel: Oops: 0002 [#1] Apr 21 16:12:44 wolverine kernel: Modules linked in: md5 ipv6 autofs4 iptable_mangle iptable_nat ipt_LOG ipt_state ip_conntrack iptable_filter ip_tables uhci_hcd ehci_hcd 8139too mii floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod Apr 21 16:12:44 wolverine kernel: CPU: 0 Apr 21 16:12:44 wolverine kernel: EIP: 0060:[<f88aed9a>] Not tainted VLI Apr 21 16:12:44 wolverine kernel: EFLAGS: 00010246 (2.6.9-5.0.5.EL) Apr 21 16:12:44 wolverine kernel: EIP is at ext3_try_to_allocate_with_rsv+0xd1/0x358 [ext3] Apr 21 16:12:44 wolverine kernel: eax: 00000000 ebx: f7e01aa8 ecx: 00158000 edx: 00000000 Apr 21 16:12:44 wolverine kernel: esi: e85bb669 edi: 00007000 ebp: 00000000 esp: e64abbd5 Apr 21 16:12:44 wolverine kernel: ds: 007b es: 007b ss: 0068 Apr 21 16:12:44 wolverine kernel: Process imapd (pid: 3502, threadinfo=e64ab000 task=e3ae38f0) Apr 21 16:12:44 wolverine kernel: Stack: 00000000 2b001580 94000000 00f69a32 00f6e4d8 00f6e4d8 01000000 00000000 Apr 21 16:12:44 wolverine kernel: 00f6e104 00f7e01a 00000070 5bf6e4d8 ecf88af2 00f5d512 68000070 4ce85bb6 Apr 21 16:12:44 wolverine kernel: e4e64abc 80c02a10 2bf500fc 68000000 00e85bb6 60f6e104 00f6e785 00000000 Apr 21 16:12:44 wolverine kernel: Call Trace: Apr 21 16:12:44 wolverine kernel: Code: 8d 98 a8 00 00 00 8b 42 14 01 c189 4c 24 04 8b 56 00 80 14 24 8b 46 34 89 44 24 14 8b 46 38 89 44 24 00 8b 04 24 8b
14 24 22 06 <18> 83 e2 01 09 c2 64 84 83 7c 24 18 00 74 20 84 ed 78 1c ff 74Apr 21 16:12:44 wolverine kernel: <1>Unable to handle kernel paging request at virtual address b9e9168a Apr 21 16:12:44 wolverine kernel: printing eip: Apr 21 16:12:44 wolverine kernel: f88aed9a Apr 21 16:12:44 wolverine kernel: *pde = 00000000 Apr 21 16:12:44 wolverine kernel: Oops: 0002 [#2] Apr 21 16:12:44 wolverine kernel: Modules linked in: md5 ipv6 autofs4 iptable_mangle iptable_nat ipt_LOG ipt_state ip_conntrack iptable_filter ip_tables uhci_hcd ehci_hcd 8139too mii floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod Apr 21 16:12:44 wolverine kernel: CPU: 0 Apr 21 16:12:44 wolverine kernel: EIP: 0060:[<f88aed9a>] Not tainted VLI Apr 21 16:12:44 wolverine kernel: EFLAGS: 00010206 (2.6.9-5.0.5.EL) Apr 21 16:12:44 wolverine kernel: EIP is at ext3_try_to_allocate_with_rsv+0xd1/0x358 [ext3] Apr 21 16:12:44 wolverine kernel: eax: 00000430 ebx: f7e014a8 ecx: 00048000 edx: 000004b0 Apr 21 16:12:44 wolverine kernel: esi: f3f0ccd1 edi: 00002f54 ebp: 00000000 esp: f5f4fbd5 Apr 21 16:12:44 wolverine kernel: ds: 007b es: 007b ss: 0068 Apr 21 16:12:44 wolverine kernel: Process cleanup (pid: 2005, threadinfo=f5f4f000 task=f594e1b0) Apr 21 16:12:44 wolverine kernel: Stack: 000004b0 09000480 64000000 00f69a37 00f6ebac 00f6ebac 00000000 00000000 Apr 21 16:12:44 wolverine kernel: 00f6f3c4 54f7e014 0000002f 5bf6ebac a0f88af2 54f64e89 d000002f 4cf3f0cc Apr 21 16:12:44 wolverine kernel: 08f5f4fc 7e000000 09c01655 d0000000 00f3f0cc 20f6f3c4 00f69e31 d4000000 Apr 21 16:12:44 wolverine kernel: Call Trace: Apr 21 16:12:44 wolverine kernel: Code: 8d 98 a8 00 00 00 8b 42 14 01 c189 4c 24 04 8b 56 00 80 14 24 8b 46 34 89 44 24 14 8b 46 38 89 44 24 00 8b 04 24 8b
14 24 22 06 <18> 83 e2 01 09 c2 64 84 83 7c 24 18 00 74 20 84 ed 78 1c ff 74at this point i was still logged in but after issuing a shutdown command..it just kept running. Same thing when the onsite person tried to shut down as well.
After the reboot everything seems fine of course but i get these errors every time that also concern me:
Apr 21 15:31:01 wolverine kernel: PCI: Probing PCI hardware Apr 21 15:31:01 wolverine kernel: PCI: Probing PCI hardware (bus 00) Apr 21 15:31:01 wolverine kernel: PCI: Using IRQ router default [1106/3128] at 0000:00:00.0 Apr 21 15:31:01 wolverine kernel: PCI: IRQ 0 for device 0000:00:0f.0 doesn't match PIRQ mask - try pci=usepirqmask Apr 21 15:31:01 wolverine kernel: PCI: IRQ 0 for device 0000:00:0f.1 doesn't match PIRQ mask - try pci=usepirqmask Apr 21 15:31:01 wolverine kernel: PCI: IRQ 0 for device 0000:00:0f.2doesn't match PIRQ mask - try pci=usepirqmask
and further down:
Apr 21 15:31:03 wolverine kernel: 8139too Fast Ethernet driver 0.9.27 Apr 21 15:31:03 wolverine kernel: eth0: RealTek RTL8139 at 0xb400, 00:40:63:c0:2c:fb, IRQ 10 Apr 21 15:31:03 wolverine kernel: PCI: IRQ 0 for device 0000:00:0f.2 doesn't match PIRQ mask - try pci=usepirqmask Apr 21 15:31:03 wolverine kernel: PCI: No IRQ known for interrupt pin C of device 0000:00:0f.2. Please try using pci=biosirq. Apr 21 15:31:03 wolverine kernel: ehci_hcd 0000:00:0f.2: Found HC with no IRQ. Check BIOS/PCI 0000:00:0f.2 setup! Apr 21 15:31:04 wolverine kernel: USB Universal Host Controller Interface driver v2.2 Apr 21 15:31:04 wolverine kernel: PCI: IRQ 0 for device 0000:00:0f.0 doesn't match PIRQ mask - try pci=usepirqmask Apr 21 15:31:04 wolverine kernel: PCI: No IRQ known for interrupt pin A of device 0000:00:0f.0. Please try using pci=biosirq. Apr 21 15:31:04 wolverine kernel: uhci_hcd 0000:00:0f.0: Found HC with no IRQ. Check BIOS/PCI 0000:00:0f.0 setup! Apr 21 15:31:04 wolverine kernel: PCI: IRQ 0 for device 0000:00:0f.1 doesn't match PIRQ mask - try pci=usepirqmask Apr 21 15:31:04 wolverine kernel: PCI: No IRQ known for interrupt pin B of device 0000:00:0f.1. Please try using pci=biosirq. Apr 21 15:31:04 wolverine kernel: uhci_hcd 0000:00:0f.1: Found HC withno IRQ. Check BIOS/PCI 0000:00:0f.1 setup!
I will repost w/the full dmesg if it's needed.
TIA,
Aaron Martinez
P.S.
I'm not a developer, but i would like to learn more about how to debug these errors, at least so that i may post better questions in the future, any pointers to info in this regard would be appreciated.-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 5 lines
P.Lavin Software Engineer, Redpine Signals ,Inc. Hyderabad. http://www.redpinesignals.com
Quoted 33 lines · Aaron P. Martinez
I am running Centos 4 with the latest kernel (updated yesterday) 2.6.9-5.0.5.EL on a P4 2.4 machine w/1 Gb ram and for the last couple weeks the machine has just been randomly hanging. I did upgrade the memory from a 512m stick to 2 1 gig sticks because the machine was regularly using 100% of the swap and would simply crawl. I know the thing to look at here obviously is the memory but i've already run memtest86+ and it reported that there was nothing wrong with the memory. I have tried running with a single stick of the 1Gb for the last couple days and tomorrow will be putting the 512 back in just for testing.
I searched the archives for the error, but as far as kernel debugging goes, i'm very new to it. I saw a lot of errors that looked similar but as i'm not exactly sure how to read all of the data from a crash I hoped i could get some help from the experts.
Generally when the machine hangs i get
__nothing__in the logs as far as a crash trace goes...the machine just seems to hang (sometimes i can ping it..other times not) and a hard reset is forced. When it comes up..the log is void of any info. Today the machine reset, I wasn't onsite, but as it was hanging i got the onsite person to give me the error:<0> kernel panic not syncing: fatal exception in interrupt <0> kernel panic not syncing:
arch/i386/kernel/irq.c:590spin_is_lockedon
initialized spinlock C03a5098I'm sure there was other messages but this is all i got from him before he needed to get the machine running again. He reset the machine and about 2 minutes later the following messages showed up in the log:
Quoted 6 lines
Unable to handle kernel paging request at virtual address b9e91c8a Apr 21 16:12:44 wolverine kernel: printing eip: Apr 21 16:12:44 wolverine kernel: f88aed9a Apr 21 16:12:44 wolverine kernel: *pde = 00000000 Apr 21 16:12:44 wolverine kernel: Oops: 0002 [#1] Apr 21 16:12:44 wolverine kernel: Modules linked in: md5 ipv6 autofs4
It's highly likely that you may have faulty memory, i suggest running for an extended period in the previous 512M configuration. If that works, try installing the new memory and resetting the BIOS so that you end up with conservative RAM timing settings.
Zwane