2.6.9 oops in poll()?
My stock debian kernel (2.6.9-1-686) crashed this morning while running bittornado (btlaunchmanycurses). At that moment only one torrent was being downloaded. It looks to me like the problem is somewhere in poll() in combination with threading.
I haven't tested it using 2.6.10 as I don't even know if I can reproduce this using 2.6.9. If I do, I'll let you know.
I've included the raw oops and the output of ksymoops below. I'm not subscribed to lkml.
===================== RAW ======================
Jan 3 07:07:26 wojas kernel: c02b5513 Jan 3 07:07:26 wojas kernel: PREEMPT
Jan 3 07:07:26 wojas kernel: Modules linked in: sch_ingress cls_u32 sch_sfq sch_cbq ip6table_filter ip6t_LOG ip6_tables ipt_limit lp msr cpuid ipt_state ipt_MASQUERADE ipt_LOG iptable_mangle iptable_filter dummy af_packet es1371 ac97_codec pci_hotplug intel_agp floppy pcspkr rtc sd_mod reiserfs ext2 dm_mod capability commoncap tsdev mousedev joydev evdev wacom usbhid psmouse usb_storage uhci_hcd usbcore eeprom lm75 i2c_sensor i2c_dev i2c_piix4 i2c_core aha152x ipv6 binfmt_misc ip_nat_ftp iptable_nat ip_conntrack_ftp ip_conntrack mga agpgart snd_pcm_oss snd_mixer_oss snd_ens1371 snd_rawmidi snd_seq_device snd_pcm snd_timer snd_page_alloc snd_ac97_codec snd gameport ipt_REJECT ip_tables ne2k_pci 8390 via_rhine mii crc32 sound soundcore sg scsi_mod parport_pc parport ide_cd cdrom ext3 jbd mbcache ide_generic piix ide_disk ide_core unix fbcon font vesafb cfbcopyarea cfbimgblt cfbfillrect
Jan 3 07:07:26 wojas kernel: CPU: 0 Jan 3 07:07:26 wojas kernel: EIP: 0060:[__func__.4+64363/135712] Not tainted VLI Jan 3 07:07:26 wojas kernel: EFLAGS: 00010246 (2.6.9-1-686) Jan 3 07:07:26 wojas kernel: EIP is at 0xc02b5513 Jan 3 07:07:26 wojas kernel: eax: 00000000 ebx: c0325c00 ecx: c61e57e0 edx: d35b85e0 Jan 3 07:07:26 wojas kernel: esi: c46082b9 edi: c61e57e4 ebp: 00000056 esp: d5f35f1c Jan 3 07:07:26 wojas kernel: ds: 007b es: 007b ss: 0068 Jan 3 07:07:26 wojas kernel: Process python (pid: 30488, threadinfo=d5f34000 task=c0f0faa0) Jan 3 07:07:26 wojas kernel: Stack: c022db99 c61e57e0 d35b85e0 00000000 00000145 c016c995 c61e57e0 00000000 Jan 3 07:07:26 wojas kernel: c4608000 d5f35f64 d5f35f68 0000001a c016ca0a 00000063 c4608008 d5f35f64 Jan 3 07:07:26 wojas kernel: d5f35f68 d5f34000 00000000 00000002 00000000 c4608000 000001ff 082b0d90 Jan 3 07:07:26 wojas kernel: Call Trace: Jan 3 07:07:26 wojas kernel: [sock_poll+41/64] sock_poll+0x29/0x40 Jan 3 07:07:26 wojas kernel: [do_pollfd+149/160] do_pollfd+0x95/0xa0 Jan 3 07:07:26 wojas kernel: [do_poll+106/208] do_poll+0x6a/0xd0 Jan 3 07:07:26 wojas kernel: [sys_poll+353/576] sys_poll+0x161/0x240 Jan 3 07:07:26 wojas kernel: [__pollwait+0/208] __pollwait+0x0/0xd0 Jan 3 07:07:26 wojas kernel: [syscall_call+7/11] syscall_call+0x7/0xb Jan 3 07:07:26 wojas kernel: Code: 79 70 74 6f 2f 63 69 70 68 65 72 2e 63 00 69 6e 63 6c 75 64 65 2f 6c 69 6e 75 78 2f 63 72 79 70 74 6f 2e 68 00 6e 61 6d 65 20 20 <20> 20 20 20 20 20 20 3a 20 25 73 0a 00 6d 6f 64 75 6c 65 20 20
===================== KSYMOOPS ======================
ksymoops 2.4.9 on i686 2.6.9-1-686. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.6.9-1-686/ (default)
-m /boot/System.map-2.6.9-1-686 (default)
Error (regular_file): read_ksyms stat /proc/ksyms failed No modules in ksyms, skipping objects
No ksyms, skipping lsmod
Jan 3 07:07:26 wojas kernel: c02b5513 Jan 3 07:07:26 wojas kernel: CPU: 0 Jan 3 07:07:26 wojas kernel: EIP: 0060:[__func__.4+64363/135712] Not tainted VLI Jan 3 07:07:26 wojas kernel: EFLAGS: 00010246 (2.6.9-1-686) Jan 3 07:07:26 wojas kernel: eax: 00000000 ebx: c0325c00 ecx: c61e57e0 edx: d35b85e0 Jan 3 07:07:26 wojas kernel: esi: c46082b9 edi: c61e57e4 ebp: 00000056 esp: d5f35f1c Jan 3 07:07:26 wojas kernel: ds: 007b es: 007b ss: 0068 Jan 3 07:07:26 wojas kernel: Stack: c022db99 c61e57e0 d35b85e0 00000000 00000145 c016c995 c61e57e0 00000000 Jan 3 07:07:26 wojas kernel: c4608000 d5f35f64 d5f35f68 0000001a c016ca0a 00000063 c4608008 d5f35f64 Jan 3 07:07:26 wojas kernel: d5f35f68 d5f34000 00000000 00000002 00000000 c4608000 000001ff 082b0d90 Jan 3 07:07:26 wojas kernel: Call Trace:
Warning (Oops_read): Code line not seen, dumping what data is available
Quoted 6 lines
Quoted 6 lines
ebx;
c0325c00<devinet_sysctl+460/4e0>
ecx;c61e57e0<pg0+5e3b7e0/3fc54400>
edx;d35b85e0<pg0+1320e5e0/3fc54400>
esi;c46082b9<pg0+425e2b9/3fc54400>
edi;c61e57e4<pg0+5e3b7e4/3fc54400>
esp;d5f35f1c<pg0+15b8bf1c/3fc54400>
Jan 3 07:07:26 wojas kernel: Code: 79 70 74 6f 2f 63 69 70 68 65 72 2e 63 00 69 6e 63 6c 75 64 65 2f 6c 69 6e 75 78 2f 63 72 79 70 74 6f 2e 68 00 6e 61 6d 65 20 20 <20> 20 20 20 20 20 20 3a 20 25 73 0a 00 6d 6f 64 75 6c 65 20 20
Using defaults from ksymoops -t elf32-i386 -a i386
Code; ffffffd5 <__kernel_rt_sigreturn+1b95/????>
00000000 <_EIP>:
Code; ffffffd5 <__kernel_rt_sigreturn+1b95/????>
0: 79 70 jns 72 <_EIP+0x72>
Code; ffffffd7 <__kernel_rt_sigreturn+1b97/????>
2: 74 6f je 73 <_EIP+0x73>
Code; ffffffd9 <__kernel_rt_sigreturn+1b99/????>
4: 2f das
Code; ffffffda <__kernel_rt_sigreturn+1b9a/????>
5: 63 69 70 arpl %bp,0x70(%ecx)
Code; ffffffdd <__kernel_rt_sigreturn+1b9d/????>
8: 68 65 72 2e 63 push $0x632e7265
Code; ffffffe2 <__kernel_rt_sigreturn+1ba2/????>
d: 00 69 6e add %ch,0x6e(%ecx)
Code; ffffffe5 <__kernel_rt_sigreturn+1ba5/????>
10: 63 6c 75 64 arpl %bp,0x64(%ebp,%esi,2)
Code; ffffffe9 <__kernel_rt_sigreturn+1ba9/????>
14: 65 gs
Code; ffffffea <__kernel_rt_sigreturn+1baa/????>
15: 2f das
Code; ffffffeb <__kernel_rt_sigreturn+1bab/????>
16: 6c insb (%dx),%es:(%edi)
Code; ffffffec <__kernel_rt_sigreturn+1bac/????>
17: 69 6e 75 78 2f 63 72 imul $0x72632f78,0x75(%esi),%ebp
Code; fffffff3 <__kernel_rt_sigreturn+1bb3/????>
1e: 79 70 jns 90 <_EIP+0x90>
Code; fffffff5 <__kernel_rt_sigreturn+1bb5/????>
20: 74 6f je 91 <_EIP+0x91>
Code; fffffff7 <__kernel_rt_sigreturn+1bb7/????>
22: 2e cs
Code; fffffff8 <__kernel_rt_sigreturn+1bb8/????>
23: 68 00 6e 61 6d push $0x6d616e00
Code; fffffffd <__kernel_rt_sigreturn+1bbd/????>
28: 65 20 20 and %ah,%gs:(%eax)
Code; 00000000 Before first symbol
2b: 20 20 and %ah,(%eax)
Code; 00000002 Before first symbol
2d: 20 20 and %ah,(%eax)
Code; 00000004 Before first symbol
2f: 20 20 and %ah,(%eax)
Code; 00000006 Before first symbol
31: 20 3a and %bh,(%edx)
Code; 00000008 Before first symbol
33: 20 25 73 0a 00 6d and %ah,0x6d000a73
Code; 0000000e Before first symbol
39: 6f outsl %ds:(%esi),(%dx)
Code; 0000000f Before first symbol
3a: 64 fs
Code; 00000010 Before first symbol
3b: 75 6c jne a9 <_EIP+0xa9>
Code; 00000012 Before first symbol
3d: 65 20 20 and %ah,%gs:(%eax)
2 warnings and 1 error issued. Results may not be reliable.
Kind regards,
Signature · 4 lines
Konrad Wojas .~. ~ [email protected] / V \ ~ /( )\ :wq GnuPG key 0x588C85B1 ^ ^
Quoted 9 lines · Konrad Wojas
My stock debian kernel (2.6.9-1-686) crashed this morning while running bittornado (btlaunchmanycurses). At that moment only one torrent was being downloaded. It looks to me like the problem is somewhere in
poll()in combination with threading.I haven't tested it using 2.6.10 as I don't even know if I can reproduce this using 2.6.9. If I do, I'll let you know.
I've included the raw oops and the output of ksymoops below. I'm not subscribed to lkml.
I don't see the Oops primary error/reason/fault message. Can you dig it up?
And it looks like you need to use
ksymoops -k /proc/kallsyms
to get the addresses converted to symbols.
Can you redo ksymoops like that, please?
Quoted 62 lines
===================== RAW ======================
Jan 3 07:07:26 wojas kernel: c02b5513 Jan 3 07:07:26 wojas kernel: PREEMPT Jan 3 07:07:26 wojas kernel: Modules linked in: sch_ingress cls_u32
sch_sfqsch_cbqip6table_filterip6t_LOGip6_tablesipt_limitlp msr cpuidipt_stateipt_MASQUERADEipt_LOGiptable_mangleiptable_filterdummyaf_packetes1371ac97_codecpci_hotplugintel_agpfloppy pcspkr rtcsd_modreiserfs ext2dm_modcapability commoncap tsdev mousedev joydev evdev wacom usbhid psmouseusb_storageuhci_hcdusbcore eeprom lm75i2c_sensori2c_devi2c_piix4i2c_coreaha152x ipv6binfmt_miscip_nat_ftpiptable_natip_conntrack_ftpip_conntrackmga agpgartsnd_pcm_osssnd_mixer_osssnd_ens1371snd_rawmidisnd_seq_devicesnd_pcmsnd_timersnd_page_allocsnd_ac97_codecsnd gameportipt_REJECTip_tablesne2k_pci8390via_rhinemii crc32 sound soundcore sgscsi_modparport_pcparportide_cdcdrom ext3 jbd mbcacheide_genericpiixide_diskide_coreunix fbcon font vesafb cfbcopyarea cfbimgblt cfbfillrectJan 3 07:07:26 wojas kernel: CPU: 0 Jan 3 07:07:26 wojas kernel: EIP: 0060:[__func__.4+64363/135712] Not tainted VLI Jan 3 07:07:26 wojas kernel: EFLAGS: 00010246 (2.6.9-1-686) Jan 3 07:07:26 wojas kernel: EIP is at 0xc02b5513 Jan 3 07:07:26 wojas kernel: eax: 00000000 ebx: c0325c00 ecx: c61e57e0 edx: d35b85e0 Jan 3 07:07:26 wojas kernel: esi: c46082b9 edi: c61e57e4 ebp: 00000056 esp: d5f35f1c Jan 3 07:07:26 wojas kernel: ds: 007b es: 007b ss: 0068 Jan 3 07:07:26 wojas kernel: Process python (pid: 30488, threadinfo=d5f34000 task=c0f0faa0) Jan 3 07:07:26 wojas kernel: Stack: c022db99 c61e57e0 d35b85e0 00000000 00000145 c016c995 c61e57e0 00000000 Jan 3 07:07:26 wojas kernel: c4608000 d5f35f64 d5f35f68 0000001a c016ca0a 00000063 c4608008 d5f35f64 Jan 3 07:07:26 wojas kernel: d5f35f68 d5f34000 00000000 00000002 00000000 c4608000 000001ff 082b0d90 Jan 3 07:07:26 wojas kernel: Call Trace: Jan 3 07:07:26 wojas kernel: [sock_poll+41/64] sock_poll+0x29/0x40 Jan 3 07:07:26 wojas kernel: [do_pollfd+149/160] do_pollfd+0x95/0xa0 Jan 3 07:07:26 wojas kernel: [do_poll+106/208] do_poll+0x6a/0xd0 Jan 3 07:07:26 wojas kernel: [sys_poll+353/576] sys_poll+0x161/0x240 Jan 3 07:07:26 wojas kernel: [__pollwait+0/208] __pollwait+0x0/0xd0 Jan 3 07:07:26 wojas kernel: [syscall_call+7/11] syscall_call+0x7/0xb Jan 3 07:07:26 wojas kernel: Code: 79 70 74 6f 2f 63 69 70 68 65 72 2e 63 00 69 6e 63 6c 75 64 65 2f 6c 69 6e 75 78 2f 63 72 79 70 74 6f 2e 68 00 6e 61 6d 65 20 20 <20> 20 20 20 20 20 20 3a 20 25 73 0a 00 6d 6f 64 75 6c 65 20 20===================== KSYMOOPS ======================
ksymoops 2.4.9 on i686 2.6.9-1-686. Options used
-V (default)
-k/proc/ksyms(default)
-l/proc/modules(default)
-o/lib/modules/2.6.9-1-686/(default)
-m/boot/System.map-2.6.9-1-686(default)Error (
regular_file):read_ksymsstat/proc/ksymsfailed No modules in ksyms, skipping objects
No ksyms, skipping lsmodJan 3 07:07:26 wojas kernel: c02b5513 Jan 3 07:07:26 wojas kernel: CPU: 0 Jan 3 07:07:26 wojas kernel: EIP: 0060:[__func__.4+64363/135712] Not tainted VLI Jan 3 07:07:26 wojas kernel: EFLAGS: 00010246 (2.6.9-1-686) Jan 3 07:07:26 wojas kernel: eax: 00000000 ebx: c0325c00 ecx: c61e57e0 edx: d35b85e0 Jan 3 07:07:26 wojas kernel: esi: c46082b9 edi: c61e57e4 ebp: 00000056 esp: d5f35f1c Jan 3 07:07:26 wojas kernel: ds: 007b es: 007b ss: 0068 Jan 3 07:07:26 wojas kernel: Stack: c022db99 c61e57e0 d35b85e0 00000000 00000145 c016c995 c61e57e0 00000000 Jan 3 07:07:26 wojas kernel: c4608000 d5f35f64 d5f35f68 0000001a c016ca0a 00000063 c4608008 d5f35f64 Jan 3 07:07:26 wojas kernel: d5f35f68 d5f34000 00000000 00000002 00000000 c4608000 000001ff 082b0d90 Jan 3 07:07:26 wojas kernel: Call Trace:Warning (
Oops_read): Code line not seen, dumping what data is availableQuoted 6 lines
Quoted 6 lines
ebx;
c0325c00<devinet_sysctl+460/4e0>
ecx;c61e57e0<pg0+5e3b7e0/3fc54400>
edx;d35b85e0<pg0+1320e5e0/3fc54400>
esi;c46082b9<pg0+425e2b9/3fc54400>
edi;c61e57e4<pg0+5e3b7e4/3fc54400>
esp;d5f35f1c<pg0+15b8bf1c/3fc54400>
Signature · 1 line
~Randy
Quoted 2 lines · Randy.Dunlap
I don't see the Oops primary error/reason/fault message. Can you dig it up?
Sorry, you're right, I looked in messages instead of kern.log:
Jan 3 07:07:26 wojas kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000 Jan 3 07:07:26 wojas kernel: printing eip: Jan 3 07:07:26 wojas kernel: c02b5513 Jan 3 07:07:26 wojas kernel: *pde = 00000000 Jan 3 07:07:26 wojas kernel: Oops: 0002 [#1] Jan 3 07:07:26 wojas kernel: PREEMPT Jan 3 07:07:26 wojas kernel: Modules linked in: sch_ingress cls_u32 sch_sfq sch_cbq ip6table_filter ip6t_LOG ip6_tables ipt_limit lp msr cpuid ipt_state ipt_MASQUERADE ipt_LOG iptable_mangle iptable_filter dummy af_packet es1371 ac97_codec pci_hotplug intel_agp floppy pcspkr rtc sd_mod reiserfs ext2 dm_mod capability commoncap tsdev mousedev joydev evdev wacom usbhid psmouse usb_storage uhci_hcd usbcore eeprom lm75 i2c_sensor i2c_dev i2c_piix4 i2c_core aha152x ipv6 binfmt_misc ip_nat_ftp iptable_nat ip_conntrack_ftp ip_conntrack mga agpgart snd_pcm_oss snd_mixer_oss snd_ens1371 snd_rawmidi snd_seq_device snd_pcm snd_timer snd_page_alloc snd_ac97_codec snd gameport ipt_REJECT ip_tables ne2k_pci 8390 via_rhine mii crc32 sound soundcore sg scsi_mod parport_pc parport ide_cd cdrom ext3 jbd mbcache ide_generic piix ide_disk ide_core unix fbcon font vesafb cfbcopyarea cfbimgblt cfbfillrect Jan 3 07:07:26 wojas kernel: CPU: 0 Jan 3 07:07:26 wojas kernel: EIP: 0060:[__func__.4+64363/135712] Not tainted VLI Jan 3 07:07:26 wojas kernel: EFLAGS: 00010246 (2.6.9-1-686) Jan 3 07:07:26 wojas kernel: EIP is at 0xc02b5513 Jan 3 07:07:26 wojas kernel: eax: 00000000 ebx: c0325c00 ecx: c61e57e0 edx: d35b85e0 Jan 3 07:07:26 wojas kernel: esi: c46082b9 edi: c61e57e4 ebp: 00000056 esp: d5f35f1c Jan 3 07:07:26 wojas kernel: ds: 007b es: 007b ss: 0068 Jan 3 07:07:26 wojas kernel: Process python (pid: 30488, threadinfo=d5f34000 task=c0f0faa0) Jan 3 07:07:26 wojas kernel: Stack: c022db99 c61e57e0 d35b85e0 00000000 00000145 c016c995 c61e57e0 00000000 Jan 3 07:07:26 wojas kernel: c4608000 d5f35f64 d5f35f68 0000001a c016ca0a 00000063 c4608008 d5f35f64 Jan 3 07:07:26 wojas kernel: d5f35f68 d5f34000 00000000 00000002 00000000 c4608000 000001ff 082b0d90 Jan 3 07:07:26 wojas kernel: Call Trace: Jan 3 07:07:26 wojas kernel: [sock_poll+41/64] sock_poll+0x29/0x40 Jan 3 07:07:26 wojas kernel: [do_pollfd+149/160] do_pollfd+0x95/0xa0 Jan 3 07:07:26 wojas kernel: [do_poll+106/208] do_poll+0x6a/0xd0 Jan 3 07:07:26 wojas kernel: [sys_poll+353/576] sys_poll+0x161/0x240 Jan 3 07:07:26 wojas kernel: [__pollwait+0/208] __pollwait+0x0/0xd0 Jan 3 07:07:26 wojas kernel: [syscall_call+7/11] syscall_call+0x7/0xb Jan 3 07:07:26 wojas kernel: Code: 79 70 74 6f 2f 63 69 70 68 65 72 2e 63 00 69 6e 63 6c 75 64 65 2f 6c 69 6e 75 78 2f 63 72 79 70 74 6f 2e 68 00 6e 61 6d 65 20 20 <20> 20 20 20 20 20 20 3a 20 25 73 0a 00 6d 6f 64 75 6c 65 20 20
Quoted 4 lines
And it looks like you need to use
ksymoops -k/proc/kallsyms
to get the addresses converted to symbols.
Can you redo ksymoops like that, please?
Doesn't really look like that's helping..
ksymoops 2.4.9 on i686 2.6.9-1-686. Options used
-V (default)
-k /proc/kallsyms (specified)
-l /proc/modules (default)
-o /lib/modules/2.6.9-1-686/ (default)
-m /boot/System.map-2.6.9-1-686 (default)
Warning (read_ksyms): no kernel symbols in ksyms, is /proc/kallsyms a valid ksyms file? No modules in ksyms, skipping objects
No ksyms, skipping lsmod
Jan 3 07:07:26 wojas kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000 Jan 3 07:07:26 wojas kernel: c02b5513 Jan 3 07:07:26 wojas kernel: *pde = 00000000 Jan 3 07:07:26 wojas kernel: Oops: 0002 [#1] Jan 3 07:07:26 wojas kernel: CPU: 0 Jan 3 07:07:26 wojas kernel: EIP: 0060:[__func__.4+64363/135712] Not tainted VLI Jan 3 07:07:26 wojas kernel: EFLAGS: 00010246 (2.6.9-1-686) Jan 3 07:07:26 wojas kernel: eax: 00000000 ebx: c0325c00 ecx: c61e57e0 edx: d35b85e0 Jan 3 07:07:26 wojas kernel: esi: c46082b9 edi: c61e57e4 ebp: 00000056 esp: d5f35f1c Jan 3 07:07:26 wojas kernel: ds: 007b es: 007b ss: 0068 Jan 3 07:07:26 wojas kernel: Stack: c022db99 c61e57e0 d35b85e0 00000000 00000145 c016c995 c61e57e0 00000000 Jan 3 07:07:26 wojas kernel: c4608000 d5f35f64 d5f35f68 0000001a c016ca0a 00000063 c4608008 d5f35f64 Jan 3 07:07:26 wojas kernel: d5f35f68 d5f34000 00000000 00000002 00000000 c4608000 000001ff 082b0d90 Jan 3 07:07:26 wojas kernel: Call Trace:
Warning (Oops_read): Code line not seen, dumping what data is available
Quoted 6 lines
Quoted 6 lines
ebx;
c0325c00<devinet_sysctl+460/4e0>
ecx;c61e57e0<pg0+5e3b7e0/3fc54400>
edx;d35b85e0<pg0+1320e5e0/3fc54400>
esi;c46082b9<pg0+425e2b9/3fc54400>
edi;c61e57e4<pg0+5e3b7e4/3fc54400>
esp;d5f35f1c<pg0+15b8bf1c/3fc54400>
Jan 3 07:07:26 wojas kernel: Code: 79 70 74 6f 2f 63 69 70 68 65 72 2e 63 00 69 6e 63 6c 75 64 65 2f 6c 69 6e 75 78 2f 63 72 79 70 74 6f 2e 68 00 6e 61 6d 65 20 20 <20> 20 20 20 20 20 20 3a 20 25 73 0a 00 6d 6f 64 75 6c 65 20 20
Using defaults from ksymoops -t elf32-i386 -a i386
Code; ffffffd5 <__kernel_rt_sigreturn+1b95/????>
00000000 <_EIP>:
Code; ffffffd5 <__kernel_rt_sigreturn+1b95/????>
0: 79 70 jns 72 <_EIP+0x72>
Code; ffffffd7 <__kernel_rt_sigreturn+1b97/????>
2: 74 6f je 73 <_EIP+0x73>
Code; ffffffd9 <__kernel_rt_sigreturn+1b99/????>
4: 2f das
Code; ffffffda <__kernel_rt_sigreturn+1b9a/????>
5: 63 69 70 arpl %bp,0x70(%ecx)
Code; ffffffdd <__kernel_rt_sigreturn+1b9d/????>
8: 68 65 72 2e 63 push $0x632e7265
Code; ffffffe2 <__kernel_rt_sigreturn+1ba2/????>
d: 00 69 6e add %ch,0x6e(%ecx)
Code; ffffffe5 <__kernel_rt_sigreturn+1ba5/????>
10: 63 6c 75 64 arpl %bp,0x64(%ebp,%esi,2)
Code; ffffffe9 <__kernel_rt_sigreturn+1ba9/????>
14: 65 gs
Code; ffffffea <__kernel_rt_sigreturn+1baa/????>
15: 2f das
Code; ffffffeb <__kernel_rt_sigreturn+1bab/????>
16: 6c insb (%dx),%es:(%edi)
Code; ffffffec <__kernel_rt_sigreturn+1bac/????>
17: 69 6e 75 78 2f 63 72 imul $0x72632f78,0x75(%esi),%ebp
Code; fffffff3 <__kernel_rt_sigreturn+1bb3/????>
1e: 79 70 jns 90 <_EIP+0x90>
Code; fffffff5 <__kernel_rt_sigreturn+1bb5/????>
20: 74 6f je 91 <_EIP+0x91>
Code; fffffff7 <__kernel_rt_sigreturn+1bb7/????>
22: 2e cs
Code; fffffff8 <__kernel_rt_sigreturn+1bb8/????>
23: 68 00 6e 61 6d push $0x6d616e00
Code; fffffffd <__kernel_rt_sigreturn+1bbd/????>
28: 65 20 20 and %ah,%gs:(%eax)
Code; 00000000 Before first symbol
2b: 20 20 and %ah,(%eax)
Code; 00000002 Before first symbol
2d: 20 20 and %ah,(%eax)
Code; 00000004 Before first symbol
2f: 20 20 and %ah,(%eax)
Code; 00000006 Before first symbol
31: 20 3a and %bh,(%edx)
Code; 00000008 Before first symbol
33: 20 25 73 0a 00 6d and %ah,0x6d000a73
Code; 0000000e Before first symbol
39: 6f outsl %ds:(%esi),(%dx)
Code; 0000000f Before first symbol
3a: 64 fs
Code; 00000010 Before first symbol
3b: 75 6c jne a9 <_EIP+0xa9>
Code; 00000012 Before first symbol
3d: 65 20 20 and %ah,%gs:(%eax)
2 warnings issued. Results may not be reliable.
Signature · 4 lines
Konrad Wojas .~. ~ [email protected] / V \ ~ /( )\ :wq GnuPG key 0x588C85B1 ^ ^
Quoted 46 lines · Konrad Wojas
Quoted 2 lines · Randy.Dunlap
I don't see the Oops primary error/reason/fault message. Can you dig it up?
Sorry, you're right, I looked in messages instead of kern.log:
Jan 3 07:07:26 wojas kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000 Jan 3 07:07:26 wojas kernel: printing eip: Jan 3 07:07:26 wojas kernel: c02b5513 Jan 3 07:07:26 wojas kernel: *pde = 00000000 Jan 3 07:07:26 wojas kernel: Oops: 0002 [#1] Jan 3 07:07:26 wojas kernel: PREEMPT Jan 3 07:07:26 wojas kernel: Modules linked in: sch_ingress cls_u32 sch_sfq sch_cbq ip6table_filter ip6t_LOG ip6_tables ipt_limit lp msr cpuid ipt_state ipt_MASQUERADE ipt_LOG iptable_mangle iptable_filter dummy af_packet es1371 ac97_codec pci_hotplug intel_agp floppy pcspkr rtc sd_mod reiserfs ext2 dm_mod capability commoncap tsdev mousedev joydev evdev wacom usbhid psmouse usb_storage uhci_hcd usbcore eeprom lm75 i2c_sensor i2c_dev i2c_piix4 i2c_core aha152x ipv6 binfmt_misc ip_nat_ftp iptable_nat ip_conntrack_ftp ip_conntrack mga agpgart snd_pcm_oss snd_mixer_oss snd_ens1371 snd_rawmidi snd_seq_device snd_pcm snd_timer snd_page_alloc snd_ac97_codec snd gameport ipt_REJECT ip_tables ne2k_pci 8390 via_rhine mii crc32 sound soundcore sg scsi_mod parport_pc parport ide_cd cdrom ext3 jbd mbcache ide_generic piix ide_disk ide_core unix fbcon font vesafb cfbcopyarea cfbimgblt cfbfillrect Jan 3 07:07:26 wojas kernel: CPU: 0 Jan 3 07:07:26 wojas kernel: EIP: 0060:[__func__.4+64363/135712] Not tainted VLI Jan 3 07:07:26 wojas kernel: EFLAGS: 00010246 (2.6.9-1-686) Jan 3 07:07:26 wojas kernel: EIP is at 0xc02b5513 Jan 3 07:07:26 wojas kernel: eax: 00000000 ebx: c0325c00 ecx: c61e57e0 edx: d35b85e0 Jan 3 07:07:26 wojas kernel: esi: c46082b9 edi: c61e57e4 ebp: 00000056 esp: d5f35f1c Jan 3 07:07:26 wojas kernel: ds: 007b es: 007b ss: 0068 Jan 3 07:07:26 wojas kernel: Process python (pid: 30488, threadinfo=d5f34000 task=c0f0faa0) Jan 3 07:07:26 wojas kernel: Stack: c022db99 c61e57e0 d35b85e0 00000000 00000145 c016c995 c61e57e0 00000000 Jan 3 07:07:26 wojas kernel: c4608000 d5f35f64 d5f35f68 0000001a c016ca0a 00000063 c4608008 d5f35f64 Jan 3 07:07:26 wojas kernel: d5f35f68 d5f34000 00000000 00000002 00000000 c4608000 000001ff 082b0d90 Jan 3 07:07:26 wojas kernel: Call Trace: Jan 3 07:07:26 wojas kernel: [sock_poll+41/64] sock_poll+0x29/0x40 Jan 3 07:07:26 wojas kernel: [do_pollfd+149/160] do_pollfd+0x95/0xa0 Jan 3 07:07:26 wojas kernel: [do_poll+106/208] do_poll+0x6a/0xd0 Jan 3 07:07:26 wojas kernel: [sys_poll+353/576] sys_poll+0x161/0x240 Jan 3 07:07:26 wojas kernel: [__pollwait+0/208] __pollwait+0x0/0xd0 Jan 3 07:07:26 wojas kernel: [syscall_call+7/11] syscall_call+0x7/0xb Jan 3 07:07:26 wojas kernel: Code: 79 70 74 6f 2f 63 69 70 68 65 72 2e 63 00 69 6e 63 6c 75 64 65 2f 6c 69 6e 75 78 2f 63 72 79 70 74 6f 2e 68 00 6e 61 6d 65 20 20 <20> 20 20 20 20 20 20 3a 20 25 73 0a 00 6d 6f 64 75 6c 65 20 20Quoted 4 lines
And it looks like you need to use
ksymoops -k/proc/kallsyms
to get the addresses converted to symbols.
Can you redo ksymoops like that, please?Doesn't really look like that's helping..
ksymoops 2.4.9 on i686 2.6.9-1-686. Options used
-V (default)
-k/proc/kallsyms(specified)
-l/proc/modules(default)
-o/lib/modules/2.6.9-1-686/(default)
-m/boot/System.map-2.6.9-1-686(default)
Quoted 3 lines
Warning (
read_ksyms): no kernel symbols in ksyms, is/proc/kallsymsa valid ksyms file? No modules in ksyms, skipping objects
No ksyms, skipping lsmod
This probably needed to use /proc/kallsyms from the dying kernel, which you most likely don't have....
I'm having trouble seeing what sock_poll() called (i.e., where EIP register points to). In the /boot/System.map-2.6.9-1-686 file, is anything near address 0xc02b5513 listed?
(or just send me that file privately)
Quoted 79 lines
Jan 3 07:07:26 wojas kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000 Jan 3 07:07:26 wojas kernel: c02b5513 Jan 3 07:07:26 wojas kernel: *pde = 00000000 Jan 3 07:07:26 wojas kernel: Oops: 0002 [#1] Jan 3 07:07:26 wojas kernel: CPU: 0 Jan 3 07:07:26 wojas kernel: EIP: 0060:[__func__.4+64363/135712] Not tainted VLI Jan 3 07:07:26 wojas kernel: EFLAGS: 00010246 (2.6.9-1-686) Jan 3 07:07:26 wojas kernel: eax: 00000000 ebx: c0325c00 ecx: c61e57e0 edx: d35b85e0 Jan 3 07:07:26 wojas kernel: esi: c46082b9 edi: c61e57e4 ebp: 00000056 esp: d5f35f1c Jan 3 07:07:26 wojas kernel: ds: 007b es: 007b ss: 0068 Jan 3 07:07:26 wojas kernel: Stack: c022db99 c61e57e0 d35b85e0 00000000 00000145 c016c995 c61e57e0 00000000 Jan 3 07:07:26 wojas kernel: c4608000 d5f35f64 d5f35f68 0000001a c016ca0a 00000063 c4608008 d5f35f64 Jan 3 07:07:26 wojas kernel: d5f35f68 d5f34000 00000000 00000002 00000000 c4608000 000001ff 082b0d90 Jan 3 07:07:26 wojas kernel: Call Trace:Warning (
Oops_read): Code line not seen, dumping what data is availableQuoted 6 lines
Quoted 6 lines
ebx;
c0325c00<devinet_sysctl+460/4e0>
ecx;c61e57e0<pg0+5e3b7e0/3fc54400>
edx;d35b85e0<pg0+1320e5e0/3fc54400>
esi;c46082b9<pg0+425e2b9/3fc54400>
edi;c61e57e4<pg0+5e3b7e4/3fc54400>
esp;d5f35f1c<pg0+15b8bf1c/3fc54400>Jan 3 07:07:26 wojas kernel: Code: 79 70 74 6f 2f 63 69 70 68 65 72 2e 63 00 69 6e 63 6c 75 64 65 2f 6c 69 6e 75 78 2f 63 72 79 70 74 6f 2e 68 00 6e 61 6d 65 20 20 <20> 20 20 20 20 20 20 3a 20 25 73 0a 00 6d 6f 64 75 6c 65 20 20 Using defaults from ksymoops -t elf32-i386 -a i386Code;
ffffffd5<__kernel_rt_sigreturn+1b95/????>
00000000 <_EIP>:
Code;ffffffd5<__kernel_rt_sigreturn+1b95/????>
0: 79 70 jns 72 <_EIP+0x72>
Code;ffffffd7<__kernel_rt_sigreturn+1b97/????>
2: 74 6f je 73 <_EIP+0x73>
Code;ffffffd9<__kernel_rt_sigreturn+1b99/????>
4: 2f das
Code; ffffffda <__kernel_rt_sigreturn+1b9a/????>
5: 63 69 70 arpl %bp,0x70(%ecx)
Code; ffffffdd <__kernel_rt_sigreturn+1b9d/????>
8: 68 65 72 2e 63 push $0x632e7265
Code;ffffffe2<__kernel_rt_sigreturn+1ba2/????>
d: 00 69 6e add %ch,0x6e(%ecx)
Code;ffffffe5<__kernel_rt_sigreturn+1ba5/????>
10: 63 6c 75 64 arpl %bp,0x64(%ebp,%esi,2)
Code;ffffffe9<__kernel_rt_sigreturn+1ba9/????>
14: 65 gs
Code; ffffffea <__kernel_rt_sigreturn+1baa/????>
15: 2f das
Code; ffffffeb <__kernel_rt_sigreturn+1bab/????>
16: 6c insb (%dx),%es:(%edi)
Code; ffffffec <__kernel_rt_sigreturn+1bac/????>
17: 69 6e 75 78 2f 63 72 imul $0x72632f78,0x75(%esi),%ebp
Code;fffffff3<__kernel_rt_sigreturn+1bb3/????>
1e: 79 70 jns 90 <_EIP+0x90>
Code;fffffff5<__kernel_rt_sigreturn+1bb5/????>
20: 74 6f je 91 <_EIP+0x91>
Code;fffffff7<__kernel_rt_sigreturn+1bb7/????>
22: 2e cs
Code;fffffff8<__kernel_rt_sigreturn+1bb8/????>
23: 68 00 6e 61 6d push $0x6d616e00
Code; fffffffd <__kernel_rt_sigreturn+1bbd/????>
28: 65 20 20 and %ah,%gs:(%eax)
Code; 00000000 Before first symbol
2b: 20 20 and %ah,(%eax)
Code; 00000002 Before first symbol
2d: 20 20 and %ah,(%eax)
Code; 00000004 Before first symbol
2f: 20 20 and %ah,(%eax)
Code; 00000006 Before first symbol
31: 20 3a and %bh,(%edx)
Code; 00000008 Before first symbol
33: 20 25 73 0a 00 6d and %ah,0x6d000a73
Code;0000000eBefore first symbol
39: 6f outsl %ds:(%esi),(%dx)
Code;0000000fBefore first symbol
3a: 64 fs
Code; 00000010 Before first symbol
3b: 75 6c jne a9 <_EIP+0xa9>
Code; 00000012 Before first symbol
3d: 65 20 20 and %ah,%gs:(%eax)2 warnings issued. Results may not be reliable.
Signature · 1 line
~Randy
Quoted 7 lines · Randy.Dunlap
This probably needed to use
/proc/kallsymsfrom the dying kernel, which you most likely don't have....I'm having trouble seeing what
sock_poll()called (i.e., where EIP register points to). In the/boot/System.map-2.6.9-1-686file, is anything near address0xc02b5513listed?
(or just send me that file privately)
Also doesn't look very helpfull to me..
c02a592a r __func__.1c02a593b r __func__.0c02a594c r __func__.8c02a5960 r llc_ouic02a59a8 r __func__.4c02c6bc8 d __pci_fixup_PCI_VENDOR_ID_S3PCI_DEVICE_ID_S3_868quirk_s3_64M c02c6bc8 D __start_pci_fixups_headerc02c6bd0 d __pci_fixup_PCI_VENDOR_ID_S3PCI_DEVICE_ID_S3_968quirk_s3_64M c02c6bd8 d __pci_fixup_PCI_VENDOR_ID_ALPCI_DEVICE_ID_AL_M7101quirk_ali7101_acpi c02c6be0 d __pci_fixup_PCI_VENDOR_ID_INTELPCI_DEVICE_ID_INTEL_82371AB_3quirk_piix4_acpi
Signature · 4 lines
Konrad Wojas .~. ~ [email protected] / V \ ~ /( )\ :wq GnuPG key 0x588C85B1 ^ ^
Quoted 5 lines · Konrad Wojas
Quoted 3 lines · Randy.Dunlap
This probably needed to use
/proc/kallsymsfrom the dying kernel, which you most likely don't have....
Quoted 7 lines
Quoted 4 lines
I'm having trouble seeing what
sock_poll()called (i.e., where EIP register points to). In the/boot/System.map-2.6.9-1-686file, is anything near address0xc02b5513listed?
(or just send me that file privately)Also doesn't look very helpfull to me..
True. Have you tested this problem on 2.6.10 yet?
Back to 2.6.9: do you normally run 2.6.9 with all of those same modules loaded? If so, please send me the /proc/modules and /proc/kallsyms files.
Anyone know what all of those __func__'s are?
Quoted 10 lines
c02a592ar __func__.1c02a593br __func__.0c02a594cr __func__.8c02a5960rllc_ouic02a59a8r __func__.4c02c6bc8d__pci_fixup_PCI_VENDOR_ID_S3PCI_DEVICE_ID_S3_868quirk_s3_64Mc02c6bc8D__start_pci_fixups_headerc02c6bd0d__pci_fixup_PCI_VENDOR_ID_S3PCI_DEVICE_ID_S3_968quirk_s3_64Mc02c6bd8d__pci_fixup_PCI_VENDOR_ID_ALPCI_DEVICE_ID_AL_M7101quirk_ali7101_acpic02c6be0d__pci_fixup_PCI_VENDOR_ID_INTELPCI_DEVICE_ID_INTEL_82371AB_3quirk_piix4_acpi
Signature · 1 line
~Randy
Quoted 12 lines · Randy.Dunlap
Quoted 11 lines · Konrad Wojas
Quoted 7 lines · Randy.Dunlap
This probably needed to use
/proc/kallsymsfrom the dying kernel, which you most likely don't have....I'm having trouble seeing what
sock_poll()called (i.e., where EIP register points to). In the/boot/System.map-2.6.9-1-686file, is anything near address0xc02b5513listed?
(or just send me that file privately)
Quoted 3 lines
Quoted 1 line
Also doesn't look very helpfull to me..
True. Have you tested this problem on 2.6.10 yet?
No, I don't even know how to reproduce this on 2.6.9.
Quoted 3 lines
Back to 2.6.9: do you normally run 2.6.9 with all of those same modules loaded? If so, please send me the
/proc/modulesand/proc/kallsymsfiles.
I'm quite sure the same modules were loaded. I've sent the files by private mail.
Regards,
Signature · 4 lines
Konrad Wojas .~. ~ [email protected] / V \ ~ /( )\ :wq GnuPG key 0x588C85B1 ^ ^
Quoted 7 lines · Konrad Wojas
Quoted 5 lines · Randy.Dunlap
Quoted 3 lines · Konrad Wojas
On Wed, Jan 05, 2005 at 10:24:09AM -0800, Randy.Dunlap wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
This probably needed to use
/proc/kallsymsfrom the dying kernel, which you most likely don't have....I'm having trouble seeing what
sock_poll()called (i.e., where EIP register points to). In the/boot/System.map-2.6.9-1-686file, is anything near address0xc02b5513listed?
(or just send me that file privately)
Sorry, I'm not getting anywhere with this.
The obvious thing is that EIP (0xc02b5513) is not pointing to code, but to data (ASCII text, as in a file name from the kernel source tree)...
There's not quite enough stack data for me to determine what happened.
Quoted 15 lines
Quoted 3 lines
Quoted 1 line
Also doesn't look very helpfull to me..
True. Have you tested this problem on 2.6.10 yet?
No, I don't even know how to reproduce this on 2.6.9.
Quoted 3 lines
Back to 2.6.9: do you normally run 2.6.9 with all of those same modules loaded? If so, please send me the
/proc/modulesand/proc/kallsymsfiles.I'm quite sure the same modules were loaded. I've sent the files by private mail.
Signature · 1 line
~Randy