Skip to contentSkip to conversation

NULL pointer dereference in sysfs_readdir

DISCUSSION Stefan Richter 4 participants 9 messages Oct 21, 2006 – Oct 31, 2006

lore
Share & tools
lore
#0 lore raw

Here is a report about lockups and/or(?) oopses while HALD is obviously traversing sysfs files of FireWire devices, seen under Fedora Core kernels 2.6.{16,17,18} on an SMP setup:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188140

Quoted 15 lines
May 17 11:49:42 malbec kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000020
May 17 11:49:42 malbec kernel:  printing eip:
May 17 11:49:42 malbec kernel: c019797d
May 17 11:49:42 malbec kernel: *pde = 14924001
May 17 11:49:42 malbec kernel: Oops: 0000 [#1]
May 17 11:49:42 malbec kernel: SMP 
May 17 11:49:42 malbec kernel: last sysfs file: /class/net/eth0/address
May 17 11:49:42 malbec kernel: Modules linked in: sunrpc xt_limit xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables ip6table_filter ip6_tables x_tables ipv6 dm_multipath lp parport_pc parport floppy nvram ohci1394 ieee1394 uhci_hcd sg ohci_hcd ehci_hcd snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_emu10k1 snd_ac97_codec snd_ac97_bus snd_usb_audio snd_usb_lib snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event i2c_piix4 snd_seq i2c_core snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_device snd_timer snd_page_alloc snd_util_mem emu10k1_gp snd_hwdep gameport snd soundcore e100 mii dm_snapshot dm_zero dm_mirror dm_mod ext3 jbd aic7xxx scsi_transport_spi sd_mod scsi_mod
May 17 11:49:42 malbec kernel: CPU:    1
May 17 11:49:42 malbec kernel: EIP:    0060:[<c019797d>]    Not tainted VLI
May 17 11:49:42 malbec kernel: EFLAGS: 00010282   (2.6.16-1.2111_FC5smp #1) 
May 17 11:49:42 malbec kernel: EIP is at sysfs_readdir+0x153/0x202
May 17 11:49:42 malbec kernel: eax: 00000000   ebx: d56cd5a0   ecx: 0000000a   edx: 00000002
May 17 11:49:42 malbec kernel: esi: e3a2e6fc   edi: e3a2e707   ebp: e2c08b14   esp: d55a7f50
May 17 11:49:42 malbec kernel: ds: 007b   es: 007b   ss: 0068
Quoted 8 lines
May 17 11:49:42 malbec kernel: Process hald (pid: 2512, threadinfo=d55a7000 task=d8bc7930)
May 17 11:49:42 malbec kernel: Stack: <0>c0171894 d55a7f98 d4c73e60 e3b28a4c e2c08b18 0000000a c0362f80 d4c73e60 
May 17 11:49:42 malbec kernel:        e3b2655c e3b265dc c0171a53 d55a7f98 c0171894 fffffff7 09e4f5d4 00000000 
May 17 11:49:42 malbec kernel:        d4c73e60 c0171ae0 09e4f754 09e4f73c 00000e80 ffffffea 0000000d 09e4f5d4 
May 17 11:49:42 malbec kernel: Call Trace:
May 17 11:49:42 malbec kernel:  [<c0171894>] filldir64+0x0/0xc3     [<c0171a53>] vfs_readdir+0x66/0x90
May 17 11:49:42 malbec kernel:  [<c0171894>] filldir64+0x0/0xc3     [<c0171ae0>] sys_getdents64+0x63/0xa5
May 17 11:49:42 malbec kernel:  [<c0103db9>] syscall_call+0x7/0xb    <0>Code: 14 00 0f 84 ad 00 00 00 89 e8 e8 b3 e8 ff ff 89 c6 31 c0 83 c9 ff 89 f7 f2 ae f7 d1 49 89 4c 24 14 8b 45 20 85 c0 74 08 8b 40 18 <8b> 50 20 eb 11 ba 02 00 00 00 a1 20 24 48 c0 e8 2e 06 fe ff 89 

A quick look at linux-2.6.19-rc2/fs/sysfs/dir.c::sysfs_readdir shows a list_move, a traversal through a list, and some accesses to data with no apparent protection by mutexes.

Is there reason to be worried...?

Signature · 3 lines
Stefan Richter
-=====-=-==- =-=- =-=-=
http://arcgraph.de/sr/
#1 lore raw
Quoted 30 lines · Stefan Richter

Here is a report about lockups and/or(?) oopses while HALD is obviously traversing sysfs files of FireWire devices, seen under Fedora Core kernels 2.6.{16,17,18} on an SMP setup:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188140

Quoted 15 lines
May 17 11:49:42 malbec kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000020
May 17 11:49:42 malbec kernel:  printing eip:
May 17 11:49:42 malbec kernel: c019797d
May 17 11:49:42 malbec kernel: *pde = 14924001
May 17 11:49:42 malbec kernel: Oops: 0000 [#1]
May 17 11:49:42 malbec kernel: SMP 
May 17 11:49:42 malbec kernel: last sysfs file: /class/net/eth0/address
May 17 11:49:42 malbec kernel: Modules linked in: sunrpc xt_limit xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables ip6table_filter ip6_tables x_tables ipv6 dm_multipath lp parport_pc parport floppy nvram ohci1394 ieee1394 uhci_hcd sg ohci_hcd ehci_hcd snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_emu10k1 snd_ac97_codec snd_ac97_bus snd_usb_audio snd_usb_lib snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event i2c_piix4 snd_seq i2c_core snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_device snd_timer snd_page_alloc snd_util_mem emu10k1_gp snd_hwdep gameport snd soundcore e100 mii dm_snapshot dm_zero dm_mirror dm_mod ext3 jbd aic7xxx scsi_transport_spi sd_mod scsi_mod
May 17 11:49:42 malbec kernel: CPU:    1
May 17 11:49:42 malbec kernel: EIP:    0060:[<c019797d>]    Not tainted VLI
May 17 11:49:42 malbec kernel: EFLAGS: 00010282   (2.6.16-1.2111_FC5smp #1) 
May 17 11:49:42 malbec kernel: EIP is at sysfs_readdir+0x153/0x202
May 17 11:49:42 malbec kernel: eax: 00000000   ebx: d56cd5a0   ecx: 0000000a   edx: 00000002
May 17 11:49:42 malbec kernel: esi: e3a2e6fc   edi: e3a2e707   ebp: e2c08b14   esp: d55a7f50
May 17 11:49:42 malbec kernel: ds: 007b   es: 007b   ss: 0068
Quoted 8 lines
May 17 11:49:42 malbec kernel: Process hald (pid: 2512, threadinfo=d55a7000 task=d8bc7930)
May 17 11:49:42 malbec kernel: Stack: <0>c0171894 d55a7f98 d4c73e60 e3b28a4c e2c08b18 0000000a c0362f80 d4c73e60 
May 17 11:49:42 malbec kernel:        e3b2655c e3b265dc c0171a53 d55a7f98 c0171894 fffffff7 09e4f5d4 00000000 
May 17 11:49:42 malbec kernel:        d4c73e60 c0171ae0 09e4f754 09e4f73c 00000e80 ffffffea 0000000d 09e4f5d4 
May 17 11:49:42 malbec kernel: Call Trace:
May 17 11:49:42 malbec kernel:  [<c0171894>] filldir64+0x0/0xc3     [<c0171a53>] vfs_readdir+0x66/0x90
May 17 11:49:42 malbec kernel:  [<c0171894>] filldir64+0x0/0xc3     [<c0171ae0>] sys_getdents64+0x63/0xa5
May 17 11:49:42 malbec kernel:  [<c0103db9>] syscall_call+0x7/0xb    <0>Code: 14 00 0f 84 ad 00 00 00 89 e8 e8 b3 e8 ff ff 89 c6 31 c0 83 c9 ff 89 f7 f2 ae f7 d1 49 89 4c 24 14 8b 45 20 85 c0 74 08 8b 40 18 <8b> 50 20 eb 11 ba 02 00 00 00 a1 20 24 48 c0 e8 2e 06 fe ff 89 
Quoted 5 lines

A quick look at linux-2.6.19-rc2/fs/sysfs/dir.c::sysfs_readdir shows a list_move, a traversal through a list, and some accesses to data with no apparent protection by mutexes.

Is there reason to be worried...?

Looking through the code I was not able to locate the exact line at which it oopses, with the above oops message.
Are you able to track down the exact pointer dereference, which causes this? By inserting printks, perhaps.

Maybe FC changed some of the structures. I couldn't find
a used structure with an interresting member at offset 00000020, at least.

Signature · 1 line
Greetings Michael.
#2 lore raw
Quoted 1 line · Michael Buesch

On Saturday 21 October 2006 10:43, Stefan Richter wrote:

...

Quoted 1 line
Quoted 1 line

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188140

...

Quoted 4 lines

Looking through the code I was not able to locate the exact line at which it oopses, with the above oops message.
Are you able to track down the exact pointer dereference, which causes this? By inserting printks, perhaps.

I need the original reporter to do this since I cannot reproduce the bug. Probably because I don't have an SMP machine yet.

Quoted 2 lines

Maybe FC changed some of the structures. I couldn't find
a used structure with an interresting member at offset 00000020, at least.

Could be struct sysfs_dirent.s_dentry if I'm counting correctly in
http://www.linux-m32r.org/lxr/http/source/include/linux/sysfs.h?v=2.6.16#L68
The trace was from 2.6.16.

Signature · 3 lines
Stefan Richter
-=====-=-==- =-=- =-=-=
http://arcgraph.de/sr/
#3 lore raw
Quoted 13 lines · Stefan Richter
Quoted 1 line · Michael Buesch

On Saturday 21 October 2006 10:43, Stefan Richter wrote:

...

Quoted 1 line
Quoted 1 line

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188140

...

Quoted 4 lines

Looking through the code I was not able to locate the exact line at which it oopses, with the above oops message.
Are you able to track down the exact pointer dereference, which causes this? By inserting printks, perhaps.

I need the original reporter to do this since I cannot reproduce the bug. Probably because I don't have an SMP machine yet.

Quoted 6 lines
Quoted 2 lines

Maybe FC changed some of the structures. I couldn't find
a used structure with an interresting member at offset 00000020, at least.

Could be struct sysfs_dirent.s_dentry if I'm counting correctly in
http://www.linux-m32r.org/lxr/http/source/include/linux/sysfs.h?v=2.6.16#L68
The trace was from 2.6.16.

Yeah, I found that offset, too, but:

There is only one usage of s_dentry
if (next->s_dentry)

But _before_ that there already comes
if (!next->s_element)

So, if "next" was NULL, it would already oops there.

Signature · 1 line
Greetings Michael.
#4 lore raw
Quoted 1 line · Michael Buesch

On Saturday 21 October 2006 23:09, Stefan Richter wrote:

...

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188140

[hard lockups, also one oops: 'Unable to handle kernel NULL pointer dereference at virtual address 00000020', 'EIP is at sysfs_readdir', 'Process hald']
...

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Maybe FC changed some of the structures. I couldn't find
a used structure with an interresting member at offset 00000020, at least.

Could be struct sysfs_dirent.s_dentry if I'm counting correctly in
http://www.linux-m32r.org/lxr/http/source/include/linux/sysfs.h?v=2.6.16#L68
The trace was from 2.6.16.

Yeah, I found that offset, too, but:

Quoted 7 lines

There is only one usage of s_dentry
if (next->s_dentry)

But _before_ that there already comes
if (!next->s_element)

So, if "next" was NULL, it would already oops there.

What if "next" became NULL afterwards? I know it's unlikely (but so is the whole bug, given that we have just one reporter despite the bug's age), but is it impossible? IOW does sysfs_readdir have any indirect mutex protection?

Dave, do you patch sysfs datatypes in FC's kernel, or types they include?

Signature · 3 lines
Stefan Richter
-=====-=-==- =-=- =-==-
http://arcgraph.de/sr/
#5 lore raw
Quoted 11 lines · Stefan Richter
Quoted 9 lines

Yeah, I found that offset, too, but:

There is only one usage of s_dentry
if (next->s_dentry)

But _before_ that there already comes
if (!next->s_element)

So, if "next" was NULL, it would already oops there.

What if "next" became NULL afterwards?

Hm, yeah. Makes kind of sense.

Quoted 4 lines

I know it's unlikely (but so is
the whole bug, given that we have just one reporter despite the bug's age), but is it impossible? IOW does sysfs_readdir have any indirect mutex protection?

I think it's protected by the BKL, but I dunno if that's sufficient here.

Signature · 1 line
Greetings Michael.
#6 lore raw
In reply to Stefan Richter
Quoted 6 lines · Stefan Richter

What if "next" became NULL afterwards? I know it's unlikely (but so is the whole bug, given that we have just one reporter despite the bug's age), but is it impossible? IOW does sysfs_readdir have any indirect mutex protection?

Dave, do you patch sysfs datatypes in FC's kernel, or types they include?

Not that I recall. <linux/sysfs.h> is untouched in the trees I just looked at (only have .18 ones handy right now)

Dave

Signature · 1 line
http://www.codemonkey.org.uk
#7 lore raw
In reply to Stefan Richter
Quoted 30 lines · Stefan Richter

Here is a report about lockups and/or(?) oopses while HALD is obviously traversing sysfs files of FireWire devices, seen under Fedora Core kernels 2.6.{16,17,18} on an SMP setup:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188140

Quoted 15 lines
May 17 11:49:42 malbec kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000020
May 17 11:49:42 malbec kernel:  printing eip:
May 17 11:49:42 malbec kernel: c019797d
May 17 11:49:42 malbec kernel: *pde = 14924001
May 17 11:49:42 malbec kernel: Oops: 0000 [#1]
May 17 11:49:42 malbec kernel: SMP 
May 17 11:49:42 malbec kernel: last sysfs file: /class/net/eth0/address
May 17 11:49:42 malbec kernel: Modules linked in: sunrpc xt_limit xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables ip6table_filter ip6_tables x_tables ipv6 dm_multipath lp parport_pc parport floppy nvram ohci1394 ieee1394 uhci_hcd sg ohci_hcd ehci_hcd snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_emu10k1 snd_ac97_codec snd_ac97_bus snd_usb_audio snd_usb_lib snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event i2c_piix4 snd_seq i2c_core snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_device snd_timer snd_page_alloc snd_util_mem emu10k1_gp snd_hwdep gameport snd soundcore e100 mii dm_snapshot dm_zero dm_mirror dm_mod ext3 jbd aic7xxx scsi_transport_spi sd_mod scsi_mod
May 17 11:49:42 malbec kernel: CPU:    1
May 17 11:49:42 malbec kernel: EIP:    0060:[<c019797d>]    Not tainted VLI
May 17 11:49:42 malbec kernel: EFLAGS: 00010282   (2.6.16-1.2111_FC5smp #1) 
May 17 11:49:42 malbec kernel: EIP is at sysfs_readdir+0x153/0x202
May 17 11:49:42 malbec kernel: eax: 00000000   ebx: d56cd5a0   ecx: 0000000a   edx: 00000002
May 17 11:49:42 malbec kernel: esi: e3a2e6fc   edi: e3a2e707   ebp: e2c08b14   esp: d55a7f50
May 17 11:49:42 malbec kernel: ds: 007b   es: 007b   ss: 0068
Quoted 8 lines
May 17 11:49:42 malbec kernel: Process hald (pid: 2512, threadinfo=d55a7000 task=d8bc7930)
May 17 11:49:42 malbec kernel: Stack: <0>c0171894 d55a7f98 d4c73e60 e3b28a4c e2c08b18 0000000a c0362f80 d4c73e60 
May 17 11:49:42 malbec kernel:        e3b2655c e3b265dc c0171a53 d55a7f98 c0171894 fffffff7 09e4f5d4 00000000 
May 17 11:49:42 malbec kernel:        d4c73e60 c0171ae0 09e4f754 09e4f73c 00000e80 ffffffea 0000000d 09e4f5d4 
May 17 11:49:42 malbec kernel: Call Trace:
May 17 11:49:42 malbec kernel:  [<c0171894>] filldir64+0x0/0xc3     [<c0171a53>] vfs_readdir+0x66/0x90
May 17 11:49:42 malbec kernel:  [<c0171894>] filldir64+0x0/0xc3     [<c0171ae0>] sys_getdents64+0x63/0xa5
May 17 11:49:42 malbec kernel:  [<c0103db9>] syscall_call+0x7/0xb    <0>Code: 14 00 0f 84 ad 00 00 00 89 e8 e8 b3 e8 ff ff 89 c6 31 c0 83 c9 ff 89 f7 f2 ae f7 d1 49 89 4c 24 14 8b 45 20 85 c0 74 08 8b 40 18 <8b> 50 20 eb 11 ba 02 00 00 00 a1 20 24 48 c0 e8 2e 06 fe ff 89 
Quoted 4 lines

A quick look at linux-2.6.19-rc2/fs/sysfs/dir.c::sysfs_readdir shows a list_move, a traversal through a list, and some accesses to data with no apparent protection by mutexes.

Most of the sysfs locking is done by parent inode's i_mutex. And for sysfs_readdir(), i_mutex is held at VFS layer in vfs_readdir().

If this is not happening during boot, could you please recreate the oops with at kdump? I guess kdump should work well on FC.

(http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28FC6KdumpKexecHowTo%29)

Quoted 1 line

Is there reason to be worried...?

Though locking looks fine to me.. but oops is a reason enough to be worried about.

Thanks
Maneesh

#8 lore raw

Maneesh Soni wrote on 2006-10-30:

Quoted 1 line · Stefan Richter

...

Quoted 6 lines
Quoted 4 lines

A quick look at linux-2.6.19-rc2/fs/sysfs/dir.c::sysfs_readdir shows a list_move, a traversal through a list, and some accesses to data with no apparent protection by mutexes.

Most of the sysfs locking is done by parent inode's i_mutex. And for sysfs_readdir(), i_mutex is held at VFS layer in vfs_readdir().

OK, thanks.

Quoted 2 lines

If this is not happening during boot, could you please recreate the oops with at kdump? I guess kdump should work well on FC.

The report came from someone else; I notified him. I cannot reproduce it, I don't have FC nor hald. BTW the reporter mentioned that the machine freezes completely much more often than merely throwing an oops. (Both caused by hald.)

Quoted 6 lines

(http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28FC6KdumpKexecHowTo%29)

Quoted 1 line

Is there reason to be worried...?

Though locking looks fine to me.. but oops is a reason enough to be worried about.

I referred to the locking. :-) Thanks for the hints,

Signature · 3 lines
Stefan Richter
-=====-=-==- =-=- =====
http://arcgraph.de/sr/