Skip to contentSkip to conversation

NFS bug?

DISCUSSION Robert Merrill 4 participants 13 messages Apr 20, 2006 – Apr 23, 2006 2 messages not in the archive VFS

lore
Share & tools
lore
#0 lore raw

we have an SMP login server we just recently switched to debian testing from FreeBSD and it's giving us a little trouble.

it mounts its /home on a seperate machine, which is still running BSD, over a NIC-to-NIC 1000BASE-T link.

We've found the following bug exists in 2.6.15 and .16: If a directory under /home is readable but not executable, a call to getdents64() on it will kill the process with an invalid operand error in __copy_from_user_ll

has this been fixed already, and is there a patch which is readily applicable?

we're not using the latest kernel, unfortunately, because it has lockd problems.

#1 lore raw
Quoted 6 lines · Robert Merrill

we have an SMP login server we just recently switched to debian testing from FreeBSD and it's giving us a little trouble.

it mounts its /home on a seperate machine, which is still running BSD, over a NIC-to-NIC 1000BASE-T link.

Quoted 6 lines

We've found the following bug exists in 2.6.15 and .16: If a directory under /home is readable but not executable, a call to getdents64() on it will kill the process with an invalid operand error in __copy_from_user_ll

has this been fixed already, and is there a patch which is readily applicable?

No idea. Can you supply us with a strace of the problem?

Quoted 1 line

we're not using the latest kernel, unfortunately, because it has lockd problems.

Care to elaborate?

Cheers,
Trond

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 4 lines · Trond Myklebust

Given that you are reporting an error with copy_from_user, then it is _definitely_ of interest to figure out what your glibc is telling the kernel to copy.

execve("./a.out", ["./a.out", "foo"], [/* 16 vars */]) = 0

uname({sys="Linux", node="soda", ...})  = 0
brk(0)                                  = 0x804a000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)

old_mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f5f000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=20397, ...}) = 0 old_mmap(NULL, 20397, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f5a000

close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/tls/libc.so.6", O_RDONLY)    = 3

read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260O\1"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=1270928, ...}) = 0 old_mmap(NULL, 1276892, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7e22000
old_mmap(0xb7f50000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12e000) = 0xb7f50000 old_mmap(0xb7f58000, 7132, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7f58000
close(3) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7e21000
mprotect(0xb7f50000, 20480, PROT_READ) = 0 set_thread_area({entry_number:-1 -> 6, base_addr:0xb7e218e0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
munmap(0xb7f5a000, 20397) = 0
open("foo", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3 fstat64(3, {st_mode=S_IFDIR|0666, st_size=512, ...}) = 0

fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
brk(0)                                  = 0x804a000
brk(0x806f000)                          = 0x806f000

fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 21), ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7f5e000
write(1, "0\n", 20
) = 2
getdents64(3, <unfinished ...>
+++ killed by SIGSEGV +++

#5 lore raw
Quoted 5 lines · Robert Merrill
Quoted 4 lines · Trond Myklebust

Given that you are reporting an error with copy_from_user, then it is _definitely_ of interest to figure out what your glibc is telling the kernel to copy.

Oh... and could you also send us the Oops/stack trace from the BUG_ON()?

Cheers,
Trond

#6 lore raw
Quoted 2 lines

Oh... and could you also send us the Oops/stack trace from the BUG_ON()?

 ------------[ cut here ]------------
kernel BUG at arch/i386/lib/usercopy.c:582!
invalid operand: 0000 [#49]
SMP
Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa

i2c_dev thermal fan button processor ac battery nfs lockd nfs_acl sunrpc ipv6 quota_v1 ide_cd cdrom generic joydev piix psmouse evdev uhci_hcd ehci_hcd parport_pc parport e1000 rtc serio_raw floppy usbcore i2c_i801 ide_core i2c_core mousedev pcspkr shpchp pci_hotplug

CPU:    2
EIP:    0060:[<c01ff157>]    Not tainted VLI
EFLAGS: 00010282   (2.6.15.7-soda0)

EIP is at __copy_from_user_ll+0x12/0xe2
eax: 00000000 ebx: 00000003 ecx: fffffffb edx: fffffffb
esi: 0804a024 edi: 00000000 ebp: 00000000 esp: f6964f84
ds: 007b es: 007b ss: 0068
Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)

Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb 000000d0
       f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003 0804a024
       00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b 000000dc
Call Trace:
Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89

c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f> 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0

#7 lore raw
Quoted 19 lines · Robert Merrill
Quoted 2 lines

Oh... and could you also send us the Oops/stack trace from the BUG_ON()?

 ------------[ cut here ]------------
kernel BUG at arch/i386/lib/usercopy.c:582!
invalid operand: 0000 [#49]
SMP
Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa

i2c_dev thermal fan button processor ac battery nfs lockd nfs_acl sunrpc ipv6 quota_v1 ide_cd cdrom generic joydev piix psmouse evdev uhci_hcd ehci_hcd parport_pc parport e1000 rtc serio_raw floppy usbcore i2c_i801 ide_core i2c_core mousedev pcspkr shpchp pci_hotplug

CPU:    2
EIP:    0060:[<c01ff157>]    Not tainted VLI
EFLAGS: 00010282   (2.6.15.7-soda0)

EIP is at __copy_from_user_ll+0x12/0xe2
eax: 00000000 ebx: 00000003 ecx: fffffffb edx: fffffffb
esi: 0804a024 edi: 00000000 ebp: 00000000 esp: f6964f84
ds: 007b es: 007b ss: 0068
Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)

Quoted 7 lines
Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb 000000d0
       f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003 0804a024
       00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b 000000dc
Call Trace:
Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89

c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f> 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0

Was there no stack trace in that Oops? AFAICS, getdents64() isn't supposed to be calling __copy_from_user_ll() at all, so you appear to have something very weird going here.

Cheers,
Trond

#8 lore raw

It's not an oops. It's a BUG_ON.

That was the entire message.

Quoted 39 lines · Trond Myklebust
Quoted 14 lines · Robert Merrill
Quoted 2 lines

Oh... and could you also send us the Oops/stack trace from the BUG_ON()?

 ------------[ cut here ]------------
kernel BUG at arch/i386/lib/usercopy.c:582!
invalid operand: 0000 [#49]
SMP
Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa

i2c_dev thermal fan button processor ac battery nfs lockd nfs_acl sunrpc ipv6 quota_v1 ide_cd cdrom generic joydev piix psmouse evdev uhci_hcd ehci_hcd parport_pc parport e1000 rtc serio_raw floppy usbcore i2c_i801 ide_core i2c_core mousedev pcspkr shpchp pci_hotplug

CPU:    2
EIP:    0060:[<c01ff157>]    Not tainted VLI
EFLAGS: 00010282   (2.6.15.7-soda0)
Quoted 6 lines

EIP is at __copy_from_user_ll+0x12/0xe2
eax: 00000000 ebx: 00000003 ecx: fffffffb edx: fffffffb
esi: 0804a024 edi: 00000000 ebp: 00000000 esp: f6964f84
ds: 007b es: 007b ss: 0068
Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)

Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb

000000d0

Quoted 1 line
       f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003

0804a024

Quoted 1 line
       00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b

000000dc

Quoted 4 lines
Call Trace:
Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89

c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f> 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0

Was there no stack trace in that Oops? AFAICS, getdents64() isn't supposed to be calling __copy_from_user_ll() at all, so you appear to have something very weird going here.

Cheers,
Trond

#9 lore raw
In reply to Trond Myklebust
Quoted 29 lines · Trond Myklebust
Quoted 19 lines · Robert Merrill
Quoted 2 lines

Oh... and could you also send us the Oops/stack trace from the BUG_ON()?

 ------------[ cut here ]------------
kernel BUG at arch/i386/lib/usercopy.c:582!
invalid operand: 0000 [#49]
SMP
Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa

i2c_dev thermal fan button processor ac battery nfs lockd nfs_acl sunrpc ipv6 quota_v1 ide_cd cdrom generic joydev piix psmouse evdev uhci_hcd ehci_hcd parport_pc parport e1000 rtc serio_raw floppy usbcore i2c_i801 ide_core i2c_core mousedev pcspkr shpchp pci_hotplug

CPU:    2
EIP:    0060:[<c01ff157>]    Not tainted VLI
EFLAGS: 00010282   (2.6.15.7-soda0)

EIP is at __copy_from_user_ll+0x12/0xe2
eax: 00000000 ebx: 00000003 ecx: fffffffb edx: fffffffb
esi: 0804a024 edi: 00000000 ebp: 00000000 esp: f6964f84
ds: 007b es: 007b ss: 0068
Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)

Quoted 7 lines
Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb 000000d0
       f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003 0804a024
       00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b 000000dc
Call Trace:
Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89

c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f> 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0

Quoted 3 lines

Was there no stack trace in that Oops? AFAICS, getdents64() isn't supposed to be calling __copy_from_user_ll() at all, so you appear to have something very weird going here.

I'd be guessing that filldir64() was passed a negative namlen.

Perhaps Robert could test this:

fs/readdir.c renamed from fs/readdir.c~a +4 -0
@@ -231,6 +231,10 @@ static int filldir64(void * __buf, const
 	buf->error = -EINVAL;	/* only used if we fail.. */
 	if (reclen > buf->count)
 		return -EINVAL;
+	if (namlen < 0) {
+		dump_stack();
+		return -EINVAL;
+	}
 	dirent = buf->previous;
 	if (dirent) {
 		if (__put_user(offset, &dirent->d_off))

_

#10 lore raw
Quoted 30 lines · Andrew Morton
Quoted 29 lines · Trond Myklebust
Quoted 26 lines · Robert Merrill
Quoted 2 lines

Oh... and could you also send us the Oops/stack trace from the BUG_ON()?

 ------------[ cut here ]------------
kernel BUG at arch/i386/lib/usercopy.c:582!
invalid operand: 0000 [#49]
SMP
Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa

i2c_dev thermal fan button processor ac battery nfs lockd nfs_acl sunrpc ipv6 quota_v1 ide_cd cdrom generic joydev piix psmouse evdev uhci_hcd ehci_hcd parport_pc parport e1000 rtc serio_raw floppy usbcore i2c_i801 ide_core i2c_core mousedev pcspkr shpchp pci_hotplug

CPU:    2
EIP:    0060:[<c01ff157>]    Not tainted VLI
EFLAGS: 00010282   (2.6.15.7-soda0)

EIP is at __copy_from_user_ll+0x12/0xe2
eax: 00000000 ebx: 00000003 ecx: fffffffb edx: fffffffb
esi: 0804a024 edi: 00000000 ebp: 00000000 esp: f6964f84
ds: 007b es: 007b ss: 0068
Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)

Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb 000000d0
       f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003 0804a024
       00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b 000000dc
Call Trace:
Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89

c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f> 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0

Quoted 5 lines
Quoted 3 lines

Was there no stack trace in that Oops? AFAICS, getdents64() isn't supposed to be calling __copy_from_user_ll() at all, so you appear to have something very weird going here.

I'd be guessing that filldir64() was passed a negative namlen.

Why would that trigger a bug in __copy_from_user_ll()? I could see it triggering errors in copy_to_user(), but not copy_from_*...

Cheers,
Trond

#11 lore raw
Quoted 11 lines
Quoted 5 lines
Quoted 3 lines

Was there no stack trace in that Oops? AFAICS, getdents64() isn't supposed to be calling __copy_from_user_ll() at all, so you appear to have something very weird going here.

I'd be guessing that filldir64() was passed a negative namlen.

Why would that trigger a bug in __copy_from_user_ll()? I could see it triggering errors in copy_to_user(), but not copy_from_*...

Cheers,
Trond

I've made that mistake before. Passing copy_from_user a negative length -> crash.

#12 lore raw
In reply to Trond Myklebust
Quoted 5 lines · Trond Myklebust
Quoted 2 lines

I'd be guessing that filldir64() was passed a negative namlen.

Why would that trigger a bug in __copy_from_user_ll()? I could see it triggering errors in copy_to_user(), but not copy_from_*...

Ah. No, I cannot see why getdents wold run copy_from_user().

I wonder why that stack trace didn't come out. Perhaps running `dmesg -n 7' prior to triggerng the crash will help. (It shouldn't, but we might have broken it).

#13 lore raw
Quoted 4 lines · Andrew Morton
Quoted 3 lines · Trond Myklebust
Quoted 2 lines

I'd be guessing that filldir64() was passed a negative namlen.

Quoted 9 lines
Quoted 2 lines

Why would that trigger a bug in __copy_from_user_ll()? I could see it triggering errors in copy_to_user(), but not copy_from_*...

Ah. No, I cannot see why getdents wold run copy_from_user().

I wonder why that stack trace didn't come out. Perhaps running `dmesg -n 7' prior to triggerng the crash will help. (It shouldn't, but we might have broken it).

Also, please check that the kernel was compiled with CONFIG_FRAME_POINTER and CONFIG_KALLSYMS.

Cheers,
Trond

#14 lore raw
Quoted 2 lines

Also, please check that the kernel was compiled with CONFIG_FRAME_POINTER and CONFIG_KALLSYMS.

Don't have the former