NFS bug?
we have an SMP login server we just recently switched to debian testing from FreeBSD and it's giving us a little trouble.
it mounts its /home on a seperate machine, which is still running BSD, over a NIC-to-NIC 1000BASE-T link.
We've found the following bug exists in 2.6.15 and .16: If a directory under /home is readable but not executable, a call to getdents64() on it will kill the process with an invalid operand error in __copy_from_user_ll
has this been fixed already, and is there a patch which is readily applicable?
we're not using the latest kernel, unfortunately, because it has lockd problems.
Quoted 6 lines · Robert Merrill
we have an SMP login server we just recently switched to debian testing from FreeBSD and it's giving us a little trouble.
it mounts its /home on a seperate machine, which is still running BSD, over a NIC-to-NIC 1000BASE-T link.
Quoted 6 lines
We've found the following bug exists in 2.6.15 and .16: If a directory under /home is readable but not executable, a call to
getdents64()on it will kill the process with an invalid operand error in__copy_from_user_llhas this been fixed already, and is there a patch which is readily applicable?
No idea. Can you supply us with a strace of the problem?
Quoted 1 line
we're not using the latest kernel, unfortunately, because it has lockd problems.
Care to elaborate?
Cheers,
Trond
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · Trond Myklebust
Given that you are reporting an error with
copy_from_user, then it is _definitely_ of interest to figure out what your glibc is telling the kernel to copy.
execve("./a.out", ["./a.out", "foo"], [/* 16 vars */]) = 0
uname({sys="Linux", node="soda", ...}) = 0
brk(0) = 0x804a000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)old_mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f5f000access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)open("/etc/ld.so.cache", O_RDONLY) = 3fstat64(3, {st_mode=S_IFREG|0644, st_size=20397, ...}) = 0 old_mmap(NULL, 20397, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f5a000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/tls/libc.so.6", O_RDONLY) = 3read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260O\1"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=1270928, ...}) = 0 old_mmap(NULL, 1276892, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7e22000old_mmap(0xb7f50000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12e000) = 0xb7f50000 old_mmap(0xb7f58000, 7132, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7f58000
close(3) = 0old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7e21000mprotect(0xb7f50000, 20480, PROT_READ) = 0 set_thread_area({entry_number:-1 -> 6, base_addr:0xb7e218e0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0munmap(0xb7f5a000, 20397) = 0open("foo", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3 fstat64(3, {st_mode=S_IFDIR|0666, st_size=512, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 brk(0) = 0x804a000 brk(0x806f000) = 0x806f000
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 21), ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7f5e000
write(1, "0\n", 20
) = 2
getdents64(3, <unfinished ...>
+++ killed by SIGSEGV +++
Quoted 5 lines · Robert Merrill
Quoted 4 lines · Trond Myklebust
Given that you are reporting an error with
copy_from_user, then it is _definitely_ of interest to figure out what your glibc is telling the kernel to copy.
Oh... and could you also send us the Oops/stack trace from the BUG_ON()?
Cheers,
Trond
Quoted 2 lines
Oh... and could you also send us the Oops/stack trace from the
BUG_ON()?
------------[ cut here ]------------ kernel BUG at arch/i386/lib/usercopy.c:582! invalid operand: 0000 [#49] SMP Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa
i2c_dev thermal fan button processor ac battery nfs lockd nfs_acl sunrpc ipv6 quota_v1 ide_cd cdrom generic joydev piix psmouse evdev uhci_hcd ehci_hcd parport_pc parport e1000 rtc serio_raw floppy usbcore i2c_i801 ide_core i2c_core mousedev pcspkr shpchp pci_hotplug
CPU: 2 EIP: 0060:[<c01ff157>] Not tainted VLI EFLAGS: 00010282 (2.6.15.7-soda0)
EIP is at __copy_from_user_ll+0x12/0xe2
eax: 00000000 ebx: 00000003 ecx: fffffffb edx: fffffffb
esi: 0804a024 edi: 00000000 ebp: 00000000 esp: f6964f84
ds: 007b es: 007b ss: 0068
Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)
Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb 000000d0
f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003 0804a024
00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b 000000dc
Call Trace:
Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f> 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0
Quoted 19 lines · Robert Merrill
Quoted 2 lines
Oh... and could you also send us the Oops/stack trace from the
BUG_ON()?------------[ cut here ]------------ kernel BUG at arch/i386/lib/usercopy.c:582! invalid operand: 0000 [#49] SMP Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa
i2c_devthermal fan button processor ac battery nfs lockdnfs_aclsunrpc ipv6quota_v1ide_cdcdrom generic joydev piix psmouse evdevuhci_hcdehci_hcdparport_pcparport e1000 rtcserio_rawfloppy usbcorei2c_i801ide_corei2c_coremousedev pcspkr shpchppci_hotplugCPU: 2 EIP: 0060:[<c01ff157>] Not tainted VLI EFLAGS: 00010282 (2.6.15.7-soda0)EIP is at
__copy_from_user_ll+0x12/0xe2
eax: 00000000 ebx: 00000003 ecx: fffffffb edx: fffffffb
esi:0804a024edi: 00000000 ebp: 00000000 esp:f6964f84
ds: 007b es: 007b ss: 0068
Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)
Quoted 7 lines
Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb 000000d0 f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003 0804a024 00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b 000000dc Call Trace: Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f> 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0
Was there no stack trace in that Oops? AFAICS, getdents64() isn't supposed to be calling __copy_from_user_ll() at all, so you appear to have something very weird going here.
Cheers,
Trond
It's not an oops. It's a BUG_ON.
That was the entire message.
Quoted 39 lines · Trond Myklebust
Quoted 14 lines · Robert Merrill
Quoted 2 lines
Oh... and could you also send us the Oops/stack trace from the
BUG_ON()?------------[ cut here ]------------ kernel BUG at arch/i386/lib/usercopy.c:582! invalid operand: 0000 [#49] SMP Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa
i2c_devthermal fan button processor ac battery nfs lockdnfs_aclsunrpc ipv6quota_v1ide_cdcdrom generic joydev piix psmouse evdevuhci_hcdehci_hcdparport_pcparport e1000 rtcserio_rawfloppy usbcorei2c_i801ide_corei2c_coremousedev pcspkr shpchppci_hotplugCPU: 2 EIP: 0060:[<c01ff157>] Not tainted VLI EFLAGS: 00010282 (2.6.15.7-soda0)Quoted 6 lines
EIP is at
__copy_from_user_ll+0x12/0xe2
eax: 00000000 ebx: 00000003 ecx: fffffffb edx: fffffffb
esi:0804a024edi: 00000000 ebp: 00000000 esp:f6964f84
ds: 007b es: 007b ss: 0068
Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb
000000d0Quoted 1 line
f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003
0804a024Quoted 1 line
00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b
000000dcQuoted 4 lines
Call Trace: Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f> 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0
Was there no stack trace in that Oops? AFAICS,
getdents64()isn't supposed to be calling__copy_from_user_ll()at all, so you appear to have something very weird going here.Cheers,
Trond
Quoted 29 lines · Trond Myklebust
Quoted 19 lines · Robert Merrill
Quoted 2 lines
Oh... and could you also send us the Oops/stack trace from the
BUG_ON()?------------[ cut here ]------------ kernel BUG at arch/i386/lib/usercopy.c:582! invalid operand: 0000 [#49] SMP Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa
i2c_devthermal fan button processor ac battery nfs lockdnfs_aclsunrpc ipv6quota_v1ide_cdcdrom generic joydev piix psmouse evdevuhci_hcdehci_hcdparport_pcparport e1000 rtcserio_rawfloppy usbcorei2c_i801ide_corei2c_coremousedev pcspkr shpchppci_hotplugCPU: 2 EIP: 0060:[<c01ff157>] Not tainted VLI EFLAGS: 00010282 (2.6.15.7-soda0)EIP is at
__copy_from_user_ll+0x12/0xe2
eax: 00000000 ebx: 00000003 ecx: fffffffb edx: fffffffb
esi:0804a024edi: 00000000 ebp: 00000000 esp:f6964f84
ds: 007b es: 007b ss: 0068
Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)Quoted 7 lines
Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb 000000d0 f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003 0804a024 00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b 000000dc Call Trace: Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f> 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0
Quoted 3 lines
Was there no stack trace in that Oops? AFAICS,
getdents64()isn't supposed to be calling__copy_from_user_ll()at all, so you appear to have something very weird going here.
I'd be guessing that filldir64() was passed a negative namlen.
Perhaps Robert could test this:
fs/readdir.c renamed from fs/readdir.c~a +4 -0
@@ -231,6 +231,10 @@ static int filldir64(void * __buf, const buf->error = -EINVAL ;/* only used if we fail.. */ if (reclen > buf->count)return -EINVAL ; +if (namlen <0 ) { +dump_stack (); +return -EINVAL ; + } dirent = buf->previous;if (dirent) {if (__put_user (offset, &dirent->d_off))
_
Quoted 30 lines · Andrew Morton
Quoted 29 lines · Trond Myklebust
Quoted 26 lines · Robert Merrill
Quoted 2 lines
Oh... and could you also send us the Oops/stack trace from the
BUG_ON()?------------[ cut here ]------------ kernel BUG at arch/i386/lib/usercopy.c:582! invalid operand: 0000 [#49] SMP Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa
i2c_devthermal fan button processor ac battery nfs lockdnfs_aclsunrpc ipv6quota_v1ide_cdcdrom generic joydev piix psmouse evdevuhci_hcdehci_hcdparport_pcparport e1000 rtcserio_rawfloppy usbcorei2c_i801ide_corei2c_coremousedev pcspkr shpchppci_hotplugCPU: 2 EIP: 0060:[<c01ff157>] Not tainted VLI EFLAGS: 00010282 (2.6.15.7-soda0)EIP is at
__copy_from_user_ll+0x12/0xe2
eax: 00000000 ebx: 00000003 ecx: fffffffb edx: fffffffb
esi:0804a024edi: 00000000 ebp: 00000000 esp:f6964f84
ds: 007b es: 007b ss: 0068
Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb 000000d0 f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003 0804a024 00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b 000000dc Call Trace: Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f> 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0
Quoted 5 lines
Quoted 3 lines
Was there no stack trace in that Oops? AFAICS,
getdents64()isn't supposed to be calling__copy_from_user_ll()at all, so you appear to have something very weird going here.I'd be guessing that
filldir64()was passed a negative namlen.
Why would that trigger a bug in __copy_from_user_ll()? I could see it triggering errors in copy_to_user(), but not copy_from_*...
Cheers,
Trond
Quoted 11 lines
Quoted 5 lines
Quoted 3 lines
Was there no stack trace in that Oops? AFAICS,
getdents64()isn't supposed to be calling__copy_from_user_ll()at all, so you appear to have something very weird going here.I'd be guessing that
filldir64()was passed a negative namlen.Why would that trigger a bug in
__copy_from_user_ll()? I could see it triggering errors incopy_to_user(), but not copy_from_*...Cheers,
Trond
I've made that mistake before. Passing copy_from_user a negative length -> crash.
Quoted 5 lines · Trond Myklebust
Quoted 2 lines
I'd be guessing that
filldir64()was passed a negative namlen.Why would that trigger a bug in
__copy_from_user_ll()? I could see it triggering errors incopy_to_user(), but not copy_from_*...
Ah. No, I cannot see why getdents wold run copy_from_user().
I wonder why that stack trace didn't come out. Perhaps running `dmesg -n 7' prior to triggerng the crash will help. (It shouldn't, but we might have broken it).
Quoted 4 lines · Andrew Morton
Quoted 3 lines · Trond Myklebust
Quoted 2 lines
I'd be guessing that
filldir64()was passed a negative namlen.
Quoted 9 lines
Quoted 2 lines
Why would that trigger a bug in
__copy_from_user_ll()? I could see it triggering errors incopy_to_user(), but not copy_from_*...Ah. No, I cannot see why getdents wold run
copy_from_user().I wonder why that stack trace didn't come out. Perhaps running `dmesg -n 7' prior to triggerng the crash will help. (It shouldn't, but we might have broken it).
Also, please check that the kernel was compiled with CONFIG_FRAME_POINTER and CONFIG_KALLSYMS.
Cheers,
Trond
Quoted 2 lines
Also, please check that the kernel was compiled with
CONFIG_FRAME_POINTERandCONFIG_KALLSYMS.
Don't have the former