Skip to contentSkip to conversation

nfs client oops, all 2.4 kernels

DISCUSSION Erik DeBill 4 participants 8 messages Sep 10, 2001 – Sep 13, 2001

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I've been running into a repeatable oops in the NFS client code, apparently related to file locking.

I've verified this with UP, SMP(2,4 procs), small (128M) and large (4G) memory under RedHat 7.1, SuSe 7.1 and SuSe 7.2. All on x86 hardware.

The problem occurs while running DB2 EEE. DB2 EEE is a clustered database, which uses NFS to share the database home directory between nodes in the cluster. This is primarily to synchronize some config files, but by default it will also store all data and logs in that same nfs mounted directory. Oopses occur on the machine (I've only been testing on a 2 node cluster) which does the NFS client mount of the home directory.

If you let it keep all it's data in the nfs mounted directory you can generate an oops very quickly once multiple clients start connecting to the db. The more clients, the faster the oops. The more thorough the job you do of removing things from NFS, the harder to reproduce.

If you run it without NFS (completely unsupported) it runs fine and I haven't been able to reproduce the oops.

It occurs with all 2.4 series kernels, at least up to 2.4.9-ac7 (which had some nfs client fixes)

I've attached the ksymoops trace from 2 different incidents. The first occurred under 2.4.9-ac7, and is pretty typical. Death in lock related code as part of sys_close(). The second is from 2.4.9 + linux-2.4.9-NFS_ALL.dif. It's a little unusual in that several oopses happened all at once.

Please let me know if there's anything I can do to help track these guys down. I can give you access to several machines with different hardware, and distros which all exhibit this bug.

Thanks,
Erik DeBill

ksymoops 2.4.0 on i686 2.4.9-ac7. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.9-ac7/ (default)
-m /boot/System.map-2.4.9-ac7 (default)

Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options.

Warning (compare_maps): mismatch on symbol md_size , md says c8a0e380, /lib/modules/2.4.9-ac7/kernel/drivers/md/md.o says c8a0e1a0. Ignoring /lib/modules/2.4.9-ac7/kernel/drivers/md/md.o entry Warning (compare_maps): mismatch on symbol mddev_map , md says c8a0db80, /lib/modules/2.4.9-ac7/kernel/drivers/md/md.o says c8a0d9a0. Ignoring /lib/modules/2.4.9-ac7/kernel/drivers/md/md.o entry

Sep  8 22:00:06 jerry kernel: WARNING: MP table in the EBDA can be UNSAFE, contact [email protected] if you experience SMP problems!
Sep  8 22:00:07 jerry kernel: cpu: 0, clocks: 1329124, slice: 664562
Sep  8 22:04:15 jerry kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Sep  8 22:04:15 jerry kernel: c0145c2b
Sep  8 22:04:15 jerry kernel: *pde = 00000000
Sep  8 22:04:15 jerry kernel: Oops: 0000
Sep  8 22:04:15 jerry kernel: CPU:    0
Sep  8 22:04:15 jerry kernel: EIP:    0010:[<c0145c2b>]
Using defaults from ksymoops -t elf32-i386 -a i386
Sep  8 22:04:15 jerry kernel: EFLAGS: 00010286
Sep  8 22:04:15 jerry kernel: eax: 00000000   ebx: 00000000   ecx: c72cc4e0   edx: c79155c8
Sep  8 22:04:15 jerry kernel: esi: c240b440   edi: 00000000   ebp: bfffba10   esp: c21a3f60
Sep  8 22:04:15 jerry kernel: ds: 0018   es: 0018   ss: 0018
Sep  8 22:04:15 jerry kernel: Process db2sysc (pid: 867, stackpage=c21a3000)
Sep  8 22:04:15 jerry kernel: Stack: c79153fc c79155c8 c240b440 00000000 c79155c8 c240b440 c0147756 c79155c8 
Sep  8 22:04:15 jerry kernel:        00000000 c21a2000 c27cc500 c0135a96 c27cc500 c240b440 00000000 c27cc500 
Sep  8 22:04:15 jerry kernel:        00000000 c27cc500 00000030 00000000 c0135b0b c27cc500 c240b440 c21a2000 
Sep  8 22:04:15 jerry kernel: Call Trace: [<c0147756>] [<c0135a96>] [<c0135b0b>] [<c0106efb>] 
Sep  8 22:04:15 jerry kernel: Code: 8b 03 8d 73 04 89 02 8b 43 04 c7 03 00 00 00 00 8b 56 04 89 
Quoted 1 line
Quoted 1 line

EIP; c0145c2b <locks_delete_lock+b/d0> <=====

Trace; c0147756 <locks_remove_posix+86/c0>
Trace; c0135a96 <filp_close+86/a0>
Trace; c0135b0b <sys_close+5b/70>
Trace; c0106efb <system_call+33/38>
Code; c0145c2b <locks_delete_lock+b/d0>
00000000 <_EIP>:
Code; c0145c2b <locks_delete_lock+b/d0> <=====
0: 8b 03 mov (%ebx),%eax <=====
Code; c0145c2d <locks_delete_lock+d/d0>
2: 8d 73 04 lea 0x4(%ebx),%esi
Code; c0145c30 <locks_delete_lock+10/d0>
5: 89 02 mov %eax,(%edx)
Code; c0145c32 <locks_delete_lock+12/d0>
7: 8b 43 04 mov 0x4(%ebx),%eax
Code; c0145c35 <locks_delete_lock+15/d0>
a: c7 03 00 00 00 00 movl $0x0,(%ebx)
Code; c0145c3b <locks_delete_lock+1b/d0>
10: 8b 56 04 mov 0x4(%esi),%edx
Code; c0145c3e <locks_delete_lock+1e/d0>
13: 89 00 mov %eax,(%eax)

3 warnings issued. Results may not be reliable.

---------------------------------------------------------------------------
these next are from a single run under 2.4.9 + the all encompassing NFS patch.

DB2 gets shut down as soon as an oops occurs, these must be from the death throes.
---------------------------------------------------------------------------

ksymoops 2.4.0 on i686 2.4.9nfs. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.9nfs/ (default)
-m /boot/System.map-2.4.9nfs (default)

Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options.

Warning (compare_maps): mismatch on symbol md_size , md says c8a0e560, /lib/modules/2.4.9nfs/kernel/drivers/md/md.o says c8a0e380. Ignoring /lib/modules/2.4.9nfs/kernel/drivers/md/md.o entry Warning (compare_maps): mismatch on symbol mddev_map , md says c8a0dd60, /lib/modules/2.4.9nfs/kernel/drivers/md/md.o says c8a0db80. Ignoring /lib/modules/2.4.9nfs/kernel/drivers/md/md.o entry

Sep 10 08:47:05 jerry kernel: WARNING: MP table in the EBDA can be UNSAFE, contact [email protected] if you experience SMP problems!
Sep 10 08:47:08 jerry kernel: cpu: 0, clocks: 1329118, slice: 664559
Sep 10 08:49:09 jerry kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Sep 10 08:49:09 jerry kernel: c0146cab
Sep 10 08:49:09 jerry kernel: *pde = 00000000
Sep 10 08:49:09 jerry kernel: Oops: 0000
Sep 10 08:49:09 jerry kernel: CPU:    0
Sep 10 08:49:09 jerry kernel: EIP:    0010:[fcntl_setlease+171/640]
Sep 10 08:49:09 jerry kernel: EIP:    0010:[<c0146cab>]
Using defaults from ksymoops -t elf32-i386 -a i386
Sep 10 08:49:09 jerry kernel: EFLAGS: 00010286
Sep 10 08:49:09 jerry kernel: eax: 00000000   ebx: 00000000   ecx: c7302b60   edx: c7926d54
Sep 10 08:49:09 jerry kernel: esi: c6b76a40   edi: 00000000   ebp: bfffba10   esp: c6a4ff60
Sep 10 08:49:09 jerry kernel: ds: 0018   es: 0018   ss: 0018
Sep 10 08:49:09 jerry kernel: Process db2sysc (pid: 769, stackpage=c6a4f000)
Sep 10 08:49:09 jerry kernel: Stack: c7926e0c c7926d54 c6b76a40 00000000 c7926d54 c6b76a40 c01488f6 c7926d54 
Sep 10 08:49:09 jerry kernel:        00000000 c6a4e000 c6a23d40 c013512d c6a23d40 c6b76a40 00000000 c6a23d40 
Sep 10 08:49:09 jerry kernel:        00000000 c6a23d40 00000030 00000000 c013519b c6a23d40 c6b76a40 c6a4e000 
Sep 10 08:49:09 jerry kernel: Call Trace: [d_validate+54/224] [sys_chdir+253/256] [sys_fchdir+107/208] [system_call+19/56] 
Sep 10 08:49:09 jerry kernel: Call Trace: [<c01488f6>] [<c013512d>] [<c013519b>] [<c0106edb>] 
Sep 10 08:49:09 jerry kernel: Code: 8b 03 8d 73 04 89 02 8b 43 04 c7 03 00 00 00 00 8b 56 04 89 
Quoted 1 line
Quoted 1 line

EIP; c0146cab <locks_delete_lock+b/d0> <=====

Trace; c01488f6 <locks_remove_posix+86/e0>
Trace; c013512d <filp_close+9d/b0>
Trace; c013519b <sys_close+5b/70>
Trace; c0106edb <system_call+33/38>
Code; c0146cab <locks_delete_lock+b/d0>
00000000 <_EIP>:
Code; c0146cab <locks_delete_lock+b/d0> <=====
0: 8b 03 mov (%ebx),%eax <=====
Code; c0146cad <locks_delete_lock+d/d0>
2: 8d 73 04 lea 0x4(%ebx),%esi
Code; c0146cb0 <locks_delete_lock+10/d0>
5: 89 02 mov %eax,(%edx)
Code; c0146cb2 <locks_delete_lock+12/d0>
7: 8b 43 04 mov 0x4(%ebx),%eax
Code; c0146cb5 <locks_delete_lock+15/d0>
a: c7 03 00 00 00 00 movl $0x0,(%ebx)
Code; c0146cbb <locks_delete_lock+1b/d0>
10: 8b 56 04 mov 0x4(%esi),%edx
Code; c0146cbe <locks_delete_lock+1e/d0>
13: 89 00 mov %eax,(%eax)

Sep 10 08:49:09 jerry kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Sep 10 08:49:09 jerry kernel: c0146cab
Sep 10 08:49:09 jerry kernel: *pde = 00000000
Sep 10 08:49:09 jerry kernel: Oops: 0000
Sep 10 08:49:09 jerry kernel: CPU:    0
Sep 10 08:49:09 jerry kernel: EIP:    0010:[fcntl_setlease+171/640]
Sep 10 08:49:10 jerry kernel: EIP:    0010:[<c0146cab>]
Sep 10 08:49:10 jerry kernel: EFLAGS: 00010286
Sep 10 08:49:10 jerry kernel: eax: 00000000   ebx: 00000000   ecx: c7302b60   edx: c7926be4
Sep 10 08:49:10 jerry kernel: esi: c6b76700   edi: 00000000   ebp: bfffba10   esp: c5d15f60
Sep 10 08:49:10 jerry kernel: ds: 0018   es: 0018   ss: 0018
Sep 10 08:49:10 jerry kernel: Process db2sysc (pid: 770, stackpage=c5d15000)
Sep 10 08:49:10 jerry kernel: Stack: c7926f20 c7926be4 c6b76700 00000000 c7926be4 c6b76700 c01488f6 c7926be4 
Sep 10 08:49:10 jerry kernel:        00000000 c5d14000 c6a23f20 c013512d c6a23f20 c6b76700 00000000 c6a23f20 
Sep 10 08:49:10 jerry kernel:        00000000 c6a23f20 00000030 00000000 c013519b c6a23f20 c6b76700 c5d14000 
Sep 10 08:49:10 jerry kernel: Call Trace: [d_validate+54/224] [sys_chdir+253/256] [sys_fchdir+107/208] [system_call+19/56] 
Sep 10 08:49:10 jerry kernel: Call Trace: [<c01488f6>] [<c013512d>] [<c013519b>] [<c0106edb>] 
Sep 10 08:49:10 jerry kernel: Code: 8b 03 8d 73 04 89 02 8b 43 04 c7 03 00 00 00 00 8b 56 04 89 
Quoted 1 line
Quoted 1 line

EIP; c0146cab <locks_delete_lock+b/d0> <=====

Trace; c01488f6 <locks_remove_posix+86/e0>
Trace; c013512d <filp_close+9d/b0>
Trace; c013519b <sys_close+5b/70>
Trace; c0106edb <system_call+33/38>
Code; c0146cab <locks_delete_lock+b/d0>
00000000 <_EIP>:
Code; c0146cab <locks_delete_lock+b/d0> <=====
0: 8b 03 mov (%ebx),%eax <=====
Code; c0146cad <locks_delete_lock+d/d0>
2: 8d 73 04 lea 0x4(%ebx),%esi
Code; c0146cb0 <locks_delete_lock+10/d0>
5: 89 02 mov %eax,(%edx)
Code; c0146cb2 <locks_delete_lock+12/d0>
7: 8b 43 04 mov 0x4(%ebx),%eax
Code; c0146cb5 <locks_delete_lock+15/d0>
a: c7 03 00 00 00 00 movl $0x0,(%ebx)
Code; c0146cbb <locks_delete_lock+1b/d0>
10: 8b 56 04 mov 0x4(%esi),%edx
Code; c0146cbe <locks_delete_lock+1e/d0>
13: 89 00 mov %eax,(%eax)

Sep 10 08:49:10 jerry kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Sep 10 08:49:10 jerry kernel: c0146cab
Sep 10 08:49:10 jerry kernel: *pde = 00000000
Sep 10 08:49:10 jerry kernel: Oops: 0000
Sep 10 08:49:10 jerry kernel: CPU:    0
Sep 10 08:49:10 jerry kernel: EIP:    0010:[fcntl_setlease+171/640]
Sep 10 08:49:10 jerry kernel: EIP:    0010:[<c0146cab>]
Sep 10 08:49:10 jerry kernel: EFLAGS: 00010286
Sep 10 08:49:10 jerry kernel: eax: 00000000   ebx: 00000000   ecx: c7302b60   edx: c7926c40
Sep 10 08:49:10 jerry kernel: esi: c6439dc0   edi: 00000000   ebp: bfffba10   esp: c643bf60
Sep 10 08:49:10 jerry kernel: ds: 0018   es: 0018   ss: 0018
Sep 10 08:49:10 jerry kernel: Process db2sysc (pid: 776, stackpage=c643b000)
Sep 10 08:49:10 jerry kernel: Stack: c7926db0 c7926c40 c6439dc0 00000000 c7926c40 c6439dc0 c01488f6 c7926c40 
Sep 10 08:49:10 jerry kernel:        00000000 c643a000 c6a23da0 c013512d c6a23da0 c6439dc0 00000000 c6a23da0 
Sep 10 08:49:10 jerry kernel:        00000000 c6a23da0 00000030 00000000 c013519b c6a23da0 c6439dc0 c643a000 
Sep 10 08:49:10 jerry kernel: Call Trace: [d_validate+54/224] [sys_chdir+253/256] [sys_fchdir+107/208] [system_call+19/56] 
Sep 10 08:49:10 jerry kernel: Call Trace: [<c01488f6>] [<c013512d>] [<c013519b>] [<c0106edb>] 
Sep 10 08:49:10 jerry kernel: Code: 8b 03 8d 73 04 89 02 8b 43 04 c7 03 00 00 00 00 8b 56 04 89 
Quoted 1 line
Quoted 1 line

EIP; c0146cab <locks_delete_lock+b/d0> <=====

Trace; c01488f6 <locks_remove_posix+86/e0>
Trace; c013512d <filp_close+9d/b0>
Trace; c013519b <sys_close+5b/70>
Trace; c0106edb <system_call+33/38>
Code; c0146cab <locks_delete_lock+b/d0>
00000000 <_EIP>:
Code; c0146cab <locks_delete_lock+b/d0> <=====
0: 8b 03 mov (%ebx),%eax <=====
Code; c0146cad <locks_delete_lock+d/d0>
2: 8d 73 04 lea 0x4(%ebx),%esi
Code; c0146cb0 <locks_delete_lock+10/d0>
5: 89 02 mov %eax,(%edx)
Code; c0146cb2 <locks_delete_lock+12/d0>
7: 8b 43 04 mov 0x4(%ebx),%eax
Code; c0146cb5 <locks_delete_lock+15/d0>
a: c7 03 00 00 00 00 movl $0x0,(%ebx)
Code; c0146cbb <locks_delete_lock+1b/d0>
10: 8b 56 04 mov 0x4(%esi),%edx
Code; c0146cbe <locks_delete_lock+1e/d0>
13: 89 00 mov %eax,(%eax)

Sep 10 08:49:10 jerry kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Sep 10 08:49:10 jerry kernel: c0146cab
Sep 10 08:49:10 jerry kernel: *pde = 00000000
Sep 10 08:49:10 jerry kernel: Oops: 0000
Sep 10 08:49:10 jerry kernel: CPU:    0
Sep 10 08:49:10 jerry kernel: EIP:    0010:[fcntl_setlease+171/640]
Sep 10 08:49:10 jerry kernel: EIP:    0010:[<c0146cab>]
Sep 10 08:49:10 jerry kernel: EFLAGS: 00010286
Sep 10 08:49:10 jerry kernel: eax: 00000000   ebx: 00000000   ecx: c7302b60   edx: c7926b88
Sep 10 08:49:10 jerry kernel: esi: c4275240   edi: 00000000   ebp: bfffba10   esp: c5d8df60
Sep 10 08:49:10 jerry kernel: ds: 0018   es: 0018   ss: 0018
Sep 10 08:49:10 jerry kernel: Process db2sysc (pid: 774, stackpage=c5d8d000)
Sep 10 08:49:10 jerry kernel: Stack: c7926b2c c7926b88 c4275240 00000000 c7926b88 c4275240 c01488f6 c7926b88 
Sep 10 08:49:11 jerry kernel:        00000000 c5d8c000 c71e30c0 c013512d c71e30c0 c4275240 00000000 c71e30c0 
Sep 10 08:49:11 jerry kernel:        00000000 c71e30c0 00000030 00000000 c013519b c71e30c0 c4275240 c5d8c000 
Sep 10 08:49:11 jerry kernel: Call Trace: [d_validate+54/224] [sys_chdir+253/256] [sys_fchdir+107/208] [system_call+19/56] 
Sep 10 08:49:11 jerry kernel: Call Trace: [<c01488f6>] [<c013512d>] [<c013519b>] [<c0106edb>] 
Sep 10 08:49:11 jerry kernel: Code: 8b 03 8d 73 04 89 02 8b 43 04 c7 03 00 00 00 00 8b 56 04 89 
Quoted 1 line
Quoted 1 line

EIP; c0146cab <locks_delete_lock+b/d0> <=====

Trace; c01488f6 <locks_remove_posix+86/e0>
Trace; c013512d <filp_close+9d/b0>
Trace; c013519b <sys_close+5b/70>
Trace; c0106edb <system_call+33/38>
Code; c0146cab <locks_delete_lock+b/d0>
00000000 <_EIP>:
Code; c0146cab <locks_delete_lock+b/d0> <=====
0: 8b 03 mov (%ebx),%eax <=====
Code; c0146cad <locks_delete_lock+d/d0>
2: 8d 73 04 lea 0x4(%ebx),%esi
Code; c0146cb0 <locks_delete_lock+10/d0>
5: 89 02 mov %eax,(%edx)
Code; c0146cb2 <locks_delete_lock+12/d0>
7: 8b 43 04 mov 0x4(%ebx),%eax
Code; c0146cb5 <locks_delete_lock+15/d0>
a: c7 03 00 00 00 00 movl $0x0,(%ebx)
Code; c0146cbb <locks_delete_lock+1b/d0>
10: 8b 56 04 mov 0x4(%esi),%edx
Code; c0146cbe <locks_delete_lock+1e/d0>
13: 89 00 mov %eax,(%eax)

3 warnings issued. Results may not be reliable.

#1 lore raw
Quoted 3 lines · On Mon, 10 Sep 2001 10:02:02 -0500 Erik DeBill <[email protected]>…

I've been running into a repeatable oops in the NFS client code, apparently related to file locking.

Sep 10 08:49:10 jerry kernel: Unable to handle kernel NULL pointer

dereference at virtual address 00000000

Quoted 8 lines
Sep 10 08:49:10 jerry kernel: c0146cab
Sep 10 08:49:10 jerry kernel: *pde = 00000000
Sep 10 08:49:10 jerry kernel: Oops: 0000
Sep 10 08:49:10 jerry kernel: CPU:    0
Sep 10 08:49:10 jerry kernel: EIP:    0010:[fcntl_setlease+171/640]
Sep 10 08:49:10 jerry kernel: EIP:    0010:[<c0146cab>]
Sep 10 08:49:10 jerry kernel: EFLAGS: 00010286
Sep 10 08:49:10 jerry kernel: eax: 00000000   ebx: 00000000   ecx: c7302b60  

edx: c7926b88

Quoted 1 line
Sep 10 08:49:10 jerry kernel: esi: c4275240   edi: 00000000   ebp: bfffba10  

esp: c5d8df60

Quoted 3 lines
Sep 10 08:49:10 jerry kernel: ds: 0018   es: 0018   ss: 0018
Sep 10 08:49:10 jerry kernel: Process db2sysc (pid: 774, stackpage=c5d8d000)
Sep 10 08:49:10 jerry kernel: Stack: c7926b2c c7926b88 c4275240 00000000

c7926b88 c4275240 c01488f6 c7926b88

Quoted 1 line
Sep 10 08:49:11 jerry kernel:        00000000 c5d8c000 c71e30c0 c013512d

c71e30c0 c4275240 00000000 c71e30c0

Quoted 1 line
Sep 10 08:49:11 jerry kernel:        00000000 c71e30c0 00000030 00000000

c013519b c71e30c0 c4275240 c5d8c000

Quoted 1 line
Sep 10 08:49:11 jerry kernel: Call Trace: [d_validate+54/224]

[sys_chdir+253/256] [sys_fchdir+107/208] [system_call+19/56]

Quoted 1 line
Sep 10 08:49:11 jerry kernel: Call Trace: [<c01488f6>] [<c013512d>]

[<c013519b>] [<c0106edb>]

Quoted 1 line
Sep 10 08:49:11 jerry kernel: Code: 8b 03 8d 73 04 89 02 8b 43 04 c7 03 00 00

00 00 8b 56 04 89

Quoted 14 lines
Quoted 2 lines
Quoted 2 lines

EIP; c0146cab <locks_delete_lock+b/d0> <=====

Trace; c01488f6 <locks_remove_posix+86/e0>
Trace; c013512d <filp_close+9d/b0>
Trace; c013519b <sys_close+5b/70>
Trace; c0106edb <system_call+33/38>
Code; c0146cab <locks_delete_lock+b/d0>
00000000 <_EIP>:
Code; c0146cab <locks_delete_lock+b/d0> <=====
0: 8b 03 mov (%ebx),%eax <=====
Code; c0146cad <locks_delete_lock+d/d0>
2: 8d 73 04 lea 0x4(%ebx),%esi
Code; c0146cb0 <locks_delete_lock+10/d0>
5: 89 02 mov %eax,(%edx)

Quoted 9 lines

Code; c0146cb2 <locks_delete_lock+12/d0>
7: 8b 43 04 mov 0x4(%ebx),%eax
Code; c0146cb5 <locks_delete_lock+15/d0>
a: c7 03 00 00 00 00 movl $0x0,(%ebx)
Code; c0146cbb <locks_delete_lock+1b/d0>
10: 8b 56 04 mov 0x4(%esi),%edx
Code; c0146cbe <locks_delete_lock+1e/d0>
13: 89 00 mov %eax,(%eax)

Hm, I am sure no nfs or fs guru, but looking at

static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)

{
        struct file_lock *fl = *thisfl_p;

        *thisfl_p = fl->fl_next;
        fl->fl_next = NULL;

        list_del(&fl->fl_link);
        INIT_LIST_HEAD(&fl->fl_link);

        fasync_helper(0, fl->fl_file, 0, &fl->fl_fasync);
        if (fl->fl_fasync != NULL){
                printk(KERN_ERR "locks_delete_lock: fasync == %p\n",
fl->fl_fasync);
                fl->fl_fasync = NULL;
        }

        if (fl->fl_remove)
                fl->fl_remove(fl);

        locks_wake_up_blocks(fl, wait);
        locks_free_lock(fl);
}

in linux/fs/locks.c I would say it fails either because thisfl_p is NULL or *thisfl_p is NULL. Try securing it via:

static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)

{
        struct file_lock *fl;
	
	if (thisfl_p == NULL || *thisfl_p == NULL)
		return;

	fl = *thisfl_p;

        *thisfl_p = fl->fl_next;
        fl->fl_next = NULL;

...
}

This is for sure not the cure, but may help your setup.

Regards,
Stephan

#2 lore raw
In reply to Erik DeBill
Quoted 23 lines · Stephan von Krawczynski
Quoted 2 lines · On Mon, 10 Sep 2001 10:02:02 -0500 Erik DeBill <[email protected]>…

I've been running into a repeatable oops in the NFS client code, apparently related to file locking.

in linux/fs/locks.c I would say it fails either because thisfl_p is NULL or *thisfl_p is NULL. Try securing it via:

static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)

{
        struct file_lock *fl;

        if (thisfl_p == NULL || *thisfl_p == NULL)
                return;

        fl = *thisfl_p;

        *thisfl_p = fl->fl_next;
        fl->fl_next = NULL;
Quoted 4 lines

...
}

This is for sure not the cure, but may help your setup.

Hi, see my post and related answers [ Oops NFS Locking in 2.4.x] I have the same Problem.
Returning on *thisfl_p == NULL don't fix the trouble... Kernel no more Oops, but process stay in wait state on IO (D).

See the answers from Trond Myklebust, I think he is right...

Regards,
Olivier.

#3 lore raw
In reply to Erik DeBill
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Erik DeBill <[email protected]> writes:

> I've been running into a repeatable oops in the NFS client > code, apparently related to file locking.

This is the same Oops as was sent in by Olivier Molteni. It's due to an incorrect assumption in the locking code. To repeat:

2 processes are calling close(), which under POSIX, tries to free all locks held by a program. The problem occurs when you `dup()' a file.

In that case, both processes can call close() at the same time, thus triggering a call to filp_close().

The race boils down to:

- locks_unlock_delete() assumes that the BKL (kernel_lock()) is sufficient to protect against *thisfl_p from disappearing beneath it due to some second process.
BUT
- The call to lock() in locks_unlock_delete() sleeps when the underlying filesystem is NFS, hence 2 processes can race despite the BKL assumption.

IMHO the correct thing to do is to detach the lock from the global linked list *before* one makes the call to f_op->lock() down to the NFS code.

This would probably entail splitting locks_delete_lock() into 2
procedures: one that does the unlinking from the global list:

        *thisfl_p = fl->fl_next;
        fl->fl_next = NULL;

        list_del(&fl->fl_link);
        INIT_LIST_HEAD(&fl->fl_link);

the other that does the rest.

Could you check if the appended patch works?

Cheers,
Trond

fs/locks.c renamed from fs/locks.c.orig +24 -8
@@ -473,21 +473,26 @@
 		fl->fl_insert(fl);
 }
 
-/* Delete a lock and then free it.
- * Remove our lock from the lock lists, wake up processes that are blocked
- * waiting for this lock, notify the FS that the lock has been cleared and
- * finally free the lock.
+/*
+ * Remove lock from the lock lists
  */
-static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)
+static inline void _unhash_lock(struct file_lock **thisfl_p)
 {
 	struct file_lock *fl = *thisfl_p;
 
 	*thisfl_p = fl->fl_next;
 	fl->fl_next = NULL;
 
-	list_del(&fl->fl_link);
-	INIT_LIST_HEAD(&fl->fl_link);
+	list_del_init(&fl->fl_link);
+}
 
+/*
+ * Wake up processes that are blocked waiting for this lock,
+ * notify the FS that the lock has been cleared and
+ * finally free the lock.
+ */
+static inline void _delete_lock(struct file_lock *fl, unsigned int wait)
+{
 	fasync_helper(0, fl->fl_file, 0, &fl->fl_fasync);
 	if (fl->fl_fasync != NULL){
 		printk(KERN_ERR "locks_delete_lock: fasync == %p\n", fl->fl_fasync);
@@ -502,6 +507,15 @@
 }
 
 /*
+ * Delete a lock and then free it.
+ */
+static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)
+{
+	_unhash_lock(thisfl_p);
+	_delete_lock(*thisfl_p, wait);
+}
+
+/*
  * Call back client filesystem in order to get it to unregister a lock,
  * then delete lock. Essentially useful only in locks_remove_*().
  * Note: this must be called with the semaphore already held!
@@ -511,12 +525,13 @@
 	struct file_lock *fl = *thisfl_p;
 	int (*lock)(struct file *, int, struct file_lock *);
 
+	_unhash_lock(thisfl_p);
 	if (fl->fl_file->f_op &&
 	    (lock = fl->fl_file->f_op->lock) != NULL) {
 		fl->fl_type = F_UNLCK;
 		lock(fl->fl_file, F_SETLK, fl);
 	}
-	locks_delete_lock(thisfl_p, 0);
+	_delete_lock(fl, 0);
 }
 
 /* Determine if lock sys_fl blocks lock caller_fl. Common functionality
@@ -1661,6 +1676,7 @@
 	while ((fl = *before) != NULL) {
 		if ((fl->fl_flags & FL_POSIX) && fl->fl_owner == owner) {
 			locks_unlock_delete(before);
+			before = &inode->i_flock;
 			continue;
 		}
 		before = &fl->fl_next;
#4 lore raw
In reply to Erik DeBill
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Trond Myklebust <[email protected]> writes:

> Could you check if the appended patch works?

The previous patch had a bug in the rewrite of locks_delete_lock() which will Oops/corrupt the inode->i_flock list. Please consign to /dev/null, and try the following instead.

(Note to self: Never attempt to write any patches *before* lunch...)

Cheers,
Trond

fs/locks.c renamed from fs/locks.c.orig +26 -8
@@ -473,21 +473,26 @@
 		fl->fl_insert(fl);
 }
 
-/* Delete a lock and then free it.
- * Remove our lock from the lock lists, wake up processes that are blocked
- * waiting for this lock, notify the FS that the lock has been cleared and
- * finally free the lock.
+/*
+ * Remove lock from the lock lists
  */
-static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)
+static inline void _unhash_lock(struct file_lock **thisfl_p)
 {
 	struct file_lock *fl = *thisfl_p;
 
 	*thisfl_p = fl->fl_next;
 	fl->fl_next = NULL;
 
-	list_del(&fl->fl_link);
-	INIT_LIST_HEAD(&fl->fl_link);
+	list_del_init(&fl->fl_link);
+}
 
+/*
+ * Wake up processes that are blocked waiting for this lock,
+ * notify the FS that the lock has been cleared and
+ * finally free the lock.
+ */
+static inline void _delete_lock(struct file_lock *fl, unsigned int wait)
+{
 	fasync_helper(0, fl->fl_file, 0, &fl->fl_fasync);
 	if (fl->fl_fasync != NULL){
 		printk(KERN_ERR "locks_delete_lock: fasync == %p\n", fl->fl_fasync);
@@ -502,6 +507,17 @@
 }
 
 /*
+ * Delete a lock and then free it.
+ */
+static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)
+{
+	struct file_lock *fl = *thisfl_p;
+
+	_unhash_lock(thisfl_p);
+	_delete_lock(fl, wait);
+}
+
+/*
  * Call back client filesystem in order to get it to unregister a lock,
  * then delete lock. Essentially useful only in locks_remove_*().
  * Note: this must be called with the semaphore already held!
@@ -511,12 +527,13 @@
 	struct file_lock *fl = *thisfl_p;
 	int (*lock)(struct file *, int, struct file_lock *);
 
+	_unhash_lock(thisfl_p);
 	if (fl->fl_file->f_op &&
 	    (lock = fl->fl_file->f_op->lock) != NULL) {
 		fl->fl_type = F_UNLCK;
 		lock(fl->fl_file, F_SETLK, fl);
 	}
-	locks_delete_lock(thisfl_p, 0);
+	_delete_lock(fl, 0);
 }
 
 /* Determine if lock sys_fl blocks lock caller_fl. Common functionality
@@ -1661,6 +1678,7 @@
 	while ((fl = *before) != NULL) {
 		if ((fl->fl_flags & FL_POSIX) && fl->fl_owner == owner) {
 			locks_unlock_delete(before);
+			before = &inode->i_flock;
 			continue;
 		}
 		before = &fl->fl_next;
#5 lore raw
In reply to Erik DeBill
Quoted 9 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Trond Myklebust <[email protected]> writes:

> Could you check if the appended patch works?

The previous patch had a bug in the rewrite of locks_delete_lock() which will Oops/corrupt the inode->i_flock list. Please consign to /dev/null, and try the following instead.

(Note to self: Never attempt to write any patches *before* lunch...)

This patch seems to work. I'll keep testing, but it's been running well for an hour or so already.

Thanks,
Erik

#6 lore raw
In reply to Erik DeBill
Quoted 3 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Trond Myklebust <[email protected]> writes:

> Could you check if the appended patch works?

Hi,

First, thank you very much for posting this patch !!

Because I'm awake and at work for too long, I have no had time to test it... Now, I MUST to sleep if I want to survive ;)) But, I promise I will test your patch ASAP and I will give you a feed back on it.

Cheers
Olivier.

#7 lore raw
In reply to Erik DeBill
Quoted 4 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Trond Myklebust <[email protected]> writes:

> Could you check if the appended patch works?

Hi,

I have just finished to test with 4 PCs doing intensive lock/unlock on several NFS files for 3 hours... it seems to work !
I will now test with the initial mail application, but if I will post again only if the test fails...

Thank's a lot again !!

Cheers,
Olivier.