Skip to contentSkip to conversation

kernel BUG at fs/locks.c:1932!

DISCUSSION Fermin Molina 4 participants 6 messages Feb 17, 2006 – Feb 27, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I run samba sharing NFS mounted shares from another machine. I'm getting the following bugs in console (and in logs), when I stop samba (but not always, I think it depends of stalled locks):

lockd: unexpected unlock status: 7
lockd: unexpected unlock status: 7
lockd: unexpected unlock status: 7

------------[ cut here ]------------
kernel BUG at fs/locks.c:1932!
invalid operand: 0000 [#1]
SMP
last sysfs file: /class/vc/vcsa5/dev
Modules linked in: nfsd exportfs parport_pc lp parport nfs lockd nfs_acl sunrpc
video button battery ac i2c_piix4 i2c_core eepro100 cpqphp e1000 e100 mii floppy ext3 jbd dm_mod cciss sd_mod scsi_mod
CPU:    5
EIP:    0060:[<c017f12f>]    Not tainted VLI
EFLAGS: 00010246   (2.6.15-1.1831_FC4smp)

EIP is at locks_remove_flock+0xc5/0xe8
eax: cbaccc78 ebx: d49d4828 ecx: f7fff180 edx: 00000000
esi: d972a980 edi: d49d4760 ebp: cbaccc78 esp: e993ef08
ds: 007b es: 007b ss: 0068
Process smbd (pid: 25624, threadinfo=e993e000 task=f7e5a000)

Stack: 00000000 00000000 00000000 00000000 00000000 d972a980 00006418 00000000
       00000000 00000000 00000000 00000000 00000000 d972a980 00000202 00000000
       00000000 ffffffff 7fffffff 00000000 00000000 00000000 00000000 00000000
Call Trace:
 [<c016a5fe>] __fput+0x9a/0x186     [<c0168e71>] filp_close+0x3e/0x62
 [<c0104035>] syscall_call+0x7/0xb
Code: c0 74 0d 3b 70 34 74 15 89 c3 8b 03 85 c0 75 f3 e8 82 ca 1a 00 83 c4 68 5b 5e 5f 5d c3 0f b6 50 38 f6 c2 02 75 11 80 e2 20 75 15 <0f> 0b 8c 07 df ad 34 c0 89 c3 eb d3 89 d8 e8 0f df ff ff eb bd

Continuing in 1 seconds.

------------------------

Feb 16 16:02:06 users kernel: ------------[ cut here ]------------
Feb 16 16:02:06 users kernel: kernel BUG at fs/locks.c:1932!
Feb 16 16:02:06 users kernel: invalid operand: 0000 [#1]
Feb 16 16:02:06 users kernel: SMP
Feb 16 16:02:06 users kernel: last sysfs file: /block/dm-2/dev
Feb 16 16:02:06 users kernel: Modules linked in: nfsd exportfs parport_pc lp parport nfs lockd nfs_acl sunrpc video button battery ac i2c_piix4 i2c_core eepro100 cpqphp e1000 e100 mii floppy ext3 jbd dm_mod cciss sd_mod scsi_mod
Feb 16 16:02:06 users kernel: CPU:    5
Feb 16 16:02:06 users kernel: EIP:    0060:[<c017f12f>]    Not tainted VLI
Feb 16 16:02:06 users kernel: EFLAGS: 00010246   (2.6.15-1.1831_FC4smp)
Feb 16 16:02:06 users kernel: EIP is at locks_remove_flock+0xc5/0xe8
Feb 16 16:02:06 users kernel: eax: f2553ee8   ebx: e6ab9238   ecx: 00b6c507   edx: 00000000
Feb 16 16:02:06 users kernel: esi: ed840180   edi: e6ab9170   ebp: f2553ee8   esp: f60a8f08
Feb 16 16:02:06 users kernel: ds: 007b   es: 007b   ss: 0068
Feb 16 16:02:06 users kernel: Process smbd (pid: 4846, threadinfo=f60a8000 task=f7cbd550)
Feb 16 16:02:06 users kernel: Stack: 00000000 00000000 00000000 00000000 00000000 ed840180 000012ee 00000000
Feb 16 16:02:07 users kernel:        00000000 00000000 00000000 00000000 00000000 ed840180 00000202 00000000
Feb 16 16:02:07 users kernel:        00000000 ffffffff 7fffffff 00000000 00000000 00000000 00000000 00000000
Feb 16 16:02:07 users kernel: Call Trace:
Feb 16 16:02:08 users kernel:  [<c016a5fe>] __fput+0x9a/0x186     [<c0168e71>] filp_close+0x3e/0x62
Feb 16 16:02:08 users kernel:  [<c0104035>] syscall_call+0x7/0xb
Feb 16 16:02:08 users kernel: Code: c0 74 0d 3b 70 34 74 15 89 c3 8b 03 85 c0 75 f3 e8 82 ca 1a 00 83 c4 68 5b 5e 5f 5d c3 0f b6 50 38 f6 c2 02 75 11 80 e2 20 75 15 <0f> 0b 8c 07 df ad 34 c0 89 c3 eb d3 89 d8 e8 0f df ff ff eb bd
Feb 16 16:02:08 users kernel: Continuing in 120 seconds.

------------------------

Feb 17 10:00:02 users nmbd[11718]: [2006/02/17 10:00:02, 0] nmbd/nmbd.c:terminate(58)
Feb 17 10:00:02 users nmbd[11718]:   Got SIGTERM: going down...
Feb 17 10:00:09 users kernel:  ------------[ cut here ]------------
Feb 17 10:00:09 users kernel: kernel BUG at fs/locks.c:1932!
Feb 17 10:00:09 users kernel: invalid operand: 0000 [#2]
Feb 17 10:00:09 users kernel: SMP
Feb 17 10:00:09 users kernel: last sysfs file: /block/dm-2/dev
Feb 17 10:00:09 users kernel: Modules linked in: nfsd exportfs parport_pc lp parport nfs lockd nfs_acl sunrpc video button battery ac i2c_piix4 i2c_core eepro100 cpqphp e1000 e100 mii floppy ext3 jbd dm_mod cciss sd_mod scsi_mod
Feb 17 10:00:09 users kernel: CPU:    0
Feb 17 10:00:09 users kernel: EIP:    0060:[<c017f12f>]    Not tainted VLI
Feb 17 10:00:09 users kernel: EFLAGS: 00010246   (2.6.15-1.1831_FC4smp)
Feb 17 10:00:09 users kernel: EIP is at locks_remove_flock+0xc5/0xe8
Feb 17 10:00:09 users kernel: eax: e7399a08   ebx: ea8b6e18   ecx: f7fff180   edx: 00000000
Feb 17 10:00:09 users kernel: esi: f7dd3e80   edi: ea8b6d50   ebp: e7399a08   esp: cf7c0f08
Feb 17 10:00:09 users kernel: ds: 007b   es: 007b   ss: 0068
Feb 17 10:00:09 users kernel: Process smbd (pid: 11799, threadinfo=cf7c0000 task=f7de7550)
Feb 17 10:00:09 users kernel: Stack: 00000000 00000000 00000000 00000000 00000000 f7dd3e80 00002e17 00000000
Feb 17 10:00:09 users kernel:        00000000 00000000 00000000 00000000 00000000 f7dd3e80 00000202 00000000
Feb 17 10:00:09 users kernel:        00000000 ffffffff 7fffffff 00000000 00000000 00000000 00000000 00000000
Feb 17 10:00:09 users kernel: Call Trace:
Feb 17 10:00:09 users kernel:  [<c016a5fe>] __fput+0x9a/0x186     [<c0168e71>] filp_close+0x3e/0x62
Feb 17 10:00:09 users kernel:  [<c0104035>] syscall_call+0x7/0xb
Feb 17 10:00:10 users kernel: Code: c0 74 0d 3b 70 34 74 15 89 c3 8b 03 85 c0 75 f3 e8 82 ca 1a 00 83 c4 68 5b 5e 5f 5d c3 0f b6 50 38 f6 c2 02 75 11 80 e2 20 75 15 <0f> 0b 8c 07 df ad 34 c0 89 c3 eb d3 89 d8 e8 0f df ff ff eb bd
Feb 17 10:00:10 users kernel: Continuing in 120 seconds.

------------------------

Feb 17 14:00:02 users nmbd[20963]: [2006/02/17 14:00:02, 0] nmbd/nmbd.c:terminate(58)
Feb 17 14:00:02 users nmbd[20963]:   Got SIGTERM: going down...
Feb 17 14:00:09 users kernel: ------------[ cut here ]------------
Feb 17 14:00:09 users kernel: kernel BUG at fs/locks.c:1932!
Feb 17 14:00:09 users kernel: invalid operand: 0000 [#3]
Feb 17 14:00:09 users kernel: SMP
Feb 17 14:00:09 users kernel: last sysfs file: /block/dm-2/dev
Feb 17 14:00:09 users kernel: Modules linked in: nfsd exportfs parport_pc lp parport nfs lockd nfs_acl sunrpc video button battery ac i2c_piix4 i2c_core eepro100 cpqphp e1000 e100 mii floppy ext3 jbd dm_mod cciss sd_mod scsi_mod
Feb 17 14:00:09 users kernel: CPU:    0
Feb 17 14:00:09 users kernel: EIP:    0060:[<c017f12f>]    Not tainted VLI
Feb 17 14:00:09 users kernel: EFLAGS: 00010246   (2.6.15-1.1831_FC4smp)
Feb 17 14:00:09 users kernel: EIP is at locks_remove_flock+0xc5/0xe8
Feb 17 14:00:09 users kernel: eax: e23d3798   ebx: f7936530   ecx: f7fff180   edx: 00000000
Feb 17 14:00:09 users kernel: esi: f7f06580   edi: f7936468   ebp: e23d3798   esp: edf05f08
Feb 17 14:00:09 users kernel: ds: 007b   es: 007b   ss: 0068
Feb 17 14:00:10 users kernel: Process smbd (pid: 21685, threadinfo=edf05000 task=ddc28550)
Feb 17 14:00:10 users rpc.mountd: authenticated unmount request from RECTOR128.udl.net:699 for /usuaris/users (/usuaris/users)
Feb 17 14:00:10 users kernel: Stack: 00000000 00000000 00000000 00000000 00000000 f7f06580 000054b5 00000000
Feb 17 14:00:10 users kernel:        00000000 00000000 00000000 00000000 00000000 f7f06580 00000202 00000000
Feb 17 14:00:10 users kernel:        00000000 ffffffff 7fffffff 00000000 00000000 00000000 00000000 00000000
Feb 17 14:00:10 users kernel: Call Trace:
Feb 17 14:00:10 users kernel:  [<c016a5fe>] __fput+0x9a/0x186     [<c0168e71>] filp_close+0x3e/0x62
Feb 17 14:00:11 users kernel:  [<c0104035>] syscall_call+0x7/0xb
Feb 17 14:00:11 users kernel: Code: c0 74 0d 3b 70 34 74 15 89 c3 8b 03 85 c0 75 f3 e8 82 ca 1a 00 83 c4 68 5b 5e 5f 5d c3 0f b6 50 38 f6 c2 02 75 11 80 e2 20 75 15 <0f> 0b 8c 07 df ad 34 c0 89 c3 eb d3 89 d8 e8 0f df ff ff eb bd
Feb 17 14:00:11 users kernel: Continuing in 120 seconds.

------------------------

I think this is a problem with file locking in NFS code, and samba triggers this bug. In fact, all samba locking over NFS works very, very bad and samba shares gets stalled constantly.

I read this http://lkml.org/lkml/2005/12/21/334 but kernel I use have this patch applied (I use kernel 2.6.15.3). The client with samba is a i386 based machine with SMP (4 processors); the NFS server is a Solaris 7.

Thanks in advance,

Signature · 5 lines
Fermin Molina Ibarz
Tècnic sistemes - ASIC
Universitat de Lleida
Tel: +34 973 702151
GPG: 0x060F857A
#1 lore raw
Quoted 10 lines · Fermin Molina

Hi,

I run samba sharing NFS mounted shares from another machine. I'm getting the following bugs in console (and in logs), when I stop samba (but not always, I think it depends of stalled locks):

lockd: unexpected unlock status: 7
lockd: unexpected unlock status: 7
lockd: unexpected unlock status: 7

------------[ cut here ]------------

Hmm... The problem here is that the server is returning an unexpected
error: it is normally supposed to return "lock granted" or "grace error", but is actually returning "stale filehandle".

Anyhow, the client should be able to deal with this without Oopsing.

The attached patch ought to fix that. Please could you give it a try?

Cheers,
Trond

Author: Trond Myklebust <[email protected]>
NLM: Ensure we do not Oops in the case of an unlock

In theory, NLM specs assure us that the server will only reply LCK_GRANTED or LCK_DENIED_GRACE_PERIOD to our NLM_UNLOCK request.

In practice, we should not assume this to be the case, and the code will currently Oops if we do.

---

 fs/lockd/clntproc.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)
fs/lockd/clntproc.c +7 -1
index 7e89655..da76592 100644
@@ -644,10 +644,16 @@ nlmclnt_unlock(struct nlm_rqst *req, str
 
 	status = nlmclnt_call(req, NLMPROC_UNLOCK);
 	nlmclnt_release_lockargs(req);
+	/*
+	 * Note: the server is supposed to either grant us the unlock
+	 * request, or to deny it with NLM_LCK_DENIED_GRACE_PERIOD. In either
+	 * case, we want to unlock.
+	 */
+	do_vfs_lock(fl);
+
 	if (status < 0)
 		return status;
 
-	do_vfs_lock(fl);
 	if (resp->status == NLM_LCK_GRANTED)
 		return 0;
 
#2 lore raw
Quoted 12 lines · Trond Myklebust
Quoted 10 lines · Fermin Molina

Hi,

I run samba sharing NFS mounted shares from another machine. I'm getting the following bugs in console (and in logs), when I stop samba (but not always, I think it depends of stalled locks):

lockd: unexpected unlock status: 7
lockd: unexpected unlock status: 7
lockd: unexpected unlock status: 7

------------[ cut here ]------------
Quoted 5 lines

Hmm... The problem here is that the server is returning an unexpected
error: it is normally supposed to return "lock granted" or "grace error", but is actually returning "stale filehandle".

Anyhow, the client should be able to deal with this without Oopsing.

This seems to be a patch that should go into 2.6.16?

Quoted 32 lines

The attached patch ought to fix that. Please could you give it a try?

Cheers,
Trond

Author: Trond Myklebust <[email protected]>
NLM: Ensure we do not Oops in the case of an unlock

In theory, NLM specs assure us that the server will only reply LCK_GRANTED or LCK_DENIED_GRACE_PERIOD to our NLM_UNLOCK request.

In practice, we should not assume this to be the case, and the code will currently Oops if we do.

---

 fs/lockd/clntproc.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)
fs/lockd/clntproc.c +4 -0
index 7e89655..da76592 100644
@@ -644,10 +644,16 @@ nlmclnt_unlock(struct nlm_rqst *req, str
 
 	status = nlmclnt_call(req, NLMPROC_UNLOCK);
 	nlmclnt_release_lockargs(req);
+	/*
+	 * Note: the server is supposed to either grant us the unlock
+	 * request, or to deny it with NLM_LCK_DENIED_GRACE_PERIOD. In either
+	 * case, we want to unlock.

fs/lockd/clntproc.c nlmclnt_unlock(struct nlm_rqst *req, str · line 659

Quoted 10 lines
fs/lockd/clntproc.c +3 -1
+	 */
+	do_vfs_lock(fl);
+
 	if (status < 0)
 		return status;
 
-	do_vfs_lock(fl);
 	if (resp->status == NLM_LCK_GRANTED)
 		return 0;
 

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#3 lore raw
Quoted 13 lines · Adrian Bunk
Quoted 12 lines · Trond Myklebust
Quoted 10 lines · Fermin Molina

Hi,

I run samba sharing NFS mounted shares from another machine. I'm getting the following bugs in console (and in logs), when I stop samba (but not always, I think it depends of stalled locks):

lockd: unexpected unlock status: 7
lockd: unexpected unlock status: 7
lockd: unexpected unlock status: 7

------------[ cut here ]------------
Quoted 9 lines
Quoted 5 lines

Hmm... The problem here is that the server is returning an unexpected
error: it is normally supposed to return "lock granted" or "grace error", but is actually returning "stale filehandle".

Anyhow, the client should be able to deal with this without Oopsing.

This seems to be a patch that should go into 2.6.16?

I'd agree and suggest that it also be send to -stable since it fixes a crash actually seen "in the wild".

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#4 lore raw
In reply to Adrian Bunk
Quoted 13 lines · Adrian Bunk
Quoted 12 lines · Trond Myklebust
Quoted 10 lines · Fermin Molina

Hi,

I run samba sharing NFS mounted shares from another machine. I'm getting the following bugs in console (and in logs), when I stop samba (but not always, I think it depends of stalled locks):

lockd: unexpected unlock status: 7
lockd: unexpected unlock status: 7
lockd: unexpected unlock status: 7

------------[ cut here ]------------
Quoted 8 lines
Quoted 5 lines

Hmm... The problem here is that the server is returning an unexpected
error: it is normally supposed to return "lock granted" or "grace error", but is actually returning "stale filehandle".

Anyhow, the client should be able to deal with this without Oopsing.

This seems to be a patch that should go into 2.6.16?

I'm still waiting to hear if it fixes the problem. In the meantime, here is a slightly cleaner version, that also fixes most of those "unexpected un/lock status" messages.

Cheers,
Trond

Author: Trond Myklebust <[email protected]>
NLM: Ensure we do not Oops in the case of an unlock

In theory, NLM specs assure us that the server will only reply LCK_GRANTED or LCK_DENIED_GRACE_PERIOD to our NLM_UNLOCK request.

In practice, we should not assume this to be the case, and the code will currently Oops if we do.

---

 fs/lockd/clntproc.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)
fs/lockd/clntproc.c +7 -2
index 220058d..970b6a6 100644
@@ -662,12 +662,18 @@ nlmclnt_unlock(struct nlm_rqst *req, str
 	 * reclaimed while we're stuck in the unlock call. */
 	fl->fl_u.nfs_fl.flags &= ~NFS_LCK_GRANTED;
 
+	/*
+	 * Note: the server is supposed to either grant us the unlock
+	 * request, or to deny it with NLM_LCK_DENIED_GRACE_PERIOD. In either
+	 * case, we want to unlock.
+	 */
+	do_vfs_lock(fl);
+
 	if (req->a_flags & RPC_TASK_ASYNC) {
 		status = nlmclnt_async_call(req, NLMPROC_UNLOCK,
 					&nlmclnt_unlock_ops);
 		/* Hrmf... Do the unlock early since locks_remove_posix()
 		 * really expects us to free the lock synchronously */
-		do_vfs_lock(fl);
 		if (status < 0) {
 			nlmclnt_release_lockargs(req);
 			kfree(req);
@@ -680,7 +686,6 @@ nlmclnt_unlock(struct nlm_rqst *req, str
 	if (status < 0)
 		return status;
 
-	do_vfs_lock(fl);
 	if (resp->status == NLM_LCK_GRANTED)
 		return 0;
 
#5 lore raw
In reply to Trond Myklebust
Quoted 12 lines · Trond Myklebust
Quoted 10 lines · Fermin Molina

Hi,

I run samba sharing NFS mounted shares from another machine. I'm getting the following bugs in console (and in logs), when I stop samba (but not always, I think it depends of stalled locks):

lockd: unexpected unlock status: 7
lockd: unexpected unlock status: 7
lockd: unexpected unlock status: 7

------------[ cut here ]------------
Quoted 7 lines

Hmm... The problem here is that the server is returning an unexpected
error: it is normally supposed to return "lock granted" or "grace error", but is actually returning "stale filehandle".

Anyhow, the client should be able to deal with this without Oopsing.

The attached patch ought to fix that. Please could you give it a try?

Sorry for the delay. I cannot try the patch in this moment, because my server is in production. I will try the patch next days, but I think I cannot reproduce the error, because I made "local" some data mounted with NFS.

In samba list, some people that have data mounted with NFS are getting similar problems. I will let they know about this patch.

Thanks a lot,

Signature · 5 lines
Fermin Molina Ibarz
Tècnic sistemes - ASIC
Universitat de Lleida
Tel: +34 973 702151
GPG: 0x060F857A