Skip to contentSkip to conversation

Oops in lockd (2.6.20.1)

DISCUSSION Jan Rekorajski 3 participants 5 messages Mar 5, 2007

Subsystems (2)
NFS client NFS server
lore
Share & tools
lore

Jan Rekorajski

author
#0 lore raw

Hi,
I have NFS server, serving /, /home and /usr to some 100 workstations, and the following oops bit me today in the morning.

yellow11 is a workstation (probably someone rebooted it with a power switch).
Server is 4xDualCore Opteron with 32GB RAM and 500GB of exported filesystems .

This kernel is heavily patched, but nothing AFAIK touches NFS or anything near it.

lockd: server yellow11 not responding, timed out
lockd: couldn't create RPC handle for yellow11
lockd: server yellow11 not responding, timed out
lockd: couldn't create RPC handle for yellow11

general protection fault: 0000 [1] SMP 
CPU 0 
Modules linked in: vroot nfsd exportfs lockd nfs_acl sunrpc tg3 ext2 mbcache dm_mod 8021q
Pid: 4114:#0, comm: lockd Not tainted 2.6.20.1 #3
RIP: 0010:[<ffffffff8807e058>]  [<ffffffff8807e058>] :lockd:nlm_release_host+0x28/0x110
RSP: 0018:ffff8107c5075e40  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0008000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0008000000000000
RBP: ffff81077ce4d080 R08: 0000000000000000 R09: 0000000000000000
R10: 2222222222222222 R11: 2222222222222222 R12: 00000000ffffffdb
R13: 000000000000000a R14: ffff8107c5075ea0 R15: ffffffff880849c0
FS:  00002b5d76afeb00(0000) GS:ffffffff805e0000(0000) knlGS:00000000f7b5a6c0
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000f7ee0ba9 CR3: 00000005d50db000 CR4: 00000000000006e0
Process lockd (pid: 4114[#0], threadinfo ffff8107c5074000, task ffff8107d8e1c100)
Stack:  ffff8107b75ab000 ffffffff8807cc1d ffff8107b75ab000 ffffffff8807cfca
0000000000000000 ffff8107b75ab038 ffff8107da3132c0 ffff8107b75ab000
ffff8106c24bd4c0 ffffffff8807e7a0 0000000000000000 ffffffff8807d032
Call Trace:
[<ffffffff8807cc1d>] :lockd:nlm_release_call+0xd/0x20
[<ffffffff8807cfca>] :lockd:__nlm_async_call+0x9a/0xc0
[<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
[<ffffffff8807d032>] :lockd:nlm_async_call+0x42/0x50
[<ffffffff8807fbae>] :lockd:nlmsvc_grant_blocked+0x12e/0x170
[<ffffffff8807fdd3>] :lockd:nlmsvc_retry_blocked+0x73/0xa0
[<ffffffff8807e8d7>] :lockd:lockd+0x137/0x280
[<ffffffff80253df8>] child_rip+0xa/0x12
[<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
[<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
[<ffffffff80253dee>] child_rip+0x0/0x12


Code: 8b 73 74 85 f6 79 25 48 c7 c1 60 47 08 88 ba 24 01 00 00 48 
RIP  [<ffffffff8807e058>] :lockd:nlm_release_host+0x28/0x110

RSP <ffff8107c5075e40>

Signature · 12 lines
Jan Rekorajski            |  ALL SUSPECTS ARE GUILTY. PERIOD!
baggins<at>mimuw.edu.pl   |  OTHERWISE THEY WOULDN'T BE SUSPECTS, WOULD THEY?
BOFH, MANIAC              |                   -- TROOPS by Kevin Rubio

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#1 lore raw
Quoted 15 lines · Jan Rekorajski

[<ffffffff8807cc1d>] :lockd:nlm_release_call+0xd/0x20 [<ffffffff8807cfca>] :lockd:__nlm_async_call+0x9a/0xc0 [<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280 [<ffffffff8807d032>] :lockd:nlm_async_call+0x42/0x50 [<ffffffff8807fbae>] :lockd:nlmsvc_grant_blocked+0x12e/0x170 [<ffffffff8807fdd3>] :lockd:nlmsvc_retry_blocked+0x73/0xa0 [<ffffffff8807e8d7>] :lockd:lockd+0x137/0x280

[<ffffffff80253df8>] child_rip+0xa/0x12
[<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
[<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
[<ffffffff80253dee>] child_rip+0x0/0x12


Code: 8b 73 74 85 f6 79 25 48 c7 c1 60 47 08 88 ba 24 01 00 00 48
RIP  [<ffffffff8807e058>] :lockd:nlm_release_host+0x28/0x110
Quoted 1 line

RSP <ffff8107c5075e40>

It seems it's dying on a bogus nlm_host pointer. In __nlm_async_call we have:

        status = rpc_call_async(clnt, msg, RPC_TASK_ASYNC, tk_ops, req);
        if (status == 0)
                return 0;
out_err:
        nlm_release_call(req);
        return status;
}

So we ended up in nlm_release_call because rpc_call_async returned an error. However, when rpc_call_async fails, it calls the rpc_call_done handler. In this case, it's nlmsvc_release_block, which ends up freeing the nlm_block object, which does nlm_release_call(block->b_call).

So it appears to me that it's a double free.

IMHO, when rpc_call_async fails, we should just return status right away.

A similar argument applies to nlmsvc_grant_blocked, where we try to release the nlm_block object if nlm_async_call() returns an error. When we get there, the block object will have been freed already.

Olaf

Signature · 11 lines
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
[email protected] |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#2 lore raw
Quoted 35 lines · Olaf Kirch
Quoted 15 lines · Jan Rekorajski

[<ffffffff8807cc1d>] :lockd:nlm_release_call+0xd/0x20 [<ffffffff8807cfca>] :lockd:__nlm_async_call+0x9a/0xc0 [<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280 [<ffffffff8807d032>] :lockd:nlm_async_call+0x42/0x50 [<ffffffff8807fbae>] :lockd:nlmsvc_grant_blocked+0x12e/0x170 [<ffffffff8807fdd3>] :lockd:nlmsvc_retry_blocked+0x73/0xa0 [<ffffffff8807e8d7>] :lockd:lockd+0x137/0x280

[<ffffffff80253df8>] child_rip+0xa/0x12
[<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
[<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
[<ffffffff80253dee>] child_rip+0x0/0x12


Code: 8b 73 74 85 f6 79 25 48 c7 c1 60 47 08 88 ba 24 01 00 00 48
RIP  [<ffffffff8807e058>] :lockd:nlm_release_host+0x28/0x110
Quoted 1 line

RSP <ffff8107c5075e40>

It seems it's dying on a bogus nlm_host pointer. In __nlm_async_call we have:

        status = rpc_call_async(clnt, msg, RPC_TASK_ASYNC, tk_ops, req);
        if (status == 0)
                return 0;
out_err:
        nlm_release_call(req);
        return status;
}

So we ended up in nlm_release_call because rpc_call_async returned an error. However, when rpc_call_async fails, it calls the rpc_call_done handler. In this case, it's nlmsvc_release_block, which ends up freeing the nlm_block object, which does nlm_release_call(block->b_call).

So it appears to me that it's a double free.

Quoted 7 lines

IMHO, when rpc_call_async fails, we should just return status right away.

A similar argument applies to nlmsvc_grant_blocked, where we try to release the nlm_block object if nlm_async_call() returns an error. When we get there, the block object will have been freed already.

Olaf

Already fixed in 2.6.21-rc1 mainline. See

http://client.linux-nfs.org/Linux-2.6.x/2.6.20/linux-2.6.20-001-fix_nlm_async_call.dif

Cheers,
Trond

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#3 lore raw
Quoted 1 line · Trond Myklebust

Already fixed in 2.6.21-rc1 mainline.

Nice :)

Olaf

Signature · 11 lines
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
[email protected] |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
author
#4 lore raw
In reply to Trond Myklebust
Quoted 3 lines · Trond Myklebust

Already fixed in 2.6.21-rc1 mainline. See

http://client.linux-nfs.org/Linux-2.6.x/2.6.20/linux-2.6.20-001-fix_nlm_async_call.dif

Thank you :)
I'll test it tomorrow and I really hope my problems with unresponsive/dying lockd will go away at last.

Jan

Signature · 12 lines
Jan Rekorajski            |  ALL SUSPECTS ARE GUILTY. PERIOD!
baggins<at>mimuw.edu.pl   |  OTHERWISE THEY WOULDN'T BE SUSPECTS, WOULD THEY?
BOFH, MANIAC              |                   -- TROOPS by Kevin Rubio

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs