nfs4 hang
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): 2212784.91274721147769.JavaMail.javamailuser@localhost
Thanks Bruce, it does appear that idmapd is somehow related to this issue. However, I'm stumped because it does exist on the client, along with all the libraries it requires (according to ldd, anyway). Is there an easy way to disable idmapd while still running nfs4 to determine if it is causing the hang?
Chris
Quoted 295 lines · ----- "J. Bruce Fields
Quoted 1 line · Christopher Hawkins
Actually this issue seems to be on the server side. I changed the
export to read only so the clients couldn't accidentally overwrite anything... and the issue persists! Will keep testing.
Probably rpc.idmapd is trying to access a file on NFSv4.
(Which doesn't work because the NFSv4 client needs idmapd to respond before it can continue, creating a deadlock.)
--b.
Quoted 4 lines
Quoted 1 line · ----- "Christopher Hawkins
I managed to make it go a little further and now the nodes will
finish
Quoted 1 line
Quoted 1 line
booting, though they hang when I start other nodes and start
using
Quoted 1 line
Quoted 1 line
files. And this was captured in the syslog from one of the nodes
where
Quoted 4 lines
Quoted 4 lines
nfs4 goes unresponsive. Not familiar enough with nfs4 to read it, however. Does this point to a culprit?
It's as if the nodes are writing all over each others
connections,
Quoted 1 line
Quoted 1 line
like maybe I have a shared directory where each node must have its
own
Quoted 2 lines
Quoted 2 lines
private directory. They each have their own
/var/liband/var/lib/nfs/rpc_pipefsmounted. Anything else that nfs clientswrite
Quoted 3 lines
Quoted 3 lines
to (aside from proc and dev)?
May 21 17:13:45 192.168.1.243 kernel: INFO: task rpciod/0:1297blocked
Quoted 4 lines
Quoted 4 lines
for more than 120 seconds.
May 21 17:13:45 192.168.1.243 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 21 17:13:45 192.168.1.243 kernel: rpciod/0 D 000000972580
Quoted 39 lines
Quoted 39 lines
1297 7 1306 340 (L-TLB)
May 21 17:13:45 192.168.1.243 kernel: de89de40 00000046 1dfa3ac0 00000097 de8ca3b4 c1984c78 00000001 0000000a May 21 17:13:45 192.168.1.243 kernel: de8ce000 1dfa3ac0 00000097 00000000 00000000 de8ce10c c1787100 f54ebac0 May 21 17:13:45 192.168.1.243 kernel: c041ebcc 00000000 00000000 00000003 00000296 c1984b00 c17875bc de89de68 May 21 17:13:45 192.168.1.243 kernel: Call Trace: May 21 17:13:45 192.168.1.243 kernel: [<c041ebcc>] __wake_up+0x2a/0x3d May 21 17:13:45 192.168.1.243 kernel: [<f8916155>] rpc_queue_upcall+0x92/0x99 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f8992b04>] nfs_idmap_id+0x18b/0x1f6 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c041f7a7>] default_wake_function+0x0/0xc May 21 17:13:45 192.168.1.243 kernel: [<f8992ba9>] nfs_map_name_to_uid+0x1b/0x1f [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898d719>] decode_getfattr+0x611/0xbbd [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898c0bd>] decode_open+0xf3/0x25e [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898e47c>] decode_getfh+0x61/0xa3 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898e4be>] nfs4_xdr_dec_open+0x0/0xa2 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898e533>] nfs4_xdr_dec_open+0x75/0xa2 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f890e755>] rpcauth_unwrap_resp+0x5b/0x60 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f89097e9>] call_decode+0x437/0x47d [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f898e4be>] nfs4_xdr_dec_open+0x0/0xa2 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f890df54>] __rpc_execute+0x7a/0x1f8 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<c0433276>] run_workqueue+0x78/0xb5 May 21 17:13:45 192.168.1.243 kernel: [<f890e0d2>]Quoted 8 lines
Quoted 8 lines
rpc_async_schedule+0x0/0x5[sunrpc]May 21 17:13:45 192.168.1.243 kernel: [<c0433b2a>] worker_thread+0xd9/0x10b May 21 17:13:45 192.168.1.243 kernel: [<c041f7a7>] default_wake_function+0x0/0xc May 21 17:13:45 192.168.1.243 kernel: [<c0433a51>] worker_thread+0x0/0x10b May 21 17:13:45 192.168.1.243 kernel: [<c0435f43>]
kthread+0xc0/0xedQuoted 1 line
Quoted 1 line
May 21 17:13:45 192.168.1.243 kernel: [<c0435e83>]
kthread+0x0/0xedQuoted 4 lines
Quoted 4 lines
May 21 17:13:45 192.168.1.243 kernel: [<c0405c53>] kernel_thread_helper+0x7/0x10 May 21 17:13:45 192.168.1.243 kernel: ======================= May 21 17:13:45 192.168.1.243 kernel: INFO: task java:13420blocked
Quoted 4 lines
Quoted 4 lines
for more than 120 seconds.
May 21 17:13:45 192.168.1.243 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 21 17:13:45 192.168.1.243 kernel: java D 000000982068
Quoted 63 lines
Quoted 63 lines
13420 1 13421 13386 (NOTLB)
May 21 17:13:45 192.168.1.243 kernel: f6bbfab0 00000086 ada06900 00000098 00000000 c06ba340 dfc6d280 0000000a May 21 17:13:45 192.168.1.243 kernel: f5781000 ada06900 00000098 00000000 00000000 f578110c c1787100 f649d3c0 May 21 17:13:45 192.168.1.243 kernel: 00000098 00000000 de7c6804 c1787100 de7c6800 f6bbfb1c c04ec500 ffffffff May 21 17:13:45 192.168.1.243 kernel: Call Trace: May 21 17:13:45 192.168.1.243 kernel: [<c04ec500>] __next_cpu+0x12/0x21 May 21 17:13:45 192.168.1.243 kernel: [<c061cf25>] __mutex_lock_slowpath+0x4d/0x7c May 21 17:13:45 192.168.1.243 kernel: [<c061cf63>] .text.lock.mutex+0xf/0x14 May 21 17:13:45 192.168.1.243 kernel: [<f89929f2>] nfs_idmap_id+0x79/0x1f6 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c041f7a7>] default_wake_function+0x0/0xc May 21 17:13:45 192.168.1.243 kernel: [<f8992ba9>] nfs_map_name_to_uid+0x1b/0x1f [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898d719>] decode_getfattr+0x611/0xbbd [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c042d5f5>] lock_timer_base+0x15/0x2f May 21 17:13:45 192.168.1.243 kernel: [<f898e025>] nfs4_xdr_dec_getattr+0x0/0x45 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898e063>] nfs4_xdr_dec_getattr+0x3e/0x45 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f890e755>] rpcauth_unwrap_resp+0x5b/0x60 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f89097e9>] call_decode+0x437/0x47d [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f898e025>] nfs4_xdr_dec_getattr+0x0/0x45 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f890df54>] __rpc_execute+0x7a/0x1f8 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f8909951>] rpc_call_sync+0x6b/0x91 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f89879f9>] nfs4_proc_getattr+0x6f/0x8b [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f897a157>] __nfs_revalidate_inode+0x120/0x249 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c061c526>] schedule+0xa22/0xa55 May 21 17:13:45 192.168.1.243 kernel: [<f8975dbe>] nfs_lookup_revalidate+0x1e8/0x3d7 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c042d5f5>] lock_timer_base+0x15/0x2f May 21 17:13:45 192.168.1.243 kernel: [<f898baea>] decode_compound_hdr+0x35/0x72 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898b9da>] decode_op_hdr+0xd/0x58 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898bf3c>] nfs4_xdr_dec_access+0x54/0x8d [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c042d5f5>] lock_timer_base+0x15/0x2f May 21 17:13:45 192.168.1.243 kernel: [<c042d769>] __mod_timer+0xda/0xe4 May 21 17:13:45 192.168.1.243 kernel: [<f890e4f8>] rpc_wake_up_next+0x11f/0x126 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<c042e0d2>] recalc_sigpending+0xe/0x20 May 21 17:13:45 192.168.1.243 kernel: [<c042e194>]Quoted 8 lines
Quoted 8 lines
sigprocmask+0xb0/0xceMay 21 17:13:45 192.168.1.243 kernel: [<f890996f>] rpc_call_sync+0x89/0x91 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f8986c9d>] _nfs4_proc_access+0xa7/0xdb [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f8976912>] nfs_access_add_cache+0xad/0x135 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c048b233>]
dput+0x22/0xedQuoted 40 lines
Quoted 40 lines
May 21 17:13:45 192.168.1.243 kernel: [<f897816c>] nfs_open_revalidate+0x186/0x1da [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c0481e46>] do_lookup+0x135/0x174 May 21 17:13:45 192.168.1.243 kernel: [<c0483652>] __link_path_walk+0x314/0xd33 May 21 17:13:45 192.168.1.243 kernel: [<c04840a9>] link_path_walk+0x38/0x95 May 21 17:13:45 192.168.1.243 kernel: [<c048446a>] do_path_lookup+0x219/0x27f May 21 17:13:45 192.168.1.243 kernel: [<c0483652>] __link_path_walk+0x314/0xd33 May 21 17:13:45 192.168.1.243 kernel: [<c04840a9>] link_path_walk+0x38/0x95 May 21 17:13:45 192.168.1.243 kernel: [<c048446a>] do_path_lookup+0x219/0x27f May 21 17:13:45 192.168.1.243 kernel: [<c0484bc4>] __user_walk_fd+0x29/0x3a May 21 17:13:45 192.168.1.243 kernel: [<c047e380>] vfs_stat_fd+0x15/0x3c May 21 17:13:45 192.168.1.243 kernel: [<c041f7a7>] default_wake_function+0x0/0xc May 21 17:13:45 192.168.1.243 kernel: [<c047e434>] sys_stat64+0xf/0x23 May 21 17:13:45 192.168.1.243 kernel: [<c042c7a5>] getnstimeofday+0x30/0xb6 May 21 17:13:45 192.168.1.243 kernel: [<c04f0b41>] copy_to_user+0x31/0x48 May 21 17:13:45 192.168.1.243 kernel: [<c04356e7>] sys_clock_gettime+0x58/0x7e May 21 17:13:45 192.168.1.243 kernel: [<c0404f17>] syscall_call+0x7/0xb May 21 17:13:45 192.168.1.243 kernel: =======================Christopher Hawkins
Cell: 315-882-2551
Fax: 716-662-0788Quoted 7 lines
Quoted 7 lines
Quoted 1 line · ----- "J. Bruce Fields
On Thu, May 20, 2010 at 03:54:28PM -0400, Christopher Hawkins
Quoted 3 lines · wrote:
Quoted 3 lines
Hello,
I have a bash script that sets up a root filesystem over NFS.
The
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
method works fine over NFS3 but hangs when using NFS4 as a
transport.
Quoted 5 lines
Hoping someone can help me figure out why?
Quoted 4 lines
In a nutshell:
1. The node boots into a ramdisk that has a few thousand files
in
Quoted 3 lines
Quoted 3 lines
it
Quoted 2 lines
already
Quoted 1 line
2. connects to an NFS server exporting / and mounts that in a
tmp
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
dir
Quoted 1 line
3. bash (on the booting node) loops through the local and
remote
Quoted 1 line
Quoted 1 line
Quoted 1 line
filesystem and creates a local symlink to anything in the nfs
root
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
that it doesn't already have.
Wild.
Quoted 1 line
Export options are: /
192.168.1.0/24(rw,no_root_squash,no_subtree_check,fsid=0)
Quoted 3 lines
Mount options are: rw,sync,noatime,rsize=1024,wsize=1024
rpcidmapd, portmap, statd are running on server and client,
and
Quoted 1 line
Quoted 1 line
Quoted 1 line
rpc_pipefsis mounted on both as well. The script currentlymakes
Quoted 1 line
Quoted 1 line
Quoted 1 line
about 6200 links, total. Nfs4 hangs about 90% of the way
through. I
Quoted 1 line
Quoted 1 line
Quoted 1 line
removed the directory it was hanging on (a simple, empty dir)
and
Quoted 4 lines
Quoted 4 lines
then
Quoted 3 lines
it just hung on the next one.
Is it possible there's some idmap deadlock? (If say an upcall
to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
idmapd
is hanging because the mapping requires access to some filethat
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
isn't
there yet?)A sysrq-T trace might help.
--b.
Quoted 1 line
Really appreciate any help or pointers on where to look for
more
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
info. Thanks!
Quoted 7 lines
Chris
--
To unsubscribe from this list: send the line "unsubscribeQuoted 7 lines
Quoted 7 lines
linux-nfs"
Quoted 3 lines
in
Quoted 2 lines
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribelinux-nfs"
Quoted 3 lines
Quoted 3 lines
in
the body of a message to [email protected] More majordomo info athttp://vger.kernel.org/majordomo-info.html
Quoted 2 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs"in
Quoted 2 lines
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Christopher Hawkins
Thanks Bruce, it does appear that idmapd is somehow related to this issue. However, I'm stumped because it does exist on the client, along with all the libraries it requires (according to ldd, anyway).
And /etc/passwd, and whatever other files it might decide it needs to open to do its work?
Quoted 3 lines
Is
there an easy way to disable idmapd while still running nfs4 to determine if it is causing the hang?
Not that I know of.
--b.
Quoted 304 lines
Chris
Quoted 285 lines · ----- "J. Bruce Fields
Quoted 1 line · Christopher Hawkins
Actually this issue seems to be on the server side. I changed the
export to read only so the clients couldn't accidentally overwrite anything... and the issue persists! Will keep testing.
Probably rpc.idmapd is trying to access a file on NFSv4.
(Which doesn't work because the NFSv4 client needs idmapd to respond before it can continue, creating a deadlock.)
--b.
Quoted 4 lines
Quoted 1 line · ----- "Christopher Hawkins
I managed to make it go a little further and now the nodes will
finish
Quoted 1 line
Quoted 1 line
booting, though they hang when I start other nodes and start
using
Quoted 1 line
Quoted 1 line
files. And this was captured in the syslog from one of the nodes
where
Quoted 4 lines
Quoted 4 lines
nfs4 goes unresponsive. Not familiar enough with nfs4 to read it, however. Does this point to a culprit?
It's as if the nodes are writing all over each others
connections,
Quoted 1 line
Quoted 1 line
like maybe I have a shared directory where each node must have its
own
Quoted 2 lines
Quoted 2 lines
private directory. They each have their own
/var/liband/var/lib/nfs/rpc_pipefsmounted. Anything else that nfs clientswrite
Quoted 3 lines
Quoted 3 lines
to (aside from proc and dev)?
May 21 17:13:45 192.168.1.243 kernel: INFO: task rpciod/0:1297blocked
Quoted 4 lines
Quoted 4 lines
for more than 120 seconds.
May 21 17:13:45 192.168.1.243 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 21 17:13:45 192.168.1.243 kernel: rpciod/0 D 000000972580
Quoted 39 lines
Quoted 39 lines
1297 7 1306 340 (L-TLB)
May 21 17:13:45 192.168.1.243 kernel: de89de40 00000046 1dfa3ac0 00000097 de8ca3b4 c1984c78 00000001 0000000a May 21 17:13:45 192.168.1.243 kernel: de8ce000 1dfa3ac0 00000097 00000000 00000000 de8ce10c c1787100 f54ebac0 May 21 17:13:45 192.168.1.243 kernel: c041ebcc 00000000 00000000 00000003 00000296 c1984b00 c17875bc de89de68 May 21 17:13:45 192.168.1.243 kernel: Call Trace: May 21 17:13:45 192.168.1.243 kernel: [<c041ebcc>] __wake_up+0x2a/0x3d May 21 17:13:45 192.168.1.243 kernel: [<f8916155>] rpc_queue_upcall+0x92/0x99 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f8992b04>] nfs_idmap_id+0x18b/0x1f6 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c041f7a7>] default_wake_function+0x0/0xc May 21 17:13:45 192.168.1.243 kernel: [<f8992ba9>] nfs_map_name_to_uid+0x1b/0x1f [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898d719>] decode_getfattr+0x611/0xbbd [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898c0bd>] decode_open+0xf3/0x25e [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898e47c>] decode_getfh+0x61/0xa3 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898e4be>] nfs4_xdr_dec_open+0x0/0xa2 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898e533>] nfs4_xdr_dec_open+0x75/0xa2 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f890e755>] rpcauth_unwrap_resp+0x5b/0x60 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f89097e9>] call_decode+0x437/0x47d [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f898e4be>] nfs4_xdr_dec_open+0x0/0xa2 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f890df54>] __rpc_execute+0x7a/0x1f8 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<c0433276>] run_workqueue+0x78/0xb5 May 21 17:13:45 192.168.1.243 kernel: [<f890e0d2>]Quoted 8 lines
Quoted 8 lines
rpc_async_schedule+0x0/0x5[sunrpc]May 21 17:13:45 192.168.1.243 kernel: [<c0433b2a>] worker_thread+0xd9/0x10b May 21 17:13:45 192.168.1.243 kernel: [<c041f7a7>] default_wake_function+0x0/0xc May 21 17:13:45 192.168.1.243 kernel: [<c0433a51>] worker_thread+0x0/0x10b May 21 17:13:45 192.168.1.243 kernel: [<c0435f43>]
kthread+0xc0/0xedQuoted 1 line
Quoted 1 line
May 21 17:13:45 192.168.1.243 kernel: [<c0435e83>]
kthread+0x0/0xedQuoted 4 lines
Quoted 4 lines
May 21 17:13:45 192.168.1.243 kernel: [<c0405c53>] kernel_thread_helper+0x7/0x10 May 21 17:13:45 192.168.1.243 kernel: ======================= May 21 17:13:45 192.168.1.243 kernel: INFO: task java:13420blocked
Quoted 4 lines
Quoted 4 lines
for more than 120 seconds.
May 21 17:13:45 192.168.1.243 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 21 17:13:45 192.168.1.243 kernel: java D 000000982068
Quoted 63 lines
Quoted 63 lines
13420 1 13421 13386 (NOTLB)
May 21 17:13:45 192.168.1.243 kernel: f6bbfab0 00000086 ada06900 00000098 00000000 c06ba340 dfc6d280 0000000a May 21 17:13:45 192.168.1.243 kernel: f5781000 ada06900 00000098 00000000 00000000 f578110c c1787100 f649d3c0 May 21 17:13:45 192.168.1.243 kernel: 00000098 00000000 de7c6804 c1787100 de7c6800 f6bbfb1c c04ec500 ffffffff May 21 17:13:45 192.168.1.243 kernel: Call Trace: May 21 17:13:45 192.168.1.243 kernel: [<c04ec500>] __next_cpu+0x12/0x21 May 21 17:13:45 192.168.1.243 kernel: [<c061cf25>] __mutex_lock_slowpath+0x4d/0x7c May 21 17:13:45 192.168.1.243 kernel: [<c061cf63>] .text.lock.mutex+0xf/0x14 May 21 17:13:45 192.168.1.243 kernel: [<f89929f2>] nfs_idmap_id+0x79/0x1f6 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c041f7a7>] default_wake_function+0x0/0xc May 21 17:13:45 192.168.1.243 kernel: [<f8992ba9>] nfs_map_name_to_uid+0x1b/0x1f [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898d719>] decode_getfattr+0x611/0xbbd [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c042d5f5>] lock_timer_base+0x15/0x2f May 21 17:13:45 192.168.1.243 kernel: [<f898e025>] nfs4_xdr_dec_getattr+0x0/0x45 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898e063>] nfs4_xdr_dec_getattr+0x3e/0x45 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f890e755>] rpcauth_unwrap_resp+0x5b/0x60 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f89097e9>] call_decode+0x437/0x47d [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f898e025>] nfs4_xdr_dec_getattr+0x0/0x45 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f890df54>] __rpc_execute+0x7a/0x1f8 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f8909951>] rpc_call_sync+0x6b/0x91 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f89879f9>] nfs4_proc_getattr+0x6f/0x8b [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f897a157>] __nfs_revalidate_inode+0x120/0x249 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c061c526>] schedule+0xa22/0xa55 May 21 17:13:45 192.168.1.243 kernel: [<f8975dbe>] nfs_lookup_revalidate+0x1e8/0x3d7 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c042d5f5>] lock_timer_base+0x15/0x2f May 21 17:13:45 192.168.1.243 kernel: [<f898baea>] decode_compound_hdr+0x35/0x72 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898b9da>] decode_op_hdr+0xd/0x58 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f898bf3c>] nfs4_xdr_dec_access+0x54/0x8d [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c042d5f5>] lock_timer_base+0x15/0x2f May 21 17:13:45 192.168.1.243 kernel: [<c042d769>] __mod_timer+0xda/0xe4 May 21 17:13:45 192.168.1.243 kernel: [<f890e4f8>] rpc_wake_up_next+0x11f/0x126 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<c042e0d2>] recalc_sigpending+0xe/0x20 May 21 17:13:45 192.168.1.243 kernel: [<c042e194>]Quoted 8 lines
Quoted 8 lines
sigprocmask+0xb0/0xceMay 21 17:13:45 192.168.1.243 kernel: [<f890996f>] rpc_call_sync+0x89/0x91 [sunrpc] May 21 17:13:45 192.168.1.243 kernel: [<f8986c9d>] _nfs4_proc_access+0xa7/0xdb [nfs] May 21 17:13:45 192.168.1.243 kernel: [<f8976912>] nfs_access_add_cache+0xad/0x135 [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c048b233>]
dput+0x22/0xedQuoted 40 lines
Quoted 40 lines
May 21 17:13:45 192.168.1.243 kernel: [<f897816c>] nfs_open_revalidate+0x186/0x1da [nfs] May 21 17:13:45 192.168.1.243 kernel: [<c0481e46>] do_lookup+0x135/0x174 May 21 17:13:45 192.168.1.243 kernel: [<c0483652>] __link_path_walk+0x314/0xd33 May 21 17:13:45 192.168.1.243 kernel: [<c04840a9>] link_path_walk+0x38/0x95 May 21 17:13:45 192.168.1.243 kernel: [<c048446a>] do_path_lookup+0x219/0x27f May 21 17:13:45 192.168.1.243 kernel: [<c0483652>] __link_path_walk+0x314/0xd33 May 21 17:13:45 192.168.1.243 kernel: [<c04840a9>] link_path_walk+0x38/0x95 May 21 17:13:45 192.168.1.243 kernel: [<c048446a>] do_path_lookup+0x219/0x27f May 21 17:13:45 192.168.1.243 kernel: [<c0484bc4>] __user_walk_fd+0x29/0x3a May 21 17:13:45 192.168.1.243 kernel: [<c047e380>] vfs_stat_fd+0x15/0x3c May 21 17:13:45 192.168.1.243 kernel: [<c041f7a7>] default_wake_function+0x0/0xc May 21 17:13:45 192.168.1.243 kernel: [<c047e434>] sys_stat64+0xf/0x23 May 21 17:13:45 192.168.1.243 kernel: [<c042c7a5>] getnstimeofday+0x30/0xb6 May 21 17:13:45 192.168.1.243 kernel: [<c04f0b41>] copy_to_user+0x31/0x48 May 21 17:13:45 192.168.1.243 kernel: [<c04356e7>] sys_clock_gettime+0x58/0x7e May 21 17:13:45 192.168.1.243 kernel: [<c0404f17>] syscall_call+0x7/0xb May 21 17:13:45 192.168.1.243 kernel: =======================Christopher Hawkins
Cell: 315-882-2551
Fax: 716-662-0788Quoted 7 lines
Quoted 7 lines
Quoted 1 line · ----- "J. Bruce Fields
On Thu, May 20, 2010 at 03:54:28PM -0400, Christopher Hawkins
Quoted 3 lines · wrote:
Quoted 3 lines
Hello,
I have a bash script that sets up a root filesystem over NFS.
The
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
method works fine over NFS3 but hangs when using NFS4 as a
transport.
Quoted 5 lines
Hoping someone can help me figure out why?
Quoted 4 lines
In a nutshell:
1. The node boots into a ramdisk that has a few thousand files
in
Quoted 3 lines
Quoted 3 lines
it
Quoted 2 lines
already
Quoted 1 line
2. connects to an NFS server exporting / and mounts that in a
tmp
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
dir
Quoted 1 line
3. bash (on the booting node) loops through the local and
remote
Quoted 1 line
Quoted 1 line
Quoted 1 line
filesystem and creates a local symlink to anything in the nfs
root
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
that it doesn't already have.
Wild.
Quoted 1 line
Export options are: /
192.168.1.0/24(rw,no_root_squash,no_subtree_check,fsid=0)
Quoted 3 lines
Mount options are: rw,sync,noatime,rsize=1024,wsize=1024
rpcidmapd, portmap, statd are running on server and client,
and
Quoted 1 line
Quoted 1 line
Quoted 1 line
rpc_pipefsis mounted on both as well. The script currentlymakes
Quoted 1 line
Quoted 1 line
Quoted 1 line
about 6200 links, total. Nfs4 hangs about 90% of the way
through. I
Quoted 1 line
Quoted 1 line
Quoted 1 line
removed the directory it was hanging on (a simple, empty dir)
and
Quoted 4 lines
Quoted 4 lines
then
Quoted 3 lines
it just hung on the next one.
Is it possible there's some idmap deadlock? (If say an upcall
to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
idmapd
is hanging because the mapping requires access to some filethat
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
isn't
there yet?)A sysrq-T trace might help.
--b.
Quoted 1 line
Really appreciate any help or pointers on where to look for
more
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
info. Thanks!
Quoted 7 lines
Chris
--
To unsubscribe from this list: send the line "unsubscribeQuoted 7 lines
Quoted 7 lines
linux-nfs"
Quoted 3 lines
in
Quoted 2 lines
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribeQuoted 10 lines
linux-nfs"
Quoted 3 lines
Quoted 3 lines
in
the body of a message to [email protected] More majordomo info athttp://vger.kernel.org/majordomo-info.html
Quoted 2 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs"in
Quoted 2 lines
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · J. Bruce Fields
Quoted 3 lines · Christopher Hawkins
Thanks Bruce, it does appear that idmapd is somehow related to this issue. However, I'm stumped because it does exist on the client, along with all the libraries it requires (according to ldd, anyway).
Quoted 8 lines
And
/etc/passwd, and whatever other files it might decide it needs to open to do its work?Quoted 3 lines
Is
there an easy way to disable idmapd while still running nfs4 to determine if it is causing the hang?Not that I know of.
Why would you need to? 'Magic sysrq-trigger t' will show you if the idmapper is deadlocked inside the NFS filesystem (as will the equivalent 'echo t >/proc/sysrq-trigger').
Trond