Skip to contentSkip to conversation

NFSv4 poops itself

DISCUSSION Jeff Garzik 6 participants 15 messages Jul 27, 2007 – Aug 6, 2007

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Background:

Server: x86-64 dual core Intel, kernel 2.6.23-rc1 (my home fileserver)
Exporting NFS/NFSv4 mounts. Client count: 1 Uptime: 4 days

Client: x86-64 dual core Intel, kernel 2.6.23-rc1 (my main workstation)
NFS mount setup:
pretzel:/ on /g type nfs4 (rw,noatime,proto=tcp,addr=10.10.10.1)
Uptime: 4 days

Home directory mounted via NFSv4.

Problem:

My workstation has been happily talking to my file server for several days without incident. An hour ago, my numeric keypad stopping working (unrelated problem... USB or X bug?). The solution to the keypad problem is usually to log out of X and log back in, or worse case, reboot.

So, I log out, and log back in. At first, a few shell windows open and successfully initialize themselves (read bash profile over NFS, etc.) Then, as more shell windows open, things start hanging. I can easily switch to console and ssh to the fileserver, so it is clear this is an NFS hang.

No adverse messages at all on the client.

On the server, I see NFSv4 spamming dmesg with hundreds of thousands of messages:

Jul 27 08:20:53 pretzel kernel: NFSD: preprocess_seqid_op: old stateid!
Jul 27 08:21:24 pretzel last message repeated 167966 times
Jul 27 08:21:55 pretzel last message repeated 173628 times
Jul 27 08:21:55 pretzel kernel: NFSD: preprocess_seqid_op: old stateid!
Jul 27 08:22:26 pretzel last message repeated 171286 times
Jul 27 08:23:27 pretzel last message repeated 344461 times
Jul 27 08:23:30 pretzel last message repeated 18656 times

I rebooted the client, the problem disappeared, and everything is happy again... but clearly NFSv4 shat itself. And now I am worried this will happen again.

In all my quite-heavy use of NFSv4 I've never seen this behavior before,
so I would call this a regression.

I always run vanilla linux-2.6.git self-built kernels on both client and server.

Jeff

#1 lore raw
Quoted 38 lines · Jeff Garzik

Background:

Server: x86-64 dual core Intel, kernel 2.6.23-rc1 (my home fileserver)
Exporting NFS/NFSv4 mounts. Client count: 1 Uptime: 4 days

Client: x86-64 dual core Intel, kernel 2.6.23-rc1 (my main workstation)
NFS mount setup:
pretzel:/ on /g type nfs4 (rw,noatime,proto=tcp,addr=10.10.10.1)
Uptime: 4 days

Home directory mounted via NFSv4.

Problem:

My workstation has been happily talking to my file server for several days without incident. An hour ago, my numeric keypad stopping working (unrelated problem... USB or X bug?). The solution to the keypad problem is usually to log out of X and log back in, or worse case, reboot.

So, I log out, and log back in. At first, a few shell windows open and successfully initialize themselves (read bash profile over NFS, etc.) Then, as more shell windows open, things start hanging. I can easily switch to console and ssh to the fileserver, so it is clear this is an NFS hang.

No adverse messages at all on the client.

On the server, I see NFSv4 spamming dmesg with hundreds of thousands of messages:

Jul 27 08:20:53 pretzel kernel: NFSD: preprocess_seqid_op: old stateid!
Jul 27 08:21:24 pretzel last message repeated 167966 times
Jul 27 08:21:55 pretzel last message repeated 173628 times
Jul 27 08:21:55 pretzel kernel: NFSD: preprocess_seqid_op: old stateid!
Jul 27 08:22:26 pretzel last message repeated 171286 times
Jul 27 08:23:27 pretzel last message repeated 344461 times
Jul 27 08:23:30 pretzel last message repeated 18656 times
Quoted 6 lines

I rebooted the client, the problem disappeared, and everything is happy again... but clearly NFSv4 shat itself. And now I am worried this will happen again.

In all my quite-heavy use of NFSv4 I've never seen this behavior before,
so I would call this a regression.

Yup. Bruce has reported the same bug so it is under investigation. I'll keep you posted when I think we have a fix.

Cheers
Trond

#9 lore raw
In reply to Jeff Garzik

Hi,

Am Friday 27 July 2007 14:53 schrieben Sie:

Quoted 42 lines

Background:

Server: x86-64 dual core Intel, kernel 2.6.23-rc1 (my home fileserver)
Exporting NFS/NFSv4 mounts. Client count: 1 Uptime: 4 days

Client: x86-64 dual core Intel, kernel 2.6.23-rc1 (my main workstation)
NFS mount setup:
pretzel:/ on /g type nfs4 (rw,noatime,proto=tcp,addr=10.10.10.1)
Uptime: 4 days

Home directory mounted via NFSv4.

Problem:

My workstation has been happily talking to my file server for several days without incident. An hour ago, my numeric keypad stopping working (unrelated problem... USB or X bug?). The solution to the keypad problem is usually to log out of X and log back in, or worse case, reboot.

So, I log out, and log back in. At first, a few shell windows open and successfully initialize themselves (read bash profile over NFS, etc.) Then, as more shell windows open, things start hanging. I can easily switch to console and ssh to the fileserver, so it is clear this is an NFS hang.

No adverse messages at all on the client.

On the server, I see NFSv4 spamming dmesg with hundreds of thousands of messages:

Jul 27 08:20:53 pretzel kernel: NFSD: preprocess_seqid_op: old stateid!
Jul 27 08:21:24 pretzel last message repeated 167966 times
Jul 27 08:21:55 pretzel last message repeated 173628 times
Jul 27 08:21:55 pretzel kernel: NFSD: preprocess_seqid_op: old stateid!
Jul 27 08:22:26 pretzel last message repeated 171286 times
Jul 27 08:23:27 pretzel last message repeated 344461 times
Jul 27 08:23:30 pretzel last message repeated 18656 times

I rebooted the client, the problem disappeared, and everything is happy again... but clearly NFSv4 shat itself. And now I am worried this will happen again.

Quoted 5 lines

In all my quite-heavy use of NFSv4 I've never seen this behavior before,
so I would call this a regression.

I always run vanilla linux-2.6.git self-built kernels on both client and server.

me too, my server has 2.6.18-? (openSUSE 10.2). On the client (2.6.23-rc1-mm1), I also see (shortly before the hang)

Jul 26 13:09:19 fb07-iapwap2 kernel: =================================
Jul 26 13:09:19 fb07-iapwap2 kernel: [ INFO: inconsistent lock state ]
Jul 26 13:09:19 fb07-iapwap2 kernel: 2.6.23-rc1-mm1 #1
Jul 26 13:09:19 fb07-iapwap2 kernel: ---------------------------------
Jul 26 13:09:19 fb07-iapwap2 kernel: inconsistent {softirq-on-W} -> 
{in-softirq-W} usage.
Jul 26 13:09:19 fb07-iapwap2 kernel: hald/3873 [HC0[0]:SC1[1]:HE1:SE0] takes:
Jul 26 13:09:19 fb07-iapwap2 kernel:  (rpc_credcache_lock){-+..}, at: 
[<c01dc166>] _atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel: {softirq-on-W} state was registered at:
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013d4f7>] mark_lock+0x77/0x630
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013c094>] add_lock_to_list+0x44/0xc0
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013e8af>] 
__lock_acquire+0x65f/0x1020
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013db0e>] mark_held_locks+0x5e/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c012448d>] local_bh_enable+0x7d/0x130
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013f2cf>] lock_acquire+0x5f/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01dc166>] 
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c02d156a>] _spin_lock+0x2a/0x40
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01dc166>] 
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01dc166>] 
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c02d156a>] _spin_lock+0x2a/0x40
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcecf770>] put_rpccred+0x60/0x110 
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcecf840>] 
rpcauth_unbindcred+0x20/0x60 [sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcece1f4>] rpc_put_task+0x44/0xb0 
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcec8ffd>] rpc_call_sync+0x2d/0x40 
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dced680d>] rpcb_register+0x10d/0x1c0 
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dced06ef>] svc_register+0x8f/0x160 
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013dca5>] 
trace_hardirqs_on+0xc5/0x170
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dced0fc5>] __svc_create+0x1d5/0x200 
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dced1160>] svc_create+0x10/0x20 
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcf69602>] nfs_callback_up+0x82/0x120 
[nfs]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcf43efd>] nfs_get_client+0x17d/0x390 
[nfs]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c0175f59>] kmem_cache_alloc+0x79/0xd0
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013dca5>] 
trace_hardirqs_on+0xc5/0x170
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcf44144>] nfs4_set_client+0x34/0x1a0 
[nfs]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcf448d1>] 
nfs4_create_server+0x61/0x3c0 [nfs]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c017657f>] __kmalloc+0xdf/0x120
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013dca5>] 
trace_hardirqs_on+0xc5/0x170
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013e987>] 
__lock_acquire+0x737/0x1020
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01e22cd>] copy_from_user+0x3d/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01636e2>] strndup_user+0x62/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcf4d647>] nfs4_get_sb+0x3c7/0x610 
[nfs]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013e987>] 
__lock_acquire+0x737/0x1020
Jul 26Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c015d536>] 
get_page_from_freelist+0x316/0x460
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013dca5>] 
trace_hardirqs_on+0xc5/0x170
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c015e0be>] __alloc_pages+0x4e/0x370
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c017b380>] vfs_kern_mount+0xa0/0x120
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c0190b1d>] do_mount+0x21d/0x840
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01667c7>] 
handle_mm_fault+0x4a7/0x690
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c02d1494>] _spin_unlock+0x14/0x20
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c0166773>] 
handle_mm_fault+0x453/0x690
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c0135c24>] up_read+0x14/0x30
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c0116ddd>] do_page_fault+0x1ed/0x6e0
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c018f597>] 
copy_mount_options+0xb7/0x150
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01911b2>] sys_mount+0x72/0xb0
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01040b2>] syscall_call+0x7/0xb
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<ffffffff>] 0xffffffff
Jul 26 13:09:19 fb07-iapwap2 kernel: irq event stamp: 240896
Jul 26 13:09:19 fb07-iapwap2 kernel: hardirqs last  enabled at (240896): 
[<c017592e>] kmem_cache_free+0x7e/0xb0
Jul 26 13:09:19 fb07-iapwap2 kernel: hardirqs last disabled at (240895): 
[<c01758fc>] kmem_cache_free+0x4c/0xb0
Jul 26 13:09:19 fb07-iapwap2 kernel: softirqs last  enabled at (240446): 
[<c0124255>] do_softirq+0x45/0x50
Jul 26 13:09:19 fb07-iapwap2 kernel: softirqs last disabled at (240887): 
[<c0124255>] do_softirq+0x45/0x50
Jul 26 13:09:19 fb07-iapwap2 kernel:
Jul 26 13:09:19 fb07-iapwap2 kernel: other info that might help us debug this:
Jul 26 13:09:19 fb07-iapwap2 kernel: no locks held by hald/3873.
Jul 26 13:09:19 fb07-iapwap2 kernel:
Jul 26 13:09:19 fb07-iapwap2 kernel: stack backtrace:
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013ccbc>] print_usage_bug+0x15c/0x170
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013d8da>] mark_lock+0x45a/0x630
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013e857>] __lock_acquire+0x607/0x1020
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013e987>] __lock_acquire+0x737/0x1020
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013f2cf>] lock_acquire+0x5f/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01dc166>] 
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02d156a>] _spin_lock+0x2a/0x40
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01dc166>] 
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01dc166>] 
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<dcecf770>] put_rpccred+0x60/0x110 
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<dcf67e10>] 
nfs_free_delegation_callback+0x10/0x20 [nfs]
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c012fa47>] 
__rcu_process_callbacks+0x67/0x1c0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c012fbaf>] 
rcu_process_callbacks+0xf/0x20
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0124293>] tasklet_action+0x33/0x70
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01241b5>] __do_softirq+0x55/0xb0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0124255>] do_softirq+0x45/0x50
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0112490>] 
smp_apic_timer_interrupt+0x40/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0104ad9>] 
apic_timer_interrupt+0x29/0x38
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0104ae3>] 
apic_timer_interrupt+0x33/0x38
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02d18a2>] _spin_unlock_irq+0x22/0x30
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02cecb4>] 
__sched_text_start+0x1c4/0x390
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02cec29>] 
__sched_text_start+0x139/0x390
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0127feb>] __mod_timer+0x8b/0xa0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02cf63a>] schedule_timeout+0x4a/0xc0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0127b40>] process_timeout+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0186197>] do_sys_poll+0x237/0x320
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0186d70>] __pollwait+0x0/0xf0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>] 
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>] 
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>] 
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>] 
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>] 
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>] 
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>] 
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>] 
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>] 
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>] 
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>] 
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>] 
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02d1444>] _read_unlock+0x14/0x20
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02cbae5>] 
unix_stream_sendmsg+0x1a5/0x350
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0263727>] sock_aio_write+0x117/0x140
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013e987>] __lock_acquire+0x737/0x1020
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0179874>] do_readv_writev+0x144/0x1c0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0263610>] sock_aio_write+0x0/0x140
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0136ac1>] do_gettimeofday+0x31/0xf0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0104108>] restore_nocheck+0x12/0x15
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013dca5>] 
trace_hardirqs_on+0xc5/0x170
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01e25af>] copy_to_user+0x3f/0x70
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01862a8>] sys_poll+0x28/0x40
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01040b2>] syscall_call+0x7/0xb
Jul 26 13:09:19 fb07-iapwap2 kernel:  =======================

marc

#10 lore raw

(please don't drop CC's when you reply to email; you are cutting relevant people out of the loop)

Quoted 19 lines · Marc Dietrich

me too, my server has 2.6.18-? (openSUSE 10.2). On the client (2.6.23-rc1-mm1), I also see (shortly before the hang)

Jul 26 13:09:19 fb07-iapwap2 kernel: =================================
Jul 26 13:09:19 fb07-iapwap2 kernel: [ INFO: inconsistent lock state ]
Jul 26 13:09:19 fb07-iapwap2 kernel: 2.6.23-rc1-mm1 #1
Jul 26 13:09:19 fb07-iapwap2 kernel: ---------------------------------
Jul 26 13:09:19 fb07-iapwap2 kernel: inconsistent {softirq-on-W} -> {in-softirq-W} usage.
Jul 26 13:09:19 fb07-iapwap2 kernel: hald/3873 [HC0[0]:SC1[1]:HE1:SE0] takes:
Jul 26 13:09:19 fb07-iapwap2 kernel:  (rpc_credcache_lock){-+..}, at: [<c01dc166>] _atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel: {softirq-on-W} state was registered at:
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013d4f7>] mark_lock+0x77/0x630
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013c094>] add_lock_to_list+0x44/0xc0
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013e8af>] __lock_acquire+0x65f/0x1020
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013db0e>] mark_held_locks+0x5e/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c012448d>] local_bh_enable+0x7d/0x130
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013f2cf>] lock_acquire+0x5f/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01dc166>] _atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c02d156a>] _spin_lock+0x2a/0x40
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01dc166>] _atomic_dec_and_lock+0x16/0x60
Quoted 8 lines
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01dc166>] _atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c02d156a>] _spin_lock+0x2a/0x40
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcecf770>] put_rpccred+0x60/0x110 [sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcecf840>] rpcauth_unbindcred+0x20/0x60 [sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcece1f4>] rpc_put_task+0x44/0xb0 [sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcec8ffd>] rpc_call_sync+0x2d/0x40 [sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dced680d>] rpcb_register+0x10d/0x1c0 [sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dced06ef>] svc_register+0x8f/0x160 [sunrpc]

[continues]

That's quite interesting. I wonder if I enable lock debugging and tracing, if new stuff will appear.

Jeff

#11 lore raw
Quoted 45 lines · Jeff Garzik

(please don't drop CC's when you reply to email; you are cutting relevant people out of the loop)

Quoted 39 lines · Marc Dietrich

me too, my server has 2.6.18-? (openSUSE 10.2). On the client (2.6.23-rc1-mm1), I also see (shortly before the hang)

Jul 26 13:09:19 fb07-iapwap2 kernel: =================================
Jul 26 13:09:19 fb07-iapwap2 kernel: [ INFO: inconsistent lock state ]
Jul 26 13:09:19 fb07-iapwap2 kernel: 2.6.23-rc1-mm1 #1
Jul 26 13:09:19 fb07-iapwap2 kernel: ---------------------------------
Jul 26 13:09:19 fb07-iapwap2 kernel: inconsistent {softirq-on-W} -> 
{in-softirq-W} usage.
Jul 26 13:09:19 fb07-iapwap2 kernel: hald/3873 [HC0[0]:SC1[1]:HE1:SE0] takes:
Jul 26 13:09:19 fb07-iapwap2 kernel:  (rpc_credcache_lock){-+..}, at: 
[<c01dc166>] _atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel: {softirq-on-W} state was registered at:
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013d4f7>] mark_lock+0x77/0x630
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013c094>] add_lock_to_list+0x44/0xc0
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013e8af>] 
__lock_acquire+0x65f/0x1020
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013db0e>] mark_held_locks+0x5e/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c012448d>] local_bh_enable+0x7d/0x130
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013f2cf>] lock_acquire+0x5f/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01dc166>] 
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c02d156a>] _spin_lock+0x2a/0x40
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01dc166>] 
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01dc166>] 
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c02d156a>] _spin_lock+0x2a/0x40
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcecf770>] put_rpccred+0x60/0x110 
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcecf840>] 
rpcauth_unbindcred+0x20/0x60 [sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcece1f4>] rpc_put_task+0x44/0xb0 
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcec8ffd>] rpc_call_sync+0x2d/0x40 
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dced680d>] rpcb_register+0x10d/0x1c0 
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dced06ef>] svc_register+0x8f/0x160 
Quoted 1 line

[sunrpc]

Quoted 1 line

[continues]

That particular hang in rpciod_down we do have a fix for, but it is not related to the issue you were seeing Jeff.

Trond

The commit 4ada539ed77c7a2bbcb75cafbbd7bd8d2b9bef7b lead to the unpleasant possibility of an asynchronous rpc_task being required to call rpciod_down() when it is complete. This again means that the rpciod workqueue may get to call destroy_workqueue on itself -> hang...

Change rpciod_up/rpciod_down to just get/put the module, and then create/destroy the workqueues on module load/unload.

---

 net/sunrpc/sched.c |   57 +++++++++++++++++++++-------------------------------
 1 files changed, 23 insertions(+), 34 deletions(-)
net/sunrpc/sched.c +23 -34
index b5723c2..954d7ec 100644
@@ -50,8 +50,6 @@ static RPC_WAITQ(delay_queue, "delayq");
 /*
  * rpciod-related stuff
  */
-static DEFINE_MUTEX(rpciod_mutex);
-static atomic_t rpciod_users = ATOMIC_INIT(0);
 struct workqueue_struct *rpciod_workqueue;
 
 /*
@@ -961,60 +959,49 @@ void rpc_killall_tasks(struct rpc_clnt *clnt)
 	spin_unlock(&clnt->cl_lock);
 }
 
+int rpciod_up(void)
+{
+	return try_module_get(THIS_MODULE) ? 0 : -EINVAL;
+}
+
+void rpciod_down(void)
+{
+	module_put(THIS_MODULE);
+}
+
 /*
- * Start up the rpciod process if it's not already running.
+ * Start up the rpciod workqueue.
  */
-int
-rpciod_up(void)
+static int rpciod_start(void)
 {
 	struct workqueue_struct *wq;
-	int error = 0;
-
-	if (atomic_inc_not_zero(&rpciod_users))
-		return 0;
-
-	mutex_lock(&rpciod_mutex);
 
-	/* Guard against races with rpciod_down() */
-	if (rpciod_workqueue != NULL)
-		goto out_ok;
 	/*
 	 * Create the rpciod thread and wait for it to start.
 	 */
 	dprintk("RPC:       creating workqueue rpciod\n");
-	error = -ENOMEM;
 	wq = create_workqueue("rpciod");
-	if (wq == NULL)
-		goto out;
-
 	rpciod_workqueue = wq;
-	error = 0;
-out_ok:
-	atomic_inc(&rpciod_users);
-out:
-	mutex_unlock(&rpciod_mutex);
-	return error;
+	return rpciod_workqueue != NULL;
 }
 
-void
-rpciod_down(void)
+static void rpciod_stop(void)
 {
-	if (!atomic_dec_and_test(&rpciod_users))
-		return;
+	struct workqueue_struct *wq = NULL;
 
-	mutex_lock(&rpciod_mutex);
+	if (rpciod_workqueue == NULL)
+		return;
 	dprintk("RPC:       destroying workqueue rpciod\n");
 
-	if (atomic_read(&rpciod_users) == 0 && rpciod_workqueue != NULL) {
-		destroy_workqueue(rpciod_workqueue);
-		rpciod_workqueue = NULL;
-	}
-	mutex_unlock(&rpciod_mutex);
+	wq = rpciod_workqueue;
+	rpciod_workqueue = NULL;
+	destroy_workqueue(wq);
 }
 
 void
 rpc_destroy_mempool(void)
 {
+	rpciod_stop();
 	if (rpc_buffer_mempool)
 		mempool_destroy(rpc_buffer_mempool);
 	if (rpc_task_mempool)
@@ -1048,6 +1035,8 @@ rpc_init_mempool(void)
 						      rpc_buffer_slabp);
 	if (!rpc_buffer_mempool)
 		goto err_nomem;
+	if (!rpciod_start())
+		goto err_nomem;
 	return 0;
 err_nomem:
 	rpc_destroy_mempool();
#12 lore raw
In reply to Marc Dietrich
Quoted 223 lines · Marc Dietrich

Hi,

Am Friday 27 July 2007 14:53 schrieben Sie:

Quoted 42 lines

Background:

Server: x86-64 dual core Intel, kernel 2.6.23-rc1 (my home fileserver)
Exporting NFS/NFSv4 mounts. Client count: 1 Uptime: 4 days

Client: x86-64 dual core Intel, kernel 2.6.23-rc1 (my main workstation)
NFS mount setup:
pretzel:/ on /g type nfs4 (rw,noatime,proto=tcp,addr=10.10.10.1)
Uptime: 4 days

Home directory mounted via NFSv4.

Problem:

My workstation has been happily talking to my file server for several days without incident. An hour ago, my numeric keypad stopping working (unrelated problem... USB or X bug?). The solution to the keypad problem is usually to log out of X and log back in, or worse case, reboot.

So, I log out, and log back in. At first, a few shell windows open and successfully initialize themselves (read bash profile over NFS, etc.) Then, as more shell windows open, things start hanging. I can easily switch to console and ssh to the fileserver, so it is clear this is an NFS hang.

No adverse messages at all on the client.

On the server, I see NFSv4 spamming dmesg with hundreds of thousands of messages:

Jul 27 08:20:53 pretzel kernel: NFSD: preprocess_seqid_op: old stateid!
Jul 27 08:21:24 pretzel last message repeated 167966 times
Jul 27 08:21:55 pretzel last message repeated 173628 times
Jul 27 08:21:55 pretzel kernel: NFSD: preprocess_seqid_op: old stateid!
Jul 27 08:22:26 pretzel last message repeated 171286 times
Jul 27 08:23:27 pretzel last message repeated 344461 times
Jul 27 08:23:30 pretzel last message repeated 18656 times

I rebooted the client, the problem disappeared, and everything is happy again... but clearly NFSv4 shat itself. And now I am worried this will happen again.

Quoted 5 lines

In all my quite-heavy use of NFSv4 I've never seen this behavior before,
so I would call this a regression.

I always run vanilla linux-2.6.git self-built kernels on both client and server.

me too, my server has 2.6.18-? (openSUSE 10.2). On the client (2.6.23-rc1-mm1), I also see (shortly before the hang)

Jul 26 13:09:19 fb07-iapwap2 kernel: =================================
Jul 26 13:09:19 fb07-iapwap2 kernel: [ INFO: inconsistent lock state ]
Jul 26 13:09:19 fb07-iapwap2 kernel: 2.6.23-rc1-mm1 #1
Jul 26 13:09:19 fb07-iapwap2 kernel: ---------------------------------
Jul 26 13:09:19 fb07-iapwap2 kernel: inconsistent {softirq-on-W} ->
{in-softirq-W} usage.
Jul 26 13:09:19 fb07-iapwap2 kernel: hald/3873 [HC0[0]:SC1[1]:HE1:SE0] takes:
Jul 26 13:09:19 fb07-iapwap2 kernel:  (rpc_credcache_lock){-+..}, at:
[<c01dc166>] _atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel: {softirq-on-W} state was registered at:
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013d4f7>] mark_lock+0x77/0x630
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013c094>] add_lock_to_list+0x44/0xc0
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013e8af>]
__lock_acquire+0x65f/0x1020
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013db0e>] mark_held_locks+0x5e/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c012448d>] local_bh_enable+0x7d/0x130
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013f2cf>] lock_acquire+0x5f/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01dc166>]
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c02d156a>] _spin_lock+0x2a/0x40
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01dc166>]
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01dc166>]
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c02d156a>] _spin_lock+0x2a/0x40
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcecf770>] put_rpccred+0x60/0x110
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcecf840>]
rpcauth_unbindcred+0x20/0x60 [sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcece1f4>] rpc_put_task+0x44/0xb0
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcec8ffd>] rpc_call_sync+0x2d/0x40
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dced680d>] rpcb_register+0x10d/0x1c0
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dced06ef>] svc_register+0x8f/0x160
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013dca5>]
trace_hardirqs_on+0xc5/0x170
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dced0fc5>] __svc_create+0x1d5/0x200
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dced1160>] svc_create+0x10/0x20
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcf69602>] nfs_callback_up+0x82/0x120
[nfs]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcf43efd>] nfs_get_client+0x17d/0x390
[nfs]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c0175f59>] kmem_cache_alloc+0x79/0xd0
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013dca5>]
trace_hardirqs_on+0xc5/0x170
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcf44144>] nfs4_set_client+0x34/0x1a0
[nfs]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcf448d1>]
nfs4_create_server+0x61/0x3c0 [nfs]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c017657f>] __kmalloc+0xdf/0x120
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013dca5>]
trace_hardirqs_on+0xc5/0x170
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013e987>]
__lock_acquire+0x737/0x1020
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01e22cd>] copy_from_user+0x3d/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01636e2>] strndup_user+0x62/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<dcf4d647>] nfs4_get_sb+0x3c7/0x610
[nfs]
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013e987>]
__lock_acquire+0x737/0x1020
Jul 26Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c015d536>]
get_page_from_freelist+0x316/0x460
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c013dca5>]
trace_hardirqs_on+0xc5/0x170
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c015e0be>] __alloc_pages+0x4e/0x370
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c017b380>] vfs_kern_mount+0xa0/0x120
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c0190b1d>] do_mount+0x21d/0x840
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01667c7>]
handle_mm_fault+0x4a7/0x690
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c02d1494>] _spin_unlock+0x14/0x20
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c0166773>]
handle_mm_fault+0x453/0x690
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c0135c24>] up_read+0x14/0x30
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c0116ddd>] do_page_fault+0x1ed/0x6e0
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c018f597>]
copy_mount_options+0xb7/0x150
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01911b2>] sys_mount+0x72/0xb0
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<c01040b2>] syscall_call+0x7/0xb
Jul 26 13:09:19 fb07-iapwap2 kernel:   [<ffffffff>] 0xffffffff
Jul 26 13:09:19 fb07-iapwap2 kernel: irq event stamp: 240896
Jul 26 13:09:19 fb07-iapwap2 kernel: hardirqs last  enabled at (240896):
[<c017592e>] kmem_cache_free+0x7e/0xb0
Jul 26 13:09:19 fb07-iapwap2 kernel: hardirqs last disabled at (240895):
[<c01758fc>] kmem_cache_free+0x4c/0xb0
Jul 26 13:09:19 fb07-iapwap2 kernel: softirqs last  enabled at (240446):
[<c0124255>] do_softirq+0x45/0x50
Jul 26 13:09:19 fb07-iapwap2 kernel: softirqs last disabled at (240887):
[<c0124255>] do_softirq+0x45/0x50
Jul 26 13:09:19 fb07-iapwap2 kernel:
Jul 26 13:09:19 fb07-iapwap2 kernel: other info that might help us debug this:
Jul 26 13:09:19 fb07-iapwap2 kernel: no locks held by hald/3873.
Jul 26 13:09:19 fb07-iapwap2 kernel:
Jul 26 13:09:19 fb07-iapwap2 kernel: stack backtrace:
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013ccbc>] print_usage_bug+0x15c/0x170
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013d8da>] mark_lock+0x45a/0x630
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013e857>] __lock_acquire+0x607/0x1020
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013e987>] __lock_acquire+0x737/0x1020
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013f2cf>] lock_acquire+0x5f/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01dc166>]
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02d156a>] _spin_lock+0x2a/0x40
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01dc166>]
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01dc166>]
_atomic_dec_and_lock+0x16/0x60
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<dcecf770>] put_rpccred+0x60/0x110
[sunrpc]
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<dcf67e10>]
nfs_free_delegation_callback+0x10/0x20 [nfs]
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c012fa47>]
__rcu_process_callbacks+0x67/0x1c0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c012fbaf>]
rcu_process_callbacks+0xf/0x20
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0124293>] tasklet_action+0x33/0x70
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01241b5>] __do_softirq+0x55/0xb0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0124255>] do_softirq+0x45/0x50
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0112490>]
smp_apic_timer_interrupt+0x40/0x80
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0104ad9>]
apic_timer_interrupt+0x29/0x38
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0104ae3>]
apic_timer_interrupt+0x33/0x38
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02d18a2>] _spin_unlock_irq+0x22/0x30
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02cecb4>]
__sched_text_start+0x1c4/0x390
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02cec29>]
__sched_text_start+0x139/0x390
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0127feb>] __mod_timer+0x8b/0xa0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02cf63a>] schedule_timeout+0x4a/0xc0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0127b40>] process_timeout+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0186197>] do_sys_poll+0x237/0x320
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0186d70>] __pollwait+0x0/0xf0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>]
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>]
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>]
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>]
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>]
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>]
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>]
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>]
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>]
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>]
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>]
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c011aa50>]
default_wake_function+0x0/0x10
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02d1444>] _read_unlock+0x14/0x20
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c02cbae5>]
unix_stream_sendmsg+0x1a5/0x350
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0263727>] sock_aio_write+0x117/0x140
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013e987>] __lock_acquire+0x737/0x1020
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0179874>] do_readv_writev+0x144/0x1c0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0263610>] sock_aio_write+0x0/0x140
Quoted 8 lines
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0136ac1>] do_gettimeofday+0x31/0xf0
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c0104108>] restore_nocheck+0x12/0x15
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c013dca5>]
trace_hardirqs_on+0xc5/0x170
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01e25af>] copy_to_user+0x3f/0x70
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01862a8>] sys_poll+0x28/0x40
Jul 26 13:09:19 fb07-iapwap2 kernel:  [<c01040b2>] syscall_call+0x7/0xb
Jul 26 13:09:19 fb07-iapwap2 kernel:  =======================

Doesn't appear related ... could you rebuild with CONFIG_FRAME_POINTER and test + resend the above log? The trace up there looks quite ugly.

[ Unrelated, but I just realized from lib/Kconfig.debug that stuff in there "select"s LOCKDEP, which itself "selects" frame pointers (on some archs). But I thought "select" didn't chain properly like that? There's whole lots of depending and selecting and non-user-visible options out there in that file -- and I'm not sure all those selects and depends are really even required to successfully build (?) ]

Satyam

#2 lore raw
In reply to Trond Myklebust
Quoted 31 lines · Trond Myklebust
Quoted 30 lines · Jeff Garzik

Background:

Server: x86-64 dual core Intel, kernel 2.6.23-rc1 (my home fileserver)
Exporting NFS/NFSv4 mounts. Client count: 1 Uptime: 4 days

Client: x86-64 dual core Intel, kernel 2.6.23-rc1 (my main workstation)
NFS mount setup:
pretzel:/ on /g type nfs4 (rw,noatime,proto=tcp,addr=10.10.10.1)
Uptime: 4 days

Home directory mounted via NFSv4.

Problem:

My workstation has been happily talking to my file server for several days without incident. An hour ago, my numeric keypad stopping working (unrelated problem... USB or X bug?). The solution to the keypad problem is usually to log out of X and log back in, or worse case, reboot.

So, I log out, and log back in. At first, a few shell windows open and successfully initialize themselves (read bash profile over NFS, etc.) Then, as more shell windows open, things start hanging. I can easily switch to console and ssh to the fileserver, so it is clear this is an NFS hang.

No adverse messages at all on the client.

On the server, I see NFSv4 spamming dmesg with hundreds of thousands of messages:

Quoted 7 lines
Quoted 7 lines
Jul 27 08:20:53 pretzel kernel: NFSD: preprocess_seqid_op: old stateid!
Jul 27 08:21:24 pretzel last message repeated 167966 times
Jul 27 08:21:55 pretzel last message repeated 173628 times
Jul 27 08:21:55 pretzel kernel: NFSD: preprocess_seqid_op: old stateid!
Jul 27 08:22:26 pretzel last message repeated 171286 times
Jul 27 08:23:27 pretzel last message repeated 344461 times
Jul 27 08:23:30 pretzel last message repeated 18656 times

Jeff and Bruce, could you please try to reproduce the problem after either applying patches 001 to 004 or just the single NFS_ALL patch from

http://client.linux-nfs.org/Linux-2.6.x/2.6.23-rc1/

Cheers
Trond

#3 lore raw
Quoted 5 lines · Trond Myklebust

Jeff and Bruce, could you please try to reproduce the problem after either applying patches 001 to 004 or just the single NFS_ALL patch from

   http://client.linux-nfs.org/Linux-2.6.x/2.6.23-rc1/

It's not easily reproducible here. Any hints on triggering the behavior?

Jeff

#4 lore raw
Quoted 9 lines · Jeff Garzik
Quoted 5 lines · Trond Myklebust

Jeff and Bruce, could you please try to reproduce the problem after either applying patches 001 to 004 or just the single NFS_ALL patch from

   http://client.linux-nfs.org/Linux-2.6.x/2.6.23-rc1/

It's not easily reproducible here. Any hints on triggering the behavior?

I suspect that the problem may involve mixing O_RDWR with O_RDONLY and/or O_WRONLY file access. Bruce might have a better suggestion, though.

#13 lore raw
In reply to Satyam Sharma

Hi again,

Am Freitag, 27. Juli 2007 16:58 schrieb Satyam Sharma:

Quoted 15 lines · Marc Dietrich
Quoted 7 lines

Hi,

(...)

me too, my server has 2.6.18-? (openSUSE 10.2). On the client (2.6.23-rc1-mm1), I also see (shortly before the hang)

Quoted 6 lines
Jul 26 13:09:19 fb07-iapwap2 kernel: =================================
Jul 26 13:09:19 fb07-iapwap2 kernel: [ INFO: inconsistent lock state ]
Jul 26 13:09:19 fb07-iapwap2 kernel: 2.6.23-rc1-mm1 #1
Jul 26 13:09:19 fb07-iapwap2 kernel: ---------------------------------
Jul 26 13:09:19 fb07-iapwap2 kernel: inconsistent {softirq-on-W} ->

(cut away ugly dump)

Quoted 2 lines

Doesn't appear related ... could you rebuild with CONFIG_FRAME_POINTER and test + resend the above log? The trace up there looks quite ugly.

this seems to be better ...

Jul 27 19:43:33 fb07-iapwap2 kernel: =================================
Jul 27 19:43:33 fb07-iapwap2 kernel: [ INFO: inconsistent lock state ]
Jul 27 19:43:33 fb07-iapwap2 kernel: 2.6.23-rc1-mm1 #1
Jul 27 19:43:33 fb07-iapwap2 kernel: ---------------------------------
Jul 27 19:43:33 fb07-iapwap2 kernel: inconsistent {softirq-on-W} -> 
{in-softirq-W} usage.
Jul 27 19:43:33 fb07-iapwap2 kernel: kdm/5909 [HC0[0]:SC1[1]:HE1:SE0] takes:
Jul 27 19:43:33 fb07-iapwap2 kernel:  (rpc_credcache_lock){-+..}, at: 
[<c01dba67>] _atomic_dec_and_lock+0x17/0x60
Jul 27 19:43:33 fb07-iapwap2 kernel: {softirq-on-W} state was registered at:
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c013e800>] 
__lock_acquire+0x650/0x1030
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c013f241>] lock_acquire+0x61/0x80
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c02db8dc>] _spin_lock+0x2c/0x40
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c01dba67>] 
_atomic_dec_and_lock+0x17/0x60
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dceca5fd>] put_rpccred+0x5d/0x100 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dceca6c1>] 
rpcauth_unbindcred+0x21/0x60 [sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcec8fd4>] rpc_put_task+0x44/0xa0 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcec3fe0>] rpc_call_sync+0x30/0x40 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dced172b>] rpcb_register+0xdb/0x180 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcecb5b3>] svc_register+0x93/0x160 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcecbebe>] __svc_create+0x1ee/0x220 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcecc053>] svc_create+0x13/0x20 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcf62752>] nfs_callback_up+0x82/0x120 
[nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcf3df36>] nfs_get_client+0x176/0x390 
[nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcf3e181>] nfs4_set_client+0x31/0x190 
[nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcf3e983>] 
nfs4_create_server+0x63/0x3b0 [nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcf47426>] nfs4_get_sb+0x346/0x5b0 
[nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c017aa74>] vfs_kern_mount+0x94/0x110
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c01900c2>] do_mount+0x1f2/0x7d0
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c0190706>] sys_mount+0x66/0xa0
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c01040de>] syscall_call+0x7/0xb
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<ffffffff>] 0xffffffff
Jul 27 19:43:33 fb07-iapwap2 kernel: irq event stamp: 4916
Jul 27 19:43:33 fb07-iapwap2 kernel: hardirqs last  enabled at (4916): 
[<c017518a>] kmem_cache_free+0x8a/0xc0
Jul 27 19:43:33 fb07-iapwap2 kernel: hardirqs last disabled at (4915): 
[<c0175152>] kmem_cache_free+0x52/0xc0
Jul 27 19:43:33 fb07-iapwap2 kernel: softirqs last  enabled at (4888): 
[<dcec9ff1>] rpc_wake_up_next+0x81/0x180 [sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel: softirqs last disabled at (4905): 
[<c01241e7>] do_softirq+0x47/0x50
Jul 27 19:43:33 fb07-iapwap2 kernel:
Jul 27 19:43:33 fb07-iapwap2 kernel: other info that might help us debug this:
Jul 27 19:43:33 fb07-iapwap2 kernel: 1 lock held by kdm/5909:
Jul 27 19:43:33 fb07-iapwap2 kernel:  #0:  (&inode->i_mutex/1){--..}, at: 
[<c0182efd>] do_unlinkat+0x7d/0x140
Jul 27 19:43:33 fb07-iapwap2 kernel:
Jul 27 19:43:33 fb07-iapwap2 kernel: stack backtrace:
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c010506a>] 
show_trace_log_lvl+0x1a/0x30
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0105c92>] show_trace+0x12/0x20
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0105da5>] dump_stack+0x15/0x20
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c013cc53>] print_usage_bug+0x153/0x160
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c013d849>] mark_lock+0x449/0x620
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c013e7b4>] __lock_acquire+0x604/0x1030
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c013f241>] lock_acquire+0x61/0x80
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c02db8dc>] _spin_lock+0x2c/0x40
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c01dba67>] 
_atomic_dec_and_lock+0x17/0x60
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dceca5fd>] put_rpccred+0x5d/0x100 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcf60fb3>] 
nfs_free_delegation_callback+0x13/0x20 [nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c012fa3a>] 
__rcu_process_callbacks+0x6a/0x1c0
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c012fba2>] 
rcu_process_callbacks+0x12/0x30
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0124228>] tasklet_action+0x38/0x80
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0124135>] __do_softirq+0x55/0xc0
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c01241e7>] do_softirq+0x47/0x50
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0124615>] irq_exit+0x35/0x40
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0112463>] 
smp_apic_timer_interrupt+0x43/0x80
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0104b0f>] 
apic_timer_interrupt+0x33/0x38
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcecab53>] 
rpcauth_lookup_credcache+0x83/0x1c0 [sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcecb1a8>] unx_lookup_cred+0x8/0x10 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dceca8fe>] rpcauth_bindcred+0x5e/0xe0 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcec3e87>] rpc_call_setup+0x57/0x60 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcec3edb>] rpc_do_run_task+0x4b/0xb0 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcec3fce>] rpc_call_sync+0x1e/0x40 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcf53cba>] 
_nfs4_proc_lookupfh+0x6a/0xb0 [nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcf55b89>] 
_nfs4_proc_lookup+0x49/0x1d0 [nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcf55d42>] nfs4_proc_lookup+0x32/0x60 
[nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcf3fe69>] nfs_lookup+0xa9/0x260 [nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcf4226c>] 
nfs_atomic_lookup+0x10c/0x180 [nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0180190>] lookup_hash+0x90/0xb0
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0182f05>] do_unlinkat+0x85/0x140
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0182fd0>] sys_unlink+0x10/0x20
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c01040de>] syscall_call+0x7/0xb
Jul 27 19:43:33 fb07-iapwap2 kernel:  =======================

this kernel is 2.6.23-rc1-mm1 with Trond's fix from this thread and this one.
http://marc.info/?l=linux-kernel&m=118546628202419&w=2.

Problem (hang) is easy reproduceable here by just launching kde and open konsole.

Quoted 9 lines

[ Unrelated, but I just realized from lib/Kconfig.debug that stuff in there "select"s LOCKDEP, which itself "selects" frame pointers (on some archs). But I thought "select" didn't chain properly like that? There's whole lots of depending and selecting and non-user-visible options out there in that file -- and I'm not sure all those selects and depends are really even required to successfully build (?) ]

Satyam

marc

#8 lore raw
In reply to Trond Myklebust

Hi,

Am Freitag, 27. Juli 2007 19:09 schrieb Trond Myklebust:

Quoted 7 lines · Trond Myklebust

(...)
Jeff and Bruce, could you please try to reproduce the problem after either applying patches 001 to 004 or just the single NFS_ALL patch from

   http://client.linux-nfs.org/Linux-2.6.x/2.6.23-rc1/

as posted earlier, 001-002 does not fix it. Then I also applied 003+004 and it seems to be fixed now. kde boots up and I can start some programs, which I couldn't before.

The "inconsistent lock state" error still exists, but it really seems to be unrelated.

Quoted 2 lines

Cheers
Trond

thanks

Marc

#14 lore raw
In reply to Marc Dietrich
Quoted 140 lines · Marc Dietrich

Hi again,

Am Freitag, 27. Juli 2007 16:58 schrieb Satyam Sharma:

Quoted 15 lines · Marc Dietrich
Quoted 7 lines

Hi,

(...)

me too, my server has 2.6.18-? (openSUSE 10.2). On the client (2.6.23-rc1-mm1), I also see (shortly before the hang)

Quoted 6 lines
Jul 26 13:09:19 fb07-iapwap2 kernel: =================================
Jul 26 13:09:19 fb07-iapwap2 kernel: [ INFO: inconsistent lock state ]
Jul 26 13:09:19 fb07-iapwap2 kernel: 2.6.23-rc1-mm1 #1
Jul 26 13:09:19 fb07-iapwap2 kernel: ---------------------------------
Jul 26 13:09:19 fb07-iapwap2 kernel: inconsistent {softirq-on-W} ->

(cut away ugly dump)

Quoted 2 lines

Doesn't appear related ... could you rebuild with CONFIG_FRAME_POINTER and test + resend the above log? The trace up there looks quite ugly.

this seems to be better ...

Jul 27 19:43:33 fb07-iapwap2 kernel: =================================
Jul 27 19:43:33 fb07-iapwap2 kernel: [ INFO: inconsistent lock state ]
Jul 27 19:43:33 fb07-iapwap2 kernel: 2.6.23-rc1-mm1 #1
Jul 27 19:43:33 fb07-iapwap2 kernel: ---------------------------------
Jul 27 19:43:33 fb07-iapwap2 kernel: inconsistent {softirq-on-W} -> 
{in-softirq-W} usage.
Jul 27 19:43:33 fb07-iapwap2 kernel: kdm/5909 [HC0[0]:SC1[1]:HE1:SE0] takes:
Jul 27 19:43:33 fb07-iapwap2 kernel:  (rpc_credcache_lock){-+..}, at: 
[<c01dba67>] _atomic_dec_and_lock+0x17/0x60
Jul 27 19:43:33 fb07-iapwap2 kernel: {softirq-on-W} state was registered at:
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c013e800>] 
__lock_acquire+0x650/0x1030
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c013f241>] lock_acquire+0x61/0x80
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c02db8dc>] _spin_lock+0x2c/0x40
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c01dba67>] 
_atomic_dec_and_lock+0x17/0x60
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dceca5fd>] put_rpccred+0x5d/0x100 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dceca6c1>] 
rpcauth_unbindcred+0x21/0x60 [sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcec8fd4>] rpc_put_task+0x44/0xa0 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcec3fe0>] rpc_call_sync+0x30/0x40 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dced172b>] rpcb_register+0xdb/0x180 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcecb5b3>] svc_register+0x93/0x160 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcecbebe>] __svc_create+0x1ee/0x220 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcecc053>] svc_create+0x13/0x20 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcf62752>] nfs_callback_up+0x82/0x120 
[nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcf3df36>] nfs_get_client+0x176/0x390 
[nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcf3e181>] nfs4_set_client+0x31/0x190 
[nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcf3e983>] 
nfs4_create_server+0x63/0x3b0 [nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<dcf47426>] nfs4_get_sb+0x346/0x5b0 
[nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c017aa74>] vfs_kern_mount+0x94/0x110
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c01900c2>] do_mount+0x1f2/0x7d0
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c0190706>] sys_mount+0x66/0xa0
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<c01040de>] syscall_call+0x7/0xb
Jul 27 19:43:33 fb07-iapwap2 kernel:   [<ffffffff>] 0xffffffff
Jul 27 19:43:33 fb07-iapwap2 kernel: irq event stamp: 4916
Jul 27 19:43:33 fb07-iapwap2 kernel: hardirqs last  enabled at (4916): 
[<c017518a>] kmem_cache_free+0x8a/0xc0
Jul 27 19:43:33 fb07-iapwap2 kernel: hardirqs last disabled at (4915): 
[<c0175152>] kmem_cache_free+0x52/0xc0
Jul 27 19:43:33 fb07-iapwap2 kernel: softirqs last  enabled at (4888): 
[<dcec9ff1>] rpc_wake_up_next+0x81/0x180 [sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel: softirqs last disabled at (4905): 
[<c01241e7>] do_softirq+0x47/0x50
Jul 27 19:43:33 fb07-iapwap2 kernel:
Jul 27 19:43:33 fb07-iapwap2 kernel: other info that might help us debug this:
Jul 27 19:43:33 fb07-iapwap2 kernel: 1 lock held by kdm/5909:
Jul 27 19:43:33 fb07-iapwap2 kernel:  #0:  (&inode->i_mutex/1){--..}, at: 
[<c0182efd>] do_unlinkat+0x7d/0x140
Jul 27 19:43:33 fb07-iapwap2 kernel:
Jul 27 19:43:33 fb07-iapwap2 kernel: stack backtrace:
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c010506a>] 
show_trace_log_lvl+0x1a/0x30
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0105c92>] show_trace+0x12/0x20
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0105da5>] dump_stack+0x15/0x20
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c013cc53>] print_usage_bug+0x153/0x160
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c013d849>] mark_lock+0x449/0x620
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c013e7b4>] __lock_acquire+0x604/0x1030
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c013f241>] lock_acquire+0x61/0x80
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c02db8dc>] _spin_lock+0x2c/0x40
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c01dba67>] 
_atomic_dec_and_lock+0x17/0x60
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dceca5fd>] put_rpccred+0x5d/0x100 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcf60fb3>] 
nfs_free_delegation_callback+0x13/0x20 [nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c012fa3a>] 
__rcu_process_callbacks+0x6a/0x1c0
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c012fba2>] 
rcu_process_callbacks+0x12/0x30
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0124228>] tasklet_action+0x38/0x80
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0124135>] __do_softirq+0x55/0xc0
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c01241e7>] do_softirq+0x47/0x50
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0124615>] irq_exit+0x35/0x40
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0112463>] 
smp_apic_timer_interrupt+0x43/0x80
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0104b0f>] 
apic_timer_interrupt+0x33/0x38
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcecab53>] 
rpcauth_lookup_credcache+0x83/0x1c0 [sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcecb1a8>] unx_lookup_cred+0x8/0x10 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dceca8fe>] rpcauth_bindcred+0x5e/0xe0 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcec3e87>] rpc_call_setup+0x57/0x60 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcec3edb>] rpc_do_run_task+0x4b/0xb0 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcec3fce>] rpc_call_sync+0x1e/0x40 
[sunrpc]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcf53cba>] 
_nfs4_proc_lookupfh+0x6a/0xb0 [nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcf55b89>] 
_nfs4_proc_lookup+0x49/0x1d0 [nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcf55d42>] nfs4_proc_lookup+0x32/0x60 
[nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcf3fe69>] nfs_lookup+0xa9/0x260 [nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<dcf4226c>] 
nfs_atomic_lookup+0x10c/0x180 [nfs]
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0180190>] lookup_hash+0x90/0xb0
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0182f05>] do_unlinkat+0x85/0x140
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c0182fd0>] sys_unlink+0x10/0x20
Jul 27 19:43:33 fb07-iapwap2 kernel:  [<c01040de>] syscall_call+0x7/0xb
Jul 27 19:43:33 fb07-iapwap2 kernel:  =======================
Quoted 5 lines

this kernel is 2.6.23-rc1-mm1 with Trond's fix from this thread and this one.
http://marc.info/?l=linux-kernel&m=118546628202419&w=2.

Problem (hang) is easy reproduceable here by just launching kde and open konsole.

That is very odd. I suspect the memory shrinker, but am at a loss to explain why that would be called from a softirq context.

Could you please see if you can reproduce the above problem with 2.6.23-rc1 + the 4 patches?

Cheers
Trond

#5 lore raw
In reply to Trond Myklebust
Quoted 6 lines · Trond Myklebust
Quoted 5 lines · Jeff Garzik
Quoted 4 lines · Trond Myklebust

Jeff and Bruce, could you please try to reproduce the problem after either applying patches 001 to 004 or just the single NFS_ALL patch from

Quoted 8 lines
Quoted 4 lines
Quoted 1 line
   http://client.linux-nfs.org/Linux-2.6.x/2.6.23-rc1/

It's not easily reproducible here. Any hints on triggering the behavior?

I suspect that the problem may involve mixing O_RDWR with O_RDONLY and/or O_WRONLY file access. Bruce might have a better suggestion, though.

I haven't done any more work to identify when exactly the problem is triggered, but I did confirm that I could reproduce the problem reliably with current git, but *not* with current git plus your 4-patch nfs-all.

--b.

#6 lore raw
Quoted 12 lines · J. Bruce Fields
Quoted 11 lines · Trond Myklebust
Quoted 9 lines · Jeff Garzik
Quoted 5 lines · Trond Myklebust

Jeff and Bruce, could you please try to reproduce the problem after either applying patches 001 to 004 or just the single NFS_ALL patch from

   http://client.linux-nfs.org/Linux-2.6.x/2.6.23-rc1/

It's not easily reproducible here. Any hints on triggering the behavior?

Quoted 7 lines
Quoted 3 lines

I suspect that the problem may involve mixing O_RDWR with O_RDONLY and/or O_WRONLY file access. Bruce might have a better suggestion, though.

I haven't done any more work to identify when exactly the problem is triggered, but I did confirm that I could reproduce the problem reliably with current git, but *not* with current git plus your 4-patch nfs-all.

And that result still holds after using the patched kernel for several more hours.

--b.

#7 lore raw
Quoted 17 lines · J. Bruce Fields
Quoted 16 lines · J. Bruce Fields
Quoted 14 lines · Trond Myklebust
Quoted 9 lines · Jeff Garzik
Quoted 5 lines · Trond Myklebust

Jeff and Bruce, could you please try to reproduce the problem after either applying patches 001 to 004 or just the single NFS_ALL patch from

   http://client.linux-nfs.org/Linux-2.6.x/2.6.23-rc1/

It's not easily reproducible here. Any hints on triggering the behavior?

I suspect that the problem may involve mixing O_RDWR with O_RDONLY and/or O_WRONLY file access. Bruce might have a better suggestion, though.

Quoted 6 lines
Quoted 3 lines

I haven't done any more work to identify when exactly the problem is triggered, but I did confirm that I could reproduce the problem reliably with current git, but *not* with current git plus your 4-patch nfs-all.

And that result still holds after using the patched kernel for several more hours.

Thanks Bruce!
Trond