nfsv4 in linux 2.6.13 (-ck7)
This oops seems to occur during heavy i/o load over nfsv4.
nfs-utils version 1.0.7
OOps follows, what other information is needed?
[kernel] Unable to handle kernel paging request at 0000000000100108 RIP:
[kernel] <ffffffff80185e98>{generic_drop_inode+56}
[kernel] PGD 34e3b067 PUD 34e68067 PMD 0
[kernel] CPU 0
[kernel] Modules linked in: nvidia snd_seq_midi snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_pcm_oss snd_mixer_oss snd_seq_oss snd_seq_midi_event snd_seq snd_emu10k1 snd_rawmidi snd_seq_device snd_ac97_codec snd_pcm snd_timer snd_page_alloc snd_util_mem snd_hwdep snd
[kernel] Pid: 179, comm: kswapd0 Tainted: P 2.6.13-ck7
[kernel] RIP: 0010:[<ffffffff80185e98>] <ffffffff80185e98>{generic_drop_inode+56}
[kernel] RSP: 0018:ffff81003fcd7b68 EFLAGS: 00010246
[kernel] RAX: 0000000000100100 RBX: ffff81001a58c950 RCX: 0000000000200200
[kernel] RDX: ffff81001a58c960 RSI: ffff81003eb84000 RDI: ffff81001a58c950
[kernel] RBP: ffff81001a58c950 R08: 00000000fffffffa R09: ffff81001a58ca68
[kernel] R10: 0000000000000001 R11: ffffffff80185e60 R12: 0000000000000000
[kernel] R13: ffff81001a58c7d0 R14: ffff81001a58c860 R15: ffff81003f1f5200
[kernel] FS: 0000000040800960(0000) GS:ffffffff80494800(0000) knlGS:0000000056160040
[kernel] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[kernel] CR2: 0000000000100108 CR3: 0000000034de7000 CR4: 00000000000006e0
[kernel] Process kswapd0 (pid: 179, threadinfo ffff81003fcd6000, task ffff81003fcb2760)
[kernel] Stack: ffff81003f1f5c00 ffffffff801d7a25 00000001803e8238 ffff81003fcd7c18
[kernel] ffffffffffffffff ffff81003fcd7c18 ffff81003fcd7c00 ffff81001a58c938
[kernel] 0000000000000000 0000000000000000
[kernel] Call Trace:<ffffffff801d7a25>{__nfs_revalidate_inode+261} <ffffffff8014e5df>{find_get_pages_tag+31}
[kernel] <ffffffff8015781a>{pagevec_lookup_tag+26} <ffffffff8014e00e>{wait_on_page_writeback_range+206}
[kernel] <ffffffff801f11ba>{nfs_do_return_delegation+42} <ffffffff801f12e5>{nfs_inode_return_delegation+197}
[kernel] <ffffffff801d8a10>{nfs4_clear_inode+32} <ffffffff80184cfe>{clear_inode+158}
[kernel] <ffffffff8018594e>{dispose_list+94} <ffffffff80185b82>{shrink_icache_memory+434}
[kernel] <ffffffff8015806b>{shrink_slab+219} <ffffffff80159517>{balance_pgdat+695}
[kernel] <ffffffff801597a8>{kswapd+312} <ffffffff80143b30>{autoremove_wake_function+0}
[kernel] <ffffffff80143b30>{autoremove_wake_function+0} <ffffffff8010f30e>{child_rip+8}
[kernel] <ffffffff80159670>{kswapd+0} <ffffffff8010f306>{child_rip+0} [kernel]
[kernel] Code: 48 89 48 08 48 89 01 48 8b 05 aa 43 26 00 48 89 50 08 48 89 [kernel] RIP <ffffffff80185e98>{generic_drop_inode+56} RSP <ffff81003fcd7b68>
Signature · 2 lines
Gabriel A. Devenyi [email protected]
Quoted 4 lines · * Gabriel A. Devenyi ([email protected]) wrote:
This oops seems to occur during heavy i/o load over nfsv4.
[kernel] Unable to handle kernel paging request at 0000000000100108 RIP:
[kernel] <ffffffff80185e98>{generic_drop_inode+56}
There have been a couple recent reports of this, and a fix is in the works.
See the recent thread here:
http://lkml.org/lkml/2005/9/25/44
Quoted 1 line
[kernel] Modules linked in: nvidia
^^^^^^
Quoted 1 line
[kernel] Pid: 179, comm: kswapd0 Tainted: P 2.6.13-ck7
Tainted kernel, when sending bug reports please be sure bug happens w/out tainted kernel.
thanks,
-chris
Quoted 10 lines · Chris Wright
Quoted 4 lines · * Gabriel A. Devenyi ([email protected]) wrote:
This oops seems to occur during heavy i/o load over nfsv4.
[kernel] Unable to handle kernel paging request at 0000000000100108 RIP:
[kernel] <ffffffff80185e98>{generic_drop_inode+56}There have been a couple recent reports of this, and a fix is in the works.
See the recent thread here:
Quoted 8 lines
http://lkml.org/lkml/2005/9/25/44
Quoted 1 line
[kernel] Modules linked in: nvidia
^^^^^^Quoted 1 line
[kernel] Pid: 179, comm: kswapd0 Tainted: P 2.6.13-ck7
Tainted kernel, when sending bug reports please be sure bug happens w/out tainted kernel.
Of course, my apologies, however, this is a fs error, is it even conceivable that something such as the nvidia kernel driver could affect this?
Quoted 4 lines
thanks,
-chris
Signature · 2 lines
Gabriel A. Devenyi [email protected]
Quoted 3 lines · * Gabriel A. Devenyi ([email protected]) wrote:
Of course, my apologies, however, this is a fs error, is it even conceivable that something such as the nvidia kernel driver could affect this?
In this case it's not very likely since others are seeing same problem under load. However, a binary module can corrupt any kernel memory. So as a general rule all bets are off with a binary module loaded.
thanks,
-chris
Quoted 3 lines · Chris Wright
In this case it's not very likely since others are seeing same problem under load. However, a binary module can corrupt any kernel memory. So as a general rule all bets are off with a binary module loaded.
Thanks, I'll keep that in mind for next time. With regards to the patch in the other thread, should I try and patch the client, the server or both?
Signature · 2 lines
Gabriel A. Devenyi [email protected]
Quoted 3 lines · * Gabriel A. Devenyi ([email protected]) wrote:
Thanks, I'll keep that in mind for next time. With regards to the patch in the other thread, should I try and patch the client, the server or both?
Client side AFAIK. May want to check with the nfs folks to see if they've got any specific testing they'd find useful.
thanks,
-chris
Quoted 2 lines · Chris Wright
Client side AFAIK. May want to check with the nfs folks to see if they've got any specific testing they'd find useful.
Well the patch seems to have cleared this problem up, do you happen to know where the NFS folks can be located so I can provide further testing/feedback? Thanks.
Signature · 2 lines
Gabriel A. Devenyi [email protected]
Quoted 7 lines · * Gabriel A. Devenyi ([email protected]) wrote:
Quoted 2 lines · Chris Wright
Client side AFAIK. May want to check with the nfs folks to see if they've got any specific testing they'd find useful.
Well the patch seems to have cleared this problem up, do you happen to know where the NFS folks can be located so I can provide further testing/feedback? Thanks.
They're at [email protected]
thanks,
-chris
on den 12.10.2005 Klokka 19:03 (-0400) skreiv Gabriel A. Devenyi:
Quoted 36 lines
This oops seems to occur during heavy i/o load over nfsv4.
nfs-utils version 1.0.7
OOps follows, what other information is needed?
[kernel] Unable to handle kernel paging request at 0000000000100108 RIP:
[kernel] <ffffffff80185e98>{generic_drop_inode+56}
[kernel] PGD34e3b067PUD34e68067PMD 0
[kernel] CPU 0
[kernel] Modules linked in: nvidiasnd_seq_midisnd_emu10k1_synthsnd_emux_synthsnd_seq_virmidisnd_seq_midi_emulsnd_pcm_osssnd_mixer_osssnd_seq_osssnd_seq_midi_eventsnd_seqsnd_emu10k1snd_rawmidisnd_seq_devicesnd_ac97_codecsnd_pcmsnd_timersnd_page_allocsnd_util_memsnd_hwdepsnd
[kernel] Pid: 179, comm: kswapd0 Tainted: P 2.6.13-ck7
[kernel] RIP: 0010:[<ffffffff80185e98>] <ffffffff80185e98>{generic_drop_inode+56}
[kernel] RSP: 0018:ffff81003fcd7b68 EFLAGS: 00010246
[kernel] RAX: 0000000000100100 RBX:ffff81001a58c950RCX: 0000000000200200
[kernel] RDX:ffff81001a58c960RSI:ffff81003eb84000RDI:ffff81001a58c950
[kernel] RBP:ffff81001a58c950R08:00000000fffffffaR09:ffff81001a58ca68
[kernel] R10: 0000000000000001 R11:ffffffff80185e60R12: 0000000000000000
[kernel] R13:ffff81001a58c7d0R14:ffff81001a58c860R15:ffff81003f1f5200
[kernel] FS: 0000000040800960(0000) GS:ffffffff80494800(0000) knlGS:0000000056160040
[kernel] CS: 0010 DS: 0018 ES: 0018 CR0:000000008005003b
[kernel] CR2: 0000000000100108 CR3:0000000034de7000CR4:00000000000006e0
[kernel] Process kswapd0 (pid: 179, threadinfoffff81003fcd6000, taskffff81003fcb2760)
[kernel] Stack:ffff81003f1f5c00ffffffff801d7a2500000001803e8238ffff81003fcd7c18
[kernel] ffffffffffffffffffff81003fcd7c18ffff81003fcd7c00ffff81001a58c938
[kernel] 0000000000000000 0000000000000000
[kernel] Call Trace:<ffffffff801d7a25>{__nfs_revalidate_inode+261} <ffffffff8014e5df>{find_get_pages_tag+31}[kernel] <ffffffff8015781a>{pagevec_lookup_tag+26} <ffffffff8014e00e>{wait_on_page_writeback_range+206} [kernel] <ffffffff801f11ba>{nfs_do_return_delegation+42} <ffffffff801f12e5>{nfs_inode_return_delegation+197} [kernel] <ffffffff801d8a10>{nfs4_clear_inode+32} <ffffffff80184cfe>{clear_inode+158} [kernel] <ffffffff8018594e>{dispose_list+94} <ffffffff80185b82>{shrink_icache_memory+434} [kernel] <ffffffff8015806b>{shrink_slab+219} <ffffffff80159517>{balance_pgdat+695} [kernel] <ffffffff801597a8>{kswapd+312} <ffffffff80143b30>{autoremove_wake_function+0} [kernel] <ffffffff80143b30>{autoremove_wake_function+0} <ffffffff8010f30e>{child_rip+8} [kernel] <ffffffff80159670>{kswapd+0} <ffffffff8010f306>{child_rip+0}
Quoted 3 lines
[kernel]
[kernel] Code: 48 89 48 08 48 89 01 48 8b 05 aa 43 26 00 48 89 50 08 48 89 [kernel] RIP <ffffffff80185e98>{generic_drop_inode+56} RSP <ffff81003fcd7b68>
Does the attached patch fix it for you?
Cheers,
Trond
NFS: Fix Oopsable/unnecessary i_count manipulations in nfs_wait_on_inode()
Oopsable since nfs_wait_on_inode() can get called as part of iput_final().
Unnecessary since the caller had better be damned sure that the inode won't disappear from underneath it anyway.
--- inode.c | 2 -- 1 files changed, 2 deletions(-)
fs/nfs/inode.c +0 -2
@@ -877,12 +877,10 @@ static int nfs_wait_on_inode(struct inodsigset_t oldmask;int error; -atomic_inc (&inode->i_count);rpc_clnt_sigmask (clnt, &oldmask); error =wait_on_bit_lock (&nfsi->flags,NFS_INO_REVALIDATING , nfs_wait_schedule,TASK_INTERRUPTIBLE );rpc_clnt_sigunmask (clnt, &oldmask); -iput (inode);return error; }
Quoted 4 lines · Trond Myklebust
Does the attached patch fix it for you?
Cheers,
Trond
This patch http://www.citi.umich.edu/projects/nfsv4/linux/kernel-patches/2.6.13-1/linux-2.6.13-001-NFS_ALL_MODIFIED.dif
I found already fixed my problem, I applied your patch on top and everything seems to be working fine.
Signature · 2 lines
Gabriel A. Devenyi [email protected]