another block layout oops
Subsystems (2)
2.6.36-rc6-pnfs+ from benny's tree of this morning, block layout mount, connectathon general "Large Compile" test fails. This worked with the previous 2.6.36-rc3.
collect2: ld terminated with signal 9 [Killed]
BUG: unable to handle kernel NULL pointer dereference at 00000090 IP: [<e0c2e06e>] nfs4_proc_layoutget+0x4b/0x159 [nfs] *pde = 00000000 Oops: 0000 [#1] last sysfs file: /sys/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/PNP0C0A:00/power_supply/BAT0/energy_full Modules linked in: blocklayoutdriver nfs lockd fscache nfs_acl auth_rpcgss sunrpc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm microcode snd_timer snd pcspkr i2c_piix4 soundcore pcnet32 i2c_core snd_page_alloc mii [last unloaded: speedstep_lib]
Pid: 1385, comm: ld Not tainted 2.6.36-rc6-pnfs+ #10 /VirtualBox
EIP: 0060:[<e0c2e06e>] EFLAGS: 00210202 CPU: 0
EIP is at nfs4_proc_layoutget+0x4b/0x159 [nfs]
EAX: 00000000 EBX: de471300 ECX: 00000004 EDX: df8631c0
ESI: ffffd8e8 EDI: df625d84 EBP: df625dac ESP: df625d58
DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0069
Process ld (pid: 1385, ti=df624000 task=df8631c0 task.ti=df624000)
Stack: df625d68 df625d84 de978000 de471334 00000000 de81ad00 df625d84 e0c42414 <0> de471300 00000000 00000001 e0c4c258 de471300 de471334 00000000 000000c8 <0> 00000001 00000000 de471300 df625ddc de471318 df625dec e0c3fb7d de978000 Call Trace: [<e0c3fb7d>] ? pnfs_update_layout+0x24e/0x29d [nfs] [<e0c23d45>] ? nfs_readpage_async+0x109/0x13b [nfs] [<e0c23e75>] ? nfs_readpage+0xfe/0x123 [nfs] [<c107eac0>] ? generic_file_aio_read+0x3e3/0x538 [<e0c1a372>] ? nfs_file_read+0x94/0xbe [nfs] [<c10ab3e3>] ? do_sync_read+0x8e/0xc9 [<c1146067>] ? fsnotify_perm+0x44/0x50 [<c11460c4>] ? security_file_permission+0x27/0x2b [<c10ab4bb>] ? rw_verify_area+0x9d/0xc0 [<c10ab355>] ? do_sync_read+0x0/0xc9 [<c10aba04>] ? vfs_read+0x82/0xde [<c10abafe>] ? sys_read+0x40/0x62 [<c132425c>] ? syscall_call+0x7/0xb Code: 00 8b 80 60 01 00 00 89 45 b4 31 c0 f3 ab 8d 43 34 89 45 b8 8d 45 d8 89 45 b0 8d 45 bc 89 45 ac 8b 43 24 b9 04 00 00 00 8b 7d b0 <8b> 80 90 00 00 00 8b 90 60 01 00 00 31 c0 f3 ab 8b 45 b8 8b 7d EIP: [<e0c2e06e>] nfs4_proc_layoutget+0x4b/0x159 [nfs] SS:ESP 0068:df625d58 CR2: 0000000000000090 ---[ end trace 5333af79b6361d78 ]---
Jim, would you mind retesting with pnfs-all-2.6.36-rc6-2010-09-30? Not that there's any possible fix there, but a fresh Oops could help, if you can reproduce it.
Thanks,
Benny
Quoted 46 lines · Jim Rees
2.6.36-rc6-pnfs+ from benny's tree of this morning, block layout mount, connectathon general "Large Compile" test fails. This worked with the previous 2.6.36-rc3.
collect2: ld terminated with signal 9 [Killed]
BUG: unable to handle kernel NULL pointer dereference at 00000090 IP: [<e0c2e06e>] nfs4_proc_layoutget+0x4b/0x159 [nfs] *pde = 00000000 Oops: 0000 [#1] last sysfs file: /sys/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/PNP0C0A:00/power_supply/BAT0/energy_full Modules linked in: blocklayoutdriver nfs lockd fscache nfs_acl auth_rpcgss sunrpc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm microcode snd_timer snd pcspkr i2c_piix4 soundcore pcnet32 i2c_core snd_page_alloc mii [last unloaded: speedstep_lib]Pid: 1385, comm: ld Not tainted 2.6.36-rc6-pnfs+ #10 /VirtualBox
EIP: 0060:[<e0c2e06e>] EFLAGS: 00210202 CPU: 0
EIP is atnfs4_proc_layoutget+0x4b/0x159[nfs]
EAX: 00000000 EBX:de471300ECX: 00000004 EDX:df8631c0
ESI:ffffd8e8EDI:df625d84EBP:df625dacESP:df625d58
DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0069
Process ld (pid: 1385, ti=df624000 task=df8631c0 task.ti=df624000)Stack: df625d68 df625d84 de978000 de471334 00000000 de81ad00 df625d84 e0c42414 <0> de471300 00000000 00000001 e0c4c258 de471300 de471334 00000000 000000c8 <0> 00000001 00000000 de471300 df625ddc de471318 df625dec e0c3fb7d de978000 Call Trace: [<e0c3fb7d>] ? pnfs_update_layout+0x24e/0x29d [nfs] [<e0c23d45>] ? nfs_readpage_async+0x109/0x13b [nfs] [<e0c23e75>] ? nfs_readpage+0xfe/0x123 [nfs] [<c107eac0>] ? generic_file_aio_read+0x3e3/0x538 [<e0c1a372>] ? nfs_file_read+0x94/0xbe [nfs] [<c10ab3e3>] ? do_sync_read+0x8e/0xc9 [<c1146067>] ? fsnotify_perm+0x44/0x50 [<c11460c4>] ? security_file_permission+0x27/0x2b [<c10ab4bb>] ? rw_verify_area+0x9d/0xc0 [<c10ab355>] ? do_sync_read+0x0/0xc9 [<c10aba04>] ? vfs_read+0x82/0xde [<c10abafe>] ? sys_read+0x40/0x62 [<c132425c>] ? syscall_call+0x7/0xb Code: 00 8b 80 60 01 00 00 89 45 b4 31 c0 f3 ab 8d 43 34 89 45 b8 8d 45 d8 89 45 b0 8d 45 bc 89 45 ac 8b 43 24 b9 04 00 00 00 8b 7d b0 <8b> 80 90 00 00 00 8b 90 60 01 00 00 31 c0 f3 ab 8b 45 b8 8b 7d EIP: [<e0c2e06e>] nfs4_proc_layoutget+0x4b/0x159 [nfs] SS:ESP 0068:df625d58 CR2: 0000000000000090 ---[ end trace 5333af79b6361d78 ]-----
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Benny Halevy wrote:
Jim, would you mind retesting with pnfs-all-2.6.36-rc6-2010-09-30? Not that there's any possible fix there, but a fresh Oops could help, if you can reproduce it.
Will do, probably after an important meeting I have at 6:00 this evening.
Quoted 7 lines · Jim Rees
Benny Halevy wrote:
Jim, would you mind retesting with pnfs-all-2.6.36-rc6-2010-09-30? Not that there's any possible fix there, but a fresh Oops could help, if you can reproduce it.
Will do, probably after an important meeting I have at 6:00 this evening.
There is a problem with the LAYOUTGET error handling, which is probably what Jim is hitting (the block servers are much more likely to send RETRYLATER). I'll send in a fix tomorrow morning.
Fred
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Fred Isaman
Quoted 6 lines · Jim Rees
Benny Halevy wrote:
Jim, would you mind retesting with pnfs-all-2.6.36-rc6-2010-09-30? Not that there's any possible fix there, but a fresh Oops could help, if you can reproduce it.
Quoted 5 lines
Quoted 1 line
Will do, probably after an important meeting I have at 6:00 this evening.
There is a problem with the LAYOUTGET error handling, which is probably what Jim is hitting (the block servers are much more likely to send RETRYLATER). I'll send in a fix tomorrow morning.
One problem I can see is that nfs4_layoutget_release frees calldata (a.k.a. lgp) which is reused later if we retry.
We should either keep a reference count on it or clone it internally in _nfs4_proc_layoutget for each call. Since the calls are essentially synchronous the caller and allocator (e.g. send_layoutget) can just free the call data (or dereference, if we keep a refcount).
Same for layoutcommit and layoutreturn.
Benny
Quoted 8 lines
Fred
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Jim, does the recently submitted PATCH 5/7 fix this?
=46red
On Fri, Oct 1, 2010 at 3:58 AM, Benny Halevy <[email protected]> wrot= e:
Quoted 5 lines · Fred Isaman
Quoted 4 lines · Jim Rees
Quoted 3 lines
Benny Halevy wrote:
=A0Jim, would you mind retesting with pnfs-all-2.6.36-rc6-2010-09-3=
0?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
=A0Not that there's any possible fix there, but a fresh Oops could =A0help, if you can reproduce it.
Will do, probably after an important meeting I have at 6:00 this ev=
ening.
Quoted 3 lines
Quoted 3 lines
There is a problem with the LAYOUTGET error handling, which is probably what Jim is hitting =A0(the block servers are much more lik=
ely
Quoted 7 lines
Quoted 1 line
to send RETRYLATER). =A0I'll send in a fix tomorrow morning.
One problem I can see is that
nfs4_layoutget_releasefrees calldata (a.k.a. lgp) which is reused later if we retry.We should either keep a reference count on it or clone it internally in
_nfs4_proc_layoutgetfor each call. =A0Since the calls are essenti=
ally
Quoted 13 lines
synchronous the caller and allocator (e.g.
send_layoutget) can just free the call data (or dereference, if we keep a refcount).Same for layoutcommit and layoutreturn.
Benny
Quoted 5 lines
Fred
Quoted 2 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs=
" in
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the body of a message to [email protected]
More majordomo info at =A0http://vger.kernel.org/majordomo-info.htm=
l
Quoted 3 lines
Quoted 1 line
Quoted 1 line
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" =
in
Quoted 3 lines
the body of a message to [email protected]
More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
Fred Isaman wrote:
Jim, does the recently submitted PATCH 5/7 fix this?
I think benny's merged this, right? I'll fetch the new pnfs-all-latest and test, if I can. My 3GB fedora root partition filled up doing a yum update and now it looks unrecoverable.
Quoted 5 lines · Jim Rees
Fred Isaman wrote:
Jim, does the recently submitted PATCH 5/7 fix this?
I think benny's merged this, right?
Yep, its in pnfs-submit at least.
Fred
Quoted 8 lines
I'll fetch the new pnfs-all-latest and
test, if I can. My 3GB fedora root partition filled up doing a yum update and now it looks unrecoverable.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): AANLkTi=T6PY6MDzbyWikSYOi7HoMWAsEarw-3k=S1+Bu-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Fred Isaman wrote:
Jim, does the recently submitted PATCH 5/7 fix this?
Yes. Benny's pnfs-all-latest has that patch, and fixes the oops. Thanks.