Skip to contentSkip to conversation

Kernel crashes when repeatedly trying to mount nfs share that is failing

Bug 24302 Trond Myklebust 3 participants 7 messages Dec 8, 2010 – Dec 9, 2010 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw
Quoted 14 lines · On Wed, 2010-12-08 at 18:30 +0000, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=24302

Andrew Morton <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

          Component|Other                       |NFS
         AssignedTo|[email protected] |[email protected]
                   |rg                          |
Quoted 3 lines

--- Comment #8 from Andrew Morton <[email protected]> 2010-12-08 18:30:32 --- OK, I reassigned it to NFS. If that was wrong then at least the NFS guys should be able to help point things in the right direction.

<Switching to email interface. Please do not edit the bugzilla entry directly, since that will lose the above Cc information>

Chuck,

Stefan appears to be hitting a panic in the nfs_umount() call from nfs_walk_authlist(). Can you take a look, please?

Cheers
Trond

#3 lore raw
Quoted 20 lines · Trond Myklebust
Quoted 14 lines · On Wed, 2010-12-08 at 18:30 +0000, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=24302

Andrew Morton <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

         Component|Other                       |NFS
        AssignedTo|[email protected] |[email protected]
                  |rg                          |
Quoted 3 lines

--- Comment #8 from Andrew Morton <[email protected]> 2010-12-08 18:30:32 --- OK, I reassigned it to NFS. If that was wrong then at least the NFS guys should be able to help point things in the right direction.

Quoted 8 lines

<Switching to email interface. Please do not edit the bugzilla entry directly, since that will lose the above Cc information>

Chuck,

Stefan appears to be hitting a panic in the nfs_umount() call from nfs_walk_authlist(). Can you take a look, please?

Recv'd. I'll have a look.

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#4 lore raw

Hi Stefan-

Quoted 27 lines · Chuck Lever
Quoted 24 lines · Trond Myklebust
Quoted 14 lines · On Wed, 2010-12-08 at 18:30 +0000, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=24302

Andrew Morton <[email protected]> changed:

          What    |Removed                     |Added
----------------------------------------------------------------------------
         Component|Other                       |NFS AssignedTo|[email protected] |[email protected] |rg                          |

Quoted 3 lines

--- Comment #8 from Andrew Morton <[email protected]>  2010-12-08 18:30:32 --- OK, I reassigned it to NFS.  If that was wrong then at least the NFS guys should be able to help point things in the right direction.

<Switching to email interface. Please do not edit the bugzilla entry directly, since that will lose the above Cc information>

Quoted 6 lines
Quoted 4 lines

Chuck,

Stefan appears to be hitting a panic in the nfs_umount() call from nfs_walk_authlist(). Can you take a look, please?

Recv'd.  I'll have a look.

Apologies in advance for the attachment.  There are a few other clean ups that can be done, but this seems to be the minimal fix.  Please try this and let us know if it addresses your panic.

The faulty commit went in a while back, so this patch is probably appropriate for stable kernels (back to 2.6.31, where this was likely introduced).

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Tested-by
#5 lore raw
Quoted 18 lines · Chuck Lever

Hi Stefan-

Quoted 14 lines · Chuck Lever
Quoted 11 lines · Trond Myklebust
Quoted 9 lines · On Wed, 2010-12-08 at 18:30 +0000, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=24302

Andrew Morton <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

Component|Other |NFS

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

AssignedTo|[email protected] |[email protected] |rg |

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

--- Comment #8 from Andrew Morton <[email protected]>

2010-12-08 18:30:32 --- OK, I reassigned it to NFS.  If that was wrong

then at least the NFS guys should be able to help point things in the right direction.

<Switching to email interface. Please do not edit the bugzilla entry directly, since that will lose the above Cc information>

Missed to read the above, so I have to duplicate things a bit, sorry. Removed the bugzilla cc, so the report does not get spammed.

Quoted 18 lines
Quoted 8 lines
Quoted 6 lines

Chuck,

Stefan appears to be hitting a panic in the nfs_umount() call from nfs_walk_authlist(). Can you take a look, please?

Recv'd. I'll have a look.

Apologies in advance for the attachment. There are a few other clean ups that can be done, but this seems to be the minimal fix. Please try this and let us know if it addresses your panic.

Quoted 3 lines

The faulty commit went in a while back, so this patch is probably appropriate for stable kernels (back to 2.6.31, where this was likely introduced).

I can confirm that this was the root cause of the crash. The sha referenced as the causing commit, got in with a 2.6.32-rc so I added the stable info accordingly. I also added my tested-by and slightly modified the bug reference format (Chuck, I hope this is ok).

-Stefan

Quoted 1 line

From dcc0a9d5d9490680ea9faa7b90de3224c3aba7e3 Mon Sep 17 00:00:00 2001

From: Chuck Lever <[email protected]>
Date: Wed, 8 Dec 2010 19:07:12 -0500
Subject: [PATCH] NFS: Fix panic after nfs_umount()

After a few unsuccessful NFS mount attempts in which the client and server cannot agree on an authentication flavor both support, the client panics. nfs_umount() is invoked in the kernel in this case.

Turns out this particular UMNT RPC invocation causes the RPC client to write off the end of the rpc_clnt's iostat array. This is because the mount client's nrprocs field is initialized with the count of defined procedures (two: MNT and UMNT), rather than the size of the client's proc array (four).

The fix is to use the same initialization technique used by most other upper layer clients in the kernel.

Introduced by commit 0b524123, which failed to update nrprocs when it added support for UMNT.

---
 fs/nfs/mount_clnt.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
fs/nfs/mount_clnt.c +2 -2
index 59047f8..50552c5 100644
@@ -503,13 +503,13 @@ static struct rpc_procinfo mnt3_procedures[] = {
 
 static struct rpc_version mnt_version1 = {
 	.number		= 1,
-	.nrprocs	= 2,
+	.nrprocs	= ARRAY_SIZE(mnt_procedures),
 	.procs		= mnt_procedures,
 };
 
 static struct rpc_version mnt_version3 = {
 	.number		= 3,
-	.nrprocs	= 2,
+	.nrprocs	= ARRAY_SIZE(mnt_procedures),
 	.procs		= mnt3_procedures,
 };
 
Signature · 1 line
1.7.0.4
#6 lore raw
Quoted 55 lines · Stefan Bader
Quoted 18 lines · Chuck Lever

Hi Stefan-

Quoted 14 lines · Chuck Lever
Quoted 11 lines · Trond Myklebust
Quoted 9 lines · On Wed, 2010-12-08 at 18:30 +0000, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=24302

Andrew Morton <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

Component|Other |NFS

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

AssignedTo|[email protected] |[email protected] |rg |

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

--- Comment #8 from Andrew Morton <[email protected]>

2010-12-08 18:30:32 --- OK, I reassigned it to NFS.  If that was wrong

then at least the NFS guys should be able to help point things in the right direction.

<Switching to email interface. Please do not edit the bugzilla entry directly, since that will lose the above Cc information>

Missed to read the above, so I have to duplicate things a bit, sorry. Removed the bugzilla cc, so the report does not get spammed.

Quoted 18 lines
Quoted 8 lines
Quoted 6 lines

Chuck,

Stefan appears to be hitting a panic in the nfs_umount() call from nfs_walk_authlist(). Can you take a look, please?

Recv'd. I'll have a look.

Apologies in advance for the attachment. There are a few other clean ups that can be done, but this seems to be the minimal fix. Please try this and let us know if it addresses your panic.

Quoted 8 lines
Quoted 3 lines

The faulty commit went in a while back, so this patch is probably appropriate for stable kernels (back to 2.6.31, where this was likely introduced).

I can confirm that this was the root cause of the crash. The sha referenced as the causing commit, got in with a 2.6.32-rc so I added the stable info accordingly. I also added my tested-by and slightly modified the bug reference format (Chuck, I hope this is ok).

Looks good to me. I assume, since Trond still owns the kernel.org bugzilla, that he will close this loop? Let me know if I need to do anything else.

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#7 lore raw
Quoted 62 lines · Chuck Lever
Quoted 59 lines · Stefan Bader
Quoted 18 lines · Chuck Lever

Hi Stefan-

Quoted 14 lines · Chuck Lever
Quoted 11 lines · Trond Myklebust
Quoted 9 lines · On Wed, 2010-12-08 at 18:30 +0000, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=24302

Andrew Morton <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

Component|Other |NFS

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

AssignedTo|[email protected] |[email protected] |rg |

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

--- Comment #8 from Andrew Morton <[email protected]>

2010-12-08 18:30:32 --- OK, I reassigned it to NFS.  If that was wrong

then at least the NFS guys should be able to help point things in the right direction.

<Switching to email interface. Please do not edit the bugzilla entry directly, since that will lose the above Cc information>

Missed to read the above, so I have to duplicate things a bit, sorry. Removed the bugzilla cc, so the report does not get spammed.

Quoted 21 lines
Quoted 8 lines
Quoted 6 lines

Chuck,

Stefan appears to be hitting a panic in the nfs_umount() call from nfs_walk_authlist(). Can you take a look, please?

Recv'd. I'll have a look.

Apologies in advance for the attachment. There are a few other clean ups that can be done, but this seems to be the minimal fix. Please try this and let us know if it addresses your panic.

The faulty commit went in a while back, so this patch is probably appropriate for stable kernels (back to 2.6.31, where this was likely introduced).

Quoted 7 lines
Quoted 4 lines

I can confirm that this was the root cause of the crash. The sha referenced as the causing commit, got in with a 2.6.32-rc so I added the stable info accordingly. I also added my tested-by and slightly modified the bug reference format (Chuck, I hope this is ok).

Looks good to me. I assume, since Trond still owns the kernel.org bugzilla, that he will close this loop? Let me know if I need to do anything else.

Actually someone spotted that the second array_size should probably use mnt_procedures3 instead of mnt_procedures. They happen to be the same size, so testing is ok.

-Stefan

#8 lore raw
Quoted 68 lines · Stefan Bader
Quoted 67 lines · Chuck Lever
Quoted 63 lines · Stefan Bader
Quoted 18 lines · Chuck Lever

Hi Stefan-

Quoted 14 lines · Chuck Lever
Quoted 11 lines · Trond Myklebust
Quoted 9 lines · On Wed, 2010-12-08 at 18:30 +0000, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=24302

Andrew Morton <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

Component|Other |NFS

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

AssignedTo|[email protected] |[email protected] |rg |

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

--- Comment #8 from Andrew Morton <[email protected]>

2010-12-08 18:30:32 --- OK, I reassigned it to NFS.  If that was wrong

then at least the NFS guys should be able to help point things in the right direction.

<Switching to email interface. Please do not edit the bugzilla entry directly, since that will lose the above Cc information>

Missed to read the above, so I have to duplicate things a bit, sorry. Removed the bugzilla cc, so the report does not get spammed.

Quoted 18 lines
Quoted 8 lines
Quoted 6 lines

Chuck,

Stefan appears to be hitting a panic in the nfs_umount() call from nfs_walk_authlist(). Can you take a look, please?

Recv'd. I'll have a look.

Apologies in advance for the attachment. There are a few other clean ups that can be done, but this seems to be the minimal fix. Please try this and let us know if it addresses your panic.

Quoted 3 lines

The faulty commit went in a while back, so this patch is probably appropriate for stable kernels (back to 2.6.31, where this was likely introduced).

I can confirm that this was the root cause of the crash. The sha referenced as the causing commit, got in with a 2.6.32-rc so I added the stable info accordingly. I also added my tested-by and slightly modified the bug reference format (Chuck, I hope this is ok).

Quoted 5 lines
Quoted 2 lines

Looks good to me. I assume, since Trond still owns the kernel.org bugzilla, that he will close this loop? Let me know if I need to do anything else.

Actually someone spotted that the second array_size should probably use mnt_procedures3 instead of mnt_procedures. They happen to be the same size, so testing is ok.

Oops. Copy-n-paste error, I thought I had fixed that. Good catch.

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com